Charlie Alfred’s Weblog

September 6, 2009

SCRUM and Architecture – Do they mix?

Filed under: Uncategorized — charliealfred @ 9:08 pm

SCRUM is a popular agile development methodology that was first popularized in the early 1990’s by Ken Schwaber and others.  It features short (2-3 week) development sprints, development team accountability, and separation of the chickens (management stakeholders) from the pigs (developers).

Software architecture is a high-level approach to the conception of a system.  It was initialy popularized in the mid 1980’s by the writings of David Garlan and Mary Shaw.  The Software Engineering Institute (SEI), affiliated with  Carnegie Mellon University has been a major force in spreading the software architecture mantra.  Today, virtually every software development group of 12 people or larger has a software architect on staff.

However, during the past 20 years, SCRUM and Software Architecture have lived a somewhat tenuous existence.  Like other agile methods (such as XP), SCRUM is not a believer in big design efforts up front.  Planning mainly consists of choosing a team, and capturing and prioritizing a number of user stories.  From that point, the development team and product manager select the set of stories to work on during each sprint, and the developers figure out how to design the solution.

The SCRUM methodology is intended to have fast reaction times to changes in  requirements or priorities.  During a sprint, no course changes are permitted, but between sprints, anything goes.  If the changes are significant, the development team relies on refactoring existing software and leveraging automated unit tests to ensure no regressions.

This article (located at https://charliealfred.wordpress.com/scrum-and-architecture-partners-or-adversaries/) explores the relationship between SCRUM  and Software Architecture, primarily from the perspective of a recent project that tried to blend both.

Advertisements

14 Comments

  1. There is this perception that Agile (and its variation SCRUM) methodology is anti-architecture. I claim that this is a false perception that is propagated by those who lack a formal education in software engineering and, therefore, neither appreciates nor recognizes a good architecture.

    The second to the last principle from http://agilemanifesto.org/principles.html says this. The best architectures, requirements, and designs
    emerge from self-organizing teams. How can anyone claim that Agile is anti-architecture when it’s founding document advocates for architecture?

    Comment by Glenn — November 6, 2009 @ 1:41 pm

  2. Hi Glenn,

    Thanks for the comment. You raise a very interesting and valid point. My intent in this article was *not* to say that agile is anti-architecture. My intent was to say that many people operate as if it is, and this is a shame.

    There’s a saying that goes, “In theory, theory and practice are the same. In practice, they are not.”

    I admit that the statement that agile projects often orphan software architecture is based on observations and anecdotal evidence. And I can’t quite tell from your comment whether you perceive me to be in the “agile is anti-architecture camp” or the “architecture and agile are necessary partners” (like Reese’s peanut butter and chocolate.”

    Personally, I believe that agile and software architecture need to co-exist. I certainly hope that this message was sufficiently clear in my article. My observation is that in spite of the second to last Agile Manifesto principle or the “System Metaphor” XP best practice, a large percentage of developers just don’t think and work that way. And I accept that your experience may tell you otherwise.

    Comment by charliealfred — November 6, 2009 @ 2:41 pm

  3. […] This post was mentioned on Twitter by kashyapa, Sathyam Karyala. Sathyam Karyala said: SCRUM and Architecture – Do they mix? http://is.gd/iJc9f. Use 10/90 rule. Great article! […]

    Pingback by Tweets that mention SCRUM and Architecture – Do they mix? « Charlie Alfred’s Weblog -- Topsy.com — December 14, 2010 @ 10:07 am

  4. Well….Scrum has more to do with Project Management Space . Progressive Planning
    Tracking and Controlling …while Architecture should be part Inception in Project
    pLANNING ..BEFORE GOING FOR SPRINT ….Inital Sprint may validate the Candidate
    Architecture .

    Sprint is NOT JUMP TO CODE BYPASSING DESIGN…

    Comment by Rajeev Pandey — September 13, 2011 @ 3:47 am

    • I’m not sure I completely agree with your premise that “architecture = before going to sprint”. Many important architecture decisions are made during development when unforseen situations and tradeoffs arise that cannot be resolved by local design decisions. In these cases, I believe it is still architecture’s responsibility to resolve system-wide tradeoffs.

      Charlie

      Comment by charliealfred — September 13, 2011 @ 7:57 am

      • I believe he meant that the initial (ideal) arquitecture design & documentation is meant to be made before going for a sprint; and of course, if the architecture was wrong/missed something/something arised during low level design or development, then an architectural decision must be made (a tradeoff), which of course would require participation of the architect and maybe modification of the architecture.

        Comment by Marcel Valdez — December 1, 2011 @ 11:36 pm

  5. Thanks, Marcel. I agree that an initial architecture should be formulated before beginning sprints. I think the theme of this thread is more about the software life cycle boundaries of SCRUM. As I understand Rajeev;s original post, and as you clarify, initial/ideal architecture falls outside SCRUM as part of project planning. I maintain that it should be part of SCRUM and the architect role is more of a pig than a chicken.

    I assert this because in my experience, when architecture is formulated separately from development, it is rare that a high percentage of the intent and rationale are communicated well and fully understood by developers. In fact, I would assert that very few things (including SCRUM itself) ever have their rationale and intent fully communicated and understood. This is inherent in the human condition – our ability to express ourselves or perceive others with clear fidelity. Some (often significant) distortion is always added somewhere in the communication channel.

    IMHO, when the architect and developers jointly participate in formulating the architecture as pigs in early sprints, there is a much higher likelihood that intent and rationale will be understood correctly. This occurs naturally as part of identifying and prioritizing challenges, assessing the pros and cons of alternatives, and choosing strategies and principles for the solution. When the developers have a clearer and deeper understanding of the architecture strategy, then they have a much more effective way to assess the impact on the architecture of story changes during sprints and when it is appropriate to reengage the architect (if not continuously involved) or when/how to make adjustments that are in the spirit of the initial/ideal architecture.

    I contrast this approach with the “formulate the architecture elsewhere and hand it off to the developers” model which is a holdover from the waterfall model. My view is that UML models, written documents, and Powerpoint presentations leak meaning like a sieve. As systems get more complex, the likelihood that the developers will make optimal choices of when to proceed independently vs. when to escalate an architectural refactoring discussion decays.

    Thanks for the comment,
    Charlie

    Comment by charliealfred — December 3, 2011 @ 1:45 pm

    • I understand what you’re trying to say, and I do know you’re way more experienced than I am. With that told, I ask you, what about “conceptual integrity”? A subject that Fred Brooks and the authors of Software Architecture in Practice 2nd Ed, confirm is a point that if sustained in a project is the difference between failure or success.
      Conceptual integrity, according to that literature, can only be achieved if the architect or achitecting team are the only ones that can make the end decision about any and all modification to the architecture, and the development team faithfully implement the architecture, going all the way to a common syntax and semantic analogy. I’ll cite them (Software Architecture in Practice 2nd Ed, 2005):
      (On definition and importance of conceptual integrity)
      “Conceptual integrity is the underlying theme or vision that unifies the design of the system at all levels. The architecture should do similar things in similar ways. Fred Brooks writes emphatically that a system’s conceptual integrity is of overriding importance, and that systems without it fail:
      I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas. [Brooks 75]
      Brooks was writing primarily about the way systems appear to their users, but the point is equally valid for the architectural layout. What Brooks’s idea of conceptual integrity does for the user, architectural integrity does for the other stakeholders, particularly developers and maintainers.”
      ….
      (On scope and detail of the architecture)
      “Correctness and completeness are essential for the architecture to allow for all of the system’s requirements and runtime resource constraints to be met. A formal evaluation, as prescribed in Part Three, is once again the architect’s best hope for a correct and complete architecture.
      Buildability allows the system to be completed by the available team in a timely manner and to be open to certain changes as development progresses. It refers to the ease of constructing a desired system and is achieved architecturally by paying careful attention to the decomposition into modules [¿Product Backlog?], judiciously assigning of those modules to development teams, and limiting the dependencies between the modules (and hence the teams). The goal is to maximize the parallelism that can occur in development.”

      “• The architecture should feature a small number of simple interaction patterns (see Chapter 5). That is, the system should do the same things in the same way throughout. This will aid in understandability, reduce development time, increase reliability, and enhance modifiability. It will also show conceptual integrity in the architecture, which, while not measurable, leads to smooth development.”

      (On project failure due to lack of conceptual integrity)
      “Unfortunately, there are no significant improvements we can make to the UCMEdit model. The system was not well designed in that the mapping from functionality to software structure is complex. This makes the abstraction of functionally coherent high-level elements within UCMEdit’s architecture impossible. However, we can take advantage of what we have learned to suggest improvements and to document what we know. The latter is especially important since we have discovered that UCMEdit lacks the conceptual integrity that often brings about intuitive understanding.”

      (On project success due to conceptual integrity and authority of architects)
      The architecture served as the foundation of the approach, both technically and culturally. In some sense, it became the tangible thing whose creation and instantiation were the ultimate goal. Because of its importance, the architecture was highly visible. A small, elite architecture team had the authority as well as the responsibility for it. As a consequence, the architecture achieved the “conceptual integrity” cited by [Brooks 95] as the key to any quality software venture.

      Because I know most of the projects where you are the architect have been a success, my question is how do you cope (in SCRUM) with all the dilemmas that in that literature are solved with the three factors: Architect-only Decision Making (with feedback possible, but finally the architect making the decisions), Architect authority and Conceptual Integrity, but are very hard to apply in SCRUM.

      By the way, I’m using your blog for a research in my Master’s Degree studies. Thank you for sharing.

      Comment by Marcel Valdez — December 4, 2011 @ 4:52 pm

      • Marcel, thank you for your comments and questions. Yes, you have my permission to cite any content on this blog. It is pubic domain. Best of luck on your Masters Project. What university are you attending? While it is true that many of my projects have been successful, the irony is that I have learned the most from the ones that failed to some extent!

        As Bass, Clements and Kazman, and Brooks before them, have correctly asserted, Conceptual Integrity is an essential characteristic of systems that exhibit long term success. I certainly agree. Christopher Alexander, the seminal force behind the Design Patterns movement (see https://charliealfred.wordpress.com/200/) has said that good architecture is recognizable by the presence of the “quality without a name.” This nameless quality represents a seemless fit between a building (or village or city), its use, and its surroundings. IMHO, conceptual integrity represents those aspects of a system which allow its elements to have the same harmony and enable the system to exhibit Alexander’s “quality without a name.” This represents the dual nature of architecture – it harmonizes what is within the system with what is outside.

        I think that there are three important things to realize about Conceptual Integrity:

        1. While Conceptual Integrity emerges from architecture, it is a property of the system. i.e. Conceptual Integrity in an “as-conceived’ architecture is only as effective as is appears in the “as-implemented” architecture.
        2. The antithesis of Conceptual Integrity is what Martin Fowler calls “Technical Debt”, which accumulates as a result of short-term compromises which sacrifice Conceptual Integrity for expediency.
        3. Architecture represents the earliest decisions which have the greatest impact on realizing a system’s quality attributes (according to Clements, Kazman, and Klien).
        4. By definition, architecture represents only a subset of the decisions made to create a system. Design and implementation account for the remainder.

        Combining these four statements, it is clear that Conceptual Integrity is only realized when the design and implementation are consistent with an architecture that has it. In other words, it is extremely difficult to recover Conceptual Integrity from an architecture that doesn’t provide it. And it is pretty easy to leak Conceptual Integrity from a system (architecture) that has it by making bad decisions.

        So, now to the issue Architect-only decision making. Architects are responsible for defining policies and rules that (if correct AND if followed) have a major effect on the quality of a system. In this light, architects can be viewed as legislators and judges in a democratic government. If the laws are good and citizens observe them, then society benefits. However, there are a lot of good laws that are broken daily. Consider laws regulating highway speed limits and prohibiting driving under the inflence laws or texting while driving. The architectural intent (safety) is good. Some drivers break the laws, and gain some satisfaction without causing any harm. Other drivers do the same, but get caught by the police. Other drivers get into accidents that damage propery and injure or kill people.

        I believe that this analogy holds for architects and developers. Architects do not make all of the design and implementation decisions, they just constrain them with policies and rules. If designers and developers understand and comply with the rules, the Conceptual Integrity present in a good architecture can be realized in the implementation. If they don’t, the Architect’s only recourse is to function as police and judge. This can be done in several ways:

        a) by participating as a member of the development team and interacting directly with developers,
        b) by evaluating the architectural impact of a significant change in requirements and adjusting the policies and rules,
        c) by answering questions asked by developers,
        d) by participating in design/implementation reviews,
        e) by investigating and correcting problems, after the fact

        I think these five remedies are fundamentally the same in an agile/SCRUM process as they are in a waterfall process.

        Finally, let’s consider the issue of Architect authority. If the architect has the respect and trust of the designers and developers, this occurs naturally. While this is the ideal way, it requires an architect who possesses a great deal of conceptual, domain, technical, communication and interpersonal skills. Yet, it does happen. It is also possible for an organization or project to mandate this. Yet as we saw above, passing and enforcing laws will never guarantee that people comply with them. The best you can do is to try to correct all of the violations that you can find.

        I hope this helps answer your question. I see Conceptual Integrity as critical, but don’t think that achieving it is any easier or more difficult using an agile process.

        Best Regards,
        Charlie

        Comment by charliealfred — December 4, 2011 @ 11:57 pm

      • I’m attending Instituto Tecnológico y de Estudios Superiores de Monterrey (ITESM) in México, and I’m currently studying for a Master’s Degree in Software and Information Technology, I started having this confusion because I’m my Software Architecture class we were told one thing and in my Software Methodologies and Disciplines class we were told the opposite. The project is for the latter class, and its about integrating Watts Humphrey’s Team Software Process/Personal Software Process (TSP/PSP) with SCRUM, I just delivered the project; and I wish I had found your blog earlier, because I couldn’t defend my posture (of not doing incremental requirements and architecture) due to lack of time and had to define the Software Development Process with incremental requirements elicitation and architecture definition, but had I found your blog earlier I would’ve been able to convince my teacher to be more flexible with the definition of the SW development process.

        I find it interesting (and feel a bit more sane, that I’m not alone in that position) that you also think that requirements and architecture cannot always (and in fact, less often than it is thought) be elicited incrementaly.

        I’m currently reading reading Gerald Weinberg’s “An Introduction to General Systems Thinking”, and plan to then read “Rethinking Systems Analysis & Design”, what other General Systems thinking books do you reccomend?

        You are officially on my must-read-entire-blog list now, I’m wondering if you have written any books or plan to write any, I have a feeling you have alot to teach, and I see you are very good at explaining yourself textually and understanding other’s writing too (even my poor writing and explanations). Finally, if I convinced one of the event organizers in my campus to invite you to give a conference on Systems Architecture and Agile Methodologies here, with enough prior planning, would you come? and what are your conditions.

        Thank you for your time, and lengthy and elegant responses. I’ve gained alot from them.

        Comment by Marcel Valdez — December 5, 2011 @ 1:11 am

      • Replied to your email

        Comment by charliealfred — December 6, 2011 @ 10:27 pm

    • By the way, someone proposed that the architecture should be created “in pieces”, for example: Have a product backlog with Login, Home, Reports, etc; then when the Login was chosen to be developed during a Sprint, the requirements and architecture for the Login should be created.

      I think that would lead to alot of rework, due to the fact that if you do that, you’ll be encountering issues that weren’t contemplated before and would require modifications to the architecture that could affect other (already built) parts of the system.

      Finally, if you do answer my question, I’d like to cite you.

      Thank you again, its great that you actually answer your blog comments, specially in such an important matter to me.

      Comment by Marcel Valdez — December 4, 2011 @ 5:01 pm

      • As an ardent follower of “systems thinking”, I try to be VERY careful considering incremental requirements and architecture. This is because virtually all complex systems are inherently populated with dependencies between elements, many of which are difficult to see. These interdependencies frequently trigger the Law of Unintended Consequences.

        Now having said this, incremental requirements and architecture do occur regularly: in multiple releases of the same system. On the surface, different sprints just appear to be a more frequent case of multiple releases. However, I don’t really buy this argument. The inherent time separation between multiple releases is much larger than the time separation between sprints. As a result, the ability to anticipate related functional requirements and architecture challenges is much better in the case of sprints.

        The incremental requirements and architecture approach is most effective when the capability under consideration is highly independent of others and the interfaces between this capability and its collaborators are stable and well-defined. As it turns out, these characteristics SEEM to occur more frequently than they actually do. To illustrate why, let’s consider one of your examples above: login.

        On the surface, authentication seems to be an ideal candidate for incremental architecture. The capability requires accepting the identity of a principal and a set of secure credentials, then determining if they match. The interface is straightforward and seems fairly stable. Implementation details seem to be able to be encapsulated behind the interface:

        a) the format of the credentials,
        b) the type of encryption used,
        c) how and where the authentication data is stored (and retrieved)
        d) how the authentication data is administered

        Yet, you are correct that thorny issues MAY be lurking.

        Suppose the application client wants to cache authentication state (e.g. principal, last login time) to enhance performance? The client handles an application request, extracts the principal ID (e.g. login name) and looks it up in its cache. If it doesn’t find a match, it calls the login service and adds an entry to the cache. If it does find a cache hit, then it calculates the time since last login and if it is less than some configurable time, it bypasses the login service and handles the request.

        One problem with this approach is that it doesn’t handle the case where login permission is revoked by the administrator. This could occur in a company where an employee resigns or is terminated. Unless the login API supports an asynchronous way for the login service to revoke credentials from the application, this scenario is a problem.

        Best Regards,
        Charlie

        Comment by charliealfred — December 5, 2011 @ 12:37 am

  6. […] SCRUM and Architecture – Do they mix? […]

    Pingback by The Role of a Software Architect in a Scrum Team | 1095 Days To Freedom — January 17, 2012 @ 11:16 am


RSS feed for comments on this post.

Create a free website or blog at WordPress.com.

%d bloggers like this: