Complexity-Driven #2

In the previous topic, we discussed the effect of complexity on process.  The more complex an environment is, and the more severe the risks and rewards, the fewer degrees of freedom there are in the solution.

There are more ways to cook a tasty chicken dish than to reach the summit of Mt. Everest and return safely.  There are more ways to design an Instant Messaging system than there are to design a jet engine.

As George Orwell wrote in Animal Farm, “All animals are equal, but some are more equal than others.”  A similar statement can be made about software development.  All software problems have complexity, but some are more complex than others.   The less complex problems have been solved already, or might not be worth solving.

What is Complexity-Driven Development?

Well, complexity-driven development is pretty much what it sounds like.  You start by discovering the most significant complexities you face, then you assess their consequences, then you come up with a strategy to address them, starting with the most dominant ones.

Sounds pretty simple, right?  That would be nice, but it isn’t real.  So what are the obstacles?  I’ll begin with seven, and the list is incomplete:

1.    Every software development project is a “system of systems”, as illustrated in the following diagram.  The deployment and development environments are systems of their own.  Within each system, processes, organization/work management, and technology systems exist.  Also, the interaction effects are high.

2.     We have incomplete knowledge about the problem.  After 30 years in the business, I have yet to work on a significant project where I didn’t understand substantially more about the problem when it was done than I understood 3 months into it.

3.    The problem usually changes over time.  Sometimes this change is gradual, sometimes abrupt.  But even when it is gradual, often it is difficult to detect that it is happening.  Toyota’s erosion of GM’s market share in the 1970’s was gradual, but it flew under the radar.  By the time the change was noticed, it was abrupt.

4.    When the problem doesn’t change over time, it changes in space.  Somebody designs a cell phone.  It’s difficult enough to get good reception in a lot of places.  Then somebody decides they need text messaging and a camera (or a video camera) in their cell phone.  Then the government decides that cell phones need GPS receivers for 911 calls, and somebody decides that it’s pointless to have a Garmin or Tom Tom, if their phone can give directions.  Then cell phones need to play music and pickup satellite TV signals.

5.    We complicate our incomplete knowledge with conceptual distance issues.  Every project contains many different subject matters.  What complicates this is the spread between the maximum and the average level of knowledge:

§ A given subject matter expert has a much greater mastery of their area than the average mastery of other team members

§ A given subject matter expert has a much greater mastery of their area than they do about the other subject matters.

6.    Next, there is the age old battle of “good, fast, cheap” pick any two.  Test-driven development has helped relax the constraints on this equation, as the earlier bugs are found and fixed, the cheaper it is to fix them, and the faster things go.  But even when you have squeezed out the inefficiencies, the tradeoffs remain.  Consider refactoring vs. architectural stability.  The former incurs an expense when an unexpected change occurs.  The latter requires some up-front effort for situations that may not materialize.  The debate over which is better resembles the debate over whether a convertible is better than a hardtop.

7.    Projects have many stakeholders, they are measured on different things, and sometimes care more about their local value model than the global one.  This point is a generalization of the previous bullet.  However, the organizational politics of software development transforms good, fast, and cheap into “all things are easy for the person(s) who don’t actually have to do the work.”

Discovering Complexity

It turns out that complexity isn’t all that tough to discover, as long as you (plural) actually understand what’s going on, and have an open mind about things.  Humans seem to have a pretty good instinct for complexity.  If it hurts your brain to think about, it’s probably complex.   If it makes your stomach queasy to think about it, there probably is a personal consequence for an unacceptable outcome.  If it hurts your head and makes your stomach queasy, then you know you’ve found it.

Unfortunately, instinct has at least three dangerous foes:

Naivety – is the state where someone doesn’t know something, and is blissfully unaware that they don’t know it (Pointy-Haired Boss).

Stubborn – is the state where someone thinks they know something and refuses to listen to advice that is counter to their belief.

Delusional – is the state where someone actually knows something, then persuades themselves that the logical outcome won’t occur.

As a result, it certainly would be handy to have a more systematic way to discover complexity.  Instinct is great, but it is even more powerful when combined with a little rigor.

A Systematic Way to Discover Complexity

Crime Scene Investigators have a technique called “walking the grid” which they use to collect evidence at a crime scene.  The purpose of this grid is to help ensure that they don’t overlook something important.

I’m going to propose a technique for discovering complexity in a system that I call SDC.  Depending on my frame of mind, this acronym either stands for Sources, Drivers, and Categories or Seeking Dominant Complexities.  For now we will use the first.

Sources – are the areas of activity which give rise to complexity.  There are eight that stand out:

1.     Organization and Policies in the Deployment Environment

2.    Technology(ies) currently used in the Deployment Environment

3.    Work Management / Processes in the Deployment Environment

4.    Interaction effects in the Deployment Environment

5.    Organization and Policies in the Development Environment

6.    Technology(ies) currently used in the Development Environment

7.    Work Management / Processes in the Development Environment

8.    Interaction effects in the Development Environment

Drivers – are the factors that cause complexity.  Four are particularly important:

1.     Constraints (physical, regulatory, contractual, policy, etc.)

2.    Uncertainty and unknowns

3.    Inherent Dependencies (X requires Y) and Interdependencies (X and Y need each other)

4.    Interaction effects among the other three drivers

Categories – categories are discrete areas where you can look to identity and discover complexities.  The figure below illustrates categories of complexity in the technology arena:

 

1.     Functional complexity is the simplest to understand.  The system is designed to offer certain features that perform tasks for the user.  Functional complexity indicates how complex this set is.  A 747 cockpit has high functional complexity.  A flashlight is low.

2.    Informational complexity represents the inherent complexity of data or object models, typically represented by database schemas, XML schemas for messages, or programmatic object models.

3.    Control complexity represents the difficulty of coordinating or orchestrating the interactions of components of the system during execution, and the controllable environment so that the system achieves its goals.  Realtime control systems for air traffic control, semiconductor fabrication, subway operation, and beer brewing are a few examples of systems with high control complexity. 

4.    Algorithmic complexity represents the difficulty of performing computations necessary to achieve a precise result.  Routing and scheduling systems, wireless network receivers, digital signal processing and portfolio risk management are systems with high algorithmic complexity.

5.    Contextual complexity exists when a system is used in very different ways, or creates different types of value for its stakeholders depending on the particular context or situation.  Context refers to a particular set of environmental factors or intended uses that alter the nature of the problem that the system must address.  Consider a Sports Utility Vehicle (SUV).  Some buyers want to drive off road over hilly and rocky terrain.  Others buy it for its size and storage capacity.  Others buy it for its four-wheel drive and handling in snowy or icy conditions.  Satisfying all of these users is quite difficult, due to the inherent tradeoffs.

Situational complexity is very similar to contextual complexity, except that instead of variation in space (place to place), the variation is sporadic, in time or location.  A web-enabled software product that continues to function without a network connection handles a form of situational complexity.

One of the important things to notice about contextual/situational complexity in the above diagram is that it spans the previous four types.  What this means is that variation can occur in each of the types: functional, informational, control, and algorithmic.

6.    Evolutionary complexity is uncertainty and variation over time.  Cell phones are a recent poster child for evolutionary complexity.  Devices that started out with the goal of providing mobile voice communications, rapidly picked up photography, video, music, TV, GPS, personal information management, and web browsing.  To make matters more complex, cell phones also have significant context-sensitive variation.

Evolutionary complexity is what agile methods tout that they manage effectively  To a large degree, they do.  Short development iterations provide opportunities to shift priorities or change direction.  Also, refactoring enables developers to repair the ravages of time on system architecture without changing the feature set.

But, refactoring alone may not be sufficient in many scenarios.  As the scale of the system grows, the complexity that is inherent in the problem spreads its tenticles and creates coupling in unexpected places.  Software is the utimate game of Twister, and just when you can least afford it, “Right foot red” is called, and you fall to the ground with somebody else sprawled across your chest.

As the above diagram shows, evolutionary complexity spans the five previous types.  Functional, informational, control, and algorithmic complexies can all evolve, but so can contextual and situational complexity.

7.    Systemic complexity is the interaction of two or more types of complexity.  An auto pilot system on an aircraft has significant functional and control complexities.   If these complexities interact in a way that amplifies complexity, then it would have systemic complexity.

Adapting Categories for Organizational & Work Management

The category descriptions provided above are appropriate for technology complexity.  A few adjustments need to be made when considering organizational and work management complexity.  In people-intensive systems:

o   Control complexity takes on a slightly different meaning.  Here, it deals with the issues of governance and coordination, instead.  Same basic idea, different execution.

o  Algorithmic complexity has marginal applicability.  A few groups (like NFL defenses) need to make realtime assessments of an evolving situation and make complimentary adjustments.  Instead, for organizational and work management sources, a more relevant category is the need for learning.

A Method for Assessing Complexity

This section outlines an approach for assessing complexity.  The essence of the approach is to identify the consequences of complexity, assess the relative significance of these consequences, and use this information to rank the complexities from most to least dominant.

1.     Accurately define the scope of your system.  Failing to do this will activate the 8^th and most fatal form of complexity – confusion.

2.    Next, identify each of the discrete subject matters that affect success in your system.  This step is essential, because without it, naivety will embed itself in some part of your problem and attack you in your sleep, visiously and mercilessly.

3.    Next, consider each complexity source (process, technology, and work management for both deployment and development environments) one at a time, and record all complexities that seem significant:

a.    Evaluate each of the complexity categories (with adjustments for organizational and work management sources, as discussed above).

b.    For each category, consider each of the complexity drivers: constraints, uncertainty, and interdependency.

4.    For each complexity identified in step 3, attempt to identify one or more consequences that might be suffered if the complexity is not managed properly.

5.    Assess each consequence in terms of its impact on the project.  Estimate an optimistic, likely, and pessimistic impact, then use the PERT formula to weight them: W =  (O + 4*L + P)/6.

6.    Prioritize the consequences according to the weighted impact calculated in step 5.

7.    Assess each complexity identified in step 3 against the consequences from step 6.  For each complexity

a.    Rate the impact of the complexity on each consequence, using a scale of 0 (none) to 4 (major).

b.    Multiply the impact rating times the priority of the complexity and sum the values for each complexity.

8.    Rank the complexities according to its impact on the set of consequences.

Architecture’s Role in Addressing Dominant Complexities

Dominant complexities are a synonym for architecture challenges.  If I can identify the complexities that have the greatest impact on the value created by my system, then I am in excellent position to figure out how best to address them.

At its core, architecture is about making the most effective early design decisions about a system.  Each early decision specifies constraints that are binding on subsequent decisions.  If I can be sure that my earliest decisions are intended to handle my most significant challenges, then I stand a better chance of getting the result I seek.

One important thing to note is that architectural decisions have side effects.  When I make a decision to solve one problem, I create constraints that limit my degrees of freedom on another.  Sometimes, I can “look ahead” and anticipate this.  Other times, I don’t discover the impact until I work my way down to those lower priority challenges.  When I do notice this effect, it makes sense to do some architectural refactoring: revisit the earlier decision in light of its impact on later challenges.  Perhaps there is an alternate approach that provides a better tradeoff.

Conclusion

It is very important to realize that architecture, as used here, addresses a much wider spectrum than just technology.  The challenges that we identify, span deployment and development environment, and cover organizational, technology, and work management sources.  In short, these challenges include every area that significantly affects the success of the project.  If the challenges have this sort of coverage, and the approaches do a good job of addressing them, then we’ve done as much preparation as is possible.

It is important to note that this type of planning is not prohibitively expensive.  Assuming that experts in all significant subject matters are available, the process should should take no more than 6-8 weeks to complete.  This is the 90-10 rule applied to planning: focus your attention on the 10% of the issues that dictate 90% of the results.