Charlie Alfred’s Weblog

Complexity-Driven #1

This is the first part in a four part series on a topic I call complexity- driven development.  This part discusses why this topic matters.  The second part describes what complexity-driven development is and how to do it.  The third pard presents a case study which readers can use to practice applying this approach.  Finally, the fourth part is the author’s application of the approach described in part 2 to the case study described in part 3.

The Effect of Complexity on Process

 

By age 20, the young man was an accomplished climber.  He had already summitted several mountains in the United States, including Pike’s Peak, Mt. Whittier, and Mt. McKinley.  He dreamed about climbing to the summit of Mt. Everest, the world’s tallest peak at 29,012’ in elevation.  He aspired to reach the summit of the tallest mountain on each of the seven continents.

He spoke with the owner-operator of one of the tours that help mountaineers from all over the world to attempt to reach the summit of Mt. Everest.  The young man asked:

“What is the best way to reach the summit of Everest?”

The expedition guide replied,

“There are only two ways to the summit: via the Northeast Ridge or via the South face.  Our teams use the northeast approach.  In either case, the climbing strategies are limited.  Safety is a huge constraining factor.  There are enormous risks.”

“Dozens of people have perished on Mt. Everest.  Some have fallen into crevices that are 3,000 feet deep, others have been buried by avalanches, others have become weak and disoriented by the lack of oxygen above 26,000 feet (30% of the oxygen at sea level), others have been trapped in surprise storms, or reached the summit too late in the day, and have gotten lost in the darkness on their descent.”

We start at the common base camp at 16,000’.  Then when the weather forecast is storm-free, we climb to the advanced base camp at 25,000 and stay overnight to acclimate to the oxygen.  We leave the advance base camp at around midnight, in parties of 2-3,  including a Sherpa guide.

We try to leave early enough to avoid congestion with other climbing parties.  There are many places in the ascent where climbing is single file, and a slow climber can cause significant delays.  This is a big problem, because climbing is slow at this altitude, and we need enough time to climb from the advanced base camp to the summit, and get back safely before dark.

Inexperienced climbers often don’t realize it, but the descent to camp is more risky than the climb to the summit.  A number of people have successfully reached the summit, only to perish on the way down!”

The same young man was at a party where he had the opportunity to meet the head coach of a successful NFL team.  He asked the coach, “what is the best game plan for winning a big football game?

The young man thought the coach would say how important it was to have an All-Pro quarterback, or two strong wide receivers, or a dominating pass rush.  But instead, the coach answered,

“Football is a 60 minute game of 11 men against 11 men.  You have to find the places where you match up well against your opponent, and find ways to exploit those.  And you have to find the places where your opponent matches up well against you and protect those.

You have to make the fewer mistakes, and you have to avoid giving up the big play.  But most importantly, you have to adjust to the situation at hand.  You play tighter defense on third down, because you have a chance to get your defense off the field.  The risk of a big play is still there, but the reward is higher.” 

What is the Relevance to Software Development?

Suppose you were to ask a hundred software developers “what is the best process for developing software?  I fully expect that 80% would respond that agile methods are by far the best approach.  Some would prefer XP, others would prefer SCRUM, but most would say that software development is full of unknowns, and short iterations are the only effective ways to navigate them.

In 2003, IEEE Computer published an excellent debate between Barry Boehm and Kent Beck, titled “Agility Though Discipline, A Debate”, (http://csse.usc.edu/csse/TECHRPTS/2003/usccse2003-518/usccse2003-518.pdf).  The gist of the article was whether an “agile approach to software” was better than a “planned approach”, or perhaps some mixture was the best.  In his rebuttal, Kent said:

“Once there was a dinosaur and a furry little mammal, a bit like a mouse. Then an ice age began. The dinosaur couldn’t regulate its internal temperature, and so it died. The mouse, with its more adaptable metabolism, survived.

The dinosaur and the mouse weren’t in competition. They ate different things, lived in different places. Certainly, the passing of the dinosaur opened up many opportunities for the furry little creature to thrive, but the passing of dinosaur had nothing to do with the mouse. If it hadn’t been a mouse taking over, it would have been cockroaches.

The agilists are reacting to the business, technology, and social environment as we see it. If your climate matches the conditions under which Barry Boehm’s advice works, well, by all means, follow it. If it’s getting chilly, though, think about what else might work better.”

I firmly believe that the most important advice in this passage is the phrase “If your climate matches the conditions under which…”.

In the introduction, we have a young mountaineer asking what is the best way to climb Mt. Everest.  There are lots of ways to climb, if you don’t care to reach the summit, and there are several ways to summit, if you don’t care to live to tell anyone else about the experience.  The hostile climate and the peril combine to eliminate most of the degrees of freedom from the solution space. 

Later, the same young man asked a successful NFL head coach what was the best game plan for a big game.  The answer was not as specific as the one for climbing Everest.  There are more ways to win an NFL football game than there are ways to climb Mt. Everest successfully.  However, there were important similarities: exploit matchups, avoid mistakes, avoid the big play, and deal with the situation (down, distance, yardline, score, and time remaining).

If your climate matches the conditions under which…”

There are two ways to deal with climate: prepare for the climate you expect, and adjust to the climate you get.  Both are necessary, and neither is more important than the other.

Effectiveness is doing the right things, and doing these things well.

One Size Does Not Fit All 

Goldilocks said, “This bed is too hard.  This bed is too soft.  But this bed is just right.”  She made similar observations about the firmness of the chairs and the temperature the porridge.

Even at a young age, Golidlocks understood the importance of properly fitting an approach to the problem at hand.  If Goldilocks was a software architect, she would know that the question is not:

o      Should I use waterfall or should I use agile?

o      Should I document my architecture and design approaches, including their rationale, or should I just annotate the code and let the generated documentation tell the story?

o      Should I design for flexibility or should I keep it simple, and refactor later on when its necessary?

o      Should I create an architecture or should I let it evolve?

o      Should I integrate and test continuously, or should I integrate and test periodically?

These questions simply cannot be answered in the absolute, for every project.  One size does not fit all.  Software project teams have a wide range of tools and techniques at their disposal, and the goal is to choose and execute the ones that best fit the problem that they face.

Complexity-Driven Development is a Meta Process

At this point, the obvious question is:

 “If it is true that one-size doesn’t fit all, then why bother talking about complexity-driven development?  Isn’t this just another process?”

The answer is that complexity-driven development is not a development process for projects to follow.  Rather, it is a meta-process, a process to find the most suitable process for the problem at hand.  Complexity-driven development does not tell you what to do, it helps you determine where to look in order to find the most significant challenges.

Complexity-driven developmnent has three fundamental premises, which appear to be true for all projects:

1.     Any systems development project today, that creates value, has complexity to overcome.

2.    Challenges occur when complex areas have consequences (i.e. they alter the outcome in a way that changes value).

3.    Focusing on overcoming the dominant complexities (a.k.a. the most significant challenges in the problem space) is on the critical path to success.

Once you discover and assess the dominant complexities for a problem, the creative part begins.  This is where you reach into your bag of tools and tricks, and pull out the things that will help you overcome these challenges.

Golfers on the P.G.A. Tour carry 14 clubs in their bag.  Excluding the putter, they might have 4 or 5  ways to hit each club: high, low, left-to-right, right-to-left, or even backhanded.  During an 18 hole round, they might not hit every club in their bag.  But eventually, the shot to be played and the conditions will call for every shot they have practiced, and a few more that they need to invent.

Software developers also have a large variety of tools and techniques at their disposal.  Design patterns, visual modeling tools, code generation tools, requirements traceability tools, automated test tools, static code analyzers, and profilers are some of them.  So are 4-week interations, feature (or story) backlogs, pair programming, daily standup meetings, and test-driven development.  And so are documented architectures, product release roadmaps, project cost estimates, and schedules.

There is a time and a place for each of these mechanisms.  And just like a P.G.A. professional is more likely to use certain clubs (driver, wedge, and putter) than others, some of these things occur in a high percentage of projects.  Frequency is not the right metric here, efficacy is.

The following two articles will continue to develop the theme of complexity-driven development.  The next one will describe the “what” of the approach – what to do and what things to focus on.  The third article will use a real-world example to illustrate “how.” 

 

 

Advertisements

3 Comments

  1. […] Complexity-Driven #1 […]

    Pingback by New Articles on Complexity-Driven Development « Charlie Alfred’s Weblog — December 8, 2008 @ 8:37 pm

  2. Your approach is really great, I especially loved the grouping of the process issues with the technical tools. The idea of any process not being a silver bullet is long being discussed, thus choosing one for the current conditions is not a hidden knowledge, but using some kind of framework to choose one is a great idea, a process to combine whole processes.

    Comment by Alp Timurhan Çevik — May 10, 2009 @ 6:41 pm

    • Hi Alp,

      Thanks for taking a look at this article. I have a strong opinion that the single biggest challenge facing software development today is rapidly increasing complexity:

      1. Problems are getting more complex; as the easier ones get solved, harder ones take their place
      2. Technologies, languages, and platforms used to solve problems are more complex
      3. We have an information explosion, which forces people to specialize more and more
      4. As people specialize, domains become narrower and deeper, making communication/understanding harder

      Complexity is the sort of problem that must be met head-on. It will not go away on its own, and will not be solved as a side-effect of some other decision. In my view, you have to start by understanding:

      o where are the places your system will be used (contexts)? how are the key conditions similar or different? what are the perspectives of your stakeholders (value, pain points, priorities)?

      o what are the most significant challenges (obstacles and risks that impact value) in each context, and how are they prioritized?

      o are the prioritized challenges by context consistent or different? For example, if a car buyer wants a sports car and another wants a pickup truck, these clearly are two very different contexts.

      Prioritized challenges by context is the only effective way I know to contain and partition complexity. I would love to hear about others, though 🙂

      Besr Regards,
      Charlie

      Comment by charliealfred — May 11, 2009 @ 12:20 pm


RSS feed for comments on this post. TrackBack URI

Blog at WordPress.com.

%d bloggers like this: