Charlie Alfred’s Weblog

Conceptual Distance

Conceptual Distance Illustrated

Consider the following situation.  A native of Shaghai, who speaks several dialects of Chinese fluently, is at the 2008 Olympics and is trying to ask directions from a person from Bejing.  The Bejing native knows the city well, but the Shaghai native does not.  The directions given by the Bejing native contain references to streets and landmarks that the Shanghai native doesn’t understand.  This is a simple example of conceptual distance – in one dimension.  Fluency in spoken Chinese is not an obstacle, but expertise in the geography of Bejing is.

Now, let’s alter the situation a bit.  An American person, who speaks no Chinese, is trying to ask directions from a Bejing native who speaks a tiny bit of English.  This is not sufficient to get the American to their destination.  Conceptual distance now constrains the situation in two dimensions: language fluency and geographical knowledge.

Next, imagine that a person (say, from Japan) who speaks fluently in both English and Chinese enters the scene.  Also imagine that the Japanese person knows little about Bejing geography.  This creates a situation that is the equivalent of the first.  The Japanese person helps the American and Chinese people to understand each other’s words, but is unable to add much value to the knowledge gap that exists about Bejing geography.

This relatively simple example illustrates the essence of the conceptual distance problem.  In practical terms, conceptual distance is about the level of difficulty of comprehending communication in one or more related subject matters.  It is partly about communication (expressing and receiving), partly about complexity of subject matters, and partly about proficiency (of the parties to the conversation).  It occurs in language translation, it occurs in discussions about specialized topics, and it occurs in spades in software system design.

As we will see, conceptual distance tends to increase as subject matters get more complex.  It also tends to increase as a problem includes more subject matters, and these subject matters interact with each other.  As subject matters grow in complexity, people often tend to do one of two things.  The first reaction is to specialize – acquire in depth knowledge in a narrow area.  The second is to generalize – acquire more shallow knowledge across a broad area.  Neither strategy truly addresses conceptual distance.  There is a joke (I wish I knew who to attribe it to) that highlights this:

“Do you know what the difference is between an engineer and a salesperson?”

“An engineer spends their entire life learning more and more about less and less, until eventually they know everything about nothing.”

“While a sales person spends their entire life learning less and less about more and more, until eventually they know nothing about everything.” 

Conceptual Distance Defined

Linguistics has a term called “semantic distance”.  It is a measure of how closely related two words in a language are.  “May” and “might” have a much shorter conceptual distance than “college” and “geriatric”.

If semantic distance is a measure of how closely related two words are, then conceptual distance can be seen as an assessment of how closely related two concepts are.  In other words, if 100 individuals understand X very well, what percentage of them will also understand Y, and vice versa.  To illustrate, consider the following concept pairs:

Concept X

Concept Y

Conceptual Distance

Pre-Compiled Query

Integrity Constraint

Low

Relational View

Stored Procedure

Medium-Low

Outer Join

XSLT Transform

Medium

Database Trigger

Ajax or Flex

Medium

  • Both concepts in the first row are related to relational database management systems (RDBMS).  Since both concepts are relatively elementary, it is reasonable to expect that somebody who understands one will also understand the other.  Further, it is reasonable to expect that somebody who knows a little bit about an RDBMS is pretty likely to understand both (or learn them quickly).
  • Both concepts in the second row are also related to an RDBMS.  The difference is that the two concepts are slightly more advanced.  Views are powerful, but simple, data hiding techniques for queries.  However, their use in insert and update operations is much more complex.  Similarly, stored procedures are a simple abstraction, but require an in-depth understanding of when to use them and how to design them effectively.
  • The concepts in the third and fourth rows cross software disciplines.  Some database developers also understand XML/XSLT.  Other DB developers understand web-based user interface technologies.  Some cross over all three.  However, some very good database developers might only have a surface understanding of Javascript, XML, or XSL transforms.

Next, let’s repeat this experiment, but this time associate software with non-software disciplines:

Concept X

Concept Y

Conceptual Distance

Thread Priority Inversion

Vacuum Chamber Control

Moderately-High

Stored Procedures

Radiology

High

Interface Inheritance

Japanese Linguistics

High

XSD Target Namespace

Credit Default Swap

High

  • In the first row, thread priority inversion, and how to avoid it, is a relatively advanced topic in realtime operating systems.  Vacuum chamber control is a physics problem, involving relatively advanced electro-mechanical control of actuators/sensors for pumps and vents.  While some experts understand both concepts, this understanding required a significant leasrning investment in both domains.
  • In the last three rows, the subject matter in column Y is far removed from software.  Someone who has become expert in Radiology, Linguists, or Swap & Derivative markets is unlikely to have more than a passing knowledge about software.  Similarly, someone who has made their career in software development is unlikely to have more than a passing knowledge in radiology, linguistics, or financial instruments (I can personally attest to this).

Why is conceptual distance important?

There are four basic reasons why conceptual distance is important. 

1.     The first reason why conceptual distance is important is that systems are becoming significantly more complex.  In other words, the easy ones have already been built.  Nobody I know is spending much time writing text editors, email readers, or file transfer programs.  Yet, it wasn’t too long ago when many people were focusing a significant amount of effort on exactly these types of things.

2.    The second reason is that as system development efforts become more complex, specialization increases.  In other words, when expertise is demanded, experts must be supplied.  When time is a major constraint and the needed expertise broadens, specialization is a natural response.  As an illustration, consider the rise of specialization over the past 30 years in medicine, computer systems, telecommunications, and many other areas.

In the computer software business, expertise frequently is required from many areas, including:

  • Market experts, who study consumer buying behavior (for products), or business processes (for IT systems) in order to increase adoption and satisfaction rates.
  • Domain experts, who understand deeply the nature of the problem being solved, how things might change, and which approaches have the best chance of being successful.
  • Compliance experts, to advise on legal and regulatory constraints and specify what actions are necessary to comply.
  • Systems experts, who understand systems, both automated and people-based, and understand how they work, how they fail, and how they change over both time and space.
  • Project managers, who know how to coordinate the efforts of many people, and understand work breakdown, structures, scheduling dependencies, risk management, communication and team dynamics.
  • Technology experts, who understand the intricacies of databases, networks, user interfaces, algorithms and a vast array of technologies, computing platforms, and programming languages.

It is important to note that these effects are not limited to computing systems.  A similar phenomenon occurs in many other areas where complex systems are deployed, such as health care, mechanical systems, telecommunications, air travel, agriculture, manufacturing, and others.

3.    Encapsulation (information hiding) is the third reason why conceptual distance is important.  Each area of specialization exposes a thin interface that describes what it can do, and hides the expertise that is needed to determine how best to do it.  This often is an effective technique to permit different subject matters to collaborate without requiring each to be expert in the other.

However, in reality encapsulation is a lever.  Most times it magnifies power in our favor, but sometimes it doesn’t.  As much as we try to contain it, complexity has a way of leaking out.  Anyone who has debugged an exception condition deep inside of a component understands how encapsulation can be a fickle ally.  Parts of a system have inter-dependencies and often have conflicting interactions.  Furthermore, the outcome might depend on contextual factors. 

4.    Scalability of Knowledge Transfer.  So, the level of expertise required in many subject matters is expanding.  The number of subject matters is expanding. And encapsulation is not able to hide significant portions of the complexity between inter-dependent subject matters.  Sounds gloomy, doesn’t it?

When conceptual distance is small and not changing rapidly, old time-proven techniques work pretty well.  Establish goals, identify the key obstacles and risks, determine the architectural approaches, define the requirements, organize the work, etc.

However, in many cases, deep expertise is required across several domains, and these areas are evolving and/or interdependent.  This is where scalability of knowledge transfer comes into play.  Enough individual expertise to cover all important subject matters is necessary, but not sufficient.  A high-bandwidth way to communicate expertise between subject matter experts and have it be understood at the receiving end is of utmost importance!

Dimensions of conceptual distance

Conceptual distance appears to have three important dimensions. 

  • First, are the two concepts in the same subject, in related subjects, or in far-removed subjects?  Databases are closer to Web Services or User Interfaces than any of these are to Radiology or Linguistics.
  • Second, how basic or advanced is a concept within its domain?  Time value of money is a simple concept in finance.  The pricing and risk management issues of credit default swaps is much more advanced.
  • Third, how much is already known about each subject (by the people who need to make decisions about it).  To further magnify this, do these people know what they don’t know or are they blissfully unaware?

The following diagram illustrates four key subject matter areas in an IT application for developing new content and some cross-dependencies between these areas.

 

The following table shows how conceptual distance varies significantly depending on how context changes affect the individual subject matters.

 

Example

Conceptual Distance

Description

3.2.1 maintenance release for a software system

Low

Bug fixes improve quality but don’t change the fundamental value proposition.  Tasks are straightforward and workload is balanced.  Most bugs have simple fixes.  A few require tricky or widespread changes.

2.0 release of an existing software system

Medium

Significant improvements to the value proposition.  Some capabilities fit into the 1.0 architecture, others need significant refactoring.  Some new technologies must be integrated and pose notable risks.

1.0 release of a multi-system framework

High

Several existing products are expected to be integrated into a new, common framework to increase interoperability and reuse.  This effort has technical, product management, and project complexity.

Interestingly, there are many cases where the problem domain is a very close cousin of the subject domain.  For example, Interactive Development Environments, such as Eclipse, NetBeans, and Microsoft Visual Studio are designed by developers for use by developers.  The same is true for programming language environments, such as Ruby on Rails, Python, J2EE, and .NET, or open source software tools environments, such as Apache and Source Forge.  In each case, the system creators are delivering technology to a market that they can relate to directly.  This is a powerful, but accidental way to lower conceptual distance).

In summary, conceptual distance is important because:

  • The simpler problems have already been solved,
  • Deeper levels of expertise from more subject matters are needed to solve the more complex problems.
  • Encapsulation helps contain complexity, but cross-dependencies between subject matters can trump information hiding.
  • Scalability of knowledge transfer requires a new way for groups of subject matter experts to think about communication and comprehension.

Conceptual distance and complexity

Imagine two beams about 2 feet apart, that extend out into the distance.  The beams appear to be parallel, but in reality are about one degree off.  As you walk forward, one foot on each beam, you cannot sense that they are gradually getting further apart.  After 50 feet, they are about 10” further apart.  However, by 200 feet they are 3.5 feet further apart!  Conceptual distance is similar.  It sneaks up on you, and then when you can least afford it, it attacks.

As Frederick Brooks aptly observed in No Silver Bullet, there are two types of complexity: essential and accidental.  Essential complexity is part of the problem domain.  It is what is is.  You can’t reduce it or eliminate it.  Accidental complexity is what is added by the solution, as design decisions are made and code is written.

Conceptual distance suggests that essential and accidental complexity represent one important dimension.  The other dimension is visible vs. transparent vs invisible complexity, as illustrated below:

Visible complexity is complexity that can be recognized, communicated and understood by whomever it affects.  It occurs when conceptual distance is small – where one expert recognizes the potential for a problem and is able to explain “what” and “why” so that others can understand both the immediate and potential impact.  This implies three critical things that all must be present:

  • The subject matter expert who detects the potential problem must know enough about the other subject matters to recognize whether this is something that is significant
  • The subject matter expert must know enough about the other subject matters to be able to express his concerns
  • The other subject matter experts must know enough about the subject with the potential problem to understand its implications on their areas.

Transparent complexity is complexity that is noticed (or noticeable), but is not acted on, because it is not communicated or understood properly.  The Challenger shuttle disaster is a well-known case.  The Wikipedia page:  http://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster says:

NASA managers had known that contractor Morton Thiokol’s design of the SRBs contained a potentially catastrophic flaw in the O-rings since 1977, but they failed to address it properly. They also ignored warnings from engineers about the dangers of launching on such a cold day and had failed to adequately report these technical concerns to their superiors.

In this case, NASA managers did not understand the risks in the O-ring design and the impact of cold weather on these risks.  At the same time, the engineers who were aware of the risks, did not communicate them in a way that made the risk impossible to ignore.

Invisible complexity is unknowable, at least given the current state of our knowledge.  Predicting the timing and magnitide of a natural disaster falls into this category.  For example, most investment firms treat U.S. Treasury Bills as “risk-free” investments.  The argument is that the U.S. Government can always implement taxes to pay its debt obligations, therefore it will never default.  The use of “always” and “never” in this sentence is a bit worrisome.

Shrinking conceptual distance

As conceptual distance expands, misunderstandings, risk, and rework increase.  Quality declines, and the time and resources needed to design and implement a new system skyrockets.  In short, as complexity grows, our ability to deal with it doesn’t seem to keep pace.

On the surface, the problem seems to be caused by our ability to absorb and comprehend information.  Over the past century, our bodies of knowledge have exploded in both number and size.  Fifty years ago, a high school diploma was enough to qualify someone for a decent job.  Today, a bachelor’s degree might not be sufficient.

Clearly, availability and accessibility of information is not the problem.  Internet search engines put vast quantities of information at our fingertips in seconds.  No, rate of absorbption is the limiting factor.  When 10 inches of rain fall in 5 hours, some of it is absorbed into the ground, but most of it runs off and floods low-lying areas.

All subject matter experts must be strong in each of the following skills:

  • Modeling.  UML class diagrams, activity diagrams, and finite state models, are not just for software.  These are very useful techniques for communicating types, relationships, collaboration, process flow and state dependent behavior.
  • Synthesis.  Subject matter experts need to be able to begin at a system boundary and look outward to discover its purpose, contexts, and forces.  The ability to communicate why is essential to determining the appropriate what and how.
  • System statics.  Subject matter experts must be able to quicly and clearly communicate how any system is organized and how it operates.  Here, we go beyond the techniques discussed under Modeling, and focus on guiding principles of organization and operation.
  • System dynamics.  All interesting systems change in both space and time.  Subject matter experts must be able to explain how feedback loops change the forces that affect a system.  They also must be able to identify and assess external contexts and communicate how common or variable their impact is.
  • Risk Analysis.  Development of any sort is a forward looking activity, and is subject to uncertainty.  Risk is what happens when uncertainty collides with outcome.  Risk analysis involves estimating the probabilities of a set of uncertain events, and the impact on something valued of each event (positive or negative).
  • Communication.  None of the five skills mentioned above are truly useful to a group of subject matter experts, unless:
    • Each individual can communicate, by writing or speaking, in a way that enables others to comprehend the complexities.
    • More importantly, each individual is open to receiving information from others, by listening or reading, and is capable of comprehending the complexities in their message(s).
  • Self-awareness.  It is essential for each subject matter expert to knowing what they don’t know and engage somebody who does.
  • Commitment to Reducing Conceptual Distance.  Finally, subject matter experts must realize that success is a team effort, not an individual showcase.  This requires more than just cooperation.  This requires each expert to take time to describe how their subject matter related to the “big picture”: goals, opportunities, obstacles, risks, and tradeoffs.  It requires an active commitment to contribute to the “collective body of knowledge.” 

 

 

 

 

 

 

 

 

 

 

 

1 Comment

  1. Awesome story, I didn’t thought this was going to be so great when I looked at the link!!

    Comment by Authekeep — January 2, 2010 @ 1:19 am


RSS feed for comments on this post. TrackBack URI

Blog at WordPress.com.

%d bloggers like this: