Story at many levels

Book section 11.1

Invariably, large systems will have lots of interacting parts. Unless great care was taken in their design, they will be hard to understand. For example, if you look at old machines, like the ones in London’s National Museum of Science and Industry, you will see many parts intricately interconnected. After staring at these machines for a long time, you can begin to understand how they work, but the understanding does not come easily. If you look at their modern equivalents, you will see that they are better structured and that their constituent parts are encapsulated.

Both the old machines and the new ones work, so the benefit is cognitive, not technical. The systems themselves do not care if they are elegantly designed or inscrutable, but developers who work with them do. Developers prefer systems that are well-organized, not ones that are a sea of classes, modules, or components that make their heads swim.

The question is: how can you build systems that are comprehensible? The usual answer is to structure the system using a hierarchical nesting of parts. Yet this is only part of the solution, since a hierarchically nested system may still be hard to understand. For example, what if a system has many components, but just one level of nesting? Or if its modules are haphazard groupings of functions? Or if modules and components have poor encapsulation boundaries that couple them tightly and reveal their implementations?

To be comprehensible, your software should be structured so that it reveals a story at many levels. Each level of nesting tells a story about how those parts interact. A developer who was unfamiliar with the system could be dropped in at any level and still make sense of it, rather than being swamped.

Constructing the story. No simple process or set of rules will always yield a system that is comprehensible and tells a story at many levels, but here are a few general guidelines that will steer you in the right direction.

  1. Create levels of abstraction by hierarchically nesting elements (primarily modules, components, and environmental elements).
  2. Limit the number of elements at any level.
  3. Give each element a coherent purpose.
  4. Ensure that each element is encapsulated and does not reveal unnecessary internal details.

If you do this at every level of nesting, developers will see a reasonable number of elements and will infer a story about how they work together. For example, in the Yinzer example, there were just four components (see Figure 9.8). You can infer how they collaborate to solve a problem, and with the provided scenario it is even easier. You should expect that each of those components will have subcomponents or objects within it, but if those components or objects also follow the guidelines above, then you could understand them too. The result is a story at many levels.

Note that maintaining multiple levels of nesting is a bureaucratic burden. You must trade off the cognitive benefit of maintaining a story at many levels with the maintenance costs. While each project will strike its own balance, here are some rough heuristics.

At a particular level of abstraction, a reasonable number of elements is likely between 5 and 50, with 50 being quite large. So, most components should be composed of 5 to 50 sub-components (or classes), and most modules should have between 5 and 50 sub-modules (or files). When you approach 50 elements, consider refactoring to bring the number back down. Similarly, if you find you have very few elements, consider refactoring to “eliminate middle management” by combining levels.

Benefits and difficulties. Architecture models enable you to tell a story at a higher level of abstraction. When the first programs were written, the invention of subroutines allowed developers to tell a story of a master and servant routines. The master task could be understood at an abstract level without reading each of the subroutines. The invention of modules, structured programming, and object-oriented programming enabled stories to be told with increasingly large codebases. The story from the subroutine level was still there, but it was augmented with a story about what each module did. The concepts in software architecture allow you to tell a story about larger chunks — for example, that this is a 3-tier system with one tier behind a security firewall.

Having a story at many levels provides several benefits. First, developers are more able to cope with scale and can reason about modules, components, or environmental elements in huge systems. This is increasingly important as internet-scale systems are constructed by composing existing systems. Second, developers are confronted with less complexity. Large systems entail lots of moving pieces, but the story at many levels restricts how much complexity has to be comprehended at any given moment. Developers treat subcomponents as black boxes and must reason only about the components at the current level. Being “dropped into the code” at any level is possible. These benefits are cognitive, not technical, as they benefit developers and their ability to maintain the system.

There is some cost, however. Maintaining a story at many levels is a bit like gardening, since as the system evolves the story needs maintenance to keep it up. Beyond the upkeep, it requires effective encapsulation, which is difficult and is a kind of deferred gratification.

This section is extracted from the book.