Complexity Has to Earn Its Place

Discretion plays a larger role in software development than most people believe. Trade-offs are everywhere, and choosing the right ones can be the difference between elegance and crud. Good developers are knowledgeable about design patterns and the strengths and weaknesses of their programming languages. Great developers need to understand the broader context they're building within. How will this software be deployed in practice? What are the broader financial and business constraints? Sometimes a highly flexible, abstracted design is absolutely the correct choice. Other times, it is a poor fit for the actual requirements, bringing along unnecessary maintenance overhead.

Anecdote: Too Little Complexity

In one of my first jobs, I worked on a suite of enterprise anti-virus products. The products hooked into Exchange and SharePoint and dispatched content to a separate process to perform the anti-virus scanning. By the time I joined the team, the external process code had evolved piecemeal over time and was exhibiting the god object anti-pattern.

Unfortunately, the reality was even worse. The product suite supported three scanning hooks - Exchange email receipt, SharePoint document upload and manual Exchange mailbox scan. There was a separate scanning process for each of those, each with its own god object. Three separate files, each over ten thousand lines of code, with over 90% similarity. The reasons for most of the differences were lost to the passage of time and the turnover of developers. What remained was difficult to maintain. Many changes needed to be synchronized across all three god-object files.

It's unlikely that anyone would set out to reach this end state, but it's valuable to analyze the decisions that led to this point. The company started by offering only an anti-virus solution for Exchange email receipt. Eventually, they decided to expand their product portfolio by migrating their core value proposition to other workloads. They identified the potential reputational damage that could come if they broke their existing product offering. They were not confident in their testing system to catch problems, so they decided to mitigate the risk by making a separate copy of the god-object workflow for each integration. This decision may have made sense at the time, but it led to innumerable future maintenance headaches. Often, a bug fixed in one workflow was not correctly ported to all the others. Previously diagnosed and fixed bugs could and would linger in less popular workflows.

In hindsight, the "simple" choice of making multiple copies of the god-object workflow was clearly sub-optimal. All workflows were essentially pipelines with the same stages in the same order. Nearly all of the differences were in adapting the incoming data from the integration points prior to forwarding them to the main pipeline. Extracting the original pipeline into a separate component would have been the better long-term choice. Well-defined extensibility points would allow real, intentional differences if necessary.

We don't know what led to the initial decision to make the first copy. It's possible that the business was operating under intense time or other external pressures, which would make it more understandable. Nonetheless, the cost of that decision became more and more expensive as more copies were made and they diverged. While a more thoughtful alternative was readily apparent in hindsight, the cost and difficulty of implementing it increased over time. Architecture needs to be reconsidered when the assumptions that produced it change.

Anecdote: Too Much Complexity

Later on, I was working on a product that provided basic Data Loss Prevention (DLP) by extracting text from files and searching it for potentially prohibited content (e.g., personally identifiable information, like SSNs, or other private information). The product was highly flexible and modeled as a state machine. As a consequence of the state machine logic, multiple processing nodes could be reordered dynamically. While this might seem useful, this solution was a little too clever for its own good.

After spending time debugging issues with the state machine and data flows through it, the appropriateness of this abstraction came into question. Every data packet flowed through the same stages in the same order: inspect the type of file, parse it (e.g., extract files from ZIP archives), extract text content, inspect text content. The argument for the state machine was that any stage could result in a terminal error condition, so processing could proceed either to the next stage or the error stage. In reality, this could be better modeled as a pipeline with short-circuiting on error conditions.

The state machine added unnecessary complexity and maintenance costs for no benefit. It added the overhead of an orchestrator for dispatching work from stage to stage. It made debugging issues a bit more difficult since it allowed dynamic control flow. None of this was needed. The wrong abstraction was applied. By making a slight change to the adapter on top of each stage, this entire subsystem could be replaced by a for loop with an error condition check!

A very senior software architect at the company was responsible for the state machine design. We don't know if he wanted to support other scenarios where a state machine may have made more sense, but there was never any strong argument for why it was needed or might be needed in the future. Regardless, in its actual and envisioned production use, the abstraction was misapplied. It led to added costs without any benefits. Architecture needs to be kept grounded in realistic business requirements.

Anecdote: Applications to AI-Assisted Coding

Recently, I was using AI to do a light refactoring of a Nuxt Content blog site. Originally, the site was structured with all the author details embedded in each blog page. This led to quite a bit of duplication and made it quite cumbersome to change an author's avatar. As an improvement, I instructed the AI to refactor the author content into a separate Nuxt collection.

When I reviewed the AI's work, I noticed one strange change. It modified the homepage to load the new authors collection and merge it with the blog posts. This is not necessary because the author information is not displayed as part of the simple blog listing on the homepage. The AI decided to create a collection structurally identical to the original collection, with the embedded authors. This retains the flexibility to eventually display the author images. However, for my use case, where I have no desire to ever display the author images, it was sub-optimal. It added unnecessary coupling between the homepage and the authors collection, along with a (very) slight performance penalty, for no actual benefit.

If I had not reviewed the code, I likely wouldn't have noticed anything. AI dramatically reduces the cost of creating complexity, but it doesn't eliminate the cost of owning it. Unchecked, decisions that add a little complexity here and there will eventually snowball into an overly complex system. The developer is still ultimately responsible for the system delivered and needs to use discretion when using AI.

Too Little, Too Much, Just Right

Complexity isn't something we should strive to maximize or minimize. Something too complex for one project might be too simple for another. We need to use our discretion as developers to pick the right level of complexity for the problem at hand. Often there are trade-offs, but we should always be able to justify the level picked.

Eschewing abstraction where it is needed can be just as harmful as adding abstractions to handle phantom requirements. The code might work, but it is harder to understand and maintain.

Abstractions often emerge naturally during development. In the "Too Little Complexity" discussion, an abstraction wasn't necessary when there was only a single integration. The requirement emerged when a second integration was added. It's generally better to start with a simple solution, and add complexity only when the requirements justify it. In the "Too Much Complexity" discussion, this advice was not followed. Instead, a complex state machine subsystem was built when a pipeline would have been sufficient.

Conclusion

We have reviewed a handful of situations where abstractions have been omitted or misapplied. In the "Too Little Complexity" discussion, we see the danger in simplicity. In the "Too Much Complexity" discussion, we see the danger in complexity. In the "Applications to AI-Assisted Coding" discussion, we see that these architectural considerations and principles remain applicable today.

The developer needs to make decisions that will enable both present and future requirements. These requirements will guide the appropriate level of abstraction and extensibility.

In all of these examples, the solutions all worked. Nevertheless, the systems were loaded with latent technical debt. An under-abstracted system can lead to unnecessary duplication and coupling. An over-abstracted system can be harder to understand and debug. In both cases, the costs of maintenance and future enhancements are larger than necessary.

Simply "working" is necessary, but it is not sufficient for a well-designed system. For long-lived software, we also need to consider the lifetime cost of the decisions we make. Complexity can either increase or decrease that cost. When we introduce it, we need to be able to explain what requirement justifies it. Complexity has to earn its place.