How to Avoid the Second-System Effect

There comes a time in the life of every project when adding new features and fixing old bugs becomes increasingly difficult and developers are tempted to throw it all out and start over. Grand ideas of how we will do it right this time mix with elevated views of our own architectural skill and the naïveté of youth. Attempts to rewrite start well. Line counts climb, spirits stay fresh, and our developers ride their agents over untrammeled meadows of code, inventing new paradigms unburdened by the strictures of the old codebase and frolicking in their newfound freedom.

As our adventurous developers continue on their quest to the mythical land of perfect software, old bugs that were long since vanquished rear their heads. Eldritch horrors lurking beneath untested architecture choices wake and threaten to swallow the entire project. Development slows and our once-intrepid developers become weary. External pressure from customers starts to mount, and the maintenance of both the old and new codebases stretches developers thin. Debates about where to focus efforts threaten to break the bonds of the fellowship as new feature requests pile up and new ideas arise on how to make things better. Eventually the quest turns into a death march to feature parity, dragging its own baggage along, or our developers abandon it for the reliable discomfort of home.

Others have documented this process many times. The Mythical Man-Month recounts Fred Brooks' experience leading development of OS/360 for IBM, and introduces the concept of the second-system effect (wherein rewriting a successful first system leads to a second system that is bloated, over-engineered, and way behind schedule). Joel Spolsky's "Things You Should Never Do, Part I" documents the Netscape 5/6 debacle, and is the standard citation for never attempting a full rewrite.

Despite all of this, I recommended and led a complete rewrite of the Rowan scientific backend. Development had slowed under our monolithic architecture, our deployment and testing strategies, and a web of imports. The way forward was to re-architect the codebase into a monorepo of self-contained packages, which required a near-complete rewrite.

The process was difficult, but worthwhile; it has allowed us to significantly increase the pace of shipping and removed a large number of bugs. Now that the rewrite is in the rearview mirror, I wanted to share a few tips as to how we pulled this off, why you probably shouldn't try to do this, and best practices for those of you who choose to ignore the conventional wisdom.

Why us

The rewrite was easier for us than for most projects because we had a strong separation of responsibilities, a small team and codebase, and someone (me) who cares about the minutiæ of Python software development.

We had walled our scientific code off from the rest of the stack behind an independent API. That let us skip the rest of the stack and focus on what needed to change (don't worry, Corin, I won't ask to rewrite anything for at least another year). Our API had numerous semi-independent workflows, so we built the core of the new codebase and then collected easy wins by replacing workflows piecemeal. Each win raised our confidence and exposed our early mistakes while they were still cheap to fix. Each one also beat the workflow it replaced, so we and our customers saw the benefits immediately.

Our scientific workflow codebase wasn't large: a few developer-years of work from a handful of developers. We knew it intimately, along with every bug we had squashed and the meaning of arcane strings of letters like ωB97M-D3BJ/def2-TZVPPD/CPCM(MeCN).

I've long been interested in the vagaries of software development practices and package architecture (who else would dedicate a whole blog post to the history of package management in Python or the best way to set up Python packages?). I started this blog in part out of frustration with our architecture and a desire to understand how to improve it. Putting those ideas on paper and trying different monorepo layouts taught me what worked and what could go wrong.

Why you shouldn't

Though a rewrite worked for us, I'd strongly advise against it.

You likely haven't dedicated the necessary effort to improving your existing codebase. A rewrite is attractive, as you will have dedicated time to work on making better software. If instead you dedicated that time to improving your existing codebase, could you make similar improvements? Most developers don't spend enough time refactoring and improving their existing code, so a rewrite will only get them so far before they fall into the same problems.

You probably haven't put enough effort into designing the new system. By this I don't mean making an exact spec, but thinking about how to simplify the system as much as possible so that the new codebase is optimal for your future needs. Rushing headlong into a rewrite is destined to end in disaster, as you won't necessarily solve the fundamental problems. Instead, spend more time reading about software design practices and implementing them in your existing codebase to see what you can improve.

You are almost certainly not determined enough to get this right. This is a drastic change that will likely upend the organization writing the code, make people upset, and potentially end in failure. If the rewrite is something you are doing on the side, it will never be able to keep up with the pace of development on the main product. This needs to be a complete commitment, which means that the entire team needs to be on board and ready to drop everything else when the time comes.

Your customers probably don't want it. Customers don't care about the state of your code. They care about what it does, and they hate changing how they work, even for an improvement. Make a feature twice as fast but move a button, and you will hear more about the button than the speed. Throughout the rewrite they still expect the same support, bug fixes, and feature updates on the old product.

Your codebase probably isn't right for this. Ours worked because we rewrote a bounded part with verifiable inputs and outputs, and we moved it over section by section instead of pivoting wholesale. Checking our work meant running both backends and comparing the results.

You don't have the right person for this. You need someone who will spend their waking hours on the minutiæ of how this could go wrong and how to engineer for future developers, and you almost certainly haven't empowered them to do it. Developers are not vending machines where you can insert money and code comes out. Each has their own knowledge, skills, and interests. Without someone who both sees the whole problem and knows how to architect the answer, this won't work.

The failures

If I still haven't convinced you, here is a long list of failures:

• Netscape 5/6 — Netscape 4 was buggy, slow, outdated, and losing market share. Netscape 5 was behind schedule and buggy, so in late 1998 the company bet on Mozilla, a side project rewriting the browser from scratch. It took 2 years and Netscape never recovered.
• Perl 6/Raku — started in 2000 as an attempt to remove the historical warts from the language. It added gradual typing, grammars, and a new object system, but the community never adopted it. It shipped in 2015, took the name Raku in 2019, and is mostly dead.
• LaTeX3 — started around 1989 as a major rewrite. Converting the existing ecosystem proved too great an undertaking, so LaTeX2e slowly absorbed the new features, arriving in 2020 as the L3 programming layer (expl3). Typst is now a serious contender to LaTeX due to its ease of use and speed of compiling.
• GNU Hurd — started in 1990 as a modular replacement for Unix kernels using a microkernel architecture; it still has not reached 1.0 status.
• Midori — never replaced Windows; nobody even remembers it.
• Unladen Swallow — a rewrite of CPython (the reference implementation of Python) to use a JIT compiler. It was based on Python 2.6 and the Python 3 switch left it behind. CPython kept some of its improvements.
• Apple Copland — planned successor to System 7; it was so far behind schedule that Apple abandoned it and purchased NeXT. This brought Steve Jobs back to Apple, so perhaps it was a success after all.
• Windows Longhorn — so far behind schedule that Microsoft reset development in 2004, discarding years of work and restarting from the Windows Server 2003 codebase. It shipped in 2007 as Vista, a name many hope to forget.
• Boston's Big Dig — massive cost overruns and minimal improvements in traffic. Costs ballooned from $2.8 billion to $14.8 billion, it took 17 years instead of the proposed 7, and it needed major repairs as soon as it opened (WGBH podcast).

Questions to answer before considering a rewrite

• Why did your existing codebase get into the current state?
• Is management just using this to cover up for other bad decisions?
• Is management willing to spend the necessary developer time to get this right?
• Is the project manager willing to defend the developers from meddling?
• Do you have someone with the domain knowledge to design a better system, and are they fanatical about it?
• Can you ship early, or will this require the rewrite to be completed before shipping?
• Does your rewrite team have a complete grasp of the domain?

The rules

If, despite all my warnings, you insist on a similar rewrite, here are some rules:

1. Don't do it until you have exhausted all other possibilities, but don't wait any longer
   • Try refactoring (you should have been doing this already; always be reforming)
   • Try selecting only a section to rewrite
2. Include the correct people early
   • The core team needs people who understand software architecture and whose expertise spans the software domain
   • Don't include junior devs or people who are new to the company, but also be wary of people who have spent too long in the old system and are invested in the previous paradigm
3. Build the architecture, scaffolding, and tooling before writing any code
   • Getting this right accelerates everything that follows. Spend a long time here with a small team before committing to the next rule
4. Burn your boats: cut yourself off from your old codebase and, if necessary, make it read-only to prevent yourself from adding new features
   • Limit bug fixes on the old codebase to critical things or those that can easily be solved
   • This may mean saying no to revenue in the short term, but if this was important enough to do now, then it has to be worth potential lost revenue
5. Accept an imperfect rewrite
   • You won't get everything right; in fact, you will get many things wrong
   • Focus on the major architecture and feature bugs that cannot be handled via a refactor
   • Ignore things that can easily be fixed later
6. Break and delete things (judiciously)
   • This is your one opportunity to get certain things correct and remove warts; use it wisely to fix things that hinder future development velocity or to remove bugs that had become "features"
7. Make people upset, but not too much
   • Don't try to appease everyone. Stick to your convictions about what needs doing and how
8. Ship fast
   • The best way to prevent scope creep, customer frustration, and a death march
9. No new features until the rewrite is done
   • Separate what is fundamental to the rewrite from what would merely be nice. Accept only features critical to the new architecture or ones that speed up the work. Everything else, including the features the rewrite unlocks, can wait
   • Make a list ahead of time and stick to it as much as possible
10. Don't rewrite the same thing; build something better with a focus on the future
    • The value of a rewrite lies not in a better version of what you already have, but in shipping faster afterwards, with fewer bugs and more features

Retrospective

I wrote this post in May as we finished the rewrite, but publishing it then felt premature. After a summer of development, I'm confident we made the right call.

While the rewrite set us back 4–5 developer-months, our increased velocity has already made up the lost time. The rewrite decoupled much of our stack, cutting the context needed to write or refactor code. It made the packages in our monorepo consistent, so switching between them and reviewing code costs less. It killed bugs we kept engineering around, and opened new avenues for features.

The rewrite also let us radically change how we do geometry optimizations. Open-source optimizers had left us with minor bugs and unfortunate side effects, since nobody designed them for streaming results, careful control of settings, or fast methods like neural-network potentials. Starting fresh let us write our own optimizer, fixing many bugs and speeding up calculations by moving database communication to a background thread.

We expect the rewrite to continue to pay dividends in the era of AI coding. Solving a problem in the new codebase takes far fewer tokens, thanks to looser coupling and liberal use of agent files for focused context. CI now runs tests per package in parallel rather than sequentially across the whole repo, dropping from hours to minutes. A standard style guide, agent skills, and extra linters have made our code and documentation uniform, and pushed both humans and agents to write better.

The rewrite served us well, and we ship better software faster than before. If you are considering one, I'd be happy to try to talk you out of it.