Software Engineering Reflection: Why Only the Giants Survive

It's New Year's. It's a time of aspiration, and a time of reflection. While some prefer to focus on their goals, or on their successes and failures, for me the most important question is "How have I changed my mind?"

Today, I'd like to share the biggest thing I've changed my mind on in software development in the last 5 years.

I used to believe fervently in aggressive reuse of small functionality. I spoke with disgust at how the myriad apps on my machine each had their own spellchecker and login-system, and the lost brainpower that had gone into creating them. I watched teams adopt prebuilt components enthusiastically, and then target them for replacement as soon as they realized they could get a 10%-smaller download size by building their own version with fewer features. I believed in the vision set forth in Doug McIllroy's famous essay "Mass-Produced Software Components," which replaced one-size-fits-all software libraries with "library families," offering variants of a library custom-suited to an application. Ever used JSON to save something to disk, but thought that, if only you had more resources, you'd make a more-efficient binary format instead? Down with JSON; there should be a button that gives you that format.

This all was shattered one year ago, when, during a long conversation with a senior game designer about management, he told me that modern game teams had very few programmers. All they needed to do, he explained, was modify the engine.

The numbers were easy to check. God of War III? 18 programmers. Zelda: Breath of the Wild? 52 programmers, but out of a team of over 900. And Dark Messiah of Might and Magic, the AAA action game that formed my teenage obsession? 5 programmers.

How had they managed this? I spent one day studying the Unreal Engine used by game developers. And in it, I discovered a reuse utopia like the one we had pined for since the early days.

Unreal is a Giant

I found in Unreal a collection of game concepts: scenes and events and assets. Each had a WYSIWYG editor, easy to pick up yet supporting endless customizations. Whereas in my corner of the industry, "built without coding" was synonymous with "shoddy," these editors were being used to pump out bestselling games.

But what impressed me most was the marketplace. Want a minimap for your game? Pull out $30 and take your pick. Game dialogue system? Get a premium one for $130. Auto-saves? Plugin.

So there was indeed plenty a team could do without need for programmers.

Why had this happened in the games industry but not the software industry? There were certainly lots of economic reasons. Companies routinely start new products from scratch, and, if consumers purchased 5 spreadsheets every year, maybe we'd also start seeing companies selling a "spreadsheet engine."

But I also saw an architectural reason. All Unreal games had essentially the same architecture, one of "actors" and "controllers" and "pawns." By keeping these and a thousand other things constant, developers could build new components and editors for them, and expect them to be useful for a lot of games.

And that's when it hit me that, actually, we already had hit a similar level of reuse in the software industry — albeit with glue-programming in place of the code-free editors. We write but a sliver of the code that runs in our software. We get the rest not by hunting libraries, but by building off a few pieces — Linux, Postgres, even HTML itself — and hoping they have what we need. And what are the hundreds of SDKs that provide analytics and crash-reporting to apps if not just a more decentralized version of the Unreal Marketplace?

In the US, television companies sell subscriptions to TV packages giving access to hundreds of TV stations, even though each customer typically watches only a few.

And so, in today's software, we get reuse the same way the proverbial slacker gets information: not by browsing the bookstore for what we want, but by signing up for a cable package with 999 channels.

A few weeks later, I discovered a 2004 essay by Turing laureate Butler Lampson, in which he reaches a similar conclusion: "Software Components: Only the Giants Survive." A "giant" is a large component like an operating system or a spreadsheet (or a game engine). Because a system can only use a few of them, you can read a book about each. There are fewer options to search through when selecting one, and fewer combinations to test. Whereas the conventional dream of little reusable components has failed, the dream of reuse has come true in the form of giants.

Lesson from lens

Many years ago, the Haskell community slew a giant, and thereby summoned a new one. Famed Haskell programmer Edward Kmett had a huge package named category-extras. People who only wanted to use parts of it convinced him to split it up. Burned by struggles with the cross-dependencies, he made up for his mistake with his blockbuster package lens, releasing it as a monolith with over 100 modules and 30 dependencies. Build times everywhere suffered, but lens became dominant against several competitors, and is today used by over a thousand other public packages.

Heard of any major packages being broken up? I expect other communities have their own Edward Kmett stories.

The future may yet look as it did in the past. We've built general-purpose components that waste countless cycles in favor of easier development; one day, we'll want those cycles back. But, for today, the road to reuse is in one well-designed giant.

Happy coding, and Happy New Year,
Jimmy Koppel