👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of four topics from today’s subscriber-only The Pulse issue. To get full issues twice a week, subscribe here.

In October, we looked into bootstrapped companies founded by software engineers and the article resonated with many readers; I got lots of messages from bootstrapped founders following that issue. Many messages were complaints about something called “Section 174 changes.” Here’s what one founder said:

“Have you heard of a change in Section 174 of the US tax law that kicked in a few years ago? This change is making bootstrapped software businesses completely unsustainable. 

Basically, all costs related to R&D cannot be expensed, including labor for software development. These costs have to be capitalized and amortized over 5 years – or 15 if labor is done outside of the US. I have no other way to put it: this change is completely insane. Everyone I talk to says the same.

I am curious if this has come up in your discussions with other bootstrapped companies?”

I did some research, and The Wall Street Journal and a handful of other news outlets have covered this change since last March. Still, founders reaching out shared that it feels like public awareness is low about just how large a problem this tax change is. Back last April, Ben Thompson at Stratechery covered this change, also expressing his surprise on how impacted companies seemed to be blissfully unaware of this regulation:

“I’ve been very surprised that few people in tech seem to know about this issue, at least in the private conversations that I have had, even though startups are arguably the hardest hit companies of all.”

I took to social media to gather more information, and a surprising number of complaints from US tech businesses came in.

In this issue, we cover:

• An unexpectedly high tax bill out of nowhere
• A tax change never intended to make it into law
• Why did Big Tech and VCs not raise the alarm?”
• The impact of this change on US tech companies and devs working at them
• Why is forced amortization of software labor objectively bad?
• Will the US be content being a less competitive place for startups?

An unexpectedly high tax bill out of nowhere

Many US software businesses amassed surprisingly high tax bills in 2023, seemingly out of nowhere, due to a tax change which took effect in July the previous year, which many small companies knew nothing about until finalizing their 2022 returns. The change was expected to be repealed (reversed) in December 2022, so many accountants didn’t inform customers for that reason. So, businesses got a surprise when the first tax payments fell due last April.

The amendment to S174 means employing software engineers can no longer be accounted as a direct cost in the year they are paid – unlike the norm, globally. Here’s a simplified example of the change from the final tax year before the change.

Take an imaginary bootstrapped software business called “Acme Corp.” This company generates $1,000,000 of revenue per year running a SaaS service. It employs five engineers, and pays each $200,000. That is $1,000,000 paid in labor costs. For simplicity, we omit other costs like servers and hosting, even though those costs can also fall under the new R&D rules, and have to be amortized. So, how much taxable profit does this company make?

In 2021, the answer would be zero profit. In 2022, the answer was $900,000 in profits(!!) This is because from 2022, software engineer labor costs must be amortized over five years. Here is how amortization works:

• 10% amortized for the first year
• 20% amortized for years 2 to 5
• 10% for year 6

Let’s break this down:

What happens to a small, bootstrapped company without $189,000 in cash floating around with which to pay this tax? There are two options:

1. Take out a loan at a relatively high interest rate (likely at around 10% or so.) I’ve heard of small businesses needing to use personal loans to pay tax. 
2. Lay off a software engineer on $200,000 per year and use this saving to pay the bill!

Less hiring of software engineers and more software engineering layoffs are already happening at smaller US tech companies. In January last year, in a Wall Street Journal article, policy analyst, Alex Muresianu, estimated this tax code change would cost the US about 20,000 full-time software engineering jobs. And I have more real-world accounts of how US businesses are being hit, straight from the companies themselves:

• A business posted a $90K loss in 2022 by the old rules, but is taxed on a $1M profit.
• Infra startup Gruntwork is hiring less in 2024 because of the change.
• A startup that raised $1.25M and hired five employees, is likely to face a $150,000 tax bill it did not account for.
• A seed-stage startup that raised $3M and was operating at a loss became profitable due to the S174 change, so must budget for higher taxes.
• Another US company let go of 23 engineers employed in India because of the tax change. Software engineers abroad can be amortized only over 15 years.

A tax change never intended to make it into law

So, what are “Section 174 changes?” We need to go back some years to understand. In 2017, then-President, Donald Trump, signed the 2017 Tax Cuts & Jobs act, which overhauled tax codes and reduced tax – for example, it reduced the top tax bracket from 39.6% to 37%. To make the bill pass strict budgetary rules, the Senate used a process called reconciliation: adding in tax code changes that delayed tax increases. These delayed increases “balanced out” the tax reduction.

One of these changes was Section 174, set to come into effect 5 years later, in 2022. These parts deliver the blow by making it clear that software development costs need to be amortized over 5-15 years. Most experts expected Congress to push back the Section 174 amendment to a later date, or simply remove it. But Congressional negotiations to repeal the changes fell apart at the last minute in December 2022, meaning it became law.

Why did Big Tech or VCs not raise the alarm?

Why did we not hear the largest tech companies protesting the tax change? Actually, several did, but their protests failed to cut through on the news agenda.

Large tech companies tend to speak up about issues like this via coalitions, trade associations, and lobbyists. Amazon, Microsoft, Intel, Ford, Lockheed Martin, and other US companies created the US R&D Coalition in 2018 to advocate in reversing this change. This group concludes:

“[The Section 174 changes are] a dramatic shift in the tax treatment of business investments in research and innovation, and it will leave the United States with a system unlike any other in the industrialized world. 

By diminishing the near-term value of R&D expenditures, the Tax Cuts and Jobs Act will reduce incentives for companies to invest in the development of new products, ultimately hurting consumers and businesses alike.”

Here is how the S174 change impacted some companies, based on what I found in their annual reports:

• Microsoft: $4.8B additional tax paid in 2023. The company generated a $72B profit that year, so this tax increase was manageable. It’s still a very large amount!
• Netflix: around $368M in additional tax paid – also manageable with $4.4B annual profit.
• Google: the tax change was minimal, because Google was voluntarily amortizing software development expenses for most staff, already. This was for all projects that reached “technological feasibility,” which is a milestone products pass before public release.

For companies with large cash reserves, the tax change was inconvenient, but manageable. Over a five-year period, the amount of tax evens out. After five years, there can even be tax benefits to this kind of accounting.

What about VC-funded companies? For loss-making companies this change doesn’t make much of a difference. But the change does impact VC-funded companies near to break even. Most VC-funded companies close to breakeven have big-enough cash buffers with which to pay unexpected tax bills. However, these companies might reduce hiring – or even consider letting go staff. 

Impact on US tech companies and devs working at them

Assuming Section 174 stays and all US companies must amortize software developer costs over 5 years, this will have an immediate impact on early-stage and small tech companies:

• Less hiring of software engineers in the US/more layoffs. Small and medium-sized tech companies facing higher tax bills in 2023 and 2024 will hire fewer developers. Redundancies might happen for cash flow purposes to replace in-house developers with vendors, which brings us to the next likely impact:
• Firing of non-US software engineers employed by US companies. The tax change is very hostile to software developers employed abroad: their wages need to be deducted over 15 years. Unless a US company has massive cash reserves, it now makes no sense to remotely employ or contract with individual software developers. An engineer shared how their company fired 23 developers employed in India because of Section 174.
• A boon for SaaS companies and vendors? US software companies now have a strong reason to buy, instead of build, software in-house. In February last year, we covered the trend of tech companies aggressively cutting vendor spend. This tax change greatly incentivizes US companies to increase vendor spend – and either not hire more devs, or let some go!
• It makes a lot less sense to incorporate tech startups in the US. Assuming there’s a choice to incorporate a startup in the US or somewhere else, then any other country makes so much more sense. In the first few years of a startup, it’s typical to make a loss while building something that might not work out. Thanks to the amortization rules, in the US this loss could turn into a taxable profit! 
• Startup software IP will likely be moved out of the US. Foreign founded venture-backed startups have been overwhelmingly incorporated in Delaware, US, until now. The Delaware company would then hold the intellectual property (IP) of the startup. With the Section 174 changes, the simplest way to not be subject to US amortization requirements is to move the IP to a foreign subsidiary – and then this subsidiary can employ developers, without being subject to the 15-year amortization requirements. This is what newly founded VC-funded startups are doing, already! As usual, this is not legal or business advice. Consult a professional for guidance on this matter.
• Creative workarounds, like “inversion buying.” Say that you run a US company with a Canadian subsidiary, with most developers working in Canada. With Section 174 changes, Canadian developers need to be amortized over 15 years (!). A tempting workaround is to have the Canadian subsidiary buy the US company! Now this is a Canadian company, and Section 174 doesn’t apply to developers in Canada. However, such workarounds can be expensive and complicated from a tax perspective, due to Passive Foreign Investment Company (PFIC) rules. Still, plenty of founders are exploring setups like this.

Tech companies are now forced to understand – and use – R&D credits. The S174 amendment forcibly categorizes pretty much all software development as R&D. The upside of this that R&D credits do offset the tax bill, but only partially:

For more details on tax credits and Section 174, Gusto created a Section 174 tax liablities caluculator. Do consider getting professional advice, as always, if you are running a business impacted by this change.

Innovation across all US software companies will take a hit if Section 174 stays. The tax change incentivizes software companies without large cash reserves to invest less in research and development. Or they can just move abroad.

But the change isn’t just bad for small software companies; it hurts even the largest ones – it is why Microsoft and Amazon are also advocating for its reversal.

Why is forced amortization of software labor objectively bad?

The concept of amortization (also referred to as depreciation) is as follows. Say you purchase a powerful server for $2,000 to serve traffic for your SaaS business. You pay $2,000 on the spot. However, the expected lifespan of this server is 4 years, after which you expect to replace it. Therefore, it is reasonable to argue that you are getting about $500 worth of “value” every year from this server, and so you amortize (depreciate) a quarter of the value of this server every accounting year, until year 4.

This kind of depreciation is sensible, as you could sell your server for around $1,000 after two years (assuming that is how its value depreciation.) Tax authorities in different countries have come up with depreciation frameworks that businesses follow along these lines. For example, in the US, the Financial Accounting Standards Board (FASB) created the Generally Accepted Accounting Principles (GAAP) provisions, which include a framework on how to deal with depreciation.

Amortizing software development over years makes sense – but only in some cases. If you have launched a software product, have customers, and you can forecast demand for this product pretty accurately, then investing developer hours to keep this SaaS running feels similar to buying hardware. The software engineer could build a new feature for the SaaS which generates revenue for years to come. So there is some argument to treat software development akin to buying a server.

But is comparing physical goods to software, accurate? Take a company that buys a server office it plans to use for 20 years. After a massive one-off investment, the only cost is maintenance, which is a pretty minimal cost.

Now take the example of creating software to generate revenue for 20 years. If you do barebones maintenance, the software becomes obsolete in a year or two! You need to keep adding new features and keep up with the competition. As you add more features, you run into tech debt and architecture issues, so you refactor the software and within a few years you rewrite the whole thing!

Building software is like continuously renovating – and rebuilding – a property. While it is tempting to see why an accountant would want to treat software development amortized like physical goods, this is simply not the reality of how software products behave, or are maintained.

Before Section 174, companies could choose how they categorized software developers, and could opt into deducting costs. This is what stable and highly profitable businesses like Google have done: for software products in production (and making money) it deducted developer costs over 5 years. For pre-launch projects, Google simply expensed those developers. This is the sensible way to run a software business, after all!

Google employing lots of devs in Switzerland starts to make a lot more sense, especially now. Ever wondered why Google has such a large software engineering center in Switzerland – despite the high cost of software engineers within Europe? Switzerland has a very powerful research and development incentive: the country allows expensing 135% of R&D-related salaries in the year they are incurred. 

Switzerland’s taxation is pretty much the polar opposite of that of the US with Section 174. Is Section 174 stays, I would not be surprised if some US companies considered following Google’s lead and explore setting up engineering offices in this country.

Will the US be content being a less competitive place for startups?

All founders whom I talked with hoped Congress would revert this disastrous tax code change, given its devastating impact on tech startups. However, this has not happened – and it’s unclear if it will.

Founders have been trying to get the attention of Congress: sending a letter signed by 1,000+ software businesses, and creating coalitions like the American Innovators Coalition. Another group formed in March this year is the Small Software Business Alliance, created by indie SaaS founder Michele Hansen, supported by over 600 small software company founders. 

I asked Michele why she thinks the general public isn't aware of the threat to US companies. She said:

“1. For years, the perception has been that tax changes impact big companies, not small ones. 

2. Tax policy issues are boring for the general public. It takes a strong social impact – for example, mass layoffs – for news like this to break into mainstream media.

3. Section 174 was never supposed to take effect, and was widely assumed that it would be fixed before it took effect. So, there was little interest in a problem likely to go away.”

Michele said there is reason for hope, though. Congress appears to have struck a deal: revert Section 174 changes on R&D expensing. Lawmakers are now looking for ways to finance this deal.

One country making software development less competitive obviously benefits other countries. Thanks to the S174 amendment, starting a tech company outside the US is far more attractive – including for raising startup capital. Still, the US has long been the global innovation hub for technology, and the country kneecapping technological innovation might let the rest of the world catch up. But I cannot see this being beneficial for innovation, as a whole, globally.

Thank you to Michele Hansen and Jacek Migdał for their insights and reviewing a draft of this section.

Updates to the article after publishing: 6 January — added a reference to Stratechery that covered this change last April

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry pulse. A roundup of recent events, with commentary. The New York Times is suing OpenAI, Canva to offer a secondary equity sale, Cloudflare receiving 3x as many applications in 2023 than a year ago, and more.
2. Uber is now worth more than all competitors, combined. Two years ago, food delivery company DoorDash and ridesharing company Lyft were worth more than Uber. Now, Uber’s worth more than DoorDash, Lyft, Didi, Zomato, Grab, Delivery Hero, Instacart, and half a dozen other competitors.
3. The largest known Ruby codebases. Stripe and Shopify almost certainly operate the world’s largest Ruby codebase. A look at other companies known to work with massive Ruby repositories.

Read the full The Pulse.

2023 was the second full year of The Pragmatic Engineer Newsletter, and this newsletter is now almost two and a half years old; the first issue came out on 26 August 2021. Thank you for being a reader, I greatly value your support.

This year, 102 newsletter issues were published, and this is number 103. You received a deepdive issue on Tuesdays, and every Thursday it was “The Pulse” – formerly The Scoop. Occasionally, there was a bonus article on Wednesdays, too, such as about the book, Tidy First?

Like last year, these newsletters all add up to 5-7 books’ worth of information. And speaking of books, this year I finally published the project I’ve been busy writing for years on the side of the newsletter: The Software Engineer’s Guidebook. This turned into a four-year-long endeavor; thank you to everyone who purchased the paperback edition. Kindle and audiobook formats are expected in early 2024.

Today’s issue is a look back at 2023, with pointers to articles you may want to reread, or discover for the first time. We cover:

1. The newsletter’s evolution
2. The most popular articles, and my personal favorites
3. New resources and templates for engineers and managers added this year
4. Tech industry pulse in 2023

Read the 2023 recap here.

For reviews of earlier years, see The Pragmatic Engineer in 2021 and in 2022.

For 3 years straight, the DevTernity conference listed non-existent software engineers representing Coinbase and Meta as featured speakers. When were they added and what could have the motivation been?

A year ago, I spent months doing an investigative report on how UK events tech company Pollen had its staff work for free, as it had run out of money but still kept operating. The company kept promising that missing wages would arrive – but they never did. Not only that, but the company stopped paying health insurance to US staff without informing them. The company went bankrupt, and the details I uncovered were troubling. A year later, the BBC made a documentary about the company called Crashed: $800M Festival Fail. 

That BBC report required more resources than I have available. My report also created potential legal jeopardy for me – and I had to review it with lawyers before publication. I decided that while I greatly respect investigative reportage, it’s not an area I would venture into, again. And I intended to keep this promise.

However, last week I saw a post on my timeline in which a good friend of mine was announced as a speaker at the DevTernity tech conference on 7-8 December. It’s a paid, online-only conference, with tickets ranging €399-798 per person ($435-870). The speaker lineup features some of the best-known speakers in the tech industry: ones like Scott Hanselman, Kelsey Hightower, Sam Newman, David Heinemeier Hansson, Dave Farley, Allen Houb and others.

I clicked to check out what his talk will be about, and to look at the speaker lineup. Most speakers were familiar, and I also checked out the others. One profile caught my attention: Anna Boyko, a core contributor to Ethereum:

There aren’t many core Ethereum contributors, and I found it curious that someone committed to working on an open source crypto platform would join Coinbase at the same time. Eager to learn more about Anna’s professional history, I turned to LinkedIn. No matches. Google. No matches, save for this conference. I looked through the Ethereum core contributors, but found no one remotely similar. Finally, I reached out to friends at Coinbase who said they never heard of anyone called ‘Anna Boyko.’

By now, I was suspicious. Could Anna be a fake profile? Surely not for such a high-profile conference? I assumed I was missing something, and I wanted to see if this could be a one-off of inventing a fake profile to make the conference lineup look more impressive than it actually is. So I looked at other conferences by the same organizer, Dev Events. And sure enough, I found yet another non-existent speaker for JDKon 2024 called Alina Prokhoda: Senior Engineer at WhatsApp, and Microsoft MVP.

But there is no such MVP, and I confirmed with my sources that no one by this name works at Meta.

I then looked at previous DevTernity conferences. In 2022, another odd profile stood out: a speaker with no Twitter handle and a weird title; Natalie Stadler “Software Craftswoman at Coinbase:”

I don’t know about you, but I cannot imagine any Coinbase employee agreeing to such a title for a conference. Still, chatting with a contact at Coinbase, I asked them if Natalie also worked there; and she did not. But it didn’t stop here. Natalie Stadler was a listed speaker on the 2021 edition of the conference, as well.

When and why were the fake speakers added to the conferences?

You’ll notice that all the fake speakers were women. Is this a coincidence? There's a timing that questions whether this could have been an accident.

To answer, first, let's meet the conference's cofounder, Julia Kirsina (Coding Unicorn). She confirmed to hold this role in 2023:

On 3 August 2021, at 5pm GMT (6pm CEST), a complaint was made on Twitter, which message called out the all-male panel for the 2021 DevTernity lineup:

"Natalia Slater, Software Craftswoman at Coinbase" was added a few hours after this complaint on the lack of diversity was made. The DevTernity website is open source, with all changes observable, so we can find the exact commit on 3 August 2021 that added the first fake speaker. It happened at 7pm GMT (8pm CEST), so only 2 hours after the complaint made:

The same commit also added Julia Kirsina, the cofounder of the conference as a speaker: but Julia delivered no talk on this conference. Given the timing, it seems highly likely that the conference's organizers invented a woman profile to address the concerns raised by Matty Stratton.

On 6 August 2021, Natalia's name was changed to "Natalia Stadler:"

Natalie Stadler was present in the 2021 conference lineup, as well as for the 2022 one.

For the DevTernity 2023 website, Natalia Stadler was no longer part of the site when it went live on 3 January 2023. Her "colleague," Anna Boyko was added on 21 January:

The original image of Anna Boyko looked like it was generated from the website This Person Doesn't Exist. This image was updated to a perhaps more realistic one on 24 January, in a commit that updated several speaker images:

Finally, Alina Prokhoda was added to the JDKon lineup on 14 October 2023:

Is there more to this, than fake Coinbase and Meta speaker profiles?

What do you do when you find four instances of non-existent speakers at four conferences held in different years? All this took about 30 minutes of research to find out, including the pinging of contacts at Coinbase and Meta. Other people must have discovered what I did with such little effort, but it seems they said nothing. My choices seem to be to stay silent and let this deceit continue: or speak up, and hopefully help end this practice.

I decided to share that this conference advertises fake speakers, and has done so for years. This statement is easily verifiable by anyone in the same way that I did. I’ve not seen this deceit at any other tech conference, and I’ve seen enough that I didn’t want to keep quiet.

So, I publicly shared on Twitter and LinkedIn that these speakers are made up. People were obviously not happy, and asked questions.

The DevTernity conference is organized “by a small team of software engineers from Singapore, Estonia, and the Netherlands,” as per the website. One of the conference’s organizers responded in public, offering an explanation that made no sense: such as stating that this was a one-off mistake, when it’s a multi-year practice.

Further disturbing details have emerged from efforts by investigative reporters. The Verge confirmed no “Julija Kirsina” graduated from The RISEBA University of Applied Science, which is listed on this speaker’s LinkedIn profile as her professional education, and The Verge could also not reach her.

Is there even a real female software engineer behind the “Coding Unicorn” account, and does a developer with the name Julia Kirsina exist? 404Media reported that the conference’s male founder seems to be heavily involved behind Julia Kirsina’s “Coding Unicorn’s” Instagram account. Despite being listed as a featured speaker at DevTernity in 2021, 2022 and 2023, Kirsina never delivered a talk at the conference. In another relevant detail, the conference organizer was previously banned from tech social media site Lobst.ers for sockpuppeting the Coding Unicorn account.

The DevTernity conference is no longer taking place: I understand this may be because several speakers canceled after learning of the deception. I hope everyone who paid for tickets and prefers a refund gets one; for an ethically run conference, this should be a given. Hopefully, this is the last time a paid tech conference advertises invented speakers with AI-generated images who don’t actually exist.

The myth of "there are not enough women speakers"

One of the organizers of the conference responded on Twitter, claiming "There have been 1000s of events chasing the same small sub-group of female speakers." This sounds like a cheap excuse. The community response to this claim has been also strong contradiction, and people sharing resources on where women speakers can be found. A couple of these:

• Women in tech speakers and organizers: a continuously updated list on GitHub. 400+ speakers.
• STEM Ladies: a list on Twitter/X with 1,600 members
• "Reply to this thread with a woman in tech you look up to:" 300+ responses
• Guests on the Hanselminutes podcast: 920+ guests working in tech (or areas related to tech), the majority from diverse backgrounds

More coverage

We've not seen anything of this kind of deceit in tech - a conference inventing speakers, including fake images - and the mainstream media covered this first-of-a-kind unethical approach to organizing a conference, extensively:

• Male tech conference founder is behind popular woman coding influencer account (404Media)
• This dev conference organizer seems addicted to making up women (The Verge)
• Meet ‘Anna Boyko’: how a fake speaker blew up DevTernity (The New Stack)
• A popular female coding influencer's Instagram is apparently run by a man (Endgadget)
• ‘What a strange tale’: Tech conference canceled after execs flee report of fake women speakers (Fortune)
• A conference organizer lists imaginary women speakers to increase the appearance of diversity at his events (Boing Boing)
• Backlash over fake female speakers shuts down developer conference (Ars Technica)
• Tech conference accused of creating fake women speakers (Skift)
• Developer conference axed after fake female profiles outcry (Bloomberg)
• DevTernity conference collapses amid claims women speakers were faked (The Register)
• Tech conference organizer accused of using fake female panelists (Business Insider)
• Fake AI-generated woman on tech conference agenda leads Microsoft and Amazon execs to drop out (Washington Post)

A shorter version of this article was shared published in The Pragmatic Engineer Newsletter on 30 November.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover thee out of nine topics from today’s subscriber-only issue: The Past and Future of Modern Backend Practices. To get full issues twice a week, subscribe here.

It’s fascinating how what is considered “modern” for backend practices keep evolving over time; back in the 2000s, virtualizing your servers was the cutting-edge thing to do; while around 2010 if you onboarded to the cloud, you were well ahead of the pack. If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications. 

How have practices considered cutting edge on the backend changed from its early days, and where is it headed in future?

To answer this big question, I turned to an industry veteran who’s been there, seen it, and done it – and is still active in the game. Joshua Burgin was an early Amazon employee, joining in 1997, back when Amazon was just an internet bookstore and cloud computing wasn’t even a speck on the horizon. 

He then worked at the casual games company Zynga, building their in-game advertising platform. After Zynga, he rejoined Amazon, and was the General Manager (GM) for Compute services at AWS, and later chief of staff, and advisor to AWS executives like Charlie Bell and Andy Jassy (Amazon’s current CEO.) Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. Joshua has remained technical while working as an executive.

Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry. Subscribe here.

In this issue, we cover:

1. Year 1 on the job: accidentally taking Amazon.com offline
2. The birth of ARPANET (1969)
3. The rise of the internet and web-based computing (1989)

In this article, we use the word “modern” to describe what larger, more established businesses consider cutting edge today. We also use it in a historical sense to describe previous generations of technology that were pioneering in their time. It’s important to note there are no one-size-fits-all choices with technology, especially for things like microservices or Kubernetes, which are still considered relatively modern at large companies today. Startups and midsized companies are often better off not jumping to adopt new technologies, especially ones which solve problems a business does not have.

With that, it’s over to Joshua:

1. Year 1 on the job: accidentally taking Amazon.com offline

My journey in backend development began with an unexpected stumble, not a carefully considered first step on a pre-planned career path. In 1997, when Amazon was a one-floor, sub-100 person startup trying to sell books online, I experienced what every engineer dreads.

Backend code I wrote and pushed to prod took down Amazon.com for several hours. This incident was my initiation into the high-stakes world of internet-scale services. Back in those days, the “cloud” wasn’t even something you could explain with hand gestures, and of course not something to which you deployed applications, and scaled on a whim. 

Our tools were simple: shell scripting, Perl (yes, really!) and hand-rolled C-code. We used a system called CVS (Concurrent Versions System) for version control, as Git did not exist until 2005 when Linus Torvalds created it. Our server setup was primitive by today's standards; we had a single, full-rack size “AlphaServer” box manufactured by Digital Equipment Corporation (DEC):

On the Alpha box, we had an “online” and “offline” directory. To update code, we flipped a symlink (a symbolic link is a file whose purpose is to point to a file or directory) to swap between the code and HTML files contained in these directories. At the time, this approach was our best effort to deliver code on the nascent web. 

Amazon.com ran on top of a monolithic C binary named “Obidos,” named after the narrowest, swiftest section of the Amazon river, in Brazil. Obidos shouldered the entire load of a business which became an online retail behemoth. 

I remember the physicality of our systems; the blinking lights of the network traffic, having to put thermometers on servers because we were using converted conference rooms as data centers, and how we put a tarp in one corner because the roof leaked! All our servers had names, including some named after Muppets characters like Bert and Ernie. 

We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. Our deployments were initially manual. Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process. 

To succeed as a software engineer, you needed to be a jack-of-all-trades. We dabbled in network engineering, database management, and system administration. Oh, and did I mention this was on top of writing the code which fueled Amazon's growth? We were practicing varying shades of DevOps before it existed as a concept.

On the day I broke the site, I learned the importance of observability. I mean “importance” as a fundamental necessity, not a buzzword. Here’s how the outage played out:

1. I pushed code from dev to staging using a script where everything worked fine.
2. I then half-manually pushed code from staging to production. I still used a script, but had to specify parameters.
3. I inadvertently specified the wrong parameters for this script. I set the source and destination Apache configuration as the same.
4. Rather than failing with an error, this encountered an existing bug in the DEC Unix “copy” (cp) command, where cp simply overwrote the source file with a zero-byte file.
5. After this zero-byte file was deployed to prod, the Apache web server processes slowly picked up the empty configuration file. And it lost pointers to all its resources!
6. Apache started to log like a maniac. The web server began diligently recording logs and errors it encountered.
7. Eventually, the excessive Apache logs exhausted the number of inodes you can have in a single directory
8. Which resulted in the servers becoming completely locked up!
9. To resolve this, we had to do a hard reboot. This involved literally pushing the power button button on the server!

Our ability to monitor what was happening for users on the website was overly simple, it turned out. It’s wasn’t that we hadn’t thought about monitoring because we automated plenty of things, such as: 

• Network monitoring: inbound/outbound traffic
• Database monitoring: for example, keeping track of order volume
• Semi-automated QA process for our “dev” and “staging” environments

Of course, we had no synthetic monitoring back then; as in simulating website usage, and thereby getting notified when things went wrong before users experienced it. Instead, we were regularly notified of bugs when executives’ spouses and partners encountered issues while trying to buy things! 

Looking back, I could’ve done a better job with bounds checking and input validation. By bounds checking, I mean determining whether an input variable is within acceptable bounds before use and input validation. And for input validation, I mean that I could have analyzed inputs and disallowed ones considered unsuitable. I made these improvements after the outage, as part of the post-mortem process, which at Amazon is known as a “COE” (Correction of Errors,) and uses the “5 whys methodology” developed originally at Toyota, which is still common today across the US. The AWS re:invent conference in 2022 hosted a good in-depth overview of Amazon’s COE process.

“The more things change, the more they stay the same,” is one theme of this article. And this was proved true not long after I returned to Amazon in 2014, when a very large AWS service outage was caused by… you guessed it; the absence of bounds checking and parallelized execution.

The outage I caused in 97 taught me that developing robust systems goes hand-in-hand with the development of robust monitoring and observability tooling – which should exist outside of whatever code you’re writing because you can’t always anticipate how your code will operate. And of course the incident was a lesson in humility and accountability, demonstrating that how you respond to outages or problems you cause, is a reflection of your character and professional integrity. Nobody’s perfect, and reacting positively to challenges and critical feedback significantly enhances how others perceive you.

But this article isn't just about learning from the past; it's about understanding the roots of current backend engineering practices. Let’s dive into the evolution of backend development, from computing’s early days up to the present, and try and predict what the future holds. As we go on this journey, remember:

Every complex system today stands on the shoulders of lessons from earlier, formative times.

2. The birth of ARPANET (1969)

To appreciate the magnitude of progress in backend development, we need to go back to 1969 and the birth of ARPANET. This pioneering project was conceived during the Cold War between the US and the USSR. ARPANET was the world’s first major foray into networked computing. This project’s advancements laid the groundwork for many technologies still with us today, seven decades later.

The concept of a distributed network was revolutionary, back then. ARPANET was designed to maintain communications during outages. This is a principle that has since become a cornerstone of modern backend architecture. But back then, the challenge wasn't just in building a network; it was about proving distributed systems could be reliable, efficient, and secure, on a scale never attempted before.

Consistent experimentation was a hallmark of this era. Protocols and practices we now regard as rudimentary were cutting-edge breakthroughs.

Packet switching was the method ARPANET used to send data, which laid the foundation for the future internet. In the early 1960s the prevailing communication networks consisted of continuous, analog circuits primarily utilized for persistent voice telephone connections. This approach was transformed by the advent of packet switching, which introduced a digital, intermittent framework for networks. This method transmits data in discrete packets, doing so only as needed. Although this introduced the compromises of signal discontinuity and conversion overhead, packet-switching had several advantages: 

1. Digital signal could be made "error-free"
2. Software processing was upgradeable at the connection points
3. Redundancy meant networks could survive damage
4. Sharing links boosted efficiency

For more on how packet switching works, this overview on the Engineering & Technology History Wiki is a good start, and it also provides details on how ARPANET was originally connected:

“ARPANET was linked by leased communication lines that sent data at speeds of fifty kilobits per second. Operating on the principles of packet switching, it queued up data and subdivided it into 128-byte packets programmed with a destination and address. The computer sent the packets along whichever route would move it fastest to a destination.”

Sending digital packages allowed for robust, adaptable communication pathways that could withstand points of failure. This resistance to failure is what modern cloud services replicate in their designs for fault tolerance and high availability.

There were a lot of technical hurdles which ARPANET had to overcome. The biggest was how to create a network that operated across disparate, geographically separated nodes. To solve this, network engineers devised innovative solutions that went on to influence the development of protocols and systems which underpin today’s web. For example, this was when these protocols were invented:

• TCP/IP
• Telnet
• FTP protocol
• Network email

All these protocols were built for or, first implemented, on ARPANET!

As a backend developer, acknowledging ARPANET's influence is vital. The engineers who built ARPANET show us the importance of expecting and designing for failure, which is the core principle of backend and distributed systems development practices today. Notions of redundancy, load balancing, and fault tolerance which we work with are the direct descendants of strategies devised by ARPANET’s architects.

3. The rise of the internet and web-based computing (1989)

ARPANET's influence expanded in the 1980s when the National Science Foundation Network (NSFNet) provided access to a network of supercomputers across the US. By 1989, the modern Internet as we know it was born, The first commercial Internet Service Providers (ISPs) emerged in the US and Australia, kicking off a radical shift in global communication.

At the same time as the first ISPs were founded, in 1991 Sir Tim Berners-Lee proposed and implemented the Hypertext Transfer Protocol (HTTP.) It defined internet-based communications between a server and a client. HTTP triggered an explosion in connectivity – and we can now safely say this protocol proved to be one of two fundamental catalysts of rapid internet adoption.

What was the other driver of adoption? Well, until 1993, the most common way to access the internet was via text-based interfaces, and the Gopher browser, which looked like this:

Then in 1993 Marc Andreessen and his team built and launched the first modern web browser, Mosaic, which provided a graphical interface for browsing the internet, making it accessible to a non-technical audience. This milestone is widely considered a critical point in the digital revolution, by enabling the average person to easily and meaningfully engage with online information and resources.

Scaling challenges became more interesting on the backend side of things. As the internet's reach grew, businesses faced the daunting task of making their infrastructure available on this thing called the “internet,” and scaling it to allow more users to access their services at once. There were several parts of this challenge: 

• Serving international audiences: All code, databases, HTML web pages, customer service tools, etc., were in english only. The localization (l10n) and internationalization (i18n) projects were many months-long undertakings just to launch amazon.de (Germany.) Later, adding support for “double-byte characters” to launch amazon.jp (Japan) took just as long.
• Managing increasing, unpredictable website loads: Amazon was growing tens-of-thousands-of-percent a year by order volume (and 100s of percent in staffing.) Third-party events like “Oprah’s book club” could send traffic and orders spiking 1000s of percent. And 1998 was the beginning of Amazon’s “peak season spike,” with a significant increase in sales the day after Thanksgiving, and the Christmas holidays. Even within that period there were waves, peaks and valleys that varied by several orders of magnitude.
• Safeguarding data and systems: There weren’t common tools like public/private key encryption, role-based-access-control (RBAC), zero-trust, least-privilege and other “defense in depth” measures that we have today. For example, it used to be common to check your database password into your version control system, and at most obfuscate the password (i.e. by rotating the letters using a ROT13 cypher,) to reduce the risk of storing it plain-text. Clearly, something more secure was needed.

The early ‘90s also saw the emergence of novel solutions to challenges such as:

• Database management
• Server clustering
• Load balancing

Today, these challenges still exist but are so much easier to solve, with no shortage of open source or commercial solutions. But 30 years ago, these problem areas demanded hands-on expertise and in-house, custom development.

At Amazon.com, we had our fair share of backend infrastructure challenges. The business was founded in 1994, a year after the Mosaic browser, and grew fast. This rapid growth put constant stress on the infrastructure to just keep up; not to mention Amazon’s stated ambition of becoming a global marketplace. 

Internally, we seemed to hit another hard scaling limit every 6 months. This limit was almost always on how much we could scale out monolithic software. Each time we hit such a limit, we were pushed further into implementing distributed systems, and figuring out how to scale them up for demand.

At the time, it felt like Amazon was forever one bad customer experience away from going under. It was a high-pressure environment, and yet we needed to build systems for the present, and also for unpredictable but ever-increasing future loads.

These were three out of the nine topics covered in The Past and Future of Modern Backend Practices. The full article additionally covers:

1. The struggle for scale at Amazon and the birth of SOA (late 1990s)
2. Virtualization: the new frontier (1999-2002)
3. Cloud computing: A paradigm shift (2006-2009)
4. Containers and Kubernetes: reshaping the backend landscape (2013-2019)
5. Today’s backend ecosystem: defining “modern”
6. Envisioning the future of backend development

Read the full article.

The below topic was sent out to full subscribers of The Pragmatic Engineer, three weeks ago, in The Pulse #66. I have received several messages from people asking if they can pay to “unlock” this information for others, given how vital it is for software engineers. It is vital, and so I’m sharing this with all readers, without a paywall. In the unlikely case that you are asked to do something fishy or illegal: I hope the below will help decide how to do the right thing. Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Here are three real-life cases, where what engineers and engineering managers did had serious consequences.

FTX: an engineering director went along with the fraud

A trial related to FTX, the cryptocurrency exchange which allegedly defrauded investors of $9B, is ongoing. Day 9 of the trial of former FTX CEO Sam Bankman-Fried trial, heard testimony from Nishad Singh, who joined the business as a software engineer, and later became an engineering director. Here is software engineer and writer

Molly White

summarizes of his evidence:

“To hear Singh tell it, he didn’t even really realize what was going on at FTX and Alameda Research until September 2022 — only a month or two before everything came crashing down. (...) Several times throughout various testimonies, we’ve seen a document written by Sam Bankman-Fried, in which he describes his thinking that Alameda Research should be shut down. That document was, ultimately, how Singh learned in September 2022 that Alameda Research had taken billions of dollars of customer funds from FTX. 

This was when Gary Wang told Singh that Alameda was borrowing massive amounts of customer money from FTX — at the time, around $13 billion of it. Singh testified that he felt ‘really afraid’, and called an in-person meeting immediately. Bankman-Fried, who was sitting next to Singh at the time, ‘seemed unsurprised and made up what I understood to be a false excuse for dodging the meeting.’ Singh, Ellison, and Wang met without him, and Singh confirmed his fears: that he had not misunderstood Wang, and that Alameda had actually taken customer funds to that extent.”

Okay, so in September 2022, Singh had confirmation that something illegal was happening at the company, which he had no direct knowledge of, until then. At that point, if he wanted to avoid being an accomplice to potentially illegal activity, his options were:

1. Talk to a lawyer on how to avoid assisting a crime
2. Turn whistleblower. See the tech whistleblower guide
3. Quit the company, ensuring he did not further aid this activity 

The smart thing would have been to do #1. The profitable thing could have been to do #2 because in the US, a whistleblower may receive a whistleblower reward of between 10-30% of what the government recovers from fraudulent activities. The final choice #3 is hard, but could have meant Singh would not have had to plead guilty as he did. 

Here’s what Singh did instead: he asked for a personal meeting with Bankman-Fried and confronted him about the missing funds. However, Bankman-Fried replied there not much to worry about, and that they’d repay the funds by raising more money from investors (!!) This should have been the point at which Singh quit. Instead:

“He thought about leaving the company then, he testified, but worried that his departure could cause everything to fall apart. He felt that if he stayed, maybe he could help the companies make back what they owed.”

For the next two months, Singh tried to make things better, but it was fruitless. FTX collapsed in November 2022.

Lesson #1: when you discover fraud may be happening, do not “stay around to fix it.” Any other approach would have been better for Singh; seeking legal advice, turning whistleblower, or quitting on the spot.

To be fair, Singh didn’t seen totally clueless, and it seems he decided to profit on the developments. Days after he found about this fraud, he took a $3.7M loan from FTX (!!) to buy a house, The Verge pointed out. It’s exactly the type of thing you don’t want to do after you discover fraud.

Now, Singh is facing up to 75 years in jail thanks to his decision to aid the company after discovering the fraud. His sentence will most likely be reduced due to his plea deal, but any course of action which leads to a criminal conviction is surely a grave error of judgment.

Frank: a software engineer refuses to fake customer data

Frank was a student loan startup founded by Charlie Javice in 2016. In 2019, Javice was featured on the Forbes “30 under 30” finance list, suggesting she was a high-flying founder:

It certainly seemed like Charlie Javice was a standout founder; in 2021, JP Morgan purchased Frank for $175M. However, things turned sour quickly. JP Morgan thought it bought a startup with 5 million customers, which worked with 6,000 schools. But after the purchase, this data was found to be mostly fake.

Let’s get to a software engineer’s involvement. This April, founder Charlie Javice was arrested, and a lawsuit is ongoing between her, former Chief Growth Officer Olivier Amar, and JP Morgan. From to this lawsuit, we get an inside look at how events unfolded inside Frank.

In 2021, an engineer was asked to produce fake data for 4.2M non-existent customers. As acquisition talks were ongoing, JP Morgan wanted to validate that Frank had the nearly 5M customers it claimed. In reality, Frank had 293,000 customers, so the CEO asked an engineer to fake the data and turn this list into 4.2M members. Here’s what happened next – from the lawsuit:

“[In 2021] Javice [CEO], Amar [Chief Growth Officer] and the Director of Engineering then had a Zoom meeting during which Javice and Amar asked the Director of Engineering to help them create a synthetic list of customer data. She asked the Director of Engineering if he could help take a known set of FAFSA application data and use it to artificially augment a much larger set of anonymous data tht her systems had collected over time.

The Director of Engineering questioned whether creating and using such a data set was legal, but Javice tried to assure the engineer by claiming that this was perfectly acceptable in an investment situation and she did not believe that anyone would end up in an ‘orange jumpsuit’ over this project.”

Lesson #2: when your manager claims they don’t believe anyone would end up in an “orange jumpsuit,” assume that someone definitely could. The engineering director’s next step? They refused:

“The Director of Engineering was not persuaded and told Javice and Amar that he would not perform the task, and only would send them the file containing Frank’s actual users, which amounted to approximately 293,000 individuals at the time.”

And this engineering director played it right, as the people who are likely to go to jail and end up in orange jumpsuits are the other two people on the call, who knowingly went along with the illegal.

Pollen: an engineer told to double charge customers by the CEO

Last year, I published my first – and to date only– investigative article on how events tech startup Pollen raised $200M and then collapsed, owing months of wages to staff. In the investigation, I focused on an unusual detail: $3.2M worth of funds taken months early from customers. The incident was described internally by Pollen as a mistake, and an incident review should have followed. Even more confusing, the company blamed the payments processor Stripe for the incident.

The reality was that this was a very deliberate double charge. I could not share this fact at the time – as the company threatened me with libel after I informed them of this detail – but the BBC has now produced a documentary revealing details about this deliberate double charge that was covered up as an outage. From the documentary:

[Narrator] “Pollen initially told some customers that the problem was with their payments provider. Later, Callum [the CEO] addressed his staff who were demanding to know what happened.”

[CEO of Pollen talking] “All that happened was that a couple millions of dollars of payment plans that were due to be paid at a later month were then paid earlier. It’s being investigated. We’ve committed already that once that investigation is done, it will be shared with the company so that people understand what happened.”

[Narrator] “With over 1,500 customers impacted, rumors began to circulate about the causes of the incident.”

[Dan Taylor, managing editor at Tech.eu] “From my understanding, there was a creative code ‘malfunction’ that all of the sudden, double charged customers. But that double charge magically happened to meet Pollen’s payroll, that month. Hmm! Odd, don’t you think?”

[Narrator] “The internal investigation due to be shared with the company was never completed, but a group of Pollen staff did their own, unofficial digging. (...) The code contained in the report confirms that the customer's monthly payment plans had been manually altered, which meant that double or triple charges will take place on a single day, without the customer’s authorization.”

The engineer making this change even did a test run the day before, to ensure that this code change “correctly” double charges customers! A former Pollen software engineer appearing in the documentary also makes the point that any code changing production code in payments needs to go through code review, so whoever made this change could have not been acting alone.

Two days after the incident, a senior engineering team member sent an internal chat message to 3 colleagues, where they admit that they had run the script at the request of the CEO. Here is what this message said:

“Also want to come clean that it was me who ran a bad script - in hindsight I wasn’t knowledgeable enough to alter a subset of payment plans for Balvin [one of the events organized by Pollen]. I did this as a special request from Callum and didn’t want to raise on call to handle. It’s been a long week and I displayed a very poor form of judgement.”

In the video, a Pollen software engineer is shown the message, and he says: “I’m not sure I buy this. It seems a bit fishy.”

Lesson #3: if the CEO asks you to do something potentially illegal – document it, and consider not doing it. We don’t know what happened with the senior engineering member who carried out the code changes, following a request from the CEO. This person could have said no, like the engineering director at Frank did. The message sent a few days ago already said that this person regretted doing so, and it’s unlikely that this action was worth the risk it carried.

If you take one lesson from this, it’s that you can always say no. In these three stories, the only engineer who’s legally safe is the former engineering director at Frank who point blank refused to assist what could be an illegal request. The engineering director at FTX who stayed after he confirmed fraud was occurring is now facing jail time, while the senior engineering member at Pollen is at the mercy of the UK police, and how they deal with what could be a potential wire fraud case. 

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover three out of seven topics from today’s subscriber-only issue Three Cloud Providers, Three Outages: Three Different Responses. To get full issues twice a week, subscribe here.

It’s rare that all three major cloud providers suffer regional outages, but that’s exactly what happened between April and July:

• 25 April 2023: GCP. A Google Cloud region (europe-west-9) went offline for about a day, and a zone was offline for two weeks (europe-west-9-a.) (incident details). We did a deepdive into this incident in What is going on at Google Cloud?
• 13 June 2023: AWS. The largest AWS region (us-east-1) degraded heavily for 3 hours, impacting 104 AWS services. A joke says that when us-east-1 sneezes the whole world feels it, and this was true: Fortnite matchmaking stopped working, McDonalds and Burger King food orders via apps couldn’t be made, and customers of services like Slack, Vercel, Zapier and many more all felt the impact. (incident details). We did a deepdive into this incident earlier in AWS’s us-east-1 outage.
• 5 July 2023: Azure. A region (West Europe) partially went down for about 8 hours due to a major storm in the Netherlands. Customers of Confluent, CloudAmp, and several other vendors running services out of this region suffered disruption. (incident details). We touched on this outage in The Scoop #55: how can a storm damage fiber cables?

A regional outage is rare for any cloud provider because regions are built to be resilient. The fact each major cloud provider suffered one allows us to compare their responses, and take some learnings about best practices. We’ll also learn how this article contributed to AWS publishing its first public postmortem in two years!

Today, we cover:

• 1. Communicating during the incident
• 2. Preliminary incident details
• 3. Incident postmortem and retrospective
  • 3.1 Azure’s standout postmortem and retrospective
  • 3.2 Google Cloud’s detailed incident review
  • 3.3 Silence of AWS (until this article was about to publish)

1. Communicating during the incident

How did the cloud providers communicate during the incidents? A summary:

Google Cloud is the only cloud provider that preserved the incident communication log on the incident page, so we can see which updates occurred and when. I like that every status update followed this template:

“Summary: {short summary}
Description: {more details, based on what is known}
{When the next update can be expected}
Diagnosis: {summary if this is known}
Workaround: {possible workaround for customers}”

The incident at Google Cloud was a particularly nasty one; flames spread through a data center hosting the europe-west-9-a zone, and also some clusters from the europe-west-9-c zone. The incident caused the entire europe-west-9 region to be inaccessible for around 14 hours. There was little to tell customers beyond that they needed to fail over to other regions. Google Cloud posted regular updates sharing what they could, such as this, four hours into the incident:

“Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones if they are impacted.

We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones.”

After the first few hours, updates became pretty much copy-paste and uninformative. However, GCP kept providing them, and was clear about when the next update was due. 

While I appreciate continuous updates, these were unnecessarily verbose and robotic in tone; as if someone was resending the same template every 30 minutes. It was also hard to tell when an update contained new information. Despite that, it’s better to send updates and make them visible after an incident, than to not send them, or remove them post-incident.

AWS did the best job of sharing concise, clear and frequent-enough updates among all the cloud providers. Here are the updates from the first hour of the incident:

“[5 June 2023] 12:08 PM PDT We are investigating increased error rates and latencies in the US-EAST-1 Region.

12:19 PM PDT AWS Lambda function invocation is experiencing elevated error rates. We are working to identify the root cause of this issue.

12:26 PM PDT We have identified the root cause of the elevated errors invoking AWS Lambda functions, and are actively working to resolve this issue.

12:36 PM PDT We are continuing to experience increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. We have identified the root cause as an issue with AWS Lambda, and are actively working toward resolution. For customers attempting to access the AWS Management Console, we recommend using a region-specific endpoint (such as: https://us-west-2.console.aws.amazon.com). We are actively working on full mitigation and will continue to provide regular updates.

1:14 PM PDT We are continuing to work to resolve the error rates invoking Lambda functions. We're also observing elevated errors obtaining temporary credentials from the AWS Security Token Service, and are working in parallel to resolve these errors.

1:38 PM PDT We are beginning to see an improvement in the Lambda function error rates. We are continuing to work towards full recovery.”

Ninety minutes into the incident, an update provided a summary of progress:

“1:48 PM PDT Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard.

Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support.

We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.”

And an update inside the final hour and a half of the outage:

“2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services.

2:29 PM PDT Lambda synchronous invocation APIs have recovered. We are still working on processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services (such as SQS and EventBridge). Lambda is working to process these messages during the next few hours and during this time, we expect to see continued delays in the execution of asynchronous invocations.

2:49 PM PDT We are working to accelerate the rate at which Lambda asynchronous invocations are processed, and now estimate that the queue will be fully processed over the next hour. We expect that all queued invocations will be executed.”

Azure is the only cloud provider whose updates from during the incident cannot be viewed after it was resolved. I tracked this incident at the time and there were regular updates. However, today there is no paper trail to see their contents. 

There is plenty to like about how Azure handles public communications, but it is the only major cloud provider that makes incident notes inaccessible to public view after an outage is resolved.

2. Preliminary incident details

With any incident, mitigation comes first. An investigation starts after everything is back to normal and customers can use a service as normal. This investigation can be time consuming, so it’s hard to tell customers exactly when to expect more details. However, in these cases we aren’t talking about a small engineering team with a few users; but the largest cloud providers in the world upon whom tens of thousands of businesses rely. And these businesses expect some full details when a full investigation is complete. Here’s how the providers compared:

AWS was extremely fast in providing a summary of the incident, and did so in just 5 minutes (!!) after the incident was mitigated. Below is the update added to the status page:

​​3:42 PM PDT Between 11:49 AM PDT and 3:37 PM PDT, we experienced increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use of other AWS services. Additionally, customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS.

Customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support. As of 2:47 PM PDT, the issue initiating calls and chats to AWS Support was resolved.

By 1:41 PM PDT, the underlying issue with the subsystem responsible for AWS Lambda was resolved. At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM PDT, the backlog was fully processed. The issue has been resolved and all AWS Services are operating normally.”

Call me amazed. Credit where it is due, this communication was fast and on point. Unfortunately, it was also almost the last time AWS communicated anything publicly about this outage.

Google Cloud posted a preliminary incident review 14 days after the regional outage was resolved, at the time when the europe-west-9-a zone was still down. The fact that the zonal outage lasted 14 days makes it a little tricky to judge whether Google Cloud posted the preliminary report one day after the full outage was resolved, or 14 days after the regional outage was mitigated.

I am leaving the 14 days here, because a regional outage is far wider-reaching than a zonal outage, and Google did leave customers who were dependent on the europe-west-9 region – but not the europe-west-9-a zone – waiting two weeks for preliminary details.

Azure is the only cloud provider that publishes timelines on preliminary PIRs (production incident reviews) – and for full incident reviews:

“We endeavor to publish a ‘Preliminary’ PIR within 3 days of incident mitigation, to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a ‘Final’ PIR with additional details/learnings.”

In this case, I remember Azure did publish such a review, and it was within the 3-day timeframe. However, after the final review was published the preliminary review was no longer publicly available.

3. Incident postmortem and retrospective

Until this post, the 3 cloud providers did comparably well at handling their incidents. Post-incident is where things diverged significantly:

3.1 Azure’s standout postmortem and retrospective

On 5 July 2023, one of the worst storms to ever hit the Netherlands struck, which I can attest to, first-hand. In The Scoop #55 I shared a photo of a tree uprooted and felled by the storm winds, just a few blocks from my house. I wrote that such an event could tear cables. Well, it turns out this is exactly what happened in Azure’s case; the storm uprooted a tree which yanked a data center fiber path out from under the ground:

The full production incident review was published within two weeks of mitigation, and contained these sections:

• “What happened?”
• “What went wrong and why?“
• “How did we respond?”
• “How are we making incidents like this less likely or less impactful?“
• “How can customers make incidents like this less impactful?”

The storm severed a cable which caused 25% of network links between two West Europe data centers to become unavailable. The problem was that network capacity was already above target utilization – and packets started to drop as a result of the capacity loss. Azure responded by starting to rebalance traffic in its network, and sent technicians to repair the cable. Cable restoration took longer thanks to the extreme weather, and functionality was restored thanks to rebalancing the network.

Azure was the only provider to share dates alongside not-yet-completed action items. Not only was Azure the fastest of all providers to publish their final postmortem: but it is the only cloud provider to do this with dates as ETAs:

“How are we making incidents like this less likely or less impactful?

- We have repaired the impacted networking links, in partnership with our dark fiber provider in the Netherlands. (Completed)

- Within 24 hours of the incident being mitigated we brought additional capacity online, on the impacted network path. (Completed)

- Within a week of the incident, we are 90% complete with our capacity augments that will double capacity in our West Europe region to bring utilization within our design targets. (Estimated completion: July 2023)

- As committed in a previous Post Incident Review (PIR), we are working towards auto-declaring regional incidents to ensure customers get notified more quickly (Estimated completion: August 2023).”

Now this is what I call self-imposed accountability!

Azure took the incident review further than other cloud providers, by holding a post-incident video discussion. This discussion is like a video retrospective, and Azure describes it like this:

“In addition to providing a written post-incident review after major outages, we also now host these retrospective conversations. You’re about to watch a recording of a livestream, where we invited impacted customers through Azure Service Health to join our panel of experts in a live Q&A.”

The video discussion is 23 minutes long, and well worth a watch for anyone interested in reliability or networking:

This video retrospective featured several people:

• The hosts: David Steele (Senior Program Manager on the Post-Incident Transparency team) and Sami Kubba (Principal PM Manager and Communications Team Lead)
• Dave Moltz: Head of the Networking team
• Jitendra Padhye: Partner Software Engineering Manager
• Frank Rey: Partner Technical Program Manager

The video retrospective was surprisingly useful, with a lot of information the PIR did not contain. For example:

• Azure sees about 40 fiber cut issues per day(!), meaning a fiber cut is “business as usual.” During the recording, a fiber cut was being handled in the UK which was invisible to customers.
• The Western Europe region was “running hot” compared to other regions, in not having as much networking redundancy as Azure prefers. The team ended up adding enough capacity to create more resilience.

My impression of Azure from a reliability perspective has greatly improved after watching this incident review. It feels like the Azure team took the incident seriously, were transparent about what happened, and were very clear about what improvements they were making to avoid similar outages. The review also helped reveal the scale at which Azure operates, with more than three dozen fiber cuts occurring every day, globally!

The review closed with these words:

“At the scale at which our [Azure’s] Cloud operates, incidents are inevitable. Just as Microsoft is always learning and improving, we hope our customers and partners can learn from these two, and provide a lot of reliability guidance through the Azure Well-Architected Framework. (...)

We’re really focused on being as transparent as possible and showing up as being accountable after these major incidents.”

Azure not only talks the talk; it walks the walk. Outstanding incident transparency is a blueprint any vendor can follow – if you want to go above and beyond on transparency and accountability, that is.

3.2 Google Cloud’s detailed incident review

The good thing about Google Cloud is that it published a preliminary incident review, and followed up with a postmortem two months after the incident on 23 June, after the region went offline on 25 April. The bad thing is that these two reviews contradict one another: the preliminary postmortem makes clear that two zones operated out of one data center (!), but the final review omitted this rather significant detail. Even worse, Google Cloud did not address this contradiction, despite me asking them about it for about a month. In September, I covered the outage in depth in The Pulse #61: What is Going on at Google Cloud?, writing:

“In Paris, Google Cloud seems to have partially operated two zones from the same data center. The fire that took the “europe-west9” region offline was caused by a water leak in the battery room at a Globalswitch data center, which is where one of Google Cloud's zones operates. 

Google published a preliminary incident report two weeks after the incident, on 10 May. In this report they wrote:

- A water leak caused a fire in a Paris data center, in the battery room.

- The water leak initially impacted a portion of europe-west9-a; however, the subsequent fire required all of europe-west9-a and a portion of europe-west9-c to be temporarily powered down (...) Many regional services were affected while europe-west9-c was partially unavailable.

It’s clear the fire was in the data center where the europe-west9-a zone was based. But why would a fire in europe-west9-a require the powering down of a portion of an independent data center? The obvious explanation is that some instances of europe-west9-c were operated in the same data center location as europe-west9-a operates from! 

I asked Google if europe-west9-a and europe-west9-c are in the same building, at least partially. The company responded, but failed to answer the question.”

Ultimately, Google Cloud went through the process of providing status updates, then providing a preliminary postmortem, and then closing with a final incident review. However, in comparison with the other cloud providers:

• AWS did a better job with concise and informative status updates.
• Both Azure and AWS provided a preliminary incident report much faster than Google Cloud did. Azure had its final incident report ready in the 14 days Google took to publish a preliminary report.
• Azure was more up front in sharing the root cause of the outage, and changes they made for greater resilience. Google did a decent job with the root cause though.

Google Cloud did provide the most detailed written post-incident analysis of all three providers. It also addressed how one building could bring down a whole region, and what they were doing about it, writing:

“Google uses an internal version of regional Spanner as a back-end database to several Google Cloud services such as IAM and various control planes that manage our infrastructure and services for a region. The outage had a regional impact as this regional Spanner was not configured correctly across the three buildings in the region for it to maintain its quorum. Regional Spanner should have had one replica in each of the three buildings in the region. Instead, it had two of its three replicas in two different clusters in the building that was powered down. (...)

We are currently conducting a detailed per-region audit (and conducting any required remediation if needed) of our internal regional Spanner allocations to confirm all regions fully meet Google Cloud expectations for fault isolation to prevent this issue in the future.”

This incident is hard to separate from what looks like a deliberate design choice by Google to partially operate two zones out of one data center. This is the type of approach that would be impossible to fathom AWS doing because the company is clear that “two zones” always means two separate locations, far enough away from each other. So even if a data center goes up in flames – as happened to Google – the region is still operational.

3.3 Silence of AWS (until this article was about to publish)

AWS did not follow up with any form of postmortem for more than four months after the incident. The company maintains a page with postmortems, stating that the page contains incidents deemed significant. On average, there has been once postmortem published per year, and none since December 2021:

We can easily compare this publishing cadence with other providers:

Looking at 2023 incidents for 2023:

• Google Cloud published more than 100 production incident reviews, and is the most granular of all providers in reporting incidents
• Azure published 15 production incident reviews
• AWS published zero… until 20 October

As I was writing this article, it struck me as very odd that despite AWS’s outage having the largest customer impact and being felt across the globe, AWS was the only provider not publishing any form of postmortem. So I reached out to the company on 15 October – more than 4 months after the outage of 4 June – and asked if it would publish a postmortem. 

I said I was writing this article which compares how cloud providers respond to regional outages, and also that out of GCP, Azure, and AWS, only they (AWS) had failed to publish a postmortem for a major outage. I also asked for a definition of what “broad and significant customer impact” means: as this is the definition AWS uses in deciding when to publish a public postmortem.

On 19 October, a spokesperson from AWS responded, and asked for a bit more time. On 20 October, this person responded by linking to a new postmortem. This was the first public postmortem from AWS in two years!

It is pretty clear AWS published this postmortem because of the press inquiry. While I’m glad about this, it makes me wonder: why does it take someone writing an article about how AWS avoids publishing postmortems after major incidents, for AWS to finally publish a postmortem?

AWS’s public postmortem lacks some key details about what happened. In the face of AWS’s absence of public communication on the outage at the time, I did some digging to find out what caused such a large region to go down. From June’s The Scoop #52: AWS’s us-east-outage:

“While no postmortem yet published, I talked with current Amazon engineers who said it was a load test for AWS Lambda which caused the incident:

- The load test seems to have overloaded AWS Lambda. Many services depend on Lambda and started to degrade

- When the engineering team figured out the likely cause, they killed the load test and added more capacity to Lambda. The test was killed about an hour into the outage

- The next 2-3 hours of the outage were spent recovering degraded services. Some needed to be restarted, and the recovery of others was slower than expected”

The postmortem AWS shared gives more details on how Lambda works, and why it couldn’t handle more capacity. However, the postmortem omits mentioning the increase in load was due to a load test, and that mitigation involved stopping the load test.

It’s actually a great sign that AWS regularly does load testing; but within AWS there were plenty of questions about why it took so long to notice that capacity was pushed overboard by the test, and why it took so long to stop the test. The public postmortem reads as abstract, and I don’t get the feeling that transparency was a goal of this document.

These were three out of the seven topics covered in the subscriber-only article Three Cloud Providers, Three Outages: Three Different Responses. The full edition additionally covers:

• What is a cloud region and how does it differ between cloud providers? A recap.
• Why is AWS so opaque to the public?
• Why is Azure stepping up in transparency and accountability?
• Lessons for engineering teams from the three cloud providers

Read the full article here

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover two out of seven topics from today’s full issue on The Man Behind the Big Tech Comics. To get full issues twice a week, subscribe here.

If you ever worked at Google, I probably don’t need to introduce Manu Cornet. And if you’re not a Googler or a Xoogler, then here’s a short recap of some of his comics. For example, if you don’t know what a Xoogler is (ex Googler) or a Noogler (new Googler) or a Doogler (the pet dogs owned by Googlers) and now you’re wondering why Google has all these terms, Manu created a comic on this:

Today’s issue will be a lot more visual than usual. The topic will also be more lighthearted: Big Tech, Twitter, and comics. Manu recently published his book Twitoons: one employee’s cartoon chronicle of Twitter’s accelerated descent. I read the book: it’s funny! I still can’t decide if it’s funny because of Manu’s style, or if it's funny because it’s true.

Today’s newsletter closes with a full chapter from this book, visualizing when Elon Musk demanded all Twitter software engineers print out their code on paper (!!) and report for code review. But before we get there, we touch on how one of the most referenced Big Tech comics was created. This one.

With this, my questions and comments are in italic, and Manu’s answers in regular fonts.

As usual, none of these links are affiliate ones, and I have not been paid to mention any of them. More on this here.

Drawing the famous Microsoft, Oracle, Amazon, Facebook, Google, Apple comic

How did you come up with the “famous” organizational chart comic that you are most known for?

Initially, the punch line was meant to be Oracle. At the time, this company was suing Google over its use of Java, and I decided to make fun of how Oracle has a much larger legal department than engineering, so I drew this:

To make a good punchline, you need a buildup. So to make this a good comic, I needed to draw other companies for a comparison. So I drew four more:

I still needed one last one, to make it six and make it symmetric. Microsoft was, of course, the obvious missing one. So I researched the culture at Microsoft a little bit and came up with that — now famous — gun drawing:

After you’ve been working on an idea for a while, you completely lose track of whether it’s actually funny or not. After I finished it, I didn’t find it funny. I showed it to my partner at the time, and she didn’t find it funny either. I was very close to not releasing this cartoon, in the end! Then I just published it.

It’s ironic because this comic became probably my most popular comic, and the Microsoft drawing with the guns was the one that people found the most funny. It shows you how clueless I am about predicting what will work!

Code review on printed paper: an excerpt from the Twitoons book

A year ago, the end of October 2022 was a very turbulent time at Twitter. In the article Turmoil at Twitter, we covered both how engineers were told to print out code on paper (!!) for Elon Musk to review, as well as how firings on the spot had started. Manu was one of the several people let go after pulling a 7-day week, working through the weekend. 

Manu’s book covers this period with far more detail than we’ve accessed, along with new, previously unpublished illustrations. The below excerpt is from Part 6 of the Twitoons book, from the chapter titled “Code Review.”

Despite the long hours, we were all still painfully aware that the chopping block was dangerously near.

Regardless of how hard we worked, would the dreaded termination email still show up, on a bright Sunday, with the usual (and newly ironic) friendly chime?

Indeed, in parallel with the urgent product changes, Elon Musk kickstarted the dreaded “reduction in force.”. Given that he had clearly stated he wanted virtually no moderation on the platform, it was easy to guess what would happen to all the teams responsible for that: all fired. For engineers, it was a little trickier: he needed at least some of them to keep the site running. But which ones?

One important problem with software engineering is that evaluating workers is extremely difficult. Large technology companies spend an enormous amount of resources trying to monitor, evaluate, and eventually accurately reward contributions from their expensive workforce.

For a team that makes cars in a factory, measuring performance isn't exactly easy, but it's feasible: you measure the number of cars produced per month, you take into account things like customer return rate and mechanical failures, and with a little trial-and-error you can reach something reasonable.

For software, it's foolish to measure quantity: a shorter program (fewer lines of code) is often better than a longer one. You could measure things like bug reports or software crashes, but most projects are so complex and involve so many moving parts produced by different teams and intertwined dependencies that any such attempt usually turns out to be a wild goose chase.

So performance measurement is already difficult in normal conditions, when it serves as an input to decisions affecting an employee's role and career. Now imagine you are part of a team of engineers coming from Tesla, tasked with determining which 20% of engineers Twitter should continue employing, and which 80% it should fire. On top of that, you need to figure it out by next Monday because otherwise they all get more stock options the following week, as planned, and that's gonna cost your boss. And on top of that, consider that the engineers doing the evaluation are not only unfamiliar with the material they need to evaluate (the code that makes Twitter work); they aren't even familiar with the very industry, since the software they usually write is destined for cars. It's like asking a tricycle maker to pilot a spaceship because, well, it's all transportation.

Tesla engineers thus resorted to measuring what most experts will tell you is completely meaningless: the quantity of code contributed to Twitter by each engineer. The number of files they changed, insertions, and deletions (lines of code added and removed). As if these numbers were the only thing that mattered in one's entire tenure, sometimes their entire career.

Given the harsh conditions and time constraints the Tesla folks were under (and had, by the way, probably been under for months or years), it's hard to blame them. Meaningless, but expeditious.

The whole process was dubbed by Mr. Musk a “code review” where each Twitter engineer was asked to pick their own “most salient” code contributions, print them out on paper, and bring them to their interview, nay, their trial, by Tesla engineers. The printing out part caused so much mess and confusion that the process was eventually simplified and people could use a computer screen to showcase the reasons why they claimed they shouldn't be fired.

Let me emphasize that this is a complete perversion of the common expression “code review”: engineers usually review each other's code in a well-established practice aimed at catching bugs, making the code better, and sharing knowledge. Not at selecting people for employment termination.

Oh, and knowing Elon, he might have devised a slightly different flavor of ”reviews” for female employees?

For those who were paying close attention to the timeline of employee terminations, even that raw code quantity number didn't seem to explain it at all. My team's most productive engineer by a long run, in terms of pure quantity, was let go alongside everyone else. There were also petty little arrangements where an engineering director who knew one of Elon's goons requested that employees X and Y should be kept on the payroll, and so on. But to whoever wasn't privy to all this underground barter...

...it all seemed rather random, and it probably was.

The press, who had a good idea of what was happening at the headquarters, sent reporters to camp at the entrance in hopes of getting a quote or a video clip from fired employees. Two pranksters who claimed to be Rahul Ligma and Daniel Johnson showed up with cardboard boxes and were able to temporarily convince the media they were part of the early layoffs, starting a trail of misinformation on social media. Ironic.

As far as I was concerned, I'm glad I didn't have to submit to any of those “code review” sessions, since I was fired on November 1, a few days before the large waves of layoffs. As I was attending a team meeting about a new critically important project, I was kicked out of the meeting, my work email and calendar blanked out, and my work laptop locked itself up. I thought I may have been hacked. It was only half an hour later that I received an actual termination email. It didn't state a cause, but the apparent immediate reason was that I was trying to help my coworkers safeguard some documentation (performance reviews, praises from peers, etc.) that would be valuable should they need to seek employment. 

More broadly, did it have anything to do with my critical cartoons? It's hard to tell, since the person who fired me (my boss's boss, a.k.a. director) was also fired the following week (I actually kind of liked him, and still do). So was my direct boss. And the director's boss (I quite liked her, too). And her boss.

Anyway, it's fair to say that I was deemed a little too much of a troublemaker under the new leadership. I don't blame them.

I had been planning to make a cartoon where I would depict myself standing in front of the office entrance, alongside reporters and recruiters. After my termination, the cartoon (you wouldn't think I would stop drawing about Twitter just because I was fired, would you?) took a slightly different meaning, which I quite liked.

This is Gergely again.

If you enjoyed these comics, you can buy Manu’s book Twitoons, here — priced at a modest $12.90 — and the one on Google’s corporate culture, here. You can view more of Manu’s comics on his website. As usual, none of these links are affiliate ones, and I have not been paid to mention any of them. More on this here.

It’s been almost exactly one year that about 50% of the staff at Twitter got fired — Manu being booted a few days earlier than most. I asked Manu if he had received severance owed to him as per his contract: and he has not gotten anything just yet.

Other former Twitter engineers are in the same situation as Manu — waiting on severance overdue. In a smaller win for former emplyoees, the National Labor Relations board issues its first complaint against X Corp for illegally firing Yao Yue, a widely respected principal engineer at the company.

Zoë Schiffer at Platformer reported this in the article How one former Twitter employee could beat Elon Musk in court. Still, this case only goes to trial in January — and Elon Musk’s strategy clearly seems to be to not worry about potentially breaking regulations in the impulse of the moment: instead, deal with these years later, in court. More than 2,000 employees have filed arbitration claims against Twitter: and eventually the jury will get to all of these, and decide on severance due.

Originally, I reached out to Manu intending just to ask for permission to publish a part of his book, as it was quite relevant for last year’s events at Twitter. But as we started talking, I realized I’ve never met a real-life cartoonist before, much less a software engineer cartoonist! So our conversation turned into a lot more than just about the latest book.

In the full issue of today’s newsletter, we dive into several other topics we discussed:

1. The motivation to draw caricature comics. How Manu started to draw comics, and why.
2. Fourteen years at Google as a software engineer. Getting into Google, spending 8 years on the Gmail team; working on Android and ChromeOS, and what Manu learned from his time at Google.
3. Comics at Google. About 1/10th of Google employees subscribed to Manu’s comics. When did Manu get into the most trouble because of comics?
4. Working at Twitter. Twitter’s web app and tech stack. Internal tooling at Twitter, versus Google.
5. The best and the worst of “old Twitter”. The things “Twitter 1.0” did vey well, and where it clearly struggled — Manu’s insider view.
6. How to draw cartoons as a software engineer. The similarities and differences in drawing a good cartoon vs writing good code; Manu’s cartoon drawing toolset; and advice to get started.

Read the full article here.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover three out of eight topics from today’s deepdive into tech scaleup Chronosphere. To get full issues twice a week, subscribe here.

We don’t frequently have CEOs on The Pragmatic Engineer: in fact, today is the first issue when we’re talking with a CEO – partially – about their CEO job. I’ve always wondered what it would be like to go from a developer, to eventually become the CEO of a large and growing company.

Martin Mao travelled this path: starting his first developer job at a local startup in Australia, then making the move to the US – first being at Microsoft in Seattle, then at Amazon, and then at Uber, in Silicon Valley. He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. We covered more on this topic in the article How Uber built its observability platform.

However, Martin had not written a line of production code for the last four years, as he’s taken on the role of CEO, and heads up observability scaleup Chronosphere – at more than 250 people and growing. The company supports billions of active time series across its customer fleet. Chronosphere was actually founded by two former software engineers: Rob Skillington – the CTO of the company – and Martin, acting as CEO. It’s not common to have only engineering founders at a company: my gut feel is that at infrastructure startups, such a setup can make a lot more sense.

Today, we cover:

1. From learning to code in Australia, to working in Silicon Valley
2. Launching a startup
3. Going from developer to CEO

With this, it’s over to Martin, who tells his story and applicable learnings.

1. From learning to code in Australia, to working in Silicon Valley

How did I learn to code? In high school, in Australia, Prashant Varanasi was one of my good friends. He was learning to code, and needed help in a programming competition, because he needed a team of three. He asked for help, and taught me the basics of Visual Basic, so we could enter the competition. And we ended up doing well in it, too! Afterwards, we both won scholarships to college to study computer science. Prashant would later work as a senior software engineer at Google, and a distinguished engineer at Uber, but, of course, back then, I didn’t know this.

In college, I tried my hand at a couple of startups. I founded a text messaging startup at first. Then, I joined a startup that tried to beat Facebook, and become the #1 social media company in Australia, around 2007. At this social media startup, we put a lot of work into importing your contacts from Microsoft Messenger. But when Facebook entered Australia, they pretty much crushed us.

After graduating, I began looking for a more serious job. In 2009, the tech scene in Australia was not as vibrant as it is today: Atlassian was still small and Canva didn’t exist. Google was already on the scene, and I interned at the Google office, working on Google Docs. However, Google paid a fraction of what they did in the US, so I set my eyes on the United States.

Microsoft

In 2009, not many US tech companies were hiring, as the sector was still recovering from the 2008 crash. Microsoft, however, was, and I got an internship to work there. They accepted five interns from Australia, and I was lucky enough to be one of those five.

When I went to intern there, I thought I would spend three cool months in Seattle, and that would be it. I had no idea you could get offered a full-time job at the end of an internship, until the internship ended and Microsoft came to me, saying, “Oh, hey, here’s your job offer.” It was an offer to stay in Seattle. On top of the option to continue working at Microsoft, it was also the best monetary opportunity as well, so I took it without a second thought.

I worked on Office 365, as it was born. I joined the Microsoft Office team but was tasked to work on a new product. My team was taking the Office product distributed on CD ROMs and making it into a SaaS solution. They called it Office 365, and in 2010, this was a really exciting project to work on. We then shipped the first version, and then I felt that it was time to move on.

See, I was working at Steve Ballmer-era Microsoft. We had a target to hit a two-year release cycle. They let me code full-on for eight weeks, and then told me to stop. I asked what I should be doing next. They said, “It’s now QA time!” 

I tried to push bug fixes, but was told that it was too risky. I was, however, told that I could get those fixes ready for the next release. But the next release was not for another two years’ time!

After a few years, I thought to myself:

“What am I doing here? I’m not productive at all. I should do something else with my life.”

Amazon

I attribute a lot of things to luck, and how I ended up at Amazon was one of these. My mentor at Microsoft was a senior engineer who had moved over to Amazon and ended up running a part of the EC2 team. This was around 2013, and EC2 was one of the hottest services at Amazon.

For some reason, Amazon didn’t have any senior engineers on the part of the EC2 team that was responsible for the EC2 instance management, and for the Windows business. The Windows business was itself a billion-dollar business at the time! So my former mentor reached out and asked, “Hey, do you want to be a tech lead on EC2?” I thought about it for a while, and then decided I couldn’t turn down an opportunity like that.

And it was really interesting to work on EC2. I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. So we took on the challenge of fixing this.

With my team, we built the basics of what is now called AWS Systems Manager. This was a remote management tool, to do things like:

• SSH remotely into machines, via a browser interface
• Roll out patch updates
• Manage rolling updates for several machines

We built it, shipped it, and presented it at re:Invent – Amazon’s annual flagship event. It was a lot of fun!

After a couple of years, it felt to me the mood changed to most teams just wanting to ship new services. The focus seemed to shift to: invent something new → build a service for it → ship it. Once you do this a couple of times – building and shipping a new service – it starts to become easy enough. The discussion I had with people went along the lines of, “Oh, I know what service you want to ship. I know how to do it. It’s not that hard to do, anymore.”

It was around this time that Rob Skillington – now my cofounder at Chronosphere – called me up and convinced me to join this startup called Uber, and said it would be a crazy ride! 

Uber

The pitch to join Uber wasn’t just the business vision, but also the engineering one. The engineering pitch went something like, “Hey, here’s your chance to come in and work on a system where we don’t use the public cloud; we own our own datacenters, and we get to build infrastructure services that could one day become the size of Google’s infrastructure.” It was a pitch that was hard to say no to!

I joined in 2015, when there were a couple hundred engineers at Uber, and the infra team was made up of about 50 people.

Rob was right that Uber would be crazy. It was at Uber where we ended up building M3: Uber’s open source, large-scale metrics platform. We previously covered the building of M3 in the article How Uber Built its Observability Platform.

2. Launching a startup

At Uber, I felt like we had completed the core mission of M3. We spun up a reliable observability platform, and rolled it out. Virtually every engineer was using the platform. The roadmap, frankly, was not that exciting, compared to the challenges we had to overcome in the past.

I decided to look around. I looked at both internal and external opportunities. And of course, I also considered building something from scratch.

In 2019, Uber didn’t have a ton of opportunities, internally. The speed of execution felt slower than it had been in earlier years, and truly hard problems were hard to come by.

On the other hand, I found the idea of building a company very compelling. We originally built M3 to support Uber’s scale on a microservices/container (cloud native) architecture. It was that very architecture that created the scale, reliability, performance and cost-efficiency challenges. When we built M3, we thought only Uber and a handful of companies in the world would need something like this: such as Google, Facebook and Netflix.

Kubernetes changed the observability needs of so many companies. As Kubernetes really picked up in popularity, it was clear the majority of the world’s architecture would look like Uber’s architecture did, and that M3 would be a solution to observability needs. We just so happened to have M3 open sourced – ready to be used in the open, as well. 

We then looked at the competition. No one was in a position to solve this problem! None of the existing vendors had built and optimized for a cloud-native architecture. At scale, pricing was out of hand with all solutions.

The cloud-native observability market seemed one that would be ripe for disruption. And we just so happened to have the perfect solution, already built and tested at scale. I remember thinking that if we ever wanted to start a company, there would never be a better opportunity from a timing perspective.

Finding a cofounder

My cofounder is Rob Skillington. We are both Australian, have known each other for over a decade, and worked together closely for four years at Uber on M3. In fact, he even officiated at my wedding! 

My personal and biased belief is that starting a company with someone you know and trust really helps. Building a company is hard. There are a lot of tough moments, which might prove to be even tougher with someone you don’t know as well.

Find someone who’s interested in a different part of the business from you, and someone who brings different skill sets. It’s common to have two engineering technical cofounders, but generally one has to focus more on the business side. In our case, Rob remained technical and I focused on the business.

Finding one cofounder who is technical and one who’s interested in the business is not as easy as you might think, either, because both founders need to want to enjoy their area. I see so many technical folks that just don't want to move over to the business side. This is fine, but then you need to go and find a CEO so you can focus on what you love to do!

Fundraising

Fundraising turned out to be pretty straightforward. Firms like Sequoia and Greylock were reaching out to us, when they heard we were planning to start a company. These VC firms have these scout networks of folks who work in Big Tech whose job it is to look out for interesting tech to invest in: and this was how they found us.

We put together an 80-page business plan. This business plan was not really for the investors, but rather, to help us think about the business. In this document, we covered:

• The product
• The market
• The go-to-market (GTM) plan
• Our competitors
• … and many other things!

We worked on a few iterations of this plan, and once it was solidified, this gave us the conviction that we were onto something. With the plan in place, we sent this document – rather than the usual pitch deck – over to the VCs.

Our approach was unorthodox, but worked! The document answered all the questions they had, and we still did a pitch, of course, with a deck. We ended up pitching to a firm and partner we liked, early on: Jerry Chen at Greylock. Jerry gave us a term sheet on the spot, and we took it.

Launching the company was easier than I imagine many founders have it. After all, we already  had open source technology to use – M3 – and we also had a clear path on what we wanted to build, similar to the roadmap the observability team at Uber used to have. 

When we started out building the company, it felt a little like a continuation of our path from Uber. We were heads down building the product we wished we could have bought.

Here's the timeline of our fundraising story since the launch of the company:

• May 2019: founding the company
• Nov 2019: the first round of funding led by Greylock: Series A ($11M)
• Jan 2021: Series B ($43M)
• Oct 2021: Series C ($200M)
• Jan 2022: Series C-1 ($115M)

An interesting bit of history is how the first funding round at most startups is called either a pre-seed, or a seed round. But we didn’t have one of these! We knew what we wanted to build, and knew there was a demand for it, and we had validation from the M3 open source community on the demand. We had the technology ready in the form of the M3 project, which was a massive achievement by itself.

We needed $11M to get started. When we told Greylock that we were raising $11M, they said that that figure was too large to be called a seed round. Still, we didn’t get hung up on the name. We just called it a Series A.

3. Going from developer to CEO

With my cofounder, Rob, we quickly established that he would remain technical, and I would be on the business side. 

By the end of my time at Uber, I didn’t feel that I had to prove myself in engineering any more. I was there from the beginning when we built M3. I could still get joy out of technical stuff, but I've proven that I'm a solid engineer, and that I'm an okay engineering leader. I didn’t feel like I needed to go and prove anything more in those areas.

Still: I had this hunger to learn about how the rest of the business worked. At Uber, I wasn’t that close to the business and had no idea about the details. Being the CEO, I got to learn so much of this!

My day-to-day life now, as CEO, is very different to when I was an engineer or even an engineering manager. Right at the start, I still had GitHub access and did some reviews. But I was definitely the first and only member of the early team to become “non-technical.” I had to take care of everything else that wasn’t technical at the company. 

Now my days involve not just back-to-back meetings, but also tons of context switching! It’s like, "Oh, I have to switch context to a marketing meeting where we're talking about a demand generation strategy to one about customer success to one about how to sell to customers."

Over time, I built up a support network. Below are a few things I did to find people who could help me navigate the new problems I faced:

• Leaning on VCs. VCs don’t have as much domain knowledge, but they have a vast network. I would ping my VCs, telling them: "Hey, I'm trying to solve this marketing problem. Can you put me in touch with 3-4 people in your network that could help me walk through it externally?"
• Leaning on advisors. For people who have been helpful, I tend to keep in mind as an advisor. For example, I have a sales advisor, a marketing advisor, and people who can advise in a few other areas.
• Seeking out other founders and CEOs. I never realized how most founders and CEOs give a lot more time than you would expect. For example, one CEO in particular, from a publicly traded infrastructure company, has helped me out no end. If I text this CEO about something, he’ll make time to help and offer support, even though I know how busy he is. I notice that most such CEOs are really protective of their time, yet they’re still willing to carve out time for other founders. This makes me want to pay it forward as well, and help earlier-stage founders too.

This was three one out of the eight topics covered in this week’s The Pulse. The full edition additionally covers:

• The early days of Chronosphere
• The need to introduce a career ladder
• The impact of Uber and future plans
• Advice for software engineers
• Appendix: Chronosphere’s 80-page(!) business plan

Read the full issue here.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in today’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here.

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. We started as teammates, and in a twist, I became Willem’s manager. He was a very productive engineer at Uber – which I regularly told the manager group during performance calibrations – and he was the embodiment of what I like to call a product-minded engineer. After spending 6 years at Uber, Willem cofounded Rise Calendar. Disclaimer: I’m an investor in this company, having invested due to knowing Willem so well after working together for years.

Willem is two and a half years into running his own startup, which just launched the public beta. I thought this a great time to ask about his experience between working at a startup, and working at a large company, as it’s the second time Willem has gone from a large company to startup.

With this, it’s over to Willem. The material below is straight from him:

Background

Early in my career, I founded two startups. One was Yunoo, a personal budgeting tool for the Dutch market. Four years after we launched, the company was acquired by a mid-sized local company in the enterprise resource planning (ERP) sector. I stayed with this ERP company for a year. But I had the startup bug in me, and so founded a second startup. This last one was a great adventure; we went to the US to join the TechStars 2013 accelerator. However, this startup failed miserably.

I was still recovering from this failure in 2015, when a friend who designed the first version of Uber– Jelle Prins – pinged me, and said Uber was kickstarting an engineering office in Amsterdam. By that time, I’d been following Uber for years, and suspected that the ridesharing industry would only get bigger.

My time at Uber was hypergrowth as its finest. In 2015 when I joined, we had 6 engineers in the Amsterdam office. Six years later, we were at 250 people. For most of my time, I was part of the Rider Payments team and worked for 4 years with Gergely.

In 2021, I left Uber for Rise Calendar as a technical co-founder. Rise is a beautiful calendar app that helps you schedule and protects your week. Since then, we’ve raised a pre-seed round – when Gergely got involved as an angel investor – and have grown the team to 7 people.

Having spent about 15 years split between startups and Big Tech, here’s my honest take on the good, the bad, and the “ugly” of each environment.

Startups: the good

Learning: Startups are such amazing places to learn in! You usually have a small developer team, and for the most part, no infrastructure, security, data science, or observability teams – which are all commonplace in Big Tech. Engineers are expected to jump in where needed, even when you don’t have expertise.

So you are forced to learn about specific topics and immediately apply them in production. Just by doing this, you broaden your expertise incredibly fast in a wide range of topics.

Impact: you can make a large impact and directly influence the direction of the company, and very clearly influence the success of the company, as well. Startups are very flat organizations, meaning you are often involved in strategic decisions.

An example of making a company-wide impact at Rise was during our Christmas hackathon, when an engineer built a Calendly-like feature for anyone to book a slot in your calendar. It worked so well that we launched the feature a month later. Going from idea, to adding, to building and shipping this to all customers, is something you rarely see in bigger companies.

Startups: the bad

Financial risk: Startups tend to pay a lower base salary than Big Tech. They rarely have the annual performance-based bonuses that are typical at larger companies. Many startups give equity to engineers, and some startups hand out handsome equity packages to early employees. Uber is a prime example of this, where early engineers did very well from generous early packages.

But let’s be honest, with any startup there is significant risk. Plenty of startups fail and close, or end up being acquired or acqui-hired. Acqui-hiring often means another company takes over the startup’s team: but does it on terms where investors – or even the founders – don’t see much of a return on their investment and equity.

Stress: startups are often races against the clock. This focus on speed and execution often leads to fast context-switching and a working environment with constant time pressure. My personal experience is that startups are more stressful than Big Tech, where there’s the option to “lean back” sometimes. This is not so much the case at a smaller company. It’s important to add that this depends very much on the culture, and my experience won’t be universal!

Startups: the ugly

Failure: Startups fail, more often than not. This can lead to ugly situations where people lose their jobs pretty much overnight. Startups are usually a tighter-knit group because people feel a personal connection to the company and product, much more than is the case in Big Tech. So shutting down a product or the entire company can be a lot more emotional than a large company shutting down one of its many products.

Big Tech: The good

Specializing: Big Tech organizations have so many specialized teams! At Uber, they ranged from mobile feature teams (called program teams – here is some history about this split,) platform teams, networking teams, and infra ones.

Internal transfers: usually this is surprisingly easy! I’ve moved between teams, and so have so many engineers, like a Rider Payments engineer who moved over to the Developer Platform team. Combined with an environment with many specialized teams, and you have a place where engineers can go really deep in a domain while working on a variety of things.

Financial stability/perks: speaking for software engineers, Big Tech pays well and almost always issues performance-based cash bonuses. Then there’s the equity granted, which is more stable and lower risk compared to startup equity.

Joining a Big Tech, and building up a “financial buffer” before going to a startup, is something I’ve seen plenty of engineers do. Things may always go south for a startup, so it’s good to have reserves!

Networking: One of the most underestimated things about Big Tech is that it’s surprisingly easy to build a large network while working there! Many of my former Uber colleagues joined other Big Tech companies, or started a new business – and there’s this one guy who ended up writing this newsletter which blew up.

I will admit the relationships I built up at Big Tech have proved to be far more valuable in the long run than I expected!

Big Tech: the bad

Lack of purpose: In a startup, it’s easy to make an immediate impact for the end-user. I used to work with Engineer #1 at Uber, Conrad Whelan, who told me this in 2015:

“If an unexpected but important task takes less than 30 minutes, just get it out immediately! Push it to production.”

Back then, Uber was akin to a startup, and not (yet) a Big Tech business. At Rise, we perform between 25 and 50 deploys per day with a team of 7. Such a pace generates a ton of energy in the team. Our end users are amazed when they see a bug they reported fixed within minutes, or when a feature they suggested is shipped a few hours later. This feeling of actual impact and purpose is much reduced in Big Tech. Also, changes take longer.

Overhead and bureaucracy: on the Rider Payments team at Uber, we had the running joke: “we’re all just plumbers here 👩‍🔧.” This referred to how much “plumbing” was needed to build even a relatively simple feature. Uber had thousands of microservices, and our team regularly made changes to more than 20 of them. Many of the changes were simply to proxy a value from one service to another.

The problem? Each of those changes needed to be approved by the team owning the service. These teams were in different time zones, and so even straightforward “plumbing” changes could take days to land.

And it got worse. For more complex changes, we usually needed the teams owning the services to carry them out. So we had a quarterly planning process to ensure all project dependencies were incorporated into each team’s roadmap. But priorities changed frequently and – surprise, surprise! – we frequently found ourselves stuck and waiting on other teams.

If this seems to be becoming a rant, it’s because it is. Uber was a fast-moving company, but also a big company, so things started to take a lot more time. At a startup, when you have a monolith the same changes take hours, not days.

Big Tech: the ugly

I had to think quite a bit to come up with anything “ugly.” There are plenty of unpleasant things at larger companies, but nothing terrible. However, one thing really got under my skin.

Misaligned incentives around performance reviews. Doing work that results in a great performance review is not always the same work that best helps the company. And this can create pretty twisted, political situations. For example:

A recently joined senior staff engineer decided to propose a project to do a re-architecture of a system. This person wrote up a neat document that was well thought out, and sent it around to other senior staff engineers. But there was a problem, this engineer took an existing document that other engineers had written a few months before, copy-pasted it, changed a few words, and presented it as their own work.

Clearly, this engineer was doing this in hopes of demonstrating how quickly they got up to speed, and how valuable their own, personal contributions were. But by not involving the original authors, the proposal lacked context that was not in this document – such as why it wasn’t made, in the end.

This is just one example where pursuing personal recognition, that comes in the form of performance bonuses, can result in actions that are not in the best interest of the company. Anyone working at a large company sees things like this.

Which is better?

This is Gergely again.

Thanks Willem, for that dose of honesty! I have to admit I always suspected Willem would go back to working at – or founding – a startup. It was pretty clear that the more “Big Tech” Uber became, the more frustrating it was for him to wait on others, versus making code changes himself. And this attitude made Willem a really productive engineer, as he jumped to working on something else while waiting on another team. During performance calibration sessions, as Willem’s manager I had an easy job proving to other managers why he belonged in the top quartile in most years, thanks to his consistently high output.

Willem’s startup background really helped him get things done, and not wait on others. At the same time, now that he’s on his third startup, I cannot unsee how his Big Tech experience of structuring things, planning ahead and building reliable systems, has proved very valuable.

Spending time at a large company and a startup, is probably as good as it gets. Both environments have very different dynamics, and you need to lean on distinct skills to succeed in each. A more common path I see is devs going from startups to Big Tech because of the increase in compensation. Although not as frequent, I also see plenty of people go the other way, from Big Tech to startups. This step feels like a bigger adjustment, especially for those who have not worked at startups before.

Growing your network is indeed an underrated benefit of working at a larger company, either Big Tech or a fast-growing scaleup. You can meet so many more people at these companies, and this is even more true if you move teams, or the company is growing quickly. I echo what Willem said about how it’s only years later that you appreciate this value. Even for this newsletter, the first few interviews were all with people I met and worked with at Uber: Ganesh Srinivasan, Adam Rogal and Sophia Vincent.

A final note on Willem’s new startup. I got involved more than two years ago and assumed the team would ship something in a few months. After all, we were talking about Willem, who was a productivity machine at Uber! But this is not what happened. For over two years, Rise kept running a private beta product. Launching so slowly is quite rare for startups, Figma is one of the best-known examples for staying in private beta for a similar duration. Rise Calendar is now out of stealth mode. And it turns out there was a reason to take so much time to build: it was to opt to launch with a product that is polished.

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

• Industry pulse. A roundup of recent events in tech, with commentary.
• Are we about to see a spike in startup shutdowns? I’ve gathered evidence which suggests startup shutdowns are on the rise, and plenty of infra and developer tooling startups are also in deep trouble.
• Is down-leveling back in Big Tech? In 2021, upleveling was common across all of Big Tech. This is no longer the case, and job market dynamics are likely to make down-leveling more common.

Read the full The Pulse.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover a fresh industry trends: Cloud Developent Environments — which is analysis full subscribers have received 3 weeks ago. To stay up to date with trends across the tech industry, subscribe here.

There is a quiet revolution unfolding at tech companies: cloud development environments (CDEs). The past three months, I shared insights with full subscribers:

• Cloud development environments at tech companies (27 Jun 2023)
• Why are CDEs spiking in popularity, now? (5 Sep 2023)
• The Cloud development environment vendor landscape (26 Sep 2023)

The trend is so fresh that even Gartner — an authority in detecting trends — has only mentioned CDEs for the first time two months after The Pragmatic Engineer analyzed this category. In today’s issue, we cover why this trend is popular around now, and what are startups to watch for in this space.

Why are CDEs gaining popularity, now?

The concept of remote development environments stretches back to the dawn of computer programming. In the 1960s, expensive central computers were shared by many users, sitting at terminals. Each user had a specific amount of time to use the processor in this setup called “timesharing.”

Cloud development environments are similar to how programmers used a terminal to program a central computer, back then. A big difference is that in the case of old-school timesharing, the goal was to save on infrastructure costs, as the cost of CPU cycles was far higher than programming time. A single mainframe machine cost several times the annual salary of a programmer; the Atlas computer at Manchester University in the UK cost £50M in today’s money, the equivalent of several hundred programmers’ annual salaries. No wonder compute time was so valuable!

Today, it is compute that’s much cheaper than software engineers’ time. In the US, a software engineer working at a Big Tech company is compensated more than 100 times the cost of a high-end laptop, annually. 

So why the sudden rise in use of remote environments? I observe several trends which make this shift understandable and sensible:

Here are the reasons:

1. Larger codebases. Every day, there’s more code at a tech company, not less. This means more repositories are needed, which are fast enough to build and work with, but which increase fragmentation. At many large companies, this inevitably leads to consolidating many smaller repositories into fewer, larger ones. Which leads to the next point…

2. Monorepos. These are becoming more popular at large tech companies for a few reasons. The biggest is consistency; with a monorepo, dependencies are clear and API breakages due to outdated dependencies are absent. Also, tasks like integration testing are much easier. However, monorepos result in codebases growing large, so that even checking out the code or updating to the head can be time consuming.

3. Laptop compute power is plateauing. The available compute power of laptops and personal computers no longer increases as it did prior to 2010. Moore’s Law says the number of transistors in an integrated circuit (IC) doubles around every two years. But the most recent 10 years suggest this is no longer the case.

4. The Cloud is spreading and offering more capabilities. Since around 2010, there’s been a boom in Cloud computing. AWS launched in 2006, Azure in 2010, and GCP launched its first region in 2015. Since then, all Cloud providers have expanded their availability, and the price of Cloud compute has dropped due to competition and technological progress.

5. Low-latency networking is cheaper and more common. CDEs only make sense when latency is low. This requires high bandwidth internet connectivity for developers, and remote development environments to be located in close vicinity to servers. High-bandwidth networking is spreading rapidly, including fallback options with mobile 4G and 5G coverage. And with cloud providers launching new data centers, zones and regions, it’s now easier than ever to have cloud environments near to software engineers’ locations. 

For example, Uber runs its Devpods out of 5 data centers this way, presumably operated by a provider like Google Cloud. We previously covered Uber's transition from its own data centers to the public cloud.

6. Developer productivity investments. Assuming a team of four software engineers costs $1M/year, would you spend $10K/year to improve the efficiency of that team by 10%? The answer should be a no-brainer ‘yes,’ because the business value generated by a team costing $1M/year should already exceed $1M/year. So if the team gets 10% more efficient, it should generate at least $100K/year extra value. 

At companies with large codebases, CDEs are a pragmatic way to increase the productivity of software engineers.

7. Remote work. Covid-19 lockdowns and the rise of remote working have played a role in boosting CDEs. With remote work, engineers spend more time on video calls, which utilizes laptop resources like CPU, memory, and more. Executing a build is much slower while on a call. Plus, a CPU and memory-intensive build can impact the quality of the video call, and make the local environment much less responsive. 

LinkedIn gathered data showing how much slower video calls make common developer operations. It found video calls added a 2-4x increase in latency to a developer laptop.

8. Concern about code leaks. One downside of remote work – and hiring people whom you’ve not met in person – is that trust problems tend to arise more easily. When working from the office, it’s easy to verify that the same person you hired is the one doing the work, remotely. With full-remote work, the risk is higher that someone other than the employee accesses the codebase.

CDEs can offer an additional layer of protection to reduce the risk of source code leaking outside of the network. At the very least, far more logging is in place, and it can be easier to detect when larger parts of the codebase are accessed and copied across the network.

9. Open source VS Code Server. In 2021, Microsoft open sourced VS Code Server. This is the server code that powers VS Code remote development, and GitHub Codespaces. Thanks to this, the barrier to entry was dramatically reduced for anyone wanting to build their own cloud development environment which supports VS Code. This applies to startups offering CDE solutions, as well as companies building such a solution in house.

None of these trends alone can explain the spike in popularity of cloud development environments, but their combined effect in recent years does provide a plausible explanation. It’s also possible to posit that the “spark” which ignited this confluence of factors was the Covid-19 pandemic of 2020-21, and the accompanying rise of remote work.

CDE vendors: a definite trend

With more than 20 players in this space, let’s begin with a timeline of when the products launched. The segment really took off around 2018, although Cloud9 – which AWS acquired in 2017 – was founded in 2010.

Startups that offer remote development solutions include:

• Gitpod: feels like it might have the most momentum in the CDE space – at least right now
• Stackblitz: the only vendor that offers environments which boot in milliseconds, and the only solution that mount a micro-operating system written in WebAssembly inside the browser tab that boots in <200ms!
• DevZero: founded in 2021 by two engineers who’d worked at Uber, and the solution brings a lot of inspiration from Devpods at Uber, and the developer experience at the company
• Crafting: probably the most advanced solution of all; one that extends Kubernetes under the hood, and aims to be far more than a CDE
• Replit: probably the most popular collaborative coding environment for personal use.
• Codesandbox started out as a web IDE in around 2018, but expanded since.
• Coder: one of the few vendors that offers only a self-hosted CDE platform.
• Codeanywhere: one of the earliest cloud IDEs, founded in 2013
• Daytona: founded in 2023, and its founders are the cofounders of Codeanywhere.
• Cloudomation: almost exclusively targets Germany
• Hocus: started in November 2022, still in the building phase
• Strong Network: approaches CDEs from the cybersecurity and compliance perspective

We go into more detail for each of the above solutions — as well as products from more established companies — in the article Cloud development environment vendors.

There’s an absolute flurry of startups which provide tools for remote development. Between 2010-2020, we saw a couple of startups launch in competition with each other, but it’s since 2020 that there’s been a lot more activity, and new product launches seem to be speeding up. Here is a summary of all the vendors and products mentioned in this article:

It feels to me like we could be nearing an inflection point: after which more developers will work with remote environments at medium and large companies, than those that use only their local machine to develop. I am reinforced by this feeling as I talked to vendors who all have a pretty clear vision on what they want to build: but their products are still being built out, and I had trouble keeping track with the various innovative approaches they are all trying!

Will most developers at large and mid-sized tech companies build software on a remote machine in 2-3 years time? And will they do this in a way that’s indistinguishable from doing it locally? It’s hard to be sure, but I believe this is the direction we are headed in.

These were two out of the close to dozen topics we covered on this fresh trend. See these articles for more details:

More on why CDEs are spiking now:

• Virtual desktops vs CDEs
• The evolution of the CDE space since 2016
• The biggest challenges of the CDE space
• The future of software development in local environments

Remote development environments at tech companies:

• The idea behind CDEs
• Upsides of CDEs
• Pipedrive
• Uber
• Slack
• The downsides of CDEs

The CDE vendor landscape:

1. CDE startups and open-source solutions
2. Gitpod
3. Stackblitz
4. DevZero
5. Crafting
6. Big Tech and established companies

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in yesterday’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here.

Two weeks ago, a JavaScript runtime and toolkit called Bun was released and took the Node.js world by storm. Bun was mostly built by Jared Sumner, a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Bun has other contributors, but Jared writes the lion’s share of code.

Bun is a backend JavaScript runtime that is an alternative to Node.js and Deno. Its top focus is performance, and according to benchmarks shared in the launch video, around 10x performance increases can be observed when building packages, running code, or handling inbound requests on a server. This theoretically reduces the hardware requirements of applications.

Bun has stirred up the JavaScript space, and adoption has been rapid. Vendors like Vercel and Replit added support for Bun, as did frameworks like Ruby on Rails and Laravel Sail (PHP.) Many developers are giving Bun a go. Of course, it’s early days, but Bun keeps improving rapidly.

But how is Bun so fast? Firstly, performance was the guiding principle, and the team made smart choices to achieve this, and also tradeoffs which need to be kept in mind. Smart choices include using a more performant JavaScript engine than Node does (Bun uses Apple’s JavaScriptCore, while Node uses Google’s V8,) and using the low-level programming language Zig. There’s also a large number of performance-centric optimizations like:

• Using tagged pointers to avoid the overhead of storing extra function pointers
• Additional steps to reduce memory usage by scheduling additional garbage collector executions
• … and many more which all add up

With performance, there are also tradeoffs. For example, in its quest for performance, when running ‘bun install’, Bun doesn’t check the network to ensure that locally cached packages are correct – even when specifying to use the latest version. Doing this would introduce seconds of latency, but also ensure correctness. The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable.

Discovering these edge cases – and making sure they don’t catch you off guard – will surely be a learning curve for developers using Bun, just like with every new framework.

Bun showcases how the ‘innovator’s dilemma’ manifests in a tech ecosystem. The innovator’s dilemma comes from the book of the same title by Clayton Christerenses. The dilemma is this: a large, market-leading company has some motivation to innovate, but also a strong disincentive as well, because innovation risks undermining its existing products.

In the case of the Node ecosystem, Node is the innovator. In 2009, it was revolutionary and the majority of the JavaScript backend development community moved to this ecosystem. Now of course, Node has interest in innovating in all areas, including performance. But it needs to make sure performance improvements don’t break any Node users.

Bun has no such constraint. It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. This means it can make decisions which greatly improve performance, which Node doesn’t have the luxury of doing.

If it was not for the innovator’s dilemma there would hardly be any startups taking on large companies, or new frameworks which successfully challenge existing ones.

Bun also reveals the contrast between venture-funded open source and ‘community sweat’-funded open source. Node’s contributors are unpaid, and have been doing all work voluntarily.

I reached out to Yagiz Nizipli, who is the contributor who took on responsibility for performance work on Node. He confirmed Node has no funding, and no collaborator gets paid for their work on it. There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers. However, at the end of the day, it’s all volunteers like Yagiz who have been building Node for over a decade, helping it become the dominant JavaScript backend runtime. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. If you are one of these people: thank you!

This is in stark contrast to how Bun is funded. The framework is built by a startup called Oven, which has raised more than $7M in funding, which means the company can afford to pay all its contributors a market-rate salary, plus equity in the company.

Clearly, strong funding will create a much more focused team, and Bun contributors work on Bun full-time, while unpaid Node contributors work on Node part-time.

Venture funding comes at a cost, and the bill usually falls due later. There is barely anything to complain about in a startup like Oven paying people to work full-time on an open source framework like Bun. It’s a win for everyone; the community and the developers working on it full time.

In contrast, the goal of a VC investor is a financial return. VCs fund businesses they expect to eventually be worth a lot more than the value of their investment, and to be either sold, or go public. This means that in the mid to long-term, Bun’s funding needs to support a business. For most commercial open source companies, the approach tends to be to offer a managed, dedicated service of the open source software, or to provide additional features on top of it.

This is the approach Vercel is taking with Next.js – where they are the trusted provider to host Next.js apps. In the case of Oven, I assume the plan is to do something similar.

Of course, not all startups succeed, and most startups which fail do so by running out of money. When this happens, the startup shuts down. In the case of a closed source startup, the software is also gone. However, with open source software like Bun, it will not go anywhere even if the business ceases to exist. The project would simply need volunteer contributors, or sponsors who pay contributions.

The license of a project is key to understanding how ‘risky’ it is to use. Bun uses an extremely permissive MIT license that minimizes any risk.

Technological innovation rarely happens in a vacuum; it builds on previous technologies. Node growing into a mature ecosystem is what enabled Bun to now take it by storm, and offer an alternative with superior characteristics – in this case, performance.

Youtuber and tech investor Theo Browne posted the most balanced take I’ve seen so far on what Bun means for the ecosystem. In the video The truth about Bun, he points out the risk of an open source project being dependent on an individual:

“Yagiz is the person on the Node team focused on performance. He is a part-time, unpaid contributor to Node, working really hard to make Node performant enough to be a viable choice (...) He’s the person thanklessly maintaining this thing (...)

For Bun, the majority of code is coming from the person (Jared) who also makes the majority of business decisions [for his company]. You eventually get to the point where these things conflict. Jared’s done an incredible job of managing this thus far: but he’s mostly done this by working absurd amounts of time. That is a risk, and we need to be real about this risk.

At Node, there’s great work going on from hundreds upon hundreds of people to make Node as great as it is. Node survived for more than a decade, because they’ve managed and mitigated these risks really well”

I’m excited to see such innovation in the Node ecosystem, because innovation begets innovation. What Jared and the team have done with Bun is outstanding engineering; they’ve shown it’s possible to push performance far beyond what most engineers would assume to be possible. By building Bun to be so ridiculously fast, they’re getting a ton of warranted interest and early adoption.

But this innovation also helps the wider ecosystem, as maintainers on Node and other package managers also jump to do performance optimizations for their tools, doing so with renewed inspiration.

It will be interesting to see the impact of more venture funding pouring into this space. Node was built by a team of volunteers, becoming an admirable framework, which is ever-improving, even without direct funding. This approach has proved to be sustainable, thanks to the hundreds of involved contributors. While Bun has incredible momentum: will this be enough to make it a sustainable framework and business?

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry Pulse: Databricks going public – maybe, Square’s long outage, Slack’s redesign, and Valve blocks video games with GenAI assets.
2. The tech IPO winter has ended: For 18 months, there were no tech IPOs, with HashiCorp being the last to list on the stock market. But now Arm, Instacart and Klaviyo have broken the ice, offering hope for more to come.
3. Twilio’s iconic billboard is different, but why? For 7 years on the highway to San Francisco, Twilio had a billboard boasting: “Ask your developer.” Now, the company has replaced this famous billboard with a new message geared toward sales and marketing teams.

Read the full issue here.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of seven topics from today’s subscriber-only issue on How Big Tech does QA. To get full issues twice a week, subscribe here.

Microsoft has played an outsized role in the development and importance of quality assurance across the industry. Microsoft was the first major company to come up with a specialized testing role which went well beyond manual testing. In this issue, we cover:

• The SDET role
• Retiring of the SDET role

The SDET role

The SDET (Software Development Engineer in Test) role was one that Microsoft pioneered across the tech industry. They are software engineers who focused on writing automated tests; building and maintaining testing systems. The only difference between an SDET and a software development engineer (SDE) is that SDETs didn’t generally write production code: they wrote test code, working in the same team as SDEs.

I could not trace the exact introduction of the role, but it was most likely in the 1990s. For example, here’s a post from 2004 from a member of the Microsoft Exchange team explaining what it means to be an SDET in their organization:

“An SDET is a developer who works in a test team and not a development team. The SDET has a keen sense of a tester yet loves writing code, and lots of it.

The SDET enables the test team with the tools and processes that need to be in place for the testers to do what they do best… test the living daylights out of the product and find as many bugs as possible before it goes to the market.

An SDET has the ability to analyze the functionality and architecture of a Product and thus design and implement tools that help testers test it.

The SDET enjoys short project life cycles, designs and implements many tools and test frameworks in a single year, uses the latest technology and has plenty of room for innovation.

Though product quality is a prime concern, the SDET doesn’t have the stressful days that developers have during the end of a product life cycle. In layman’s terms… an SDET back-side is rarely on the line :)”

Microsoft had a formal career path for the SDET role. From the same post:

“There’s plenty of room for growth in [the SDET position.] If you love doing what you are doing as an SDET then you can grow to become a Test Architect. If you want to get involved in management, then you can progress towards becoming an SDET Lead and then Test Manager.

If you want to just code and not be involved with testing then you can take the path of becoming a developer. Many people take this path. If you realize that your heart belongs to testing, then you can become a tester.”

A 2:1 SDE:SDET ratio was common across Microsoft until around 2014. It was the case on my team at Skype for Xbox One, in 2012, when the team was formed. Here’s how our team was composed, based on headcount:

• 12x SDEs (software development engineers)
• 6x SDETs (software development engineer in test)
• 2x PMs (product managers)
• 1x EM (engineering manager)
• 1x SDET lead

On our project, the SDET team owned all parts of testing:

• Manually verifying that features devs built, worked as expected, including edge cases we might have not considered
• Building integration and end-to-end tests to automate checks
• Creating manual tests plans, and executing before major milestones
• Being involved during the planning phase of features, bringing ideas on edge cases and how the work would be validated
• Building solutions for tricky problems, such as how to do reliable performance benchmarking for our Skype product on the Xbox hardware

Unit tests were a source of tension, early on. Who should write them? Several experienced developers came from gaming, where developers typically don’t write automated tests, and their view was that any automation – including unit tests – should be done by the SDET teams. We cover more on how games are built in Game development basics, and go very hands-on in the issue, Building a simple game.

Those of us who’d previously built applications, or did test driven development (TDD,) felt that this approach was wrong, and that developers should write their own unit tests because unit tests and the code are tightly coupled. I was in this camp.

Having dedicated SDETs made it a tempting option for developers to “outsource” the writing of unit tests. I’m just going to say it now: without an SDET team, the question of who writes unit tests would have not been up for debate: we developers would have had to write them. This is a recurring debate I’ve seen in every team with assigned SDETs. Surprisingly, even this year I heard of a Silicon Valley-based company where a developer team has the test team write unit tests!

In our case, we settled on developers writing unit tests, with the SDET team doing everything else. This approach worked well enough, but there were a few memorable features of this setup:

• An “us and them” dynamic that created division. When us developers finished a feature, we handed it over to an SDET, who usually found issues, so the feature came back to developers to fix. This felt annoying to devs, as it created work we might have not accounted for. Over time, it started to feel like there were two teams, with different goals, which didn’t always row in the same direction.
• Ticket ping-pong between dev and test. I finish work on my feature and send it over to test (ping). The tester finds a bug and sends it back to me the next day (pong). I fix the bug, and send it over again in a few days (ping). The tester finds another bug and sends it back to me… and this back-and-forth happened more times than I’d be willing to admit!
• Devs were good at building complex systems, and could help SDET a lot. Our SDET team had been building an integration testing system for months, and progress was slow. Our team really needed this system, as manual tests were taking too long. Finally, one of the senior engineers proposed that devs join in to help build this system, as one team. Two weeks later, with the lead of experienced devs, a system was up and running. This got me thinking; would the team not work better without the dev/test division? We had just proved it did.
• The elephant in the room: some devs looked down on the SDET role. Although not everybody did, it was clear that many devs regarded SDET work as less challenging than their own. SDETs also knew they could have better career options by switching to a dev role.

Well, it turns out they didn’t have to wait that long for advancement.

Retiring the SDET role

Early 2014, I joined a new team called Skype for Web. This team was different from most teams at Skype, as they shipped a new version of the software every day, not every month.

The team consisted of 6 SDEs and 3 SDETs, on paper. In reality, we were 9 SDEs, thanks to the engineering manager and the test lead who quietly decided it made zero sense to have a dedicated test role, when we shipped new features every day. I write about this change that the team’s leadership didn’t broadcast, in the issue How Big Tech runs tech projects and the curious absence of Scrum:

“When I joined the Skype for Web team, we initially did two-week sprints, and followed the usual Scrum processes. We also had a split of software engineers and QA engineers. However, our shipping pace was every two weeks, but we wanted to ship more frequently.

The first thing we did was make QA a part of engineering. In the “old world”, an engineer would finish their work, check into their branch, update a ticket, and let the QA know it’s ready for review. The QA would take this ticket a day or two later, review it, and reopen the ticket if they found issues. This was a long delay.

We made a quiet, unofficial, change where all SDETs built production software as well, and all software engineers became responsible for testing their own code. Now, we no longer had to wait days for feedback before shipping code to production. However, the bi-weekly sprints and the numerous Scrum rituals became the next problem.”

We became a lot more productive by removing the SDET role from our team! SDETs still focused mainly on testing-related work, but also picked up development tasks. Just as importantly, we paired a lot! I remember pairing with SDETs to build a feature. I was good at thinking about how to make something work, and the SDET was really good at pointing out edge cases I hadn’t considered. When it came to debugging, the resourcefulness of SDETs surprised me.

On most teams across Microsoft, SDETs spent a lot of time manually testing things, and writing integration tests. But on our team there was very little manual testing, and we all built integration testing infrastructure, and monitoring infrastructure. When a developer or an SDET picked up a piece of work, they wrote all tests – unit and integration – which made sense.

The best part of this change was that there was no more “us vs them.” Arguments ceased about whether to fix a bug which an SDET had discovered, as now we did our own testing and fixed bugs which we discovered, before shipping to production.

Web teams across Microsoft started to quietly remove the SDET role. Back in 2014, our web team in the London Skype office felt ‘special,’ as the only other teams to merge the SDET function were web-based teams, of which there were not many. On every other team, SDETs kept working the way they always had.

However, it wasn’t only web teams within the Skype division which merged these roles for better efficiency. Web teams independently came to the realization that merging SDET and dev roles made them move faster, and so this happened across all of Microsoft!

In the middle of 2014, Microsoft formally retired the SDET role and introduced the SE role. The inspiration was apparently a larger web team at Microsoft, Bing. From Ars Technica, in 2014:

“At Bing, the task of creating programmatic tests was moved onto developers, instead of dedicated testers. QA still exists and is still important, but it performs end-user style "real world" testing, not programmatic automated testing. This testing has been successful for Bing, improving the team's ability to ship changes without harming overall software quality.”

In July 2014, Microsoft announced that they will execute their largest layoffs to that date, letting go 18,000 staff of the 127,000 employees at the company. 12,500 of the cuts were for the Nokia division. As part of this layoff, a large number of SDET roles were also eliminated. This happened around the same time as the SDET role was announced to be retired, and existing SDETs needed to move over to the software development engineer (SDE) track over the next several months. The SDE role was also renamed to SE – Software Engineer.

How did this transition work out? From what I gather, it went fine. The change made a lot of sense for teams that ship on a daily basis. And teams within Microsoft that ship weekly or monthly are increasingly rare, as Microsoft also leans into the software-as-a-service (SaaS) model. Of course, Microsoft continues to be a vendor for the Windows operating system family, and the Surface tablet. These are both areas where the approach to quality needs to be different to that of SaaS products.

A good account of the change came from the Visual Studio Team Services team in 2017, three years after this change. Reflecting on it, Brian Harry — currently Technical Fellow at Microsoft — wrote:

“Two years ago [in 2015], we had 10’s of thousands of tests. They were written by ‘testers’ to test code written by ‘developers’. While there were some advantages of this model – like clearly measurable and controllable investment in test, expertise and career growth in the testing discipline, etc., there were also many disadvantages – lack of accountability on the developers, slow feedback cycles (introduce bug, find bug, fix bug), developers had little motivation to make their code “testable”, divergence between code architecture and test architecture made refactoring and pivoting very hard/expensive, and more. (...)

Full testing would take the better part of a day to run, many more hours to “analyze the results” to identify false failures, and days or weeks to repair all the tests that were broken due to some legitimate change in the product. (...) Two years ago, we started on a path to completely redo testing.

We combined the dev and test orgs into a consolidated ‘engineering’ org. For the most part, we eliminated the distinction between people who code and people who test. That’s not to say every person does an identical amount of each, but every person does some of everything and is accountable for the quality of what they produce. We also set out to completely throw away our 10’s of thousands of tests that took 8 years to create, and replace them with new tests that were done completely differently.”

This team took stock of the type of tests they had in place and decided they didn’t like that there were few small unit tests, but lots of complex and hard to maintain end-to-end tests. So they changed this:

Here’s another visualization on how the team’s tests changed over a two-year timeframe:

So, was all this work worth it in the end? According to Brian, yes it was. Writing at the time, he said:

“We are starting to reap the benefits in improved quality, agility and engineer satisfaction.”

We covered the QA approach at one out of seven Big Tech companies from the article How Big Tech does QA. The full edition additionally covers details for:

• Google
• Google
• Meta
• Apple
• Amazon
• Uber
• Netflix

Read the full issue here.

Related issues, which overlap with testing:

• How Big Tech runs tech projects and the curious absence of Scrum. How projects are organized does have an impact on how straightforward – or not – it is to do manual testing on a regular cadence
• Healthy oncall practices – while not all Big Tech companies have a dedicated QA function, engineers are almost always oncall, and this forces some more focus on quality
• Shipping to production
• Dealing with a low-quality engineering culture at Big Tech
• What is a senior software engineer at Big Tech?

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of for topics from the past newsletter issue Game Development Basics. To get the full issues, twice a week, subscribe here.

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. I’d love to get a glimpse into the game development world, and how it compares to the software engineering approaches at tech companies.”

How games are built is a fascinating subject for me, and my interest derives from having worked with former game developers. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. It was by talking with them that I realized just how different the game development world is. Video games are very complex pieces of software, built in different conditions and under very different constraints than many other business applications are.

To answer the question of how the video game development sector works, I turned to Tommy Thompson, director of AI and Games, who’s a veteran of game development and of teaching it. Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. He also runs the highly popular AI and Games YouTube channel, where he delves into how AI makes for better games, and games make for better AI.

I met Tommy in 2010 at Scott Logic, a UK software consultancy where we were both software developers on neighbouring desks. As we reconnected more than a decade later, I discovered just how much our paths diverged right after that job; Tommy spent the next ten years in gaming, while I moved on to tech companies.

In this issue, we cover:

• Getting started in games development
• Games are software, but in a non-standard way
• Typical roles within games development
• How games typically get built
• The game development lifecycle

As a heads up, Tommy is hosting his third annual AI and Games jam, 15-22 September. Over a week, participants build their own game utilizing AI in some form. Learn more or sign up if you’re interested.

With this, it’s over to Tommy:

Getting started in games development

Video game development is undoubtedly one of the largest entertainment sectors in the world. It’s a $200+ billion industry, which thrived during the strictest of lockdowns throughout the Covid-19 pandemic, when people stayed indoors and checked out the latest instalments of action-packed games such as DOOM and The Last of Us, to more relaxing experiences like Bugsnax and Animal Crossing. Or perhaps they jumped on their favourite ‘live service’ games, playing the likes of Dota 2, Fall Guys or Fortnite with friends, to enjoy social interactions at a time when we couldn’t meet up in person.

Game development is a sector that hosts a myriad of challenges. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process. But even in 2022, the biggest challenge for anyone keen to learn more is that gaming is a sector which is often shrouded in mystery.

Getting started in video game development, or even just understanding how it all works, is often a difficult first step. Perhaps unlike any other software development field, the tools, technologies, practices and pipelines that are typical for a video game production are not well communicated or documented to the wider world. Naturally, this leads to a gap between the creator and the consumer, as wild and inaccurate hot takes on social media regarding the recent leaks of Grand Theft Auto 6 footage show. Still, even for a typical software engineer, it can be difficult to figure out how your skills apply in this industry, and how it really works.

Back in 2010, I was working as a software engineer in the banking sector, and Gergely and I used to work in the same office. It was around that time our paths diverged, with mine leading into the world of video game development. I had actually transitioned into a career as a university lecturer in computer science. But soon after, I was teaching game development on a bachelor's degree course. Despite my enthusiasm, and some earlier work making crude games as part of my PhD research, it became readily apparent that despite being a video game player, I still knew very little of how game development worked. This is of course a problem when you’re supposed to be teaching it!

So, over the past 10 years I’ve been on a journey to learn more about game development, with my experiences in teaching resulting in me becoming course director of several games development courses at UK universities, starting a YouTube channel that marries my interests in artificial intelligence (AI) and video game development (perhaps unsurprisingly called ‘AI and Games’) and developing my own game on PC and Xbox. I now work as a developer and consultant on artificial intelligence in the video games industry; working with companies ranging from big hitters such as Intel and Ubisoft, to smaller game studios around the world, and contributing to the production of several game projects over the past few years.

In this article, my focus is on providing a high-level overview of how game development works, and how software engineers fit into that space. Naturally, there’s a lot of ground to cover, and I hope that even at this introductory level, you find it quite insightful. So, let’s start with the basics.

Games are software, but in a non-standard way

Video games can often feel far removed from traditional software, given the modes of interaction and style of presentation are often unique to this space. But in essence, it’s business as usual. As a software engineer working on a game, your task is to facilitate the introduction of new features, integrate them within a large and ever-changing code base, clamp down on bugs and other emergent problems, and ensure it runs smoothly on a myriad of different platforms with varying hardware specifications. I’d argue that my time working as a mobile developer prepared me more for my time in games, than anything else.

All this sounds a lot like a regular software development gig, but the big difference compared to most software projects, is the highly multidisciplinary nature of the gaming medium. As a game programmer, you’re part of a much larger team, catering to a variety of different needs and project requirements.

Typical roles

On a typical games project, programmers work alongside designers, artists, animators, writers, sound designers and other disciplines. It’s usually a large and diverse team working together to make the project a reality. This is undoubtedly one of the most exciting, and equally challenging, aspects of any given project. It’s a highly collaborative space, as you work together to bring the vision of the game to life. Being able to feed off the creative energies of others around you is incredibly satisfying. I often explain this working relationship as that artists make it pretty, while designers and programmers make it work.

If anything, maintaining consistency of vision and preventing feature creep is one of the biggest problems you face while working on a game. This becomes the responsibility of the directors and producers.

The director is typically at the helm of the creative vision. Directors work with different teams to paint the broader picture, making executive decisions, all the while trying to maintain consistency.

Producers work on ensuring requirements are established, tasks are properly identified, scoped and budgeted, that team synergy is effective, and whether the director’s vision is actually attainable given the time and money available.

Compared to traditional software projects, a game director acts akin to a product manager, with an emphasis upon hands-on work, leading team members below them, and focusing on the creative aspects of the project. A producer occupies an interesting middle ground between product and project manager. They’re responsible for keeping the project on track, collaborating with leadership and stakeholders – such as an external publisher – but also they keep a close eye on the product performance and work to ensure the game ships.

Depending on the size of a studio, a producer may also be responsible for business management, such as liaising with external partners and handling finances. In a worst-case scenario, the director is also the producer. But it’s generally agreed that separating the creative direction from the project management helps keep things on track.

Perhaps unsurprisingly, this whole process was heavily affected by the global health crisis and the need to work from home. It led to a number of new challenges, for which the industry is only now beginning to understand the most effective solutions.

How games typically get built

If what I’ve described of multi-disciplinary teams working together sounds exciting or intriguing, what typically makes many scream in terror is when I talk about how games are typically built. It’s an onerous, complicated and often counter-intuitive process, given so many games emerge more as happy accidents, than the products of a carefully supervised process.

Prototyping. Games start out in a prototype stage, typically built using existing art assets, or with cobbled-together solutions for the purpose of establishing the basics of functionality and behaviour. Prototypes are critical for getting a game signed off into full production, or obtaining funding from an external partner; given the ‘fun’ of the experience should be made evident and also that there is a market for the product.

Should a game move into full production, it typically becomes a mishmash of different creative processes operating at once. Programmers are building core systems, artists are establishing the visual aesthetic and style of the game, and designers are establishing gameplay, long-term progression and retention.

Integration testing is often a difficult aspect of game development. Integration testing is challenging because so many disciplines are working in parallel. This is because assets from different teams get pushed into the build at different stages, often breaking much of the game as it’s being created. Identifying and helping resolve these issues is often led by dedicated quality assurance teams – assuming the team size or budget permits. Quality assurance might also be pushed out to the latter stages of development.

In full production, you’re essentially trying to “capture lightning in a bottle” for a second time - the first time was during prototyping. Only this time, the game is significantly more complex and potentially is even being built in an entirely different game engine.

Managing feature creep is a very real problem. Games take a long time to make and the industry is an evolving and reactive beast. As new games get a foothold in the market, they can often force changes upon in-production titles in an effort to stay competitive and relevant. Trends emerge in specific genres when a new idea captures the imaginations of players. A recent example would be the adoption of ‘ping’ systems in first-person shooters, brought about by the success of Apex Legends.

New trends emerging can lead to small deviations from an existing production schedule, or an entire revamp of a concept in order to pursue a new trend. Arguably the biggest example of this in recent years is Fortnite by Epic Games. Fortnite started out as a third-person wave defence shooter, only to become a battle royale in a style akin to PlayerUnknown’s Battlegrounds, to significant critical acclaim and financial reward.

But of course, a success story like Fortnite isn’t the reality for most projects. Games can spend years in something of a ‘fugue state,’ whereby the core of the game fails to take shape until the closing months as core design and direction are not nailed down sufficiently.

Tech debt can manifest quite easily as systems are built for one purpose, only for them later to be used for another. For some large-scale projects, the initial ‘fun’ aspect is only recaptured in the final 20% of the production process, which often leads to difficult working conditions for developers in the closing weeks – if not months – of a given project.

There has been a growing emphasis over the past decade for game studios to embrace better work practices from traditional software development. Practices include:

• Test-driven development
• Automated build processes
• Agile methodologies

All these processes are introduced to better catch many of the problems that emerge earlier in production. The real challenge is then adapting many of these ideas to work effectively within this multi-disciplinary environment.

The Game Development Cycle

For games, the traditional software development lifecycle never really fits. This is because you're often working to build or rebuild the same game multiple times during production. In some instances, the game continues to be developed even after it ships.

In this section, I’ll give a breakdown of the overall process. As a caveat, this breakdown isn’t universal, experiences will vary, and practicalities will shift depending on the type of project and the size of the studio. But it at least paints a picture of how the process typically pans out. For reference, it’s worth keeping in mind that your average game will typically take somewhere in the region of 2-5 years to make.

The game development life cycle consist of these three phases:

• Pre-production
• Production
• Release

This was one out of the four topics covered in the article Game Development Basics. The rest of the article covers:

1. The game development cycle, in-depth. What happens in the pre-production, production and release steps?
2. Indie vs AAA production. What are the major differences between how indie games are built vs how AAA titles come together?
3. Game development tools & services. Some of the custom tools used in game development: source control approaches, game engines and middlewares.

Read the full article here.

I recommend checking out Tommy’s game development YouTube channel, AI and Games. If you are interested in games development, you can follow Tommy on LinkedIn as well or subscribe to his newsletter, also called AI and Games.

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of five topics from today’s subscriber-only The Pulse issue. To get full issues twice a week, subscribe here.

Before we start: plenty of people who subscribe to the newsletter utilize a learning and development budget their company provides. In what is neat: The Pragmatic Engineer is now the first-ever newsletter listed on Learnerbly. Learnerbly is an L&D platform hundreds of tech companies use, as it makes administering these budgets much simpler — and it’s also a lot easier for people to request books, courses, or newsletters. If your company already has Learnerbly, you can now request the full newsletter with just one click.

Davidson Fellipe is a lead software engineer based in New York, with more than 15 years of experience. In April this year he was unfortunately impacted by job cuts at the company where he worked, and found himself on the job market. A few months later, he signed an offer to start a new company, and shared brief details of his job search on LinkedIn:

I was interested in learning more about how Davidson went about his job search, and his experience of being in the jobs market, so I reached out and he kindly shared more context and advice for navigating the current market.

Davidson shared that he eventually received one engineering manager offer and two individual contributor offers. He accepted the senior engineer offer.

How long was your job search?

‘My search lasted for about three months, until I received the first offer.

‘Starting in April 2023, I applied to a few positions, mostly targeting engineering manager roles. During that time, I dedicated a significant effort to applying using referrals whenever possible. The period from April to mid-May was challenging: I found myself in hiring freezes and canceled processes. For some applications, it took 2-3 weeks before I had a call with the hiring manager. At this time, more than half of my applications were rejected during the resume review, even with referrals.

‘If you attempt to apply everywhere, you can get a lot more frustrated. Instead of applying broadly, I focused on engineering manager positions for frontend, product engineering, and developer experience (front-end.) Over time, I gradually opened myself up to individual contributor roles, but only for frontend positions (senior, lead, staff, and founding engineer roles.)’

Which tools did you use to stay organized and to apply for so many positions?

‘To keep track of all my applications, I created a spreadsheet with columns like:

• Company name
• Position applied for
• Application date
• Status
• Last stage

‘Other tools that helped:

• Teal application tracker and Simplify for applications. Simplify helped me a lot to automatically fill applications in the most popular applicant tracking systems (ATS.) Another useful option was for me to have an "Identity" ready in 1Password, with the most common information requested by the ATS.
• Notion: for every company I talked to, I took notes on Notion to record what we talked about.
• Crafting tailored resumes: I dedicated time to crafting two, different one-page resumes, highlighting accomplishments, and showing my past experiences. I did one as a manager, and the other as an engineer: with the goal to improve my chances of converting interviews. For instance, my experience as a founding engineer at a startup helped secure interviews with some startups.’

How helpful did you find referrals?

‘People in my network offered referrals, and I also asked people I knew when I saw positions close to my profile at their company.

‘I also had cases when I saw a position I was interested in, but didn't know anyone at the company. In some cases, I added someone from the company via LinkedIn, and asked for a referral. I got a few interviews with this strategy!

‘I helped others impacted by layoffs, and the other way around. I shared opportunities or redirected recruiters to others I knew were let go, and people did the same for me. We were all experiencing the same challenges, after all!

‘I worked with external recruiters as well. These recruiters – which work with multiple companies – tend to have a number of opportunities that may match your skills and interests. Working with external recruiters helped in getting more interviews, especially with startups.’

How did you find the interview processes?

‘There are far more candidates for fewer positions, so the bar to go to the final round is higher.

‘Talking about startups, they are very creative with interviews! I’ve completed both technical quizzes and even cognitive aptitude tests. These types of interviews were usually an extra round before the coding interview or take-home assessment. Having at least 4 interviews or screening steps was pretty common.’

How did you manage your time and energy?

‘Job searching took a toll on my time and energy. I took a few weeks of vacation in my home country, then gradually started to increase the pace of interviews. My daily high-level plan was this:

• Morning: walk, study, review answers and tailor my resume.
• Afternoon: interviews, review job descriptions and take-home assessments that were timeboxed.
• Evening: take-home assessments that were not timeboxed; fire off applications, and in the end: recharge!’

What are interesting observations about the hiring process, and what advice would you share with job seekers?

‘What was surprising was that most hiring processes took up to 3 weeks from the conversation with the recruiter to the conversation with the hiring manager!

‘One thing that helped: New York City’s Salary Transparency Law was very helpful for not applying for jobs whose salary was below my salary expectations.

‘Advice for others: Don't underestimate the power of your LinkedIn intro and LinkedIn headline! And don't be afraid to apply if you see “500 applicants” for a role.”

This is Gergely again. Thanks a lot Davidson, for sharing the reality of your job search, and congrats on securing offers and accepting one of them. You can follow Davidson on LinkedIn or Twitter. He announced this week which company he will join.

It’s nice to hear about the positive impact of salary transparency regulations, and a good reminder that companies which publish compensation bands in their job adverts will likely attract more qualified candidates, as Davidson passed on places whose compensation was not a fit, which saved both him and the company time.

This was one out of the five topics covered in this week’s The Pulse. The full edition additionally covers:

• Are reports of StackOverflow’s fall greatly exaggerated? A blog post suggests traffic is down 50% at Stack Overflow, due to ChatGPT gaining popularity. I reached out to Stack Overflow for more details: the company admitted a drop, but it’s only 14% as per data shared with me. The company seems to be doubling down on Teams for generative AI use cases as well. Exclusive.
• What kind of migration is causing a payout outage at Booking.com? Small business hosts on the travel booking platform are waiting more than a month to be paid. Booking.com says a systems migration is the reason for the delay. I talked with engineers at the company and discovered an SAP migration is to blame. Exclusive.
• Amazon gets stricter about enforcing return to the office (RTO.) The online retail giant sent a warning email to employees “not meeting the expectation of joining colleagues in the office at least three days a week.” I talked with engineers about the response to this ominous, unfriendly email. Exclusive.
• Zoom to end remote work: an RTO turning point? Remote work tool Zoom is having most staff return to the office for two days a week. It’s a symbolic turning point which may signal how many companies will operate in a similarly hybrid way, going forward. Analysis.

Read the full The Pulse here.