Zak van der Merwe has spent his entire career at AWS building control planes. First for EC2 and now for DSQL. On the surface, the control plane looks quite boring: it records what should exist and reconciles that with what actually does. Nobody leaves school dreaming of building one, but Zak will be the first to tell you that if you like solving hard problems in distributed systems, there are few better places to be. It’s where many of those hard problems converge, and where the decisions you make determine whether a service survives its own growth.
If you’ve been following Marc Brooker’s and Marc Bowes’s writing on DSQL, this is a great companion piece that pulls back the curtain and shows what it means to build a database that was designed from the start with control plane engineers in mind.
–W
On building scalable control planes
I’ve been working at AWS for nearly fourteen years, and for almost all of that time I’ve been building control planes. It’s not the kind of career anyone maps out for themselves. Nobody leaves university thinking “I want to spend the next decade making sure the bookkeeping layer of a cloud service stays up.” But here I am, and I think the reason I’m still here is that control planes turn out to be where many of the interesting problems live, even if it takes a while to see that clearly.
Before Amazon, I worked at a telecoms company in Cape Town where we had maybe ten servers, all in a room in the back of the office, and every single one had a name. You’d SSH into them, you’d share them with your colleagues, and if something went wrong you could walk over and deal with it. That was my entire mental model of what it meant to run infrastructure. Servers were things you knew individually, took care of deliberately, and could reason about as a set because there were few enough to fit in your head.
I mention this not because it’s an unusual background but because it was so common less than two decades ago, and I think that’s what makes it worth saying out loud. Maybe your version is a small Kubernetes cluster or a handful of RDS instances where you can visualize the whole thing, you can name the parts, and when something breaks you know which part broke. That feeling of knowing your infrastructure is comfortable, and it makes the next part of the story genuinely hard to describe, because what happened when I joined EC2 was that that feeling just evaporated.
Honestly, when I started, I didn’t really understand how EC2 worked. I kept trying to map it back to what I knew. If I launch an instance and the underlying server dies, what happens? Does my VM somehow get teleported onto another host? How does the cloud create this illusion that hardware failures don’t matter? I couldn’t square any of it with what I knew about running software.
My first job at EC2 was health-checking the fleet, pinging every server and trying to figure out if it was healthy or not, and what I found was the opposite of magic. Things were failing constantly. Hosts were going down, hardware misbehaving, disks dying. I had seen the underbelly of EC2 and it was chaotic. My mental model had gone from “servers are precious things you protect” to “everything is on fire all the time.”
It took a while to shake that feeling, but what I would eventually come to realize was that these failures were tiny drops in an enormous ocean of things working fine. The system was just operating at a scale where failures were a constant, a statistical certainty rather than an emergency. And the thing that made it possible to run a service at that scale without a human responding to every failure, the thing keeping everything humming, was the control plane.
One way or another, my years at AWS have been spent working on control planes. Every AWS service has one, and I like to think of them as our unsung heroes. The better they work, the less anyone notices them. They’re the reason you don’t have to name your servers, and the reason that when hardware fails, you as a customer never have to deal with it. I’ve gotten to build control planes for two major AWS services: EC2, and DSQL. They’re nearly a decade apart, yet the hard lessons from building one led directly to the design of the other, and that’s the story I want to tell today.
What is a control plane anyway?
At this point, I probably owe you a better explanation of what I mean by control plane and why I think they’re interesting. I’ll use EC2 as an example, because that’s where I learned most of what I know.
The way I think about it is that every service has a data plane and a control plane. The data plane is the set of core capabilities, the raw computing power, the hardware, the networking. The control plane is the conduit between those capabilities and customers. It’s the thing that takes what exists physically in a data center and presents it to you in a format you can actually consume and get value from. Without the control plane, you’d be back to SSH-ing into named servers in a closet somewhere. With it, you can spin up a thousand machines with an API call and never think about where they live.
EC2 involves thousands of engineers and more features than anyone can keep track of, and yet the control plane, conceptually… is pretty simple. Stripped down, EC2 lets you rent a virtual machine (VM) in the cloud, and the control plane’s job is to set up and tear down these VMs for you.
I like the analogy of a thermostat, because it’s constantly measuring the temperature, it knows where things need to be, and it’s always nudging the system in the right direction. That’s what our control plane does. It’s a continuous loop, watching the state of the world, comparing it to what should be true, and correcting the difference. When you launch a VM, the control plane records that a VM should exist, finds a physical server in the right data center, sets up the image, configures networking, and launches it. Later, if that server disappears for any reason, the control plane notices and updates its records to reflect reality. It’s always reconciling what is with what should be.
One thing the team talked about constantly, almost to the point where it became a mantra, was that no matter what happens to the control plane, VMs that are already running need to keep working. We call this static stability, and it sounds obvious because of course running VMs should keep running. But at scale, obvious things are the hardest to protect, because every new feature, every change, every dependency is a chance to accidentally violate that guarantee. Maintaining it is the difference between an outage where customers can’t launch new resources and an outage where everything stops. Both are bad, but the second is catastrophically worse. The fact that EC2 was statically stable gave me some comfort in my early days.
The EC2 team has done a phenomenal job making bad days rare. But understanding what bad days look like shaped a lot of what I know about building control planes.
Living inside the control plane
To understand how bad days start, it helps to know how the control plane stores state. At the heart of EC2’s control plane there is a relational database. When customers call the RunInstances API to launch a VM, the most critical thing that happens is that the control plane writes a row into its database: customer X now has VM Y. That’s when the API can safely return.
In reality, a single RunInstances request triggers hundreds or thousands of internal API calls between micro and macro-services. Many of these services have their own databases recording their own state. It’s hard to exaggerate how complex this has grown over the years, but at the very bottom of all that complexity, there is a MySQL database, and what’s in that database is supposed to match reality.
The simplest way things went wrong was also the scariest. Sometimes the primary database server just died. Our solution was a hot standby, a backup server continuously replicating from the primary, ideally only milliseconds behind. When the primary failed, we’d cut over to the standby and it could limit the outage to seconds. The team earned that through years of operational practice, building tooling, writing runbooks, training on-call engineers to execute the switchover under pressure. But seconds of outage still meant pagers getting lit up at 3am and asking humans to make decisions with incomplete information. We kept asking ourselves whether the architecture could take humans out of that loop entirely.
The slower, more chronic problem was making sure our MySQL database kept up with business growth. This is pretty frustrating when you think about it, because the data plane does all the heavy lifting, like downloading VM images, configuring networking, running workloads, while the database is just keeping track of what exists. Every instance we launched meant more inserts, more updates, and more reads against the database, and eventually the bookkeeper couldn’t keep up with the workers.
So we introduced more servers replicating from the primary and used these as read replicas. Many of the EC2 APIs don’t make any changes, they just describe the state of your current resources (how many VMs do you have, and so on). We sent traffic for these read-only APIs to our new read replicas and this massively reduced the load on our primary database server. This is standard practice for any team trying to scale up a relational database. Incidentally, this fleet of read replicas is why the EC2 API is eventually consistent, and as Marc Brooker has written, this puts an unfortunate cognitive load on our customers. It’s something we wanted to do better with DSQL, which we’ll get to in a bit.
Read replicas bought us time, but every write still funneled through a single primary server, and eventually we had to shard the database. The first phase of this was visible to customers as we split each AWS region into multiple availability zones (AZs), each with their own independent control plane and separate MySQL databases. This helped with both scaling and availability, since zones fail independently and the blast radius of any single failure shrinks. It also became a fundamental building block that allows AWS customers to build architectures resilient to the loss of a single AZ. The second phase was internal: we sharded each zone into what we call cells. Both of these projects took years of engineering time because they required changes across many services. Every place in the codebase that talks to the database has to know which shard to route to. Simple lookups by primary key are straightforward, but anything else, such as joins across data that doesn’t align with your sharding boundaries, gets much trickier. Even the simplest decisions have consequences at this level. Do you shard by account or by resource? Different services choose differently depending on their access patterns, and there’s no universally right answer.
There is also a human cost to all of this that I don’t think we talk about enough. In those early years, we didn’t have the automation to handle a lot of what a modern control plane just takes care of. When a security vulnerability was discovered and the whole fleet needed to be patched, we didn’t have a system that could say “go update every host at a safe rate.” We would literally recruit the whole team, subdivide all the hosts, and assign shifts. Everyone in the Cape Town office would get a chunk. Go update every one of your hosts, report status. That’s what life looks like without a mature control plane, and it’s the kind of thing that doesn’t scale. You can patch a fleet of a few hundred hosts that way. You cannot patch a fleet of millions that way. The control plane is what eventually got humans out of that loop entirely.
If you’ve lived through this progression, the scaling cliffs, the read replica tradeoffs, the sharding projects that always take longer than you think they will, you know it’s a long and painful road, and it’s one that every team building a successful service backed by a relational database eventually walks.
Searching for Database Xanadu
After a decade working on EC2, I formed some strong opinions on what my ideal database looks like. It scales with my business without heroics. It is highly available with no downtime for updates, and no servers to babysit. My ideal database lets me leverage the power of the relational data model to model my domain and write software more productively.
As it turns out, in the early 2020s, a group of experienced engineers on the databases side of AWS were thinking about exactly how to build this type of database. These engineers were expats from services like EC2 and had felt the pain of operating relational databases firsthand. They were also looking at the lessons learned operating massive scale serverless databases like DynamoDB and dreaming up ways to apply them to relational databases.
They wanted to do for databases what EC2 and really Lambda did to servers. If you operate a traditional database with a “head node” you are in the world of “servers with names” like I was before joining EC2. The ideal database would free you from thinking about “databases with names”. Instead, it would have a control plane that takes care of all of that for you so that you can just think about your database as a logical endpoint that’s always available while it scales up and down.
Sometime around 2021, this project really started to pick up steam. We’d figured out an architecture which seemed to deliver on this promise of the ideal database. I got the opportunity to join the team and start building its control plane. This service would launch in GA as Amazon Aurora DSQL in 2025.
Let’s quickly revisit the major pain points that EC2 went through and see how life is different on DSQL—especially for control plane builders.
In DSQL, there isn’t one server running your database. DSQL spins up a Firecracker micro-VM per connection, which means every connection is its own small head node. If one fails, only that single connection is affected rather than your whole application. Nobody gets paged, no one has to decide to cut over. I don’t manage standbys anymore, because the architecture has removed humans from that painful loop entirely.
Scaling reads was another problem we spent years on at EC2, adding replicas by hand and accepting eventual consistency as the cost. DSQL adds read replicas automatically, and in fact this is one of the primary jobs of the control plane that I helped build. If your application suddenly sees a spike in read traffic, DSQL handles it, and the reads are strongly consistent, always. After years of telling customers “try again in a moment,” this property still blows my mind. It fundamentally simplifies the architecture of any control plane built on DSQL, and it removes that cognitive tax from the developers using the APIs those control planes expose.
And then there’s sharding, which was availability zones and cells at EC2 and took us years. When you build AWS control planes for major new services, you have to anticipate that sharding will become necessary, and experience has shown that it’s cheaper to do it from the start than to retrofit it later. This is an ugly dilemma, because you’re extending your time to market on a speculative future problem, and when delivery timelines get tight, I’ve seen many teams give up on sharding just to ship. DSQL removes that dilemma because it automatically partitions your workload and you don’t have to think about it. You can use all the Postgres goodies you’re used to, complex transactions, multi-table joins, secondary indexes, while knowing your database is going to scale with your needs. Many new AWS control planes over the last decade were built on DynamoDB for this same reason, but DSQL offers a world with fewer compromises. You get the scalability of DynamoDB with the relational programming model that developers actually prefer to work with.
“Self-hosting”
When it came time to choose a database for the DSQL control plane, we chose DSQL. A team that runs on its own product feels every rough edge before its customers do, but getting there meant taking on the same circular dependency we’d faced at EC2: a control plane can’t depend on the thing it controls.
We’ve seen two significant benefits from the decision to “self-host”. As customers adopt DSQL, they are creating thousands of databases, and the control plane is continuously scaling their databases up and down based on usage, often very rapidly. All of this customer activity creates “bookkeeping” work for the DSQL control plane, and the amount of this work grows with DSQL adoption. Since the DSQL control plane runs on DSQL, our bookkeeping database scales up to keep up with this increase in demand with minimal work from the team.
The other benefit is in how we deal with availability zone outages. DSQL was designed from the ground up to survive single zone failures, but just because a zone is down doesn’t mean that customer workloads stop scaling or that customers stop creating databases. In my EC2 days, zone failures were fire storms as control plane databases died and pagers went off. For the DSQL control plane, these unfortunate bad days are much less painful because the DSQL control plane’s database remains available which allows the control plane to keep doing its critical work that ensures customer databases keep chugging along.
Taking off the rose-tinted glasses
If you’re still with me, you’re probably thinking to yourself: “what’s the catch?”
As a relatively new service, there are features that we just don’t support yet. Some of these are gaps that we’re actively filling. Others are more nuanced, and we want to take our time to make sure we build the right thing. A good example is foreign key constraints. Foreign key constraints are a classic database feature that can be very useful and aren’t fundamentally hard to implement. However, foreign keys can also be dangerous at scale. We want to get this right, and that takes time.
One of the advantages of running Postgres on a single node is that it maintains the working set in memory, and cached reads are insanely fast. Real architectures are more complicated though. For example, a control plane using Postgres would run across multiple availability zones and put a connection multiplexing proxy in front of the database. These are necessary steps for availability and scale, but they increase latency. When you build on DSQL, you don’t need to manage these things yourself. You get good (though not quite single-node Postgres good) latency that remains consistent as your application scales. This is exactly what I want as a control plane builder. Yes, I want fast, but I care even more about predictable latency as my application scales.
It’s also worth being honest about where things stand for control plane builders at AWS. Migrating something like EC2’s control plane onto DSQL would take years even if we started today, and that’s okay. The ten-odd years I spent on the EC2 control plane taught me that the work that matters most tends to measure its impact in years, not quarters.
Looking around corners
We’ve spent most of this post deep in database scaling and life support. It’s a familiar shape for a lot of engineering stories. The problems we faced at EC2, how to go faster without breaking things, how to spend more of our time on the things that matter to customers, how to coordinate across a team that grew from a handful of people to thousands, and how to keep the system reliable while the ground shifted underneath us, are the same problems every engineering organization runs into as it scales. They are close cousins of the problems that produced Amazon’s original distributed computing manifesto back in 1998, and my own focus narrowed over the years to a single version of them, which was how to let individual teams fully own a piece of EC2 and move fast on their most urgent problems without expensive coordination, all while the product still felt like one coherent thing to a customer.
When I look at the broader industry today, I see echoes of that same pressure playing out at a scale I did not expect, because the arrival of agentic coding has driven the cost of writing software down to almost nothing, and that pushes the hard part of the work somewhere else. When code is cheap, the bottleneck moves to judgment, to figuring out what to build, how to ship it safely, and how to anticipate what your customers will need before they ask. That is the same shift a good control plane makes for the people who build on it, taking the invisible work of keeping infrastructure alive off their plate so they can spend their attention on their customers, only now it is happening to software development as a whole, and even a single-person team feels the need to scale out.
I am not going to pretend I know what building software will look like a year from now, because we are in the middle of a remodel and the walls are still open. What I do know is that it is much easier to move fast when you are standing on a foundation that will not crack under you, and that the problems worth spending a career on have always been the ones that need your judgment rather than your ability to keep the bookkeeping layer from falling over. My hope is that DSQL gives the next generation of builders that foundation, and gives them back the time to go look around corners for their customers, which is the part I always wished we had more room for at EC2.
And as Werner says: “Now, go build.”
