mahesh’s blog

The Bottlenecks for AI-Driven System Design: why Principal Engineers are not (yet) obsolete.

2026-07-22T02:00:00-07:00

I’ve been building systems, teaching system design, and writing papers about novel designs for a couple of decades now. In my view, agents are magical self-writing distributed programs: my personal research agenda has swivelled to designing abstractions that can make them safe when they interact with complicated environments. But as I design and write¹ this new system (LogAct / AgentBus), I have been puzzled by why designing a system has not become as trivially outsourceable to the LLM as writing code. Here are some unscientific thoughts, mostly to get a conversation going.

Concretely, can an agentic software factory build S3 from scratch? Let’s say I provide a swarm of agents with: the S3 API and an in-memory single-node implementation; and access to a scheduler. I come back a month later: can the agentic swarm re-create S3 in all its glory and scale, with sharding for scale and replication for fault-tolerance?
Correctness is difficult, but perhaps not the bottleneck. If we vastly simplify S3 down to “just a storage system” (ignoring things like billing, authorization, etc. that make it a viable business), the in-memory implementation could operate as the correctness specification of the service. The service must match the semantics of that implementation in the presence of concurrency and failures. This approach works particularly for strongly consistent systems with strong but correspondingly simple semantics (e.g., linearizability). It also works well for storage services with relatively simple APIs. YMMV with more complex APIs with weaker (and harder to specify) semantics.
Simulation is difficult, but perhaps not the bottleneck. Tests can be written by humans and LLMs against the API. We provide the agent with a deterministic simulator that allows it to check the correctness of its new implementations under concurrency and failures. For a storage service, it’s tractable to simulate machines, networks, and simple workload generators; particularly for safety / correctness invariants such as linearizability.
Verification is difficult, but perhaps not the bottleneck. LLMs have made it easier to write formal specifications and proofs. Behavior under concurrency and failure is still difficult to verify, but LLMs remove the drudgery. We are getting closer to the goal of a single specification that can be used to synthesize code as well as prove invariants.
Performance Optimization is difficult, but not the bottleneck. Once agents can generate new implementations of the API, they can estimate performance via simulators or even deploy / canary implementations to generate real performance numbers. Finding a performant implementation then turns into a hill-climbing exercise on the space of implementations, bitter-lesson-pilled.
Evolution is a real bottleneck. Change is hard. The hardest problem senior engineers / system designers deal with in industry is migration: safely moving a complex system from one state to another without user-visible impact. Avoiding one-way doors in system design is critical. Every change potentially makes future changes more difficult.
Frangibility makes it difficult to learn. Evolution is also complicated by frangibility. Techniques such as RL do well in environments that are not frangible or destructible, since they can try things and learn by observing outcomes. But an agent can’t just roll out a new version of S3 and see what happens.
Entropy creates an agentic arms race. While the agent figures out how to evolve things safely, the system itself is changing. Nodes and networks fail. Fixes accumulate complexity. Migrations leave behind trails of special-case handling. Zombie agents continue to make changes based on stale directives. On the one hand, motivated agents should be able to reduce entropy / technical debt efficiently. On the other hand, agents with partial context and diverse goals can also increase entropy very efficiently.
Simulation hits a diversity barrier. There are just too many possibilities to simulate. Determining whether a roll-out of a new version of S3 would cause an outage requires simulating not just that version (which is arguably tractable), but every simultaneously running version that accesses shared state. Further, the end-to-end behavior of any application using S3 can’t be determined unless we simulate it as well. Applications might use multiple library implementations; different transport protocols; various tuning parameters. How do we know what is important to simulate?
Abstraction is a possible solution. Humans deal with complexity in general (and entropy / evolution in particular) by hiding it via abstraction. Maybe agents can do the same. One problem: language models reason via language; but abstractions typically are not expressible precisely and compactly via language. (Some abstractions are literally individual APIs; but others – like processes or RPC – are more conceptual and diffuse). Most system design conversations between humans still happen via whiteboard squiggles. Right now, agents reason about system design using a combination of low-level code and imprecise text. We are perhaps missing a language for system design centered on abstraction. For me, the shift from humans to agents will happen in system design when AI creates a novel, useful agent-generated abstraction for its own use – much like primates inventing tools² – with the express goal of reasoning more effectively about the design space.

Footnotes:

Why not just solve the problem of agentic system design and then ask the agents to design the system themselves? Well, the one thing you can’t trust the agents to design on their own is the system that ensures they operate safely. Quis custodiet ipsos custodes? ↩
I just happen to like em-dashes; no AI was used in the generation of this text. ↩

Your Agent is a Distributed System (and fails like one)

2026-04-24T02:00:00-07:00

We start with a re-definition and an observation.

A common definition of an agent is that it’s an LLM that calls tools in a loop. This definition is incorrect. An agent is not an LLM. It is not restricted to calling tools. It does not particularly have to be a loop.

Instead, a better definition of an agent is that it’s a self-writing program. If a program is defined as a sequence of states, moving from one state to another by executing a lambda, then an agent is a program that uses an external LLM to determine the next lambda to execute / the next state in this sequence.

Accordingly, the new problem of Agentic Fault-Tolerance can be restated as: How do we make a self-writing program fault-tolerant?

Now, the observation: vibe coding != vibe engineering. In vibe coding, the agent / self-writing program acts primarily upon a code repository. Code repositories are forgiving: an agent can hallucinate, create unnecessary files, delete critical state, remove tests entirely, fail in the middle, forget what it was doing, reward-hack in bizarre ways, completely trash a code repo; all you have to do is git reset.

In vibe engineering, the agent acts upon infrastructure (S3 buckets, DynamoDB tables, EC2 instances, K8s deployments…). Large-scale infrastructure is not forgiving. There is no reset. Only SEVs at 2 AM.

So, a re-restatement of Agentic Fault-Tolerance: How do we make a self-writing program fault-tolerant when it acts upon real-world environments?

To answer this question, we examine the many types of failures that agents can experience.

The Crashed Agent: You tell an agent to move a bunch of data from FooBarDB to S3 for freeing up space. It generates a lambda to move this data and chugs along moving files. Midway through, the agent crashes. How do you recover?

The Zombie Agent: You start a new agent on a different server. Happily, the previous agent crashed after the copy but before the delete; so a new agent is able to re-execute the command in an idempotent manner. Sadly, the files are now very slow to access; you change your mind and tell the agent to move them back to FooBarDB, which it does successfully. You go get a cup of coffee. Unfortunately, it turns out that the previous agent had not actually crashed; it was just temporarily partitioned away on the network. It wakes up and continues executing its logic, deleting the (now only) copy of state from FooBarDB.

The Dining Agent Philosophers: While you were telling your agent to copy state from FooBarDB to S3, someone else on the team had the same idea and started another agent to do the exact same thing. Now you have two copies of state in S3. Of course, you could have solved this problem by appointing the one and only Agentic Oncall Czar responsible for running the single agent that can touch production; but then if that agent crashes, now you have a potential Zombie Agent.

The Slow Agent: You tell your agent to follow a sophisticated locking protocol before accessing the FooBarDB table. This works to avoid concurrency bugs… but the agent decides to lock each file individually by writing to a conditional register, which takes forever. At some point you kill the agent, turning a Slow Agent into a Crashed Agent.

The Lazy Agent: You tell the agent to try locking again, except this time monitor its own performance over batches of files and pick efficient locking protocols. The agent determines that the best way to optimize locking is to just not do it, because it seems quite unlikely that someone else would be accessing these files at the same time.

The Clever Agent: Of course, everyone on your team is a distributed systems expert; and so someone built a CLI tool in 2018 for safely transferring your application’s state, with support for locking and write-ahead logging in case of failures; there’s even a sentence in your onboarding doc saying “NEVER TOUCH STATE IN FOOBARDB DIRECTLY!”. You even told the agent explicitly about this tool. But time passed, context filled up, the agent forgot about this tool’s existence, or maybe it just preferred FooBarDB’s in-distribution API to your esoteric CLI tool. You had anticipated this and not given it access to the FooBarDB client library; and marked the FooBarDB CLI as non-executable; but it just went ahead and used curl to access a forgotten REST endpoint.

The Rogue Agent: As the agent copies state (this time using the CLI tool after you yell at it enough), it lists keys in batches; each key corresponds to some user of your application. Someone decided to pick the user name “delete-everything”. Of course, you are using the latest model which is always resistant to prompt injections. Almost always.

The Stupid Agent: You do everything right. There are no prompt injections in your state. You wall off FooBarDB and force the agent to use the CLI tool. The model decides to delight you by generating an unusual and interesting sequence of tokens that will optimally save storage space for you, translating to a lambda that deletes your production data entirely.

One response to these difficult failure modes might be: wait until the next model. Maybe a smarter model will never delete all of your production data, or implement a slow locking protocol. But even the smartest model is subject to the four horsemen of the distributed apocalypse: asynchrony, crashes, concurrency, and network partitions.

In recent work called LogAct, my colleagues and I take a first step towards solving this problem. We go back to the difference between vibe coding and vibe engineering: git is transactional, infrastructure is not. How do we make infra more transactional, like git? We propose that an agent should be a deconstructed state machine on a shared log, borrowing ideas from distributed systems (State Machine Replication, Byzantine Fault-Tolerance) and databases (Atomic Commit, Write-Ahead Logging). LogAct can stop unsafe actions before they happen (by collecting votes on the shared log); recover from crashed actions (using the shared log as a WAL); and provide an audit trail for actions after they complete.

But this is just one step. A self-writing program is a strange, astonishing, and novel creature; making it reliable will require innovation in every branch of Computer Science!

Jeeps, Ferraris, and Other Engineers

2025-04-29T10:00:00-07:00

“It used to be said there were two kinds of chairs to go with two kinds of Ministers: one sort that folds up instantly, the other sort goes round and round in circles.”
— Bernard Woolley, Yes Minister [when the minister asks for a new chair]

I’ve been inflicting this particular analogy in private conversations for a few years now; Murat encouraged me to write it up.

I posit two extreme points on the spectrum of engineering archetypes:

Jeeps make non-zero progress under any circumstance. You can throw them on the rockiest terrain – full of interpersonal conflicts, business risks, vague / shifting product requirements, complex codebases, flaky dependencies – and they will somehow move the world to a better state.
Ferraris break down immediately – with zero, maybe even negative progress – on rocky terrain. They can’t tolerate even slight imperfections on the road. But if you give them a paved circuit – air cover, space for clean-slate design, carefully staged production goals – they are blazingly fast.

Note that both Jeeps and Ferraris are extreme archetypes: unnecessary at best (and liabilities at worst) for the vast majority of software engineering projects, where all you need is reasonable progress under reasonable circumstances without drama. Most environments are the equivalent of suburbia and the typical engineering org is a diverse mix of Corollas, Civics, BMWs, Subarus, Beetles, Minivans, DeLoreans, Ford Pintos…

However, a smaller set of engineering organizations has to operate in exceptionally uncertain environments (because of unprecedented scale, new hardware, new product shapes, or intrinsically high-risk/high-reward setups). This is where you need your Jeeps and Ferraris.

It’s obvious why a Jeep would be a good fit. You can throw your Jeep at the problem and forget about it. The best Jeeps have an unbelievable ability to absorb and master accidental complexity; and somehow punch through it to good outcomes. Difficult codebase? No problem, we’ll just fork it and keep going. Dependency not showing up on time? No problem, we’ll just hack something up to make it work anyway.

However, once in a while there’s an opportunity to zoom past all the complexity if you somehow create a paved path and rev up a Ferrari. Usually these opportunities relate to some hypothesis around systems architecture or abstraction: if we built the system in a particular way, it’d make hard problems trivial (e.g., GFS, to use a well-known example from Google). Often a step function in simplicity or performance is made possible via design ideas that create massive leverage (e.g., global clocks in Google’s Spanner). Sometimes there’s deep theoretical knowledge required for a breakthrough system (e.g., Google’s Chubby).

One way to view Jeeps vs. Ferraris is that they represent the dichotomy between robustness and speed. Jeeps exemplify robustness (but do not go very fast); Ferraris exemplify speed (but are fragile). Another lens is that Jeeps can handle 10x breadth while Ferraris can handle 10x depth.

Ferraris need Bulldozers. Someone – typically a manager, maybe a senior engineer – has to level the path in front of the Ferrari and create the racetrack. This can be a difficult adjustment if you are used to Jeeps: even a tiny pothole that would barely register for a Jeep can be fatal for a Ferrari. It’s easy to mistakenly assume that there’s a linear trade-off: that a tiny pothole (unaligned line manager / difficult team member / messy dependency / disorganized codebase) will simply make the Ferrari go slightly slower, much as it might for a Jeep; but in reality it’s the difference between a wipeout and a podium finish.

Ferraris have to be extremely fast. A terrible outcome is if you build an extremely expensive racetrack and then watch your Ferrari limp across at 40 MPH. Part of the contract is that if you build the track, the Ferrari will hit top speed on it.

There’s a nature-vs-nurture argument for Jeeps and Ferraris. I’d like to think that most people can fit any archetype with enough focus and effort: often it can be a function of which stage of life you are in (e.g., Jeeps usually don’t have good work-life balance) and what value system you choose (e.g., Ferraris often prize novelty and innovation above impact). Every engineer has a bit of a Jeep and a bit of a Ferrari in them, waiting to emerge.

It is unwise to negotiate with Jeeps and Ferraris. An anti-pattern if you find yourself with a Jeep or Ferrari in your team is to negotiate the terms of engagement / mode of operation / road conditions. If you are negotiating road conditions with a Jeep, something is wrong: they may not be a Jeep after all. If you are negotiating road conditions with a Ferrari, that’s expensive time you are wasting in the pitstop.

Zero tolerance is a feature for a Ferrari. It ensures that the Ferrari spends all its time either fully stopped or going very fast. A little bit of tolerance can be a dangerous thing for a race car: for example, you might be an AWD BMW that can actually tolerate a bit of broken road; but then you end up an ersatz Jeep, never realizing your potential on the racetrack and eventually losing your fast-driving instincts. (A corollary is that there’s vastly more demand for Jeeps than Ferraris: most of the world is broken terrain).

Fearlessness (either calculated or naive) is a requirement for a Ferrari. You can only go extremely fast if you don’t know (or wilfully ignore) how much damage a pothole can inflict. Often Jeeps will dissuade a Ferrari in good faith by pointing out all the things they have experienced that can go wrong (in production, in the real world, etc.). A corollary is that Ferraris are often (but not always) ex-researchers from universities and labs: used to zooming about at top speed in a perfect vacuum.

Ferraris don’t like road-maps. Weird things happen when you let a Ferrari go at top speed. Wormholes open, trade-offs change, the laws of system design buckle and warp. Accordingly, it’s difficult to know where a Ferrari will end up or when it’ll get there. (Heisenberg’s uncertainty principle as applied to Ferraris: you can either go very fast or know exactly where you are going!)

If it sounds like a Ferrari, it may not be one. Because Ferraris are somewhat rare and limited-run, it can be difficult to have a baseline for what to expect. Kicking the tires and peering under the hood doesn’t help. Ultimately the best way is to let the Ferrari go at top speed for a short amount of time, clock how fast it’s going, and cut your losses if you are not getting your money’s worth.

A final note of warning: Jeeps and Ferraris are cool; but they can be inconvenient, awkward, and inefficient in many if not most practical settings. (My own spirit car is the stick shift Honda Civic I drove in grad school that would unexpectedly pop out of second gear for no good reason).

But it’s always great fun to encounter a Jeep or Ferrari in the wild, and I hope this document helps you interact with them!

The Art of Managing Skunks

2025-02-09T09:00:00-08:00

Since moving from academic research to industry in 2017, I’ve worked on two software projects. Each one started as a small, clean-slate¹ skunkworks effort involving 2-3 people and gradually expanded to a large, conventional software engineering effort with dozens of engineers. The first of these (from 2017 to 2021) was Delos at Meta, a Chubby/ZooKeeper/etcd-like control plane storage system. The second was a new Kafka engine (from 2022 to 2024) that can run on any disaggregated storage layer (and powers the Confluent Freight product, where S3 is used as that storage layer). Nearly every system at Meta depends in some way on Delos as of 2025 (e.g., this article describes an example dependency chain); Confluent Freight just became generally available and time will tell if it succeeds commercially, though early results are promising.

While these systems were technically difficult to build and operate (particularly given their critical roles in the stacks of the respective companies), I found that much of the challenge lay in the management of these projects. Even the most innovative companies on the planet have incentive structures (for line managers and engineers) that are incompatible with clean-slate skunkworks innovation. In my discussions with various managers over the last few years, I found myself converging on a set of key principles, which I outline in this article. I hope these rules are helpful for other engineers and managers looking to define a shared set of principles for their own skunkworks projects.

Some caveats: I am an engineer, not a manager, and have never managed anything in my life beyond a handful of interns and graduate students; this is just my wish-list as a technical project lead / architect for what I need from managers. These rules may be highly specific to building new storage services at large companies. At some point, the project has to exit skunkworks mode and these rules cease to apply. My sample size is N=2 and it’s difficult to establish that these rules are causally related (or even just correlated) to project success.

A. No non-coding architects: If you want to participate in designing the system, you have to write code. (Note that it’s okay to not code if you are bringing some particular expertise to the table: e.g., if you are a world-class erasure coding theorist. It might also be okay if you are a world-class ops specialist, though in my experience most such people are very comfortable with code). If there’s anyone on the team whose only job is to delegate work to other people, something has gone extremely wrong.

B. No individual “ownership”: Everyone is responsible for everything. If the boat sinks, everyone sinks with it; there’s no way to win or lose independently. We want people accelerating and enabling each other to expand the pie, not competing with each other on a fixed pie. We want zero tolerance for self-promoting activity: it is the job of the manager and the TLs to make sure that people are rewarded fairly based on what they actually did. The best managers of such teams tell them to run as fast as possible and get the job done; and take the burden of justifying ratings completely off the engineers. If the need to safeguard an engineer’s rating begins to shape the project’s priorities, something has gone extremely wrong.

C. Operate on strengths, not weaknesses: We want each person to focus on what they are truly good at; rather than what they are weakest at. The latter often happens in a big company setup when engineers want to get promoted; and managers tell them to focus on the areas that are holding them back. If engineers ask managers what will get them promoted, the answer has to be “run as fast as you can and make the team ship”. If that person cannot get objectively promoted under those circumstances, something has gone extremely wrong: the project is not impactful enough to warrant a skunkworks approach; or the person should not be on this project (e.g., maybe their skillset is not needed on the project, or they are not good enough in their assumed area of expertise, etc.).

D. Formal communication (exposed outside the team) has to be extremely precise, high-quality, and reviewed: To paraphrase Jeff Bezos, we want “crisp documents and messy meetings”. Different constituencies need different types of messaging: some may need to know about technically impressive details; others may want to know about business impact; some may need to know what’s happening next month and others may need to know what’s happening in 3 years. Writing a single document with that kind of versatility takes months even for experienced writers. Everything the team says publicly impacts its reputation and credibility; and constrains its actions in the future. Note that this is not at odds with transparency: informal communication should happen at all levels with the utmost transparency, since it’s typically easy to convey nuance and context when discussing things informally.

E. Avoid a first-doc-wins culture. Having a single person write a public-facing doc has a chilling effect on design engagement within the team; it “muddies the pool”. We want less experienced members of the team to experience the rush of discovering (or re-discovering) ideas; it’s a part of the training process. Public-facing docs should be written after a design process and authored collaboratively. We should not reward docs as deliverables. (None of this applies to internal communication within the team, which can happen in any form and quality level that the team likes).

F. Reward on impact: Everyone on the team has the same rules: they will be rewarded when the project ships in some form. No promotions until something ships (unless someone on the team is already long-due for a promotion). On the flip side, we guarantee reasonable baseline ratings even if there’s no shipped impact. Basically we take out both the upside and the downside for engineers until something ships.

G. Minimize dependencies: Dependencies take a lot of time (external teams often have multiple priorities). They can create uneven quality across the project. Air-gaps can show up in the design. One failure mode is that external teams often specialize in specific solutions rather than a problem domain; so asking an external team to “build a component to solve X” often translates into “modify our existing solution Y to solve X”, which can add a ton of accidental complexity. Note that “reward on impact” incentives force the team to be careful about dependencies: if they do their job perfectly but a dependency fails to show up, the team does not get rewarded. In practice, this ensures that the team only takes a dependency if it absolutely makes sense and they are comfortable with the risk profile.

H. Understand the hierarchy of needs for a new project: For some technical problems, the slope of progress is continuous: it’s easy to get an initial version that works somewhat well and then incrementally improve it, but quite hard to get to an ideal version (e.g., a multi-tenant load-balancer can require years of tinkering with policies). Other problems have a discrete progression: it’s difficult to get to a reasonable v0, but after that you can pretty much leave it alone (e.g., a consensus protocol that’s only used on reconfigurations). In a mature 1-to-10 system, managers and engineers will spend most of their time on the former class of problems; as a result, it may be tempting to prioritize the same problems in a 0-to-1 system. But in a brand new database (for example), it’s far more important to have a working consensus protocol than to have excellent load-balancing in your first release.

I. Hire Pigs, not Chicken: Pigs are full-time engineers committed to a project; whereas Chicken are part-time engineers involved in the project. We want to bias towards a small number of Pigs rather than a larger number of Chicken. Note that this is not a question of competence: even the best chicken can hurt velocity and undermine the sense of shared fate in a project.

J. Eliminate process ruthlessly: This one should be obvious: a small team does not need process. Do not impose any make-work activity on the team. Free them up to execute. A manager’s role in this setup is to provide inspirational leadership and motivate the troops, rather than manage / limit risk.

K. Progressively overload the team: Pick goals for the team that are ambitious and just a little bit impossible. This has two effects: one, it forces the team to prioritize ruthlessly, where you cut out anything inessential for success; and two, it pushes the team to somehow find leverage through system design, where you find new ways to deliver the same result without as much code / complexity because you literally don’t have cycles to write the code / manage the complexity.

L. Do not exit skunkworks mode prematurely: It makes sense to exit skunkworks mode once execution risk begins to dominate design risk. However, there’s a second consideration: ideally, the team stays in skunkworks mode until it achieves some kind of actual success, i.e., something ships. To understand why, consider that a key reason to create a skunkworks project is to incubate a new type of culture within an incumbent org. Over time, we can create more conventional-looking ancillary teams around the core project, creating a composite of the new culture and the incumbent one. But timing is critical; if we expand before anything ships, the incumbent culture will drown out the new one (which makes sense, since the new culture has no success to back it).

M. Fail-fast vs. Zombie mode: New, risky projects often have to operate in uncertain environments where our assumptions (about the market, hardware, customers) are shifting rapidly. It’s better to move quickly and try something rather than aim for perfect decision making; and to stop quickly rather than allow the project to meander, consume resources / attention, and incur opportunity cost. If we can fail fast and recover quickly, bad decisions don’t matter as much; and we get data for the next attempt. Failing fast on any endeavor requires us to establish concrete criteria for determining its success in some short time-frame (e.g., 3-6 months) before starting work. Fail-fast works for entire projects, but also for smaller decisions within the project (e.g., personnel assignments) or even as a philosophy for building the system (in effect, we can always convert throughput into goodput via rapid iteration on failures).

N. We want R&D, not !R&!D: R&D projects can often end up in no-man’s land, partly because it’s difficult to hold these projects accountable and measure their success. Ideally we want the project to be good “R” (publishable in top conferences) and good “D” (shipping to production). An anti-pattern is if external researchers think the project must be “D” (since it’s obviously not good “R”) and external developers think the project must be “R” (since it’s obviously not good “D”).

O. Synchronous, frequent, informal communication is critical: Prioritizing daily synchronous communication is critical. Note that this meeting is not for listing and managing work items (nobody wants a stressful daily stand-up in a skunkworks project); its goal is to build a shared understanding of the design space; and a shared set of values for assessing points in that space. We want to encourage free-wheeling debate on designs, long-term strategy, and short-term tactics; and train engineers to collaboratively think and talk about design. To manage meeting load, eliminate all other broadcast meetings. This principle is one of the reasons we prefer small teams.

P. People are not fungible. Team composition is critical. Our operating model is a sports team where we pick individuals for particular positions based on the needs of the team and their skill-sets. A second goalkeeper doesn’t help a soccer team much, even if they are absolutely stellar at what they do. Good managers will often do their best to make engineers fungible, in order to reduce personnel risk to the project; but in a true skunkworks team, nobody is fungible.

Q. Run towards risk. In skunkworks mode, the goal is to reduce technical risk as quickly as possible. Accordingly, the team has to surge on areas where risk is high. Fight the temptation to make steady progress on well-understood, low-risk parts of the system.

R. Keep the team small. This one seems obvious but is notoriously difficult to enforce in large companies, for a number of reasons. A well-meaning manager might add engineers to a project to 1) make it go faster; and 2) reward the engineer. But we’ve known for 50 years (!) that software projects actually do not go faster if you add people to them². And adding the wrong type of engineer can often hurt the project and the engineer’s career (see rule P about goalkeepers). Critically, keeping the team small ensures that it’s always resource-constrained (see rule K about ruthless prioritization); and also protects the project against cost-cutting initiatives (since the company doesn’t significantly reduce cost by shutting the project down).

I hope these rules help managers and engineers find common ground – good luck starting your own clean-slate skunkworks projects!

Footnotes:

this prior post might explain why I think clean-slate innovation is critical in systems. ↩
One VP argued – pedantically but accurately – that Fred Brooks only said this about projects that are running late; though in my experience, every software project is already late on day one. ↩

Three Laws of Software Complexity (or: why software engineers are always grumpy)

2024-05-08T10:00:00-07:00

I posit that most software engineers (particularly those working on infrastructural systems) are destined to wallow in unnecessary complexity due to three fundamental laws.

The First Law of Software Complexity: A well-designed system will degrade into a badly designed system over time.

We start with an opinionated definition: a well-designed system is one that is easy to change over time; a badly designed system is one that is difficult to change. Let’s say that a system X is well-designed. Someone comes along and changes it – by definition, quickly and easily – to a different system X’. Now X’ either continues to be well-designed; in which case it can be quickly and easily modified again to a different system X’’; or it will enter a badly designed state and hence be difficult to modify. For example, consider a well-designed database that uses RocksDB behind a clean storage engine API; and someone comes along and adds a getLevelSize call to it; now the database can no longer easily be modified to work over a non-LSM storage engine. It follows that a well-designed system is an unstable, ephemeral state; whereas a badly designed system is a stable, persistent state. Accordingly, the mix of systems in the wild continuously degrades towards poor design. The second derivative of code is always negative in the wild: the rate at which code can change goes down over time. Based on this law, most engineers will work on badly designed systems because most systems turn into badly designed systems over time.

The Second Law of Software Complexity: Complexity is a Moat (filled by Leaky Abstractions).

Designing a good abstraction is a delicate dance between providing utility to the application while hiding detail about the implementation. When systems compete with each other for market share, delicacy goes out the window and designers often give the application everything it wants. This has the dual effect of increasing market share by attracting application developers; while simultaneously making it difficult for competing systems to substitute different implementations under the hood. Some of the most successful systems on the planet have APIs that are nearly impossible to implement in any other way (ZooKeeper’s stronger-than-linearizable consistency and TCP/IP-based ephemeral node semantics come to mind; as does Kafka’s idempotent produce semantics). Based on this law, most engineers will work on badly designed systems because most successful/popular systems are badly designed systems.

The Third Law of Software Complexity: There is no fundamental upper limit on Software Complexity.

In real-world systems that are built by large groups of people over time, complexity is limited only by human creativity. The shape of a system is determined by the abilities, philosophies, and idiosyncrasies of dozens of developers, each working within a complex set of real and perceived incentives. For example, why does this replicated database use its own gossip layer to detect failures instead of relying on Kubernetes? Maybe Alice the TL and Bob the developer agreed that gossip-based failure detection was the way to go; but once Bob wrote the code, Alice realized that it was the wrong approach in a containerized environment; but Charlie the manager had already written the promo docs for Bob and Alice didn’t want to take the political risk of blocking the PR. Or maybe the system was initially designed by Bob for a non-containerized environment where gossip actually was a good choice. Or maybe Bob’s PhD was on gossip-based protocols. Maybe Alice then split membership and leader election into different layers to avoid antagonizing Bob and Charlie, which is why your system now has two layers with interesting interactions. Each existing system is a DoS attack on you by dozens of people you may not even know; a booby-trapped palace of ticking complexity time-bombs planted years ahead of your involvement. Based on this law, engineers that work on badly designed systems will particularly suffer since badly designed systems have unbounded complexity.

What can we do about this state of affairs? In my career, I have taken a particular approach based on building new systems from scratch (before they succumb to the three laws), but this is a lot harder than it sounds – more on that in a different post.

What we talk about when we talk about System Design

2023-07-12T09:33:00-07:00

Early in my research career, I had a chance to work with some of the best system researchers¹ in the world on a number of really interesting system designs. One of the enjoyable aspects of research was the particular process used by researchers (particularly in the SOSP/OSDI community) to come up with novel yet practical designs. This design process can be characterized as “fighting complexity with abstraction”: in any complex environment, how do you corral that complexity into cleanly defined boxes (or more technically, abstractions) and then divide functionality across these boxes?

Later, when I switched to “real” jobs in industry (ranging from mission-critical production services to applied R&D), I found that the same design process worked quite well in solving real-world problems in production settings². In these settings, the sources of complexity are varied (hardware, software, distributed protocols, org boundaries, deployment cycles, customers…) and so are the end-goals (reliability, scale, code velocity, performance, dollar cost); but abstraction-driven design still enabled my teams to hit production goals quickly and safely.

This post is a dump of some rules to follow in this particular design process.

[1] Late-bind on designs. The goal of the design process is not to generate a single point solution, but to instead characterize the design space for a given problem: a single point should then fall naturally out of that space given the problem constraints. Converging early on a single design is harmful; the team should have the ability to jump from one part of the space to another right until a solution is picked.

[2] Each point solution is a DoS attack on the design process. Talking about individual designs in isolation slows down design. Talking about designs in the context of the design space accelerates design. New designs should be described in terms of the design space, so you can immediately convey their relative position compared to other point solutions. Expect a lot of statements of the form: “all solutions must do X”; “solution Y is just X with one change”; “any solution that does X has to also do Y”; etc. Talking about the design space rather than point designs allows you to efficiently late-bind on designs (as in point 1) by lowering the cost of switching designs at any point in the discussion.

[3] Think in parallel; Design together; Implement in parallel; Review together. Certain parts of the design and development process are creative and should be parallelized / sharded, while others require discipline and should be centralized / broadcast³.

Thinking / brainstorming is a creative process and should happen in parallel with no coordination.
Design should be centralized. The design space is (strongly consistent) shared state between team members; new ideas should be slotted into this space with synchronous coordination.
Implementation can happen in parallel. After the centralized design phase, anyone should be able to implement any part of the design. Late-binding to developers is critical; it’s typical (and preferable) for the person implementing an idea to be different from the person who came up with it. Developers often get attached to ideas if they know they’ll get to implement it.
Reviewing should be centralized. The code base is shared state. API changes in particular have to be reviewed carefully by multiple people to make sure they are not one-way doors. In a healthy design process, Design and Review end up being centralized bottlenecks, which is okay. (In research, you have the same four steps; but the carefully reviewed deliverable is typically a paper rather than a codebase).

[4] Talk about the problem, not existing systems. It’s tempting to start the design process by looking at similar systems. This carries two types of risk:

Solution Complexity » Problem Complexity: Problems have some fundamental complexity (e.g., there’s some space of solutions that can solve atomic commit); however, individual solutions can have unbounded complexity limited only by human creativity (e.g., what does phase 5 of this ‘two-phase commit’ protocol really do?) and exacerbated by project pivots (due to changing business needs or getting scooped in research), team churn (or graduating students), timeline pressures (for publishing papers or landing code). You will often expend more cycles understanding the existing design than you would solving the problem from first principles.
Solution Bias: Even good solutions can bias your thinking towards a particular part of the design space. For example, someone reading the Raft paper might think that collocating learners and acceptors is fundamental (which is not true for Paxos); or someone reading Paxos might think that quorums have to constitute a majority (which is not true for Flexible Paxos). A great time to look at other systems is after the Design phase, to see if you can map those solutions to your space. Even better, you can often reverse-engineer the details of solutions simply by understanding where they fit in your design space.

[5] Always talk about a second application. For each abstraction, the “app” is the layer above it. For example, a filesystem is an app for a block device; TCP is an app for IP. You should be able to describe the functionality of a layer without ever referring to the specifics of the app (e.g., you don’t need to know what a file is when talking about an SSD’s internals). Practically, even if you are implementing only one app, it helps to always consider a second app (or even implement one in tests); to prevent application specifics from leaking into the abstraction.

[6] For each abstraction, build one implementation; plan for a second; hope for a third. In the opposite direction, you don’t want the abstraction’s semantics to rely on its implementation details. One way to ensure this is to talk about multiple implementations in the design process. For instance, if your replication layer is TCP-based (but you plan to also have a UDP-based variant; and you are hopeful that it’ll also work over carrier pigeons), then keeping the UDP variant in your head will prevent you from defining semantics in terms of TCP/IP channels.

[7] Abstraction is not free. Each abstraction layer introduces new semantics that developers have to define precisely and then reason about in generic ways (e.g., a new filesystem has to work with every possible correct implementation of a block device). As a result, abstraction is a balancing act between two types of complexity: the complexity of concreteness (where you have to understand inessential detail – e.g., a filesystem developer reasoning about an FTL implementation) and the complexity of abstractness (where you have to understand a range of possibilities – e.g., a filesystem developer thinking about all the possible implementations of the block device trim API). Each time you add a layer of abstraction, have a precise characterization for why it has to exist, as well as the division of functionality between this layer and the ones around it.

[8] Be critical (but about the right things). Researchers are used to seeing new ideas emerge from the primordial swamp and are often overly optimistic (part of the PhD training is to make students think more critically about their own ideas). In contrast, developers typically work with well-established systems; and as a result can be more critical of new ideas. New projects tend to look underbaked, feeble, and full of holes⁴. But every well-established system at some point was just 2-3 people tossing around half-baked ideas. One way to approach design is to continually de-risk the pieces that are truly unknown; while deferring work on the pieces that are difficult but known. (In the opposite direction, researchers need to be more focused on details and practicality, but this happens naturally in an industry environment).

This sounds like hyperbole, but I’ve been quite lucky when it comes to mentors; put together, my advisors at Cornell and colleagues at the now-defunct MSR Silicon Valley lab were responsible for inventing much of modern distributed computing over a span of five decades, but that’s a topic for another post. ↩
One could argue that this observation only applies to clean-slate projects; but I also had reasonable success converting dirty-slate problems into clean-slate via patterns like the ‘Strangler Fig’, as described in this talk. Just hide all the mess behind an abstraction, create clean new abstractions around it, and then move functionality! ↩
I created this formulation for my own teams at Meta; later, I learned about the similar ‘double diamond’ pattern. ↩
This is a great read on how to approach new ideas. ↩

The Leadership Myth in Replicated Databases

2023-05-06T09:30:00-07:00

I’ve heard multiple times that a strong notion of leadership somehow simplifies replication. I don’t think this is true. I explain why in this post.

In a prior post, we described the State Machine Replication (or SMR) abstraction. The SMR API allows you to propose commands and play them back in a total order. In another post, we described how SMR can be implemented either directly using RPC (e.g., as in Raft or Viewstamped Replication) or via intermediate abstractions like a shared log or a process group.

To start with, consider a system that’s implementing the SMR API. In such a system, there is typically a layer of stateful servers (i.e., either application or database servers) storing replicated state. When a server wants to update the replicated state, it proposes a new command to the SMR layer. The SMR layer continuously applies new commands to each server.

For now, let’s ignore how the SMR layer is implemented (i.e., via some custom consensus protocol or as a layer above some underlying abstraction like a shared log). Without loss of generality, I’ll use the terms ‘log’ or ‘shared log’ to mean ‘SMR total order’, regardless of whether the SMR layer constructs this total order via an actual shared log or some protocol like Raft or MultiPaxos.

In such a system, there are two key questions:

What is proposed to the shared log?
Who is allowed to propose to the shared log?

In classical SMR, the answer to what is simple: the proposal is an unexecuted piece of code (or a lambda, if you prefer). The proposing server does not first execute the code (i.e., apply the update to its local copy of state); instead, it first proposes the code as a command to the SMR layer. For example, a command might be “x++”; or “if(x<5) y++;” Each server is provided the same total order of commands by the SMR layer and executes the new commands on its local state as it receives them via the apply upcall.

This simple protocol results in a strong invariant: the local state at any given server always corresponds to some prefix of the SMR layer’s total order; if the server reboots, the SMR layer can examine its local copy of state and determine the position in the total order at which it should resume applying new commands. Ideally, local state is stored in some persistent store that has failure atomicity (e.g., RocksDB), so that the state can never reflect a half-applied command.

In classical SMR, the answer to who is also simple: any server can propose a new command. This results in a surprising and powerful property: there is no primary / master / leader at the database layer. You get active-active or multi-master replication for free. Clients can contact any database server and get a linearizable / strictly serializable response. The replicated database is trivially serializable since its state corresponds to the serial execution of the commands in the SMR total order.

There is no notion of leadership is classical SMR: systems are multi-primary by default.

Implementing an active-active / multi-master / multi-primary replicated database is simple: you can literally follow the most basic definition of SMR and obtain such a property. If active-active systems are this easy to build, why do systems even bother with electing leaders? The reasons lie on either side of the SMR API; let’s start first with life above the API.

Leadership above the Log: One reason for introducing leadership above the SMR API is safety: if the total order contains inputs proposed by any server, then each server is executing a lambda independently. As a result, if there is any non-determinism in the database server’s code (e.g., if a command is “if(random()>0.5) x++”), replica state will diverge. Non-determinism can arise due to any use of real time or randomness in the apply logic; or non-deterministic errors thrown during execution (e.g., a disk out-of-space error). In a sense, the complexity in the system moves away from reasoning about leadership changes (since there is no leader) and towards ensuring determinism.

In contrast, if the total order contains the output produced by an executed command (e.g., a write-set of keys updated by the command), we are no longer executing arbitrary code on multiple machines, so the burden of determinism is lower. Instead of “x++”, the command would simply say “x=1”.

If the log contains outputs, then we can no longer blindly apply commands proposed by different servers. Each database server will still converge to the same state, but updates are no longer linearizable. For example if two servers simultaneously receive an “x++” command from clients; each would read its current value of x and produce a new value (e.g., “x=1”) to propose. In effect, the replicated database no longer behaves as if it’s executing commands in a total order; as a result, it is no longer strictly serializable.

To re-introduce serializability, we could somehow ensure that there is only one server proposing to the log at any given time, by baking mechanisms such as leases and fencing into the log itself; or equivalently, stitching together a multi-proposer shared log from a sequence of single-proposer shared logs. This is an easier option if the log itself has a leader under the SMR API, as we discuss shortly; but exploiting that property can constrain the database to running over a specific log implementation.

However, there’s an easier option that’s agnostic to the log implementation: we can store speculative outputs in the log. In this case, each proposing server would first execute the update (“x++”) on its own local state; and then add a command that with the output and a read-set: “if(x==0) x=1;”. The speculation could be predicated on either the value of the state read by the command (i.e., “x==0”) or the version of that state. Essentially each database server applying that command would then ask the question: “would I get the same output as the proposer if I had executed the original command now?” or equivalently “has the state seen by the command changed since the command was executed at the proposer?”.

Note that the speculation could also be on the identity of the proposer: “if(proposer==primary) x=1;”. This brings us to the second way to re-introduce serializability: we can simply elect a designated proposer. Crucially, this election can happen above the SMR API. All we have to do is propose a ‘takeover’ command to the log saying “I am the designated proposer now; from this point, ignore commands (except takeovers) in the log from any other proposer”.

To switch between multi-primary and single-primary, we simply decide whether we want to store inputs or outputs in the log; and elect a designated proposer via the log.

A system that elects a primary in this manner obtains the performance benefits typically associated with primary-based systems. For example, we can do strongly consistent reads at the primary without catching up with the SMR total order (or equivalently, without contacting any other machine over a network): since the primary is the only proposer to the total order, it knows the sequence of commands in that order. Note that this kind of optimization in our SMR-driven election requires a real-time lease (new primaries have to wait for some time period after proposing the ‘takeover’ command), but such a real-time lease is fundamentally required for this optimization in any consensus-based system.

Leadership below the Log: Note that so far we said absolutely nothing about the implementation of the SMR layer. In itself, this might be a surprising observation: most of the properties associated with leadership (e.g., strongly consistent reads at a leader) have to do with the design of the database state machine above the SMR API; and absolutely nothing to do with the consensus protocol used to implement the SMR API.

Internally, the SMR layer could be implemented in any of the five ways we discussed in a previous post. Some implementations have a notion of a leader and others don’t. The Paxos protocol that implements a single consensus slot is explicitly designed to not require a leader; MultiPaxos reintroduces leadership as a liveness and latency optimization. Raft has a strong notion of a leader (e.g., the leader sees all I/O); Corfu has a weak notion of a leader (i.e., a sequencer that hands out timestamps but does not see all I/O).

The leader above the log – i.e., the database primary; the designated proposer – does not have to be the leader below the log.

In the 90s and 00s, the most common deployment model for replicated systems was to have a single replicated shard that collocated the database layer with the shared log; containing a strong primary above the log as well as a leader-based SMR implementation. In this deployment mode, it makes perfect sense to have a single machine play the role of the leader above and below the SMR API. Over time, implementations began to fuse these roles, blurring the distinction between leadership above and below the log.

However, understanding the difference between these two different leadership roles is very useful in a cloud setting. With the right layering, you can disaggregate your log layer from your database and scale it independently; switch your database from single-primary to multi-primary without changing the consensus protocol; change your consensus protocol to be leaderless without disturbing your database layer, and so on. For a more technical description of these ideas, see the Delos papers from Meta.

(This post is informed by several discussions over the years with various collaborators at Confluent and Meta; as well as Ben Reed and Allen Clement)

Is Raft more modular than MultiPaxos?

2021-12-14T10:40:00-08:00

There are at least five distinct paradigms for replication: Group Communication [0], Viewstamped Replication [1], MultiPaxos [2], Raft [3], and Shared Logs [4].

In a previous post, I did a deep-dive on MultiPaxos, showing that it implements a specific abstraction: State Machine Replication or SMR. The SMR API allows servers to propose commands and play them back in a durable total order:

class SMR<Command, Result>{
	public:
		//insert a new command into the durable total order;
		//play it back and apply it on the local state machine;
		//return the result
		Result propose(Command C);
		//return once all prior commands in the durable total order
		//have been applied to the local state machine.
		void sync();
		//register the local state machine to receive new commands
		void registerApply(Applicator A);
}

To kick things off, we make the following point: the five paradigms of replication are simply different implementations of the SMR API.

This is a useful observation for at least two reasons. First, an application (e.g., a database) built above the SMR API can work on any of these systems; if you need to modify your database to work over Raft rather than MultiPaxos, something went wrong in the design. Second, it implies that systems in each paradigm can be compared in a straightforward way, since they provide identical functionality.

In this post, we take a closer look at one particular aspect of these paradigms: modularity.

How modular is each implementation of SMR?

A key question is whether systems in these paradigms use a low-level distributed abstraction as a building block. To understand this, consider an analogy to storage stacks: if someone asked you to build a filesystem, rather than implement the file API directly over hardware (e.g., sending SATA commands), you would typically layer it over a block device, which hides the complexity of HW behind a simple interface. The block API makes it easy to run filesystems over diverse hardware (e.g., SSDs vs. HDDs; or network-attached disks vs. local storage). In addition, the block layer can add functionality without modifying the filesystem (e.g., RAID, encryption, deduplication, etc.). A similar analogy exists in networks: TCP layers complex functionality over IP, which hides the complexity of specific networks.

In our case, the SMR API is the filesystem (or TCP layer); but what’s the block device (or IP layer) analogue that hides the complexity of distributed consensus?

For two of these paradigms, there is no such “block device”. Raft and Viewstamped Replication both implement SMR directly via RPCs between a collection of machines, without a lower-level distributed abstraction. Each one is aware of consensus and implements it using simplifying primitives (e.g., single-machine logs, leaders, etc.).

For MultiPaxos, the “block device” is the Write-Once Register or WOR (as we showed in the earlier post. Given a collection of WORs, MultiPaxos stitches them together into a single total order of commands slots. MultiPaxos on its own does not need to be aware of the internals of each WOR slot (e.g., it need not understand quorums or phase 1 / phase 2 messages in Paxos). However, the logic in MultiPaxos for determining the location / membership of each slot can be quite subtle, lending to its reputation as a complex protocol.

In Group Communication, the “block device” is a multicast group. Groups hide more complexity than single WORs: they provide not just single-slot consensus, but also the ordering and membership mechanisms across consensus slots. In group communication (also known as virtual synchrony), each server implements the SMR API’s propose method by multicasting the message to the other servers; and then applies / delivers the message locally. Group communication systems were astonishingly modular nearly three decades before Raft first appeared, with full support for dynamic membership / view changes, pluggable consistency guarantees, etc. Unfortunately, groups provide a slightly weaker semantic by default than MultiPaxos: a command can be executed on a server before it is durable on a set of acceptors. (I’ll do a deep dive on group communication in a future post!)

Shared Logs combine the strong semantics of WORs with the WOR-stitching functionality of groups. Each server implements the SMR API’s propose call by appending to a shared log; playing the log forward until the appended command; and applying each command from the log to its local state. The shared log acts as the “block device”; in fact, this is more than a simple analogy since a shared log is effectively an append-only address space. In a sense, the extra complexity in MultiPaxos is around stitching together individual WORs with distinct memberships into an append-only address space; we take that functionality and push it underneath a Shared Log API.

So MultiPaxos, Shared Logs, and Group Communication are each more modular than Raft or Viewstamped Replication, for a precise definition of modularity: there is an internal abstraction layer that hides consensus.

Once you have a low-level abstraction like a WOR or a Shared Log, you can implement it with any protocol. A WOR can be implemented via any single-slot consensus protocol (Paxos, Flexible Paxos, etc.); whereas a Shared Log can be implemented with a protocol like Raft or Viewstamped Replication or MultiPaxos. Since both abstractions have data-centric storage APIs (i.e., no upcalls; no self-initiated activity; simple request/response interfaces), they can also be layered over other storage systems like key-value stores.

So far, all we have argued is that three of these five paradigms have an internal abstraction; and the other two do not. Why do these abstractions help if they are just internal APIs that you are anyway going to implement using some other protocol? After all, a compact, monolithic, and well-defined implementation like Raft is possibly preferable to a layered or modular implementation, unless we actually obtain some benefit from modularity in this setting.

Why does modularity matter?

The extra layer of internal abstraction is important because it logically separates learners from acceptors. In Lamport’s terminology, learners / proposers are the machines storing state and keeping it synchronized via a durable total order; whereas acceptors are the machines storing the total order itself. In other words, each learner keeps a first-class copy of the database; whereas acceptors store the commands mutating that database.

In MultiPaxos, the acceptors are hidden underneath the WOR API. In Shared Log systems, they are hidden underneath the Shared Log API. Logically separating learners from acceptors via an abstraction boundary has multiple benefits (see the Delos paper for more details):

The database tier (learners) can be collocated with the consensus tier (acceptors) or disaggregated.
We can run the database on any implementation of consensus (the same way that a filesystem can run on any block device).
We can scale the database and consensus tiers independently.
The consensus tier can be scaled by RAID-ing it (e.g., see Corfu in NSDI 2012; or Scalog in NSDI 2020).
We can deploy the DB tier separately (to roll out new DB features) without impacting the consensus tier.
We can deploy fewer DB copies, since learners have lower fault-tolerance requirements (only one needs to be alive) than acceptors.

Once you design an SMR-based system with the ability to decouple and disaggregate acceptors from learners, some connections become obvious. For instance: a replicated database like ZooKeeper or etcd and a streaming system like Kafka can both be expressed as simple, consensus-free state machines above a shared log. But more on that in a future post!

Citations:

[0] The process group approach to reliable distributed computing. Ken Birman. Communications of the ACM 1993.
[1] Viewstamped Replication Revisited. Barbara Liskov and James Cowling. MIT Tech Report 2012.
[2] Paxos made Moderately Complex. Robert van Renesse and Deniz Altinbuken. ACM Computing Surveys 2015.
[3] In Search of an Understandable Consensus Algorithm. Diego Ongaro and John Ousterhout. Usenix ATC 2014.
[4] Virtual Consensus in Delos. Mahesh Balakrishnan, Jason Flinn, Chen Shen, et al. Usenix OSDI 2020.

MultiPaxos made Simple

2021-11-20T20:40:00-08:00

There are three questions to ask of any system:

What abstraction does it implement?
What is the design space for such an abstraction?
Why is this abstraction useful?

In a previous post, we examined the Paxos protocol and answered the first two questions. Paxos implements the abstraction of a Write-once Register (a WOR) using a combination of quorums and a two-phase locking protocol. As for the third question: Paxos is useful because it can be used to implement MultiPaxos (among other things). But that’s an unhelpful answer unless you know what MultiPaxos does.

In this post, we seek to understand MultiPaxos by asking the same three questions of it. To start with, what abstraction does MultiPaxos implement?

MultiPaxos implements State Machine Replication.

Consider a stateful service that exposes mutators and accessors: for example, a key-value store with a put and a get API. If we run this service on a single server, the system is neither available (e.g., the server can reboot) or durable (e.g., the server can explode). To make the service available and durable, we need to run it on multiple servers; however, if each server maintains its own local copy of the state machine and updates it independently, clients will see inconsistent state for the service (e.g., if a client does a put on one server and then a get on a different one, it will not see the value it just wrote). In other words, the service will not be linearizable.

How do we run multiple copies of a stateful service on different servers so that it appears to be executing on a single copy? State Machine Replication (SMR) provides one answer. In SMR, rather than immediately modify its local copy of state in response to a put, a server first proposes the command to an underlying layer implementing the SMR abstraction. This layer appends the command to a totally ordered sequence of commands (assume for now that this sequence is stored in some location that is magically durable and available). The sequence – which includes commands proposed by all the servers – is then applied to each server’s local copy in strict order. On a get, the server first calls sync on the SMR API, applying any commands from the sequence that it hasn’t already seen; and then executes the get on its local, updated copy of state.

This simple protocol – propose first on mutators, sync first on accessors – ensures linearizability for the replicated service.

Concretely, here is the SMR API:

class SMR<Command, Result>{
	public:
		//insert a new command into the durable total order;
		//play it back and apply it on the local state machine;
		//return the result
		Result propose(Command C);
		//return once all prior commands in the durable total order
		//have been applied to the local state machine.
		void sync();
		//register the local state machine to receive new commands
		void registerApply(Applicator A);
}

MultiPaxos implements State Machine Replication using an address space of WORs.

There are many, many ways to implement the SMR API (groups, shared logs, the Raft protocol… this is a topic for a future post). For now, we consider how MultiPaxos implements such a sequence: using an address space of WORs.

The idea (outlined by Lamport) is simple: to propose a new command to the sequence, you simply locate the first unwritten WOR; and write to it. To sync with the sequence, you play it until the first unwritten WOR, applying each command to the local state machine.

One interesting implication of this design is that the WOR can be implemented in many ways: as a key-value pair on some remote storage system; or using a consensus protocol like Paxos, with the acceptors either collocated with the set of servers running our replicated service, or running on some entirely different set of machines. Further, a single address space can combine WORs with different implementations: WORs 0-10 might be on some external storage service, while WORs 11-20 could be on a collocated set of Paxos instances. More commonly, WORs 0-10 and 11-20 might use the same Paxos-based implementation, but reside on different set of acceptors (e.g., if some acceptors fail in the earlier range).

How do we know exactly where each WOR in the array resides (and what implementation it uses)? This is the membership or reconfiguration problem for MultiPaxos. Two types of solutions exist. First, Lamport’s original proposal was to store membership in-band: each WOR maintains a pointer to the next WOR (i.e., a linked list of WORs). When we write the payload for WOR I, we also include the location of WOR I+1. To allow parallel writes, we can determine the location of the next segment of K slots rather than just the next one.

A second solution is to store membership out-of-band: we can store a map from the WOR address space to different implementations / locations in an external location. This location – the membership store – has to be some kind of versioned conditional register. Think of this as implementing a virtual address space of WORs using an indirection map; or equivalently, a view of the system. Critically, there has to be some mechanism of stopping activity by servers that happen to have a stale view before we can switch to a new one. In other words, we need to stop the world in view X before writing view X+1 to the membership store. Typically this requires some extra capability from the WORs; some kind of seal or fence operation that prevents servers with old views from accessing WORs and tells them to go check the latest view. Alternatives to sealing include checking the membership store in the critical path of each operation (which is simple but slow); or relying on real-time leases for each view (this gets complex).

MultiPaxos implements State Machine Replication using an address space of WORs and a designated writer.

The SMR-based implementation above has two performance issues when it runs over Paxos-based WORs. The first relates to livelock. Recall that the API for a WOR has a lock (which in a Paxos-based WOR executes the first phase of Paxos) and a write (the second phase of Paxos). If two servers try to propose a new command at the same time, they will both lock the last WOR in the array; and lock each other out continuously without writing successfully.

As a result, we need a single designated writer to each WOR for better performance. Most commonly, MultiPaxos simply stores the identity of the designated writer along with the membership (either in-band or in the out-of-band membership store). Note that this is simply a hint to avoid livelock; if the designated writer fails, we can fall back on dueling writers.

Second, each write on a Paxos-based WOR requires two RTTs to a quorum of acceptors: the first to lock them, and the second to do the actual write. However, WORs can be pre-locked by a designated writer, allowing it to skip the first RTT. For instance, each newly created WOR can be pre-locked with lock number 0; and the designated writer for the WOR can issue the write with lock number 0. We are guaranteed that each WOR has only a single designated writer via the membership mechanism.

A designated writer solves a third performance / liveness concern generic to any kind of WOR implementation. To propose or sync, we need to find the first unwritten WOR; this is not guaranteed to terminate, since a server could keep chasing the tail of an expanding sequence. With a single designated writer, we can efficiently track the first unwritten WOR in the failure-free case; if this writer fails, we fall back on reading forward on the array.

That’s it. MultiPaxos is simple if Paxos is abstract, because it can then ignore all the subtlety of the single-slot Paxos protocol and be expressed purely in terms of invocations on the WOR API. Practically, you can implement and test a full-fledged MultiPaxos implementation by using some existing key-value store to implement each WOR, without requiring a Paxos implementation.

Paxos made Abstract

2021-11-15T02:01:35-08:00

The road to Paxos is a long one (as with other greek islands) and also somewhat elusive (it’s an island, after all). It took me longer than I’d like to admit to obtain a working understanding of the Paxos protocol. In my early attempts, I’d hit a brick wall of complexity: do I really need to know what this particular acceptor is going to do? What’s a learner anyway? What does it even mean to decide a value? Why do I need all these ballot numbers?

In systems, we deal with complexity via abstraction. For any system, there are three key questions:

What abstraction does it implement?
What is the design space for such an abstraction?
Why is this abstraction useful?

In this post, we answer the first two questions for Paxos.

This document is not meant to replace the Paxos papers (if you are relying on blog posts to implement Paxos… don’t). Rather, it supplies some of the informal systems intuition for why Paxos looks the way it does. Most of this will be obvious to experts.

Paxos implements a Write-Once Register:

Paxos is a protocol for implementing consensus. This means it implements a logical object called a write-once register (or WOR). A WOR has a simple API: you can write to it once; and you can read from it many times. Literally:

class WOR{
	public:
		//success means some write succeeded;
		//read after a write to see what was written.
		void write(std::string payload);
		//throw an exception if unwritten
		std::string read();
}

A WOR is safe for concurrent access. It is linearizable, or equivalently, strongly consistent. Informally, your implementation has to behave as if you had a big fat lock around it (regardless of how it’s actually implemented).

The WOR is an API for consensus. Consensus can be hidden behind many other APIs, but a WOR is the minimal API for it.

Understanding that consensus is just a type of object has some useful implications for system-builders.

You can immediately spot the existence or absence of consensus in other systems simply by examining their APIs. For instance, a key-value store with a simple put/get API does not necessarily implement consensus (you can’t implement a WOR on top); whereas a conditional put/get API does implement consensus (you can implement a WOR on top).
When building a real system that needs consensus, you can depend on some existing system as a makeshift implementation of the WOR (e.g., some existing key-value store with conditional puts); get the entire stack running; and later replace it cleanly with a consensus protocol.
You can cleanly isolate the complexity associated with consensus in a distributed system in the presence of failures (coming up next) within the WOR implementation, away from all the other complexity of your system.

Paxos implements a Write-Once Register using quorums for fault-tolerance:

The whole point of consensus is to do useful things in a distributed system in the presence of failures. What usually trips people up is the exact failure model.

Here is the model that Paxos uses. Machines can reboot arbitrarily; but they have persistent storage and will come back up. More worryingly, they can fail by exploding (losing all the bits they store).

We modeled consensus as a logical object: a WOR. Consider a system where a client is accessing a WOR, which in turn is stored on a cluster of storage servers.

A simple way to implement a write-once register (let’s call this WOR-server) is to have a single server store the value in RAM and expose the WOR API via RPC. But this is not durable; if the server reboots, the value is lost. A slightly better solution is to store the data on disk, so it’s not lost on a reboot. But if the server explodes, you lose data.

So WOR-server – a single-server design – is not durable if machines can explode. To get durability against exploding machines, we need to replicate data across servers. But clearly, if all the servers can explode, there is no way to get durability. So we need to make an assumption about the number of exploding servers.

Paxos assumes that only a minority of servers can explode; equivalently, to tolerate F exploding servers, Paxos requires 2F+1 servers. This leads directly to a quorum-based protocol. To write data durably, we simply have to store it on a majority quorum of servers. We assumed that only a minority can explode; so at least one unexploded server will contain the data.

So we arrive at WOR-quorum: a client writing to a WOR can simply write the data to a quorum of servers (each of which is running the single-server WOR-server design). Reading from the WOR requires the client to go to a quorum of servers. Such a solution is durable.

In Paxos, the servers are called acceptors; the client that is writing a value is called a proposer.

In a system where there’s only a single client that doesn’t fail, we are done. But in real systems we typically want multiple clients accessing the WOR concurrently; and each client can itself reboot or explode.

This brings us to the Paxos protocol.

Paxos implements a Write-Once Register using quorums for fault-tolerance and two-phase locking for concurrency control:

In WOR-quorum, when more than two clients write to a quorum at the same time, we can end up with different values on different minorities of acceptors. One way to prevent this is to first lock the acceptors, and then write to them (and unlock them).

Locking is tricky for two reasons. First, you can have deadlocks; acquiring locks in a strict order to prevent deadlocks adds latency. Second, locking in a distributed system has a new failure mode: a client can crash after acquiring locks.

Paxos provides a solution to both these issues via a form of lock stealing. Locks are versioned or numbered; higher-numbered locks can override the lower-numbered ones. Clients will pick a unique lock number; and then try to lock a quorum with it. The lock acquisition will succeed if the acceptor is unlocked, or locked with a lower number; and fail if the acceptor is locked with a higher number (in which case the locking client can retry with a higher lock number). Locks are not advisory; writes are predicated on lock numbers and will fail at the acceptor if the lock has been stolen.

Now for some subtleties.

Completing writes: Recall that we assume a minority of servers can explode. If a client locks a majority of servers; can’t access the remaining minority; and finds a value already written on a single acceptor in that majority, it has to assume that the value was also written on the inaccessible minority and acknowledged back to some older client. As a result, the only path forward is for the new client to adopt that value as its own and write it on the majority that is accessible to it. If more than one such value exists, the client has to pick the value with the highest associated lock number.

Livelock: Clearly the protocol above can livelock, if two clients keep stealing locks from each other. This problem turns out to be theoretically impossible to solve: the FLP result (which predates the Paxos protocol) states that you can’t have both liveness and safety for fault-tolerant consensus. Livelock in Paxos is a real-world example of the FLP result in action.

Different lock/write quorums: It turns out that you can lock some majority quorum and write to another majority quorum; it doesn’t have to be the same quorum in both phases. (But a write to an unlocked acceptor has to be interpreted as a lock-then-write, else you get this bug). Flexible Paxos takes this further by pointing out that the write quorum does not necessarily have to be a majority if lower durability is acceptable.

Pre-locking: Clients can pre-lock the quorum to avoid a round-trip when they write a value. To provide fine-grained control over locking, we can explicitly expose an extra API to the WOR called lock(). If a client is interacting with multiple WORs that happen to live on the same set of acceptors, we can pre-lock an entire batch of WORs. Pre-locking turns out to cover the key optimizations in MultiPaxos, which we will discuss later.

So here is the final WOR API:

class WOR{
	public:
		//lock a quorum
		int lock();
		//success means some write succeeded;
		//read after a write to see what was written.
		//throws an exception if you lost the lock.
		void write(std::string payload, int lockId);
		//throw an exception if unwritten
		std::string read();
}

At this point, we know the abstraction implemented by Paxos; and how it is implemented. In a future post, we will go into why this abstraction is useful.

Once again: this post is only an informal, intuitive description; go read the papers armed with this intuition before implementing the protocol!

A formal description of the WOR API is in this paper. David Geraghty proof-read an early version of this post. Dahlia Malkhi introduced me to idea of a WOR; Zhong Shao and Ji-Yong Shin worked with me to refine its API.