I’ve been building systems, teaching system design, and writing papers about novel designs for a couple of decades now. In my view, agents are magical self-writing distributed programs: my personal research agenda has swivelled to designing abstractions that can make them safe when they interact with complicated environments. But as I design and write1 this new system (LogAct / AgentBus), I have been puzzled by why designing a system has not become as trivially outsourceable to the LLM as writing code. Here are some unscientific thoughts, mostly to get a conversation going.

  • Concretely, can an agentic software factory build S3 from scratch? Let’s say I provide a swarm of agents with: the S3 API and an in-memory single-node implementation; and access to a scheduler. I come back a month later: can the agentic swarm re-create S3 in all its glory and scale, with sharding for scale and replication for fault-tolerance?
  • Correctness is difficult, but perhaps not the bottleneck. If we vastly simplify S3 down to “just a storage system” (ignoring things like billing, authorization, etc. that make it a viable business), the in-memory implementation could operate as the correctness specification of the service. The service must match the semantics of that implementation in the presence of concurrency and failures. This approach works particularly for strongly consistent systems with strong but correspondingly simple semantics (e.g., linearizability). It also works well for storage services with relatively simple APIs. YMMV with more complex APIs with weaker (and harder to specify) semantics.
  • Simulation is difficult, but perhaps not the bottleneck. Tests can be written by humans and LLMs against the API. We provide the agent with a deterministic simulator that allows it to check the correctness of its new implementations under concurrency and failures. For a storage service, it’s tractable to simulate machines, networks, and simple workload generators; particularly for safety / correctness invariants such as linearizability.
  • Verification is difficult, but perhaps not the bottleneck. LLMs have made it easier to write formal specifications and proofs. Behavior under concurrency and failure is still difficult to verify, but LLMs remove the drudgery. We are getting closer to the goal of a single specification that can be used to synthesize code as well as prove invariants.
  • Performance Optimization is difficult, but not the bottleneck. Once agents can generate new implementations of the API, they can estimate performance via simulators or even deploy / canary implementations to generate real performance numbers. Finding a performant implementation then turns into a hill-climbing exercise on the space of implementations, bitter-lesson-pilled.
  • Evolution is a real bottleneck. Change is hard. The hardest problem senior engineers / system designers deal with in industry is migration: safely moving a complex system from one state to another without user-visible impact. Avoiding one-way doors in system design is critical. Every change potentially makes future changes more difficult.
  • Frangibility makes it difficult to learn. Evolution is also complicated by frangibility. Techniques such as RL do well in environments that are not frangible or destructible, since they can try things and learn by observing outcomes. But an agent can’t just roll out a new version of S3 and see what happens.
  • Entropy creates an agentic arms race. While the agent figures out how to evolve things safely, the system itself is changing. Nodes and networks fail. Fixes accumulate complexity. Migrations leave behind trails of special-case handling. Zombie agents continue to make changes based on stale directives. On the one hand, motivated agents should be able to reduce entropy / technical debt efficiently. On the other hand, agents with partial context and diverse goals can also increase entropy very efficiently.
  • Simulation hits a diversity barrier. There are just too many possibilities to simulate. Determining whether a roll-out of a new version of S3 would cause an outage requires simulating not just that version (which is arguably tractable), but every simultaneously running version that accesses shared state. Further, the end-to-end behavior of any application using S3 can’t be determined unless we simulate it as well. Applications might use multiple library implementations; different transport protocols; various tuning parameters. How do we know what is important to simulate?
  • Abstraction is a possible solution. Humans deal with complexity in general (and entropy / evolution in particular) by hiding it via abstraction. Maybe agents can do the same. One problem: language models reason via language; but abstractions typically are not expressible precisely and compactly via language. (Some abstractions are literally individual APIs; but others – like processes or RPC – are more conceptual and diffuse). Most system design conversations between humans still happen via whiteboard squiggles. Right now, agents reason about system design using a combination of low-level code and imprecise text. We are perhaps missing a language for system design centered on abstraction. For me, the shift from humans to agents will happen in system design when AI creates a novel, useful agent-generated abstraction for its own use – much like primates inventing tools2 – with the express goal of reasoning more effectively about the design space.

Footnotes:

  1. Why not just solve the problem of agentic system design and then ask the agents to design the system themselves? Well, the one thing you can’t trust the agents to design on their own is the system that ensures they operate safely. Quis custodiet ipsos custodes? ↩

  2. I just happen to like em-dashes; no AI was used in the generation of this text. ↩