GenAI Capacity Is a Product, Not Just an Accelerator Pool

For a long time, I noticed that capacity conversations became hardware conversations almost immediately.

How many GPUs do we have? When will the next TPU slice arrive? Can we add more replicas? What does utilization look like?

I understand why. Accelerators are expensive, visible, and difficult to acquire. When traffic is being throttled, “we need more chips” feels like the most concrete answer in the room.

But after working across ML hardware delivery, large-scale inference infrastructure, capacity planning, and GenAI serving, I have become convinced that this framing is incomplete.

A GPU or TPU pool is infrastructure. Capacity is a product.

The hardware matters, of course. But users do not consume a pile of accelerators. They consume a promise: that a particular workload can run, in a particular place, at a particular level of performance, when it matters.

Turning hardware into that promise is the real engineering problem.

This article reflects my own lessons from working on large-scale ML and GenAI systems, plus what I have explored independently in projects such as Atlas and Hyperion. I am deliberately keeping the discussion at the level of general architecture and engineering judgment rather than describing any employer’s internal systems.

The Mistake I Keep Seeing

I have seen variations of the same failure mode at several layers of infrastructure.

A team has a launch coming. Demand looks larger than expected. Capacity feels tight. The conversation narrows to supply: more machines, more accelerators, more replicas.

Sometimes supply really is the bottleneck. But even after hardware arrives, the original problem may remain:

• the allocation exists in a planning document but is not enforced in the serving path;
• the customer and the platform use different definitions of a capacity unit;
• one tenant’s burst consumes headroom intended for another;
• aggregate fleet utilization looks healthy while one model in one region is saturated;
• or the workload shape has changed so much that the original estimate no longer means anything.

In those cases, the fleet is not necessarily too small. The contract between demand and supply is unclear.

That distinction changed how I think about capacity. I no longer start with, “How many accelerators do we need?” I start with, “What exactly are we promising, and how will the system keep that promise?”

GPUs and TPUs Are Supply, Not the Product Boundary

The original title of this article used “GPU pool.” That was too narrow.

Modern ML serving fleets can include GPUs, TPUs, CPUs, and other specialized accelerators. Even within one accelerator family, different generations, memory configurations, interconnects, runtime versions, and model optimizations can produce very different results.

One TPU slice and one GPU node are not interchangeable boxes. More importantly, neither is a useful customer-facing unit by itself.

The same hardware can deliver different throughput depending on:

• the model architecture and parameter count;
• input and output sequence lengths;
• quantization and compilation choices;
• batch formation and KV-cache pressure;
• streaming behavior;
• and the latency target the service is trying to maintain.

This is why accelerator count is useful for supply planning but dangerous as the only definition of capacity.

Customers need a unit closer to the work they submit. That might be tokens per minute, concurrent generation slots, model-specific throughput units, or a normalized unit calibrated against a reference workload.

There is no perfect unit. I care more about whether the unit is understandable, enforceable, and measurable. If product planning, quota enforcement, scheduling, and billing all interpret the unit differently, the system has a correctness problem before the first request arrives.

What the Customer Is Actually Buying

When someone reserves GenAI capacity, they are not merely buying access to compute. They are buying confidence.

They want to know whether they can launch on Tuesday without being crowded out by another tenant on Wednesday. They want to know what happens when traffic exceeds the forecast. They want to distinguish their own quota issue from a platform shortage. They want an answer more useful than “the fleet was busy.”

For me, a capacity product needs to make five things explicit:

1. The entitlement: how much work the tenant may submit.
2. The scope: which model, region, project, traffic class, and time window it applies to.
3. The workload envelope: the request shapes and concurrency assumptions behind the promise.
4. The failure behavior: what happens above entitlement or during degraded supply.
5. The evidence: how both sides can verify what was attempted, admitted, served, or rejected.

Notice that hardware does not appear in this list. Hardware is how the provider fulfills the contract. It should not leak into every customer decision.

This separation is valuable internally too. It allows the platform to change model placement, accelerator type, runtime, or batching strategy without rewriting the external promise every time the implementation evolves.

Capacity and Latency Are Related, but They Are Not the Same Promise

One mistake I have learned to watch for is allowing “reserved capacity” to quietly become “guaranteed latency under any workload.”

A platform may reserve enough aggregate throughput and still produce poor tail latency. Long prompts can block short ones. A cold model can add startup delay. Safety checks can consume part of the latency budget. A request may spend more time waiting in a queue than running on an accelerator.

The contract therefore needs a workload envelope. A latency expectation without assumptions about token length, concurrency, model, and region is not really an SLO. It is a hope.

I prefer to separate the questions:

• How much work will the platform admit?
• Under what request shape?
• What latency dimensions are targets or commitments?
• What happens when the workload moves outside that envelope?

That last question matters. Real workloads do not stay inside neat boxes. Product launches succeed unexpectedly. Clients retry. Output lengths drift. Teams switch models. A useful capacity product is not one that assumes perfect forecasts; it is one that behaves predictably when the forecast is wrong.

The Product Is a Control Loop

I think of capacity as a loop rather than a stockpile:

Demand signal
    ↓
Forecast and allocation
    ↓
Entitlement configuration
    ↓
Admission control and scheduling
    ↓
Serving telemetry
    ↓
Reconciliation and supply correction
    └──────────────────────────────→ back to planning

The difficult part is that each stage runs on a different clock.

Accelerator supply may require planning weeks or months ahead. Fleet provisioning may take hours or days. Admission control makes decisions in milliseconds. Usage reconciliation operates over minutes or hours. Business commitments may last months.

I have seen teams try to solve all of these horizons with one mechanism, usually autoscaling. Autoscaling is useful, but it cannot manufacture scarce accelerators during a synchronized launch. It also cannot repair an entitlement that was never connected to the serving path.

The control loop only works when each stage has a clear responsibility and passes consistent information to the next one.

The Capacity You Can Sell Is Smaller Than the Capacity You Own

Peak benchmark throughput is not deliverable capacity.

Before traffic arrives, some supply is already spoken for:

deliverable capacity
  = physical accelerator capacity
  - failover reserve
  - rollout and maintenance headroom
  - operational safety margin
  - placement and fragmentation loss

The exact formula is less important than the discipline behind it. If every team plans against the theoretical maximum, the same headroom gets promised several times.

Placement makes this more subtle. A fleet can show moderate average utilization while a particular model, accelerator generation, or region is exhausted. A device can have available compute but not enough memory for another model replica or a larger KV cache. To a fleet-wide dashboard, the hardware looks idle. To the scheduler, it may be useless for the waiting request.

This was one of the most important shifts in my own thinking: capacity is not just quantity. It has shape, locality, and timing.

Forecasting Is About the Cost of Being Wrong

LLM demand is not well described by requests per second alone.

Input length, output length, concurrency, model mix, streaming duration, region, retries, and caching all matter. Two applications sending the same number of requests can consume radically different amounts of accelerator time and memory.

I do not think a single forecast should be treated as truth. I prefer a small set of scenarios:

• committed demand tied to a real launch or customer promise;
• expected demand based on current evidence;
• upside demand if adoption goes better than planned;
• stress demand for correlated spikes or failover.

The point is not to pick the cleverest prediction. The point is to make the consequence of forecast error visible.

Under-forecasting can cause throttling and damage a launch. Over-forecasting can strand expensive TPU or GPU capacity. The right buffer depends on accelerator lead time, uncertainty, the importance of the workload, and whether unused capacity can be safely shared.

That is why capacity planning is as much a risk decision as a modeling problem.

Admission Control Comes Before Autoscaling

When a serving system is overloaded, accepting unlimited work is usually worse than rejecting some of it early.

An unbounded queue turns a capacity shortage into a latency incident. Clients time out, retry, and create even more work. By the time autoscaling reacts, the system may already be serving requests whose callers have given up.

Admission control gives the platform a stable boundary. It decides whether a request fits the tenant’s entitlement and whether the serving system can accept the work now.

For GenAI, that decision cannot rely only on request count. A streaming request occupies capacity over time, and its final output length is unknown when it arrives. A practical gateway may need to estimate token cost, reserve quota, track concurrent work, release unused reservation, and reconcile the estimate after generation completes.

This is why I made token-aware reservations central to Atlas. Atlas is an independently built reference implementation and has not operated under sustained production traffic. I use it to make these ideas concrete, not to claim scale it has not demonstrated.

The broader lesson is one I have carried across different systems: scale behind a controlled boundary. Autoscaling works better when the platform first controls what it accepts.

Scheduling Is Product Policy Wearing an Infrastructure Hat

Schedulers are often judged by throughput and utilization. In a multi-tenant capacity product, they also enforce policy.

Imagine a fleet serving reserved, shared, and best-effort traffic. When the fleet is quiet, borrowing unused reserved capacity improves efficiency. When contention returns, the scheduler must give that capacity back quickly enough to preserve the original promise.

That raises uncomfortable questions:

• Can unused reservations be borrowed?
• How quickly can they be reclaimed?
• Can priority traffic move ahead of queued work?
• Should long and short requests share a queue?
• Is fairness measured in requests, tokens, compute time, or cost?

There is no scheduler setting that answers these questions automatically. They are product decisions expressed through infrastructure.

This is also why utilization should not be the only success metric. A fleet running near 100% may look efficient while delivering a terrible experience to the tenants it was supposed to protect. Some headroom is not waste; it is part of the reliability promise.

Observability Should Explain the Contract

GPU utilization, TPU utilization, memory, replica count, and error rate remain essential. But they are not enough to operate a capacity product.

During an incident, I want to answer questions such as:

• How much entitled traffic arrived?
• How much was admitted?
• What was throttled, and for what reason?
• Was the constraint quota, queueing, model placement, accelerator supply, or request shape?
• How different were estimated and actual token usage?
• Did the platform fail the promise, or did the workload move outside its envelope?

If the only user-visible result is a generic 429 or 503, every capacity issue becomes a support investigation.

Capacity context should travel with the request through the gateway, router, safety layers, scheduler, and inference runtime. That is the only way a trace can explain not just that a request was slow, but why.

This is an area where my interest in serving and observability naturally came together. Metrics are not decoration after a system is built. They are how the product proves that its contract is real.

Failover Capacity Cannot Be Counted Twice

Capacity promises are easiest to make when the fleet is healthy. They are tested when a zone fails, a model rollout reduces healthy replicas, or several workloads spike together.

One trap is to treat the same failover reserve as available to multiple regions or products. On paper, each service appears protected. During an actual failure, all of them attempt to consume the same backup supply.

The platform needs an explicit answer for degraded conditions: bounded queues, overload reason codes, retry guidance, circuit breakers, fallback models, regional routing, or controlled rejection.

I do not believe every system should fail closed or every workload should receive the same fallback. I do believe the behavior should be decided before the incident. Ambiguity is expensive when supply is already constrained.

The Organizational Contract Is Part of the System

Capacity crosses too many boundaries to be owned by one team.

Product teams understand launches. Infrastructure teams understand the fleet. Model teams change runtime behavior. Finance sees cost. Reliability teams see incidents. Sales and support communicate expectations to customers.

Many capacity problems are really synchronization problems between these groups.

The artifacts that help most are not glamorous: one definition of the capacity unit, one source of truth for entitlements, one launch-intake process, one mapping from demand to supply, and one set of reason codes when the platform cannot admit work.

I have learned to value this kind of operational clarity more than an impressive architecture diagram. The diagram describes the happy path. The shared definitions are what keep the system working when plans change.

The Questions I Would Ask Before Calling Capacity “Guaranteed”

I would not start by asking how many GPUs or TPUs are in the fleet. I would ask:

• What exactly is the customer entitled to, and where?
• Which workload assumptions make the promise valid?
• Can the serving path enforce the entitlement in real time?
• Are streaming requests reserved and reconciled safely?
• Is failover headroom protected from double counting?
• Can unused capacity be shared without weakening the reserved promise?
• Can an operator explain every throttle or rejection?
• Can the customer tell quota exhaustion from platform shortage?
• What happens when the forecast is wrong?

If these questions do not have clear answers, the platform may own substantial accelerator capacity without yet offering a reliable capacity product.

Final Thoughts

I started my career thinking about infrastructure largely in terms of delivery: get the hardware connected, make the software reliable, automate the manual steps, and keep launches on schedule.

Working closer to ML and GenAI serving changed that perspective. Delivery is only the beginning. The harder problem is translating heterogeneous GPU and TPU supply into a promise that customers can plan around and operators can defend under pressure.

That promise lives across forecasting, entitlement, admission control, scheduling, observability, and failure handling. If any one of those layers uses a different definition of capacity, the hardware pool cannot save the product from confusion.

So yes, accelerator supply matters. It matters enormously.

But the real product is confidence: confidence that the workload will run, that contention will be handled intentionally, and that when reality diverges from the plan, everyone can understand why.