IBM Just Served a Frontier Model for $0.11 per Million Tokens — llm-d 1.0 Made Inference 11.5 Times Cheaper
The expensive part of AI is no longer training it
For two years, the AI cost conversation was about training: bigger clusters, scarcer GPUs, larger capital budgets. In production, a different bill has quietly taken over. Every agent step, every tool call and every follow-up question is inference — and for long-context, multi-turn agent workloads, inference is now the dominant, recurring cost of running AI at all.
On October 20, at PyTorch Conference North America, the llm-d project released version 1.0 — its first stable release — and published the number that makes that shift concrete. IBM Research reported serving a frontier-class open model at a blended cost of about $0.11 per million tokens, roughly 11.5 times lower than an equivalent proprietary API stack, at peak throughput above 6.6 million output tokens per minute while 3,000 coding agents ran concurrently with zero preemptions.
The model in that deployment was GLM-5.2, a 744-billion-parameter mixture-of-experts model with 39 billion active parameters, served on widely deployed NVIDIA GPU fleets. This was not a lab demo of a small model. It was a production-shaped workload, reported by the team running it.
What llm-d actually is
llm-d is a Kubernetes-native, open-source coordination layer for distributed LLM inference. It does not replace the inference engines teams already use — vLLM and SGLang still serve the model on each machine. llm-d sits above them and turns a fleet of separate model servers into one coordinated serving pool, adding the cluster-level work the engines do not do on their own: intelligent routing, distributed KV-cache management, and split prefill/decode processing.
The project was founded in May 2025 by Red Hat, IBM Research, Google Cloud, CoreWeave and NVIDIA, under the deliberately portable vision of “any model, any accelerator, any cloud.” It joined the Cloud Native Computing Foundation as a Sandbox project in March 2026 and is Apache 2.0 licensed. By the 1.0 announcement it had grown to more than 500 contributors across more than 25 organisations, including AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago.
One precision matters here: CNCF Sandbox is the foundation’s earliest maturity tier. The “1.0” is the project’s own declaration of production readiness — stable, versioned APIs and hardened operations — not a CNCF Graduated stamp. Both things can be true, and readers pricing a platform decision should hold them separately.
Three mechanisms do most of the work
The savings are not magic, and they are not mainly about a faster model. They come from stopping the waste that a generic load balancer creates.
- Prefix-cache-aware routing. A normal balancer sends each request to a random replica, which then recomputes the request’s entire input — system prompt, tool definitions, conversation history — from scratch. llm-d reads each replica’s KV-cache state and routes the request to the machine that already holds that prefix in memory. On IBM Research’s agentic coding workload, that produced a prefix-cache hit rate approaching 98% — meaning the most expensive part of the computation was replaced by a lookup, almost every time.
- Tiered KV-cache offloading. GPU memory is fast and scarce. When it fills, llm-d spills cache entries to CPU memory or local SSD instead of discarding them, extending the effective context a fleet can hold. The project reports a 13.9x throughput improvement at 250 concurrent users versus GPU-only serving from this technique.
- Disaggregated prefill and decode. Reading a prompt (prefill) is compute-heavy; generating tokens (decode) is memory-bandwidth-heavy. Running both on the same hardware wastes both. llm-d splits them onto separate worker pools matched to each job. AWS reported 70% higher tokens per second running that pattern on SageMaker HyperPod versus standard vLLM.
The same pattern shows up independently elsewhere. On a Google Cloud agentic coding workload on Ironwood TPUs, prefix-aware routing plus host-memory offload cut median time-to-first-token by more than 85% and raised throughput 2.4x versus a plain Kubernetes Service. Google also says its GKE Inference Gateway — built on llm-d’s router — more than doubled its prefix-cache hit rate in production, from 35% to 70%, and cut time-to-first-token by more than 35%.
What 1.0 adds is not speed — it is operability
Earlier 0.x releases proved reach: the same stack running on NVIDIA, AMD and Intel hardware and Google TPUs, in public clouds and on-premises. Version 1.0 is explicitly about “Day 2” — running inference reliably after the demo works.
- Multi-tenant quality of service: priority- and fairness-aware flow control at the router, so one tenant’s bursty agent loop cannot stall everyone else’s interactive requests, with graceful load-shedding when demand exceeds capacity.
- Zero-downtime operations: a hot-standby router that takes over if the active one fails, and coordinated lockstep upgrades of prefill and decode pools so traffic only reaches a pair once both sides are compatible.
- Agent-aware serving: session-aware routing that keeps a multi-turn conversation pinned to the replica holding its context, and native support for carrying context across turns without the agent resending its full history each time.
- A stable API: the first versioned, stable configuration surface, so platform teams can standardise on it without chasing breaking changes between releases.
Alongside IBM and Google, the launch roster of reported production users included Red Hat (shipping llm-d in Red Hat AI), DigitalOcean (serving the trillion-parameter Kimi K3 across mixed AMD and NVIDIA hardware), JetBrains (its Junie coding agent), Mistral AI, Prime Intellect, AWS, and a Tesla/Red Hat case study showing 3x output tokens per second on Llama 3.1 70B.
Read the headline number carefully
$0.11 per million tokens is a blended serving cost for one organisation’s specific workload — long-context, highly repetitive agentic coding, which is close to the best case for cache-aware routing. Workloads with short, unique prompts will see far less benefit, because there is less shared prefix to reuse. The 5-to-10x cost range the project cites against commercial API pricing is the more honest planning figure; 11.5x is the reported peak, not the average.
Self-hosting also is not free in the ways an API is. You still buy or rent the GPUs, run the Kubernetes fleet, and employ the people who keep it up at 3 a.m. Version 1.0 reduces that operational tax — it does not eliminate it. Below a certain volume, a commercial API will still win on total cost, convenience and time to first request.
And the coalition’s incentives are worth naming, because they explain why this software is free. The founders sell hardware and cloud infrastructure. If the inference software layer is open and commoditised, the remaining margin flows to GPUs and cloud capacity — exactly where they sit. That does not make llm-d less useful. It makes its economics durable: the project is maintained because keeping it open serves its backers, not despite their interests. CNCF governance and the Apache 2.0 licence add the other half of the safety case — no single founder can take it proprietary or strand adopters by walking away.
What to do with this, this week
- Profile your workload before you price an alternative. If your traffic is multi-turn agents with long shared system prompts and tool definitions, you are in llm-d’s best case and the savings case is real. If it is short one-shot chat, it probably is not.
- Measure your current prefix waste. If you self-host behind round-robin balancing today, your replicas are recomputing the same prefixes constantly. Cache hit rate is the first number to instrument — it predicts most of the gain.
- Do not rip out your engine. llm-d layers over vLLM and SGLang. A pilot is an additive routing and cache layer on a fleet you already run, not a migration.
- Compare on total cost, not token price. Put GPU, operations and engineering time on your side of the ledger, and per-token price plus lock-in risk on the API side. Run both numbers monthly; the crossover point moves as your volume grows.
The bottom line
llm-d 1.0 is not a new model, and that is exactly why it matters. The frontier race has spent two years making models more capable; this release is about making them radically cheaper to actually run — by routing requests to work already done, instead of paying to redo it. An 11.5x cost drop reported in production, from the most boring layer of the stack, changes which AI products are economically possible. The next cost war in AI will not be won by the smartest model. It will be won by whoever wastes the least compute serving it.