OPEN PROBLEMS — LLM INFERENCE

Every problem below is open in the sense that people are actively publishing on it and no answer has clearly won. Where I name a system or paper, treat it as a pointer to a line of work rather than a settled result.

1. Long context is still fundamentally unaffordable

The arithmetic from Level 02 does not improve with cleverness. Llama-3-70B at 128k context needs 40 GiB of KV cache per sequence. On four H100s that is three concurrent users.

The attacks all trade something real:

Eviction — H2O, SnapKV, StreamingLLM's attention sinks. Drop tokens judged unimportant. Fast and simple, but you cannot know in advance which token a future query will need, so it is lossy in a way that is hard to characterize and worse under adversarial or retrieval-heavy use.
Compression — quantize the cache, or project it to a lower rank. Reliable to 8 bits, harder below. Quality loss concentrates in exactly the long-context tasks you bought the context for.
Architecture — MLA compresses to a latent; sliding-window and sparse attention bound what must be kept. Requires training the model that way; no help for existing checkpoints.
Offload — push cold cache to CPU DRAM or NVMe. PCIe at ~64 GB/s against HBM at 3.35 TB/s is a 50× cliff, so this only works when access is predictable.

Nobody has an approach that is simultaneously lossless, architecture-agnostic and cheap. The honest state of the art is a menu of trades.

2. Reasoning models inverted the workload

Long chain-of-thought models spend thousands of tokens thinking before answering. A request that was 2,000 prompt tokens and 200 output tokens becomes 2,000 and 20,000.

That change is not incremental — it moves nearly all the wall-clock into decode, the memory-bound half. It also means the KV cache grows for far longer per request, so sequences occupy batch slots much longer, and it makes TTFT nearly irrelevant next to total time.

Open questions: can you batch reasoning traces more aggressively given they are less latency-sensitive? Should thinking tokens use a cheaper decode path, or a smaller model, or aggressive speculation given they are never shown to the user? Can you prune a reasoning trace mid-flight when it is clearly going nowhere? There is no consensus on any of these.

3. Speculative decoding at high batch

Speculation converts spare compute into throughput, which works beautifully at batch 1 and stops working as batch grows — because at large batch you are approaching the ridge point and no longer have spare compute to convert. Worse, a rejected draft wastes work that a non-speculating batch would have spent usefully.

The open problem is adaptive speculation: choosing draft length per request, per step, based on current batch occupancy and observed acceptance rate. Some systems do simple versions. Doing it well, with a scheduler that understands the interaction between speculation and batching, is unsolved.

4. Agentic and multi-turn workloads

An agent re-sends its whole growing trajectory every step. Without prefix caching, total prefill work is quadratic in the number of steps. With it, the cache becomes the working set of the system and the interesting questions are all about cache policy:

What do you evict when the cache holds hundreds of partially-shared trajectories?
Should you keep a trajectory's cache warm while the agent waits on a slow external tool call — paying memory for latency you do not control?
How do you schedule fairly when one agent's cache is worth 50× another's?

This is a caching and scheduling problem more than a kernels problem, and it is comparatively under-explored.

5. Evaluating optimized models honestly

We are still not good at this. Perplexity is insufficient — a 4-bit model can match fp16 perplexity and fail badly on multi-step reasoning or long-context retrieval. Quality loss from quantization is not uniform: it concentrates in rare tokens, long contexts, and exactly the capabilities that are hardest to measure.

There is no standard, trusted evaluation suite for "is this quantized model still the model I tested?" — which means a great deal of production quantization is deployed on faith and spot-checks. This is arguably the most important unglamorous problem in the list.

6. Disaggregation at scale

Prefill/decode disaggregation is clearly right in principle and awkward in practice. You must move the KV cache between pools, and for a long prompt that is gigabytes over an interconnect. Open: how to place and schedule across heterogeneous pools, how to size the two pools as traffic shifts, and whether the transfer can be overlapped well enough to disappear.

7. Hardware and the widening wall

Compute has grown far faster than bandwidth for a decade, and the ridge point has drifted from 153 on an A100 to 295 on an H100. Each generation makes decode relatively worse.

FP4 and FP8 help by shrinking bytes. Larger HBM stacks help capacity. But the structural fix would be a memory-centric architecture — much more bandwidth per FLOP, or compute placed nearer the memory. Several companies are betting on variations of this. Whether any of it displaces GPUs at scale is genuinely uncertain, and it is the question that would most change everything else in this course.

8. The economics

Inference is now the dominant cost of operating a model, and the field has no shared vocabulary for cost-quality trade-offs. What is the right price for a token that took 20,000 thinking tokens to produce? How should a provider expose the latency/throughput dial to customers who do not know they want it? These are not technical questions, but they increasingly determine which technical work gets done.

A note on currency. This page reflects the state of things as of the author's knowledge and was not verified against live sources at build time. Inference moves quickly, particularly on points 2, 3 and 7. Before relying on any specific claim here, search for recent work — and if something below has obviously been solved since, that is a good sign the field is healthy.