HARDWARE TABLE — LLM INFERENCE

How to read this table

Two columns matter more than the rest.

Bandwidth sets your decode speed. Time per token at batch 1 is approximately weight_bytes / bandwidth, and nothing else in the spec sheet enters that calculation.

Ridge point is dense FLOP/s ÷ bandwidth — the arithmetic intensity at which the machine stops being memory-bound. Since decode intensity equals your batch size, the ridge point is the batch size you would need to saturate the machine's arithmetic. A lower ridge is easier to saturate, and therefore better for decode.

All compute figures below are dense BF16. Vendors usually headline the 2:4 structured sparsity number, which is double and does not apply to production LLMs.

Datacenter GPUs

GPU	memory	bandwidth	dense BF16	dense FP8	ridge point
A100 40GB SXM	40 GB HBM2e	1.56 TB/s	312 TFLOP/s	—	200
A100 80GB SXM	80 GB HBM2e	2.04 TB/s	312 TFLOP/s	—	153
H100 SXM	80 GB HBM3	3.35 TB/s	989 TFLOP/s	1979 TFLOP/s	295
H100 PCIe	80 GB HBM3	2.0 TB/s	756 TFLOP/s	1513 TFLOP/s	378
H200 SXM	141 GB HBM3e	4.8 TB/s	989 TFLOP/s	1979 TFLOP/s	206
B200	192 GB HBM3e	~8 TB/s	~2.25 PFLOP/s	~4.5 PFLOP/s	~281
L40S	48 GB GDDR6	0.86 TB/s	362 TFLOP/s	733 TFLOP/s	419
AMD MI300X	192 GB HBM3	5.3 TB/s	~1.31 PFLOP/s	~2.61 PFLOP/s	~247

Consumer

GPU	memory	bandwidth	dense BF16	ridge point
RTX 4090	24 GB GDDR6X	1.01 TB/s	165 TFLOP/s	164
RTX 3090	24 GB GDDR6X	0.94 TB/s	71 TFLOP/s	76

Verify before you rely on these. Vendors quote compute under varying assumptions and revise specifications between silicon revisions. Blackwell and MI300X figures in particular should be checked against a current datasheet — they are given here as working estimates, not authoritative values. Bandwidth figures are the most stable and the most important.

The trend that explains the whole field

	A100 (2020)	H100 (2022)	H200 (2023)	B200 (2024)
bandwidth	2.04 TB/s	3.35 TB/s	4.8 TB/s	~8 TB/s
dense BF16	312 TF/s	989 TF/s	989 TF/s	~2250 TF/s
ridge point	153	295	206	~281

Between the A100 and the H100, compute grew 3.2× while bandwidth grew 1.6×. The ridge point nearly doubled, meaning it takes twice the batch size to keep the machine busy. This is the memory wall, and it has been widening for a decade — which is why an entire subfield exists to work around it.

The H200 is the interesting counterexample: identical compute to the H100 with 43% more bandwidth, which pushes the ridge back down to 206. It is a part designed for decode.

Memory hierarchy (H100, approximate)

The roofline above concerns HBM traffic. Inside the chip there are faster tiers, and exploiting them is what Level 07 is about.

level	capacity	bandwidth	latency
registers	256 KB per SM	~100 TB/s	~1 cycle
shared memory / L1	228 KB per SM	~20 TB/s	~30 cycles
L2 cache	50 MB	~10 TB/s	~200 cycles
HBM3	80 GB	3.35 TB/s	~500 cycles
NVLink to peer GPU	—	900 GB/s	~2 µs
PCIe Gen5 x16	—	~64 GB/s	~10 µs
host DRAM	TBs	~200 GB/s	~1 µs

Shared memory is roughly 6× the bandwidth of HBM and about 16× lower latency. FlashAttention is, in one sentence, the observation that if you can keep a tile of the computation in that tier you never pay for the N×N matrix at all.

Note also the NVLink-to-PCIe gap: 14×. Tensor parallelism does two all-reduces per layer, so running TP across a PCIe link rather than NVLink is usually a mistake.

Quick calculations

TPOT_floor  = weight_bytes / bandwidth
max_batch   = (memory - weights - workspace) / (kv_per_token * seq_len)
ridge       = dense_flops / bandwidth
saturated?  = max_batch > ridge          # almost always "no"

kv_per_token = 2 * n_layers * n_kv_heads * head_dim * dtype_bytes

  Llama-3-8B  fp16:  2 x 32 x 8 x 128 x 2 = 131,072 B = 128 KiB
  Llama-3-70B fp16:  2 x 80 x 8 x 128 x 2 = 327,680 B = 320 KiB