HARDWARE TABLE

How to read this table

Two columns matter more than the rest.

Bandwidth sets your decode speed. Time per token at batch 1 is approximately weight_bytes / bandwidth, and nothing else in the spec sheet enters that calculation.

Ridge point is dense FLOP/s ÷ bandwidth — the arithmetic intensity at which the machine stops being memory-bound. Since decode intensity equals your batch size, the ridge point is the batch size you would need to saturate the machine's arithmetic. A lower ridge is easier to saturate, and therefore better for decode.

All compute figures below are dense BF16. Vendors usually headline the 2:4 structured sparsity number, which is double and does not apply to production LLMs.

Datacenter GPUs

GPU	memory	bandwidth	dense BF16	dense FP8	ridge point
A100 40GB SXM	40 GB HBM2e	1.56 TB/s	312 TFLOP/s	—	200
A100 80GB SXM	80 GB HBM2e	2.04 TB/s	312 TFLOP/s	—	153
H100 SXM	80 GB HBM3	3.35 TB/s	989 TFLOP/s	1979 TFLOP/s	295
H100 PCIe	80 GB HBM3	2.0 TB/s	756 TFLOP/s	1513 TFLOP/s	378
H200 SXM	141 GB HBM3e	4.8 TB/s	989 TFLOP/s	1979 TFLOP/s	206
B200	192 GB HBM3e	~8 TB/s	~2.25 PFLOP/s	~4.5 PFLOP/s	~281
L40S	48 GB GDDR6	0.86 TB/s	362 TFLOP/s	733 TFLOP/s	419
AMD MI300X	192 GB HBM3	5.3 TB/s	~1.31 PFLOP/s	~2.61 PFLOP/s	~247

Consumer

GPU	memory	bandwidth	dense BF16	ridge point
RTX 4090	24 GB GDDR6X	1.01 TB/s	165 TFLOP/s	164
RTX 3090	24 GB GDDR6X	0.94 TB/s	71 TFLOP/s	76

Verify before you rely on these. Vendors quote compute under varying assumptions and revise specifications between silicon revisions. Blackwell and MI300X figures in particular should be checked against a current datasheet — they are given here as working estimates, not authoritative values. Bandwidth figures are the most stable and the most important.

Google TPUs

Different chip, same wall. A TPU's compute core is a 128x128 systolic array rather than a set of SIMT cores with tensor units, but it is attached to the same generation of HBM — so the ridge point means exactly what it means above, and decode is bounded by bandwidth on both. Level 13 works through the internals; these are the numbers it uses.

TPU	memory	bandwidth	dense bf16	pod	ridge point
v4	32 GB HBM2	1.2 TB/s	275 TFLOP/s	4096, 3D torus + OCS	229
v5e	16 GB HBM2e	0.82 TB/s	197 TFLOP/s	256, 2D torus	241
v5p	95 GB HBM2e	2.77 TB/s	459 TFLOP/s	8960, 3D torus	166
v6e	32 GB HBM	~1.64 TB/s	~918 TFLOP/s	256	~560

The v5p is the best-balanced part in either table: it reaches its own arithmetic ceiling at a batch of 166, where an H100 needs 295. That is a statement about balance, not speed — the H100's ceiling is 2.2x higher. The v6e is the opposite lesson and the same one the A100-to-H100 jump teaches: compute grew 4.7x over the v5e while bandwidth doubled, so the ridge point more than doubled with it.

Interconnect is where the two families differ most. A GPU's fast fabric is an island — ~900 GB/s to every peer inside an NVLink domain of 8 (72 on an NVL72 rack), then a cliff to ~50 GB/s per GPU on InfiniBand. A TPU's ICI links run chip-to-chip in a torus at roughly 100 GB/s per link, six links per chip on a 3D-torus part, and that figure does not change whether the pod holds 64 chips or 8,960. No cliff, but a diameter: opposite corners of a large torus are many hops apart.

Same caveat, more so. TPU specifications are published less completely than GPU ones and some figures here (particularly v6e) are working estimates. The derivations in Level 13 are the defence: if a number is wrong, the geometry check — peak FLOPs against MXU count, array size and a plausible clock — will usually say so.

The trend that explains the whole field

	A100 (2020)	H100 (2022)	H200 (2023)	B200 (2024)
bandwidth	2.04 TB/s	3.35 TB/s	4.8 TB/s	~8 TB/s
dense BF16	312 TF/s	989 TF/s	989 TF/s	~2250 TF/s
ridge point	153	295	206	~281

Between the A100 and the H100, compute grew 3.2× while bandwidth grew 1.6×. The ridge point nearly doubled, meaning it takes twice the batch size to keep the machine busy. This is the memory wall, and it has been widening for a decade — which is why an entire subfield exists to work around it.

The H200 is the interesting counterexample: identical compute to the H100 with 43% more bandwidth, which pushes the ridge back down to 206. It is a part designed for decode.

Memory hierarchy (H100, approximate)

The roofline above concerns HBM traffic. Inside the chip there are faster tiers, and exploiting them is what Level 07 is about.

level	capacity	bandwidth	latency
registers	256 KB per SM	~100 TB/s	~1 cycle
shared memory / L1	228 KB per SM	~20 TB/s	~30 cycles
L2 cache	50 MB	~10 TB/s	~200 cycles
HBM3	80 GB	3.35 TB/s	~500 cycles
NVLink to peer GPU	—	900 GB/s	~2 µs
PCIe Gen5 x16	—	~64 GB/s	~10 µs
host DRAM	TBs	~200 GB/s	~1 µs

Shared memory is roughly 6× the bandwidth of HBM and about 16× lower latency. FlashAttention is, in one sentence, the observation that if you can keep a tile of the computation in that tier you never pay for the N×N matrix at all.

Note also the NVLink-to-PCIe gap: 14×. Tensor parallelism does two all-reduces per layer, so running TP across a PCIe link rather than NVLink is usually a mistake.

Quick calculations

TPOT_floor  = weight_bytes / bandwidth
max_batch   = (memory - weights - workspace) / (kv_per_token * seq_len)
ridge       = dense_flops / bandwidth
saturated?  = max_batch > ridge          # almost always "no"

kv_per_token = 2 * n_layers * n_kv_heads * head_dim * dtype_bytes

  Llama-3-8B  fp16:  2 x 32 x 8 x 128 x 2 = 131,072 B = 128 KiB
  Llama-3-70B fp16:  2 x 80 x 8 x 128 x 2 = 327,680 B = 320 KiB