Benchmarks and sources
Large model (default ranking)
gpt-oss-120b MXFP4 (59 GiB weights), text generation tok/s at empty context, single request. Needs about 59 GiB for weights plus context and OS, so devices with 64GB or less are marked 'doesn't fit'.
| Chip | tok/s | Engine | Date | Source |
|---|---|---|---|---|
| NVIDIA GB10 (DGX Spark) | 58.72 | llama.cpp CUDA, llama-bench tg32, b7946 | 2026-02-05 | source |
| Ryzen AI Max+ 395, 128GB | 52.74 | llama.cpp Vulkan, llama-bench tg32 (comment in thread) | 2025-10 | source |
| Apple M5 Max 40-core, 128GB (MacBook Pro) | 79.1 | Ollama 0.34.1 (not llama-bench) | 2026-09-16 | source |
| Apple M5 Ultra 80-core | 130 | LM Studio, spoken in video (not llama-bench) | 2026 | source |
Small model
llama.cpp llama-bench, Llama-2 7B Q4_0, tg128 (text generation tok/s); best of FA on/off as posted.
| Chip | tok/s | Backend | Posted | Source |
|---|---|---|---|---|
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | 50.95 | ROCm | 2026-02-02 | llama.cpp thread |
| NVIDIA GB10 (DGX Spark) | 57.21 | CUDA | 2025-10-15 | llama.cpp thread |
| Apple M5 Ultra 36/80 (256GB) | 179.1 | Metal | 2026-09-30 | llama.cpp thread |
| Apple M5 Max 40-core GPU (MacBook Pro 16, 128GB) | 119.08 | Metal | 2026-06-22 | llama.cpp thread |
| Apple M5 Pro (Mac mini, 64GB) | 67.59 | Metal | 2026-09-24 | llama.cpp thread |
| NVIDIA RTX 5090 | 314.0 | CUDA | 2025-12-04 | llama.cpp thread |
| NVIDIA RTX 4090 | 191.57 | CUDA | 2025-08-24 | llama.cpp thread |
| NVIDIA RTX 3090 | 172.26 | CUDA | 2025-08-01 | llama.cpp thread |
llm-speed.com community runs
Data from llm-speed.com, licensed CC BY 4.0. Changes: Aggregated to median top_decode_tps per accelerator; mixed models, runtimes and workloads. Imported 2026-10-16.
| Accelerator | Runs | Median decode tok/s | Link |
|---|---|---|---|
| RTX 5090 | 165 | 71.3 | example run |
| M3 Ultra (60-core GPU) | 21 | 80.3 | example run |
| RTX 4090 | 14 | 131.1 | example run |
| M3 Pro (18-core GPU) | 11 | 19.4 | example run |
| M4 Max (40-core GPU) | 5 | 109.1 | example run |
| RTX 3090 | 4 | 137.7 | example run |