LLM generation benchmarks on edge AI hardware
Each table compares only results with the same model-size class and batch size. Search this workload · JSON.
Fastest in the best-covered like-for-like group (LLM ≤4B, batch 1): Intel 265K Custom PC (Nvidia RTX 4090) (as tested by Jeff Geerling) at 335 tok/s C undated. 12 groups, 91 platforms.
Show data — 12 like-for-like tables
LLM ≤4B · batch 1 (64 platforms)
✓ Like-for-like: model-size class LLM ≤4B, batch 1. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Intel 265K Custom PC (Nvidia RTX 4090) (as tested by Jeff Geerling) | 335 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 262 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 13900K (Nvidia 5090) (as tested by Jeff Geerling) | 271 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 263 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 495 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core Ultra 7 368H | 250 tok/s 2nd-token throughput, input 32 tokens · b=1 | minicpm4-0.5b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM ≤4B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-368H.csv) undated |
| Ryzen 9 7900X (Nvidia 4090) (as tested by Jeff Geerling) | 237 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 4090) (as tested by Jeff Geerling) | 236 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 290 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 13900K (Nvidia 4090) (as tested by Jeff Geerling) | 216 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 209 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 394 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Ryzen 9 7950X (Nvidia 4080) (as tested by Jeff Geerling) | 204 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Ryzen 9 7950X (Nvidia 4070 Ti Super) (as tested by Jeff Geerling) | 199 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 196 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 267 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (AMD Radeon AI Pro R9700) (as tested by Jeff Geerling) | 196 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 334 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 2080 Ti) (as tested by Jeff Geerling) | 180 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 258 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 166 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 228 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Ryzen 9 5950X (Nvidia 4070) (as tested by Jeff Geerling) | 161 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 2080 Ti) (as tested by Jeff Geerling) | 155 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 258 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Mac Studio (M3 Ultra 512GB) (as tested by Jeff Geerling) | 155 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 223 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX A4000) (as tested by Jeff Geerling) | 149 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 269 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core 5 330 | 137 tok/s 2nd-token throughput, input 32 tokens · b=1 | minicpm4-0.5b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM ≤4B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_5-330.csv) undated |
| Intel Core Ultra 7 258V (Lunar Lake) | 135 tok/s 2nd-token throughput, input 32 tokens · b=1 | minicpm4-0.5b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM ≤4B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-258V.csv) undated |
| Pi CM5 - 16GB (Nvidia RTX A4000) (as tested by Jeff Geerling) | 134 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 163 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Ryzen 9 9950X (AMD 7900 XT) (as tested by Jeff Geerling) | 131 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| M1 Ultra (64 GPU Core) 128GB (as tested by Jeff Geerling) | 128 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 3060) (as tested by Jeff Geerling) | 123 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 214 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson AGX Thor | 122 tok/s generation_throughput · b=1 | Qwen3-VL-2B NVFP4 · vLLM · LLM ≤4B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Pi CM5 - 16GB (Nvidia RTX 3060) (as tested by Jeff Geerling) | 113 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 192 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core Ultra 7 255H | 110 tok/s 2nd-token throughput, input 32 tokens · b=1 | minicpm4-0.5b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM ≤4B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-255H.csv) undated |
| M1 Ultra (48 GPU Core) 64GB (as tested by Jeff Geerling) | 109 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 500+ - 16GB (AMD RX 7900 XT) (as tested by Jeff Geerling) | 109 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 315 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Dell Pro Max with GB10 | 97.93 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 129 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson Thor T4000 | 95.00 tok/s generation_throughput · b=1 | Qwen3-VL-2B NVFP4 · vLLM · LLM ≤4B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| System76 Thelio Astra (Nvidia A4000) (as tested by Jeff Geerling) | 90.92 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 244 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 500+ - 16GB (AMD RX 9070 XT) (as tested by Jeff Geerling) | 89.63 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 304 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| System76 Thelio Astra (AMD Pro W7700) (as tested by Jeff Geerling) | 89.31 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 261 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) | 88.14 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 133 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (AMD AI Pro R9700) (as tested by Jeff Geerling) | 77.54 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 281 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Minisforum MS-R1 (Nvidia RTX A2000) (as tested by Jeff Geerling) | 71.36 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 94.30 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson AGX Orin 64GB | 60.00 tok/s generation_throughput · b=1 | Qwen3-VL-2B W4A16 · vLLM · LLM ≤4B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| M1 Max Mac Studio (10 core - 64GB) (as tested by Jeff Geerling) | 59.38 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 5 - 8GB (AMD Pro W7700) (as tested by Jeff Geerling) | 56.14 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 145 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 5 - 8GB (AMD RX 6700 XT) 12GB (as tested by Jeff Geerling) | 49.01 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 94.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 5 - 8GB (AMD RX 7600) (as tested by Jeff Geerling) | 48.47 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 156 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 500+ - 16GB (Intel Arc B580) (as tested by Jeff Geerling) | 47.38 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 146 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M4 (10-core GPU, 120 GB/s) | 41.31 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 30.10 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 5 - 8GB (AMD RX 6500 XT) (as tested by Jeff Geerling) | 39.82 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 88.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| HiFive Premier P550 (AMD RX 580) (as tested by Jeff Geerling) | 36.23 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 150 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| System76 Thelio Astra (Nvidia A400) (as tested by Jeff Geerling) | 35.51 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 167 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 500+ - 16GB (Intel Arc Pro B50) (as tested by Jeff Geerling) | 29.80 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 78.50 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson Orin NX 16GB | 25.00 tok/s generation_throughput · b=1 | Cosmos Reasoning 2 2B W4A16 · vLLM · LLM ≤4B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Dell XPS 13 (Intel Core 5 320) (as tested by Jeff Geerling) | 23.88 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 14.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Framework 13 (Ryzen AI 5 340) (as tested by Jeff Geerling) CPU-only | 23.81 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 51.10 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| AmpereOne A192-32X (512GB) (as tested by Jeff Geerling) CPU-only | 23.52 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson Orin Nano 8GB | 18.00 tok/s generation_throughput · b=1 | Nemotron3 Nano 4B Q4_K_M GGUF · llama.cpp · LLM ≤4B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Pi 500+ - 16GB (Intel Arc A310 ECO) (as tested by Jeff Geerling) | 13.36 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM ≤4B | C | 50.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| MetaComputing AI PC Framework 13 Mainboard (as tested by Jeff Geerling) CPU-only | 13.10 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 29.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Radxa Dragon Q8B (as tested by Jeff Geerling) CPU-only | 12.30 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 24.30 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Minisforum MS-R1 (as tested by Jeff Geerling) CPU-only | 12.12 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 35.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| GMKtec G3 Plus (Intel N150) - 16GB (as tested by Jeff Geerling) CPU-only | 9.06 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 26.40 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 500+ - 16GB (as tested by Jeff Geerling) CPU-only | 5.55 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 13.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Raspberry Pi 5 (CPU only) CPU-only | 4.88 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 11.90 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 400 - 4GB (as tested by Jeff Geerling) CPU-only | 1.60 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 6.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Dell Optiplex 780 (C2Q Q8400) (as tested by Jeff Geerling) CPU-only | 1.09 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 146 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| DC-ROMA Mainboard II (8-core RISC-V) (as tested by Jeff Geerling) CPU-only | 0.310 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 30.60 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| HiFive Premier P550 (4-core RISC-V) (as tested by Jeff Geerling) CPU-only | 0.240 tok/s generation_throughput · b=1 | Llama 3.2:3b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM ≤4B | C | 13.50 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
LLM 5–14B · batch 1 (53 platforms)
✓ Like-for-like: model-size class LLM 5–14B, batch 1. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Apple M5 Ultra (80-core GPU, 1228 GB/s) | 179 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit c1d0e7a) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Apple M5 Max (40-core GPU, 614 GB/s) | 120 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit c1d0e7a) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Apple M3 Ultra iGPU | 116 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA off, commit 2d451c8) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Intel 265K Custom PC (Nvidia RTX 4090) (as tested by Jeff Geerling) | 92.49 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 464 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core Ultra 7 258V (Lunar Lake) | 89.29 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-8b-a1b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 5–14B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-258V.csv) undated |
| Intel Core Ultra 7 368H | 83.33 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-8b-a1b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 5–14B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-368H.csv) undated |
| Pi CM5 - 16GB (Nvidia RTX 4090) (as tested by Jeff Geerling) | 83.31 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 398 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M4 Max (40-core GPU, 546 GB/s) | 83.06 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 8e672ef) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Apple M4 Max iGPU | 75.02 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA off, commit 1ece0cb6) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Intel 265K Custom PC (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 70.74 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 495 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M4 Max (32-core GPU, 410 GB/s) | 69.95 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 8e672ef) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Pi CM5 - 16GB (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 67.74 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 476 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M5 Pro (16-core GPU, 307 GB/s) | 67.59 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 60081bb) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Apple M5 Pro (20-core GPU, 307 GB/s) | 66.33 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit c1d0e7a) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Radxa Orion O6 - 16GB (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 64.58 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 465 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA DGX Spark | 60.39 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA on, commit 657e011) · LLM 5–14B | C | 153 W spec platform spec envelope (not measured with this result) | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| AMD Ryzen AI Max+ (Strix Halo) iGPU | 54.38 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA on, commit fb4b273) · LLM 5–14B | C | 120 W spec platform spec envelope (not measured with this result) | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Intel 265K Custom PC (AMD Radeon AI Pro R9700) (as tested by Jeff Geerling) | 53.72 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 331 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Mac Studio (M3 Ultra 512GB) (as tested by Jeff Geerling) | 51.85 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 227 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M4 Pro (20-core GPU, 273 GB/s) | 50.74 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 8e672ef) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Intel 265K Custom PC (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 49.74 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 311 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Apple M4 Pro (16-core GPU, 273 GB/s) | 49.64 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 8e672ef) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| Pi CM5 - 16GB (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 48.24 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 253 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 2080 Ti) (as tested by Jeff Geerling) | 45.06 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 308 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson AGX Thor | 45.00 tok/s tok/s gen, concurrency 1 · b=1 | Qwen2.5-VL 7B n/s on page · NVIDIA (vLLM/TensorRT-LLM per page) · LLM 5–14B | D | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson Benchmarks undated |
| M1 Ultra (64 GPU Core) 128GB (as tested by Jeff Geerling) | 41.51 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 2080 Ti) (as tested by Jeff Geerling) | 41.23 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 275 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core 5 330 | 40.82 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-8b-a1b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 5–14B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_5-330.csv) undated |
| Pi CM5 - 16GB (Nvidia RTX A4000) (as tested by Jeff Geerling) | 36.21 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 163 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| M1 Ultra (48 GPU Core) 64GB (as tested by Jeff Geerling) | 35.89 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson Thor T4000 | 35.00 tok/s generation_throughput · b=1 | Qwen 3 8B NVFP4 · vLLM · LLM 5–14B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Intel Core Ultra 7 255H | 31.35 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-8b-a1b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 5–14B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-255H.csv) undated |
| NVIDIA Jetson AGX Orin 64GB | 30.00 tok/s generation_throughput · b=1 | Qwen 3 8B W4A16 · vLLM · LLM 5–14B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Intel 265K Custom PC (Nvidia RTX 3060) (as tested by Jeff Geerling) | 29.77 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 224 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 3060) (as tested by Jeff Geerling) | 29.40 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 194 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core Ultra 200 Series iGPU | 25.12 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA on, commit 4a89937) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Intel Core Ultra 300 Series iGPU | 24.45 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA off, commit 481c65f) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Apple M4 (10-core GPU, 120 GB/s) | 24.11 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Metal (commit 8e672ef) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #4167 scoreboard (via GitHub API) 2026-10-01 |
| AMD Ryzen AI 300 Series iGPU | 23.93 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA on, commit N/A) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Dell Pro Max with GB10 | 23.92 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 138 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi 5 - 16GB (AMD Pro W7700) (as tested by Jeff Geerling) | 19.90 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 164 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| AMD Ryzen AI 400 Series iGPU | 19.03 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA on, commit cb26014) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) CPU-only | 11.37 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 140 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson Orin NX 16GB | 11.00 tok/s generation_throughput · b=1 | Nemotron Nano 9B V2 W4A16 · vLLM · LLM 5–14B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| MediaTek Dimensity 9400 iGPU | 8.92 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA off, commit b9ab0a4) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Intel Core Ultra 100 Series iGPU | 8.21 tok/s tg128 · b=1 | Llama 2 7B Q4_0 · llama.cpp Vulkan (FA off, commit 1d72c84) · LLM 5–14B | C | unknown power n/a no power published | llama.cpp discussion #10879 Vulkan scoreboard (via GitHub API) 2026-10-10 |
| Pi CM5 - 16GB (AMD AI Pro R9700) (as tested by Jeff Geerling) | 7.98 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 5–14B | C | 280 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Framework 13 (Ryzen AI 5 340 16GB) (as tested by Jeff Geerling) CPU-only | 5.83 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 50.40 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Radxa Orion O6 - 16GB (as tested by Jeff Geerling) CPU-only | 4.33 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 34.70 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Minisforum MS-R1 (as tested by Jeff Geerling) CPU-only | 3.39 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 38.40 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| MetaComputing AI PC Framework 13 Mainboard (as tested by Jeff Geerling) CPU-only | 3.25 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 32.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| GMKtek G3 Plus (Intel N150) - 16GB (as tested by Jeff Geerling) CPU-only | 2.13 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 30.30 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Raspberry Pi 5 (CPU only) CPU-only | 1.20 tok/s generation_throughput · b=1 | DeepSeek R1 14b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 5–14B | C | 13.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
LLM 15–40B · batch 1 (22 platforms)
✓ Like-for-like: model-size class LLM 15–40B, batch 1. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Intel 265K Custom PC (Nvidia RTX 4090) (as tested by Jeff Geerling) | 280 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 509 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 4090) (as tested by Jeff Geerling) | 189 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 322 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (AMD Radeon AI Pro R9700) (as tested by Jeff Geerling) | 163 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 333 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 163 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 299 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 3080 Ti) (as tested by Jeff Geerling) | 144 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 434 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 4070 Ti) (as tested by Jeff Geerling) | 133 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 235 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Mac Studio (M3 Ultra 512GB) (as tested by Jeff Geerling) | 115 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 227 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| M1 Ultra (64 GPU Core) 128GB (as tested by Jeff Geerling) | 100 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX A4000) (as tested by Jeff Geerling) | 93.55 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 199 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX A4000) (as tested by Jeff Geerling) | 90.68 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 163 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Dell Pro Max with GB10 | 87.97 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 123 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel 265K Custom PC (Nvidia RTX 3060) (as tested by Jeff Geerling) | 83.96 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 229 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Pi CM5 - 16GB (Nvidia RTX 3060) (as tested by Jeff Geerling) | 79.14 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 181 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson AGX Thor | 72.60 tok/s generation_throughput · b=1 | Qwen3-30B-A3B INT4 GPTQ · Edge-LLM · LLM 15–40B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Thor T4000 | 57.00 tok/s generation_throughput · b=1 | Qwen3-30B-A3B NVFP4 · vLLM · LLM 15–40B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 55.00 tok/s generation_throughput · b=1 | Qwen3-30B-A3B W4A16 · vLLM · LLM 15–40B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Intel Core Ultra 7 258V (Lunar Lake) | 53.48 tok/s 2nd-token throughput, input 1024 tokens · b=1 | lfm2-24b-a2b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 15–40B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-258V.csv) undated |
| Intel Core Ultra 7 368H | 52.36 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-24b-a2b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 15–40B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-368H.csv) undated |
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) | 47.50 tok/s generation_throughput · b=1 | Qwen3-Coder 30B (MoE, 3.3B active) not stated per model · Ollama (OpenAI-compatible endpoint) · LLM 15–40B | C | 120 W spec platform spec envelope (not measured with this result) | Adam Taylor (Adiuvo Engineering) (manual extraction) undated |
| Pi CM5 - 16GB (AMD Radeon AI Pro R9700) (as tested by Jeff Geerling) | 19.59 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · llama.cpp (Vulkan) · LLM 15–40B | C | 318 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Intel Core Ultra 7 255H | 16.89 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2-24b-a2b INT4-MIXED · OpenVINO 2026.4.1 GenAI · LLM 15–40B | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-255H.csv) undated |
| MetaComputing AI PC Framework 13 Mainboard (as tested by Jeff Geerling) CPU-only | 8.52 tok/s generation_throughput · b=1 | gpt-oss 20b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM 15–40B | C | 27.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
LLM >40B · batch 1 (11 platforms)
✓ Like-for-like: model-size class LLM >40B, batch 1. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) | 47.80 tok/s generation_throughput · b=1 | Qwen3.8-Flash-Next 125B (MoE, 6B active) not stated · gfx1151 native engine with speculative decoding (not Ollama) · LLM >40B | C | 120 W spec platform spec envelope (not measured with this result) | Adam Taylor (Adiuvo Engineering) (manual extraction) undated |
| Mac Studio (M3 Ultra 512GB) (as tested by Jeff Geerling) | 19.89 tok/s generation_throughput · b=1 | DeepSeek R1 671b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | 262 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| NVIDIA Jetson AGX Thor | 15.00 tok/s generation_throughput · b=1 | Nemotron 3 Super 120B-A12B NVFP4 · vLLM · LLM >40B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| M1 Ultra (64 GPU Core) 128GB (as tested by Jeff Geerling) | 9.84 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| M1 Max Mac Studio (10 core - 64GB) (as tested by Jeff Geerling) | 7.25 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Dell Pro Max with GB10 | 4.71 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | 156 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| AmpereOne A192-32X (512GB) (as tested by Jeff Geerling) CPU-only | 4.18 tok/s generation_throughput · b=1 | DeepSeek R1 671b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | 477 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Ryzen 9 7900X (Nvidia 4090) (as tested by Jeff Geerling) | 3.10 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | D | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Raspberry Pi CM5 Cluster (10x 16GB) (as tested by Jeff Geerling) CPU-only | 0.850 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | 70.00 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Minisforum MS-R1 (as tested by Jeff Geerling) CPU-only | 0.770 tok/s generation_throughput · b=1 | Llama 3.1:70b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | 38.20 W measured reported in result source — measured | Jeff Geerling ai-benchmarks README (Findings tables) undated |
| Framework Mainboard Cluster (512GB) (as tested by Jeff Geerling) | 0.710 tok/s generation_throughput · b=1 | Llama 3.1:405b as in linked issue (Ollama default tags are 4-bit) · Ollama / llama.cpp (see linked issue) · LLM >40B | C | unknown power n/a no power published | Jeff Geerling ai-benchmarks README (Findings tables) undated |
LLM 5–14B · batch 8 (4 platforms)
✓ Like-for-like: model-size class LLM 5–14B, batch 8. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| NVIDIA Jetson AGX Thor | 305 tok/s tok/s gen, concurrency 8 · b=8 | Deepseek R1 7B n/s on page · NVIDIA (vLLM/TensorRT-LLM per page) · LLM 5–14B | D | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson Benchmarks undated |
| NVIDIA Jetson Thor T4000 | 200 tok/s generation_throughput · b=8 | Qwen 3 8B NVFP4 · vLLM · LLM 5–14B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 157 tok/s generation_throughput · b=8 | Qwen 3 8B W4A16 · vLLM · LLM 5–14B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Orin NX 16GB | 60.00 tok/s generation_throughput · b=8 | Nemotron Nano 9B V2 W4A16 · vLLM · LLM 5–14B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
LLM ≤4B · batch 8 (4 platforms)
✓ Like-for-like: model-size class LLM ≤4B, batch 8. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| NVIDIA Jetson AGX Thor | 523 tok/s generation_throughput · b=8 | Gemma 4 E2B NVFP4 · vLLM · LLM ≤4B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Thor T4000 | 423 tok/s generation_throughput · b=8 | Qwen3-VL-2B NVFP4 · vLLM · LLM ≤4B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 273 tok/s generation_throughput · b=8 | Cosmos Reasoning 2 2B W4A16 · vLLM · LLM ≤4B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Orin NX 16GB | 65.00 tok/s generation_throughput · b=8 | Cosmos Reasoning 2 2B W4A16 · vLLM · LLM ≤4B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
LLM 15–40B · batch 8 (3 platforms)
✓ Like-for-like: model-size class LLM 15–40B, batch 8. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| NVIDIA Jetson AGX Thor | 242 tok/s generation_throughput · b=8 | Qwen3-30B-A3B NVFP4 · vLLM · LLM 15–40B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Thor T4000 | 184 tok/s generation_throughput · b=8 | Qwen3-30B-A3B NVFP4 · vLLM · LLM 15–40B | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 174 tok/s generation_throughput · b=8 | Qwen3-30B-A3B W4A16 · vLLM · LLM 15–40B | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
LLM >40B · batch 8 (1 platforms)
✓ Like-for-like: model-size class LLM >40B, batch 8. Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| NVIDIA Jetson AGX Thor | 40.00 tok/s generation_throughput · b=8 | Nemotron 3 Super 120B-A12B NVFP4 · vLLM · LLM >40B | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
LLM >40B · batch unknown (4 platforms)
✓ Like-for-like: model-size class LLM >40B, batch unknown (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) | 62.09 tok/s tg128 · b=unknown | Qwen3-Next 80B-A3B UD-Q4_K_XL · llama.cpp Vulkan RADV b10687 · LLM >40B | C | 120 W spec platform spec envelope (not measured with this result) | StrixHaloGuide evidence (llama-bench, 20 repeats) 2026-10 |
| Apple Mac mini M4 Pro 64GB | 5.48 tok/s tg · b=unknown | 70B-class LLM (Q4) ~Q4 · as reviewed (Ollama/llama.cpp class) · LLM >40B | C | 80.00 W measured reported in result source — measured | vminstall.com review 2026 (year only) |
| NVIDIA DGX Spark | 4.56 tok/s tg · b=unknown | 70B-class LLM (Q4) ~Q4 · as reviewed (Ollama/llama.cpp class) · LLM >40B | C | 153 W measured reported in result source — measured | vminstall.com review 2026 (year only) |
| NVIDIA Jetson AGX Thor | 4.40 tok/s tg · b=unknown | 70B-class LLM (Q4) ~Q4 · as reviewed (Ollama/llama.cpp class) · LLM >40B | C | 100 W measured reported in result source — measured | vminstall.com review 2026 (year only) |
LLM 15–40B · batch unknown (1 platforms)
✓ Like-for-like: model-size class LLM 15–40B, batch unknown (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| AMD Ryzen AI Max+ 395 / PRO 395 (Strix Halo, Radeon 8060S) | 94.64 tok/s tg128 · b=unknown | Qwen3-Coder 30B-A3B UD-Q4_K_XL · llama.cpp Vulkan RADV b10687 · LLM 15–40B | C | 120 W spec platform spec envelope (not measured with this result) | StrixHaloGuide evidence (llama-bench, 20 repeats) 2026-10 |
unknown · batch 1 (8 platforms)
✓ Like-for-like: model-size class unknown, batch 1 (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Intel Core Ultra 7 368H | 227 tok/s 2nd-token throughput, input 1024 tokens · b=1 | gemma-3-270m INT4-MIXED · OpenVINO 2026.4.1 GenAI · unknown | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-368H.csv) undated |
| Intel Core Ultra 7 258V (Lunar Lake) | 200 tok/s 2nd-token throughput, input 32 tokens · b=1 | lfm2.5-350m INT8-CW · OpenVINO 2026.4.1 GenAI · unknown | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-258V.csv) undated |
| Intel Core 5 330 | 127 tok/s 2nd-token throughput, input 1024 tokens · b=1 | gemma-3-270m INT4-MIXED · OpenVINO 2026.4.1 GenAI · unknown | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_5-330.csv) undated |
| NVIDIA Jetson AGX Thor | 117 tok/s generation_throughput · b=1 | Pi 0.5 NVFP4 + FP8 · TRT · unknown | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| Intel Core Ultra 7 255H | 115 tok/s 2nd-token throughput, input 32 tokens · b=1 | distil-large-v2 INT4-MIXED · OpenVINO 2026.4.1 GenAI · unknown | D | unknown power n/a no power published | OpenVINO official LLM benchmarks (llm_models_7-255H.csv) undated |
| NVIDIA Jetson Thor T4000 | 94.00 tok/s generation_throughput · b=1 | Pi 0.5 NVFP4 + FP8 · TRT · unknown | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA DGX Spark | 59.60 tok/s generation_throughput · b=1 | Nemotron 3 Nano Omni NVFP4 · vLLM · unknown | C | 153 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 44.00 tok/s generation_throughput · b=1 | Cosmos3 Edge (Reasoner) BF16 · vLLM · unknown | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
unknown · batch 8 (4 platforms)
✓ Like-for-like: model-size class unknown, batch 8 (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| NVIDIA Jetson AGX Thor | 293 tok/s generation_throughput · b=8 | Cosmos3 Edge (Reasoner) BF16 · vLLM · unknown | C | 130 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson Thor T4000 | 284 tok/s generation_throughput · b=8 | Cosmos3 Edge (Reasoner) BF16 · vLLM · unknown | C | unknown power n/a no power published | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA DGX Spark | 191 tok/s generation_throughput · b=8 | Nemotron 3 Nano Omni NVFP4 · vLLM · unknown | C | 153 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
| NVIDIA Jetson AGX Orin 64GB | 171 tok/s generation_throughput · b=8 | Cosmos3 Edge (Reasoner) BF16 · vLLM · unknown | C | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson AI Lab benchmarks (benchmarks.json) undated |
How to read these numbers
Every number is labelled. Confidence: A MLPerf closed, audited · B documented third-party method · C community / review · D vendor-published, not independently verified · E estimate. Power basis: measured with the result · spec vendor spec/envelope · power n/a not published.
Machine-readable: /find.json (see llms.txt); full tables are in “Show data”. Confidence grades: A audited MLPerf closed · B documented third-party · C community · D vendor-published · E estimate. Unknown values are shown as unknown, never estimated. Generated 2026-10-16 from public sources.