Find edge AI hardware
Task LLM serving (MLPerf datacenter) · model-size class LLM >40B · batch unknown · 93 platforms
Not like-for-like across groups: same model-size class and batch, but 11 different model/precision combinations. Compare within a group; each group is sorted by result.
MLPerf gpt-oss-120b · mxfp4 (5)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Crusoe Cloud 64x8 MI355X (8x AMD Instinct MI355X 288GB HBM3e) | 5749440 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b mxfp4 · vLLM 0.22.1 (ROCm 7.2.2), PyTorch 2.10 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| 9x8xMI355X_2xEPYC_9575F (8x AMD Instinct MI355X 288GB HBM3e) | 1039900 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b mxfp4 · vLLM 0.22.1.dev0, PyTorch 2.10.0, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| LLM-D v0.5.0,Openshift 4.20.12,NVIDIA 8xB200-SXM-180GB (8x NVIDIA B200-SXM-180GB) | 93071 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b mxfp4 · LLM-D v0.5.0 ,vLLM 0.14.1, RHEL 9.6 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Lambda 4x GB300-288GB (4x NVIDIA GB300) | 65512 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b mxfp4 · TensorRT-LLM 1.3.0rc14 (GPT-OSS-120B); vLLM (CentML) 0.20.2rc1.dev2091+g2e1040b7e + NVIDIA Dynamo (Qwen3-VL-235B); CUDA | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| LLM-D v0.5.0,OpenShift 4.21,IBM cloud gx3d-160x1792x8h200,NVIDIA 8xH200-SXM-141GB (8x NVIDIA H200-SXM-141GB) | 28680 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b mxfp4 · LLM-D v0.5.0 ,OpenShift 4.21, RHEL 9.6 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
MLPerf gpt-oss-120b · fp4 (24)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| CoreWeave GB300 NVL72 (4x NVIDIA GB300) | 1197710 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML:mlperf-inf-mm-q3vl-v6.0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Nebius GB300 NVL72 (4x NVIDIA GB300) | 1186750 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| AMD MI355X Custom Cluster (8x AMD Instinct MI355X 288GB HBM3e (x94)) | 1031070 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · PyTorch 2.9.1+git8907517, ROCm 7.0.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| CoreWeave GB200 NVL72 (4x NVIDIA GB200) | 911566 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| NVIDIA GB200 NVL72 (4x NVIDIA GB200) | 899218 tok/s throughput_server b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| smci355-ccs-aus-m13-13 (8x AMD Instinct MI355X 288GB HBM3e) | 121818 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · PyTorch 2.10.0+git8514f05, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| G4826 (8x AMD Instinct MI355X 288GB HBM3e) | 118042 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0+git8514f05, ROCm 7.2.53211 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Dell XE9785L_8xMI355X (8x AMD Instinct MI355X 288GB HBM3e) | 117804 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| HPE ProLiant Compute XD685 (8x AMD Instinct MI355X 288GB HBM3e) | 117355 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| AS-4126GS-NMR-LCC (8x AMD Instinct MI355X 288GB HBM3e) | 117005 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.4 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| 8xMI355X_2xEPYC_9575F (8x AMD Instinct MI355X 288GB HBM3e) | 115567 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| HPE Compute XD690 (8x NVIDIA B300-SXM-270GB) | 114569 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Dell XE9780 (8x NVIDIA B300-SXM-270GB) | 109257 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, Dell Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML: | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Lenovo ThinkSystem SR680a V4 (8x NVIDIA B300-SXM-270GB) | 109141 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Nebius B300 n1 (8x NVIDIA B300-SXM-270GB) | 108529 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| NVIDIA GB200 NVL72 by HPE (4x NVIDIA GB200) | 101453 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, TensorRT-LLM 1.3.0rc0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| G8825Z5 (8x AMD Instinct MI350X 288GB HBM3e) | 97020 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · Pytorch 2.10, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Nebius B200 n1 (8x NVIDIA B200-SXM-180GB) | 89856 tok/s throughput_server b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| AS-8126GS-TNMR,vLLM (8x AMD Instinct MI350X) | 64293 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · | B 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, open division, data-centre suite |
| OneW GB300 NVL72 (4x NVIDIA GB300) | 63991 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| XE7745_8xMI350P_2xEPYC_9455 (8x AMD Instinct MI350P) | 48971 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| NVIDIA DGX B200 (8x NVIDIA B200-SXM-180GB) | 16167 tok/s throughput_server b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1 | B 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, open division, data-centre suite |
| Nebius RTX PRO 6000 n1 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 15739 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Nebius GB300 NVL72 (1x NVIDIA GB300) | 15372 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, Driver 580.105.08, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
MLPerf llama2-70b-99 · fp4 (26)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| NVIDIA GB300 NVL72 (4x NVIDIA GB300) | 1126850 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| AMD MI355X Custom Cluster (8x AMD Instinct MI355X 288GB HBM3e (x87)) | 1042110 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · PyTorch 2.9.1+git8907517, ROCm 7.0.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Cisco UCS 16xMI350X (8x AMD Instinct MI350X 288GB HBM3e) | 159932 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · vLLM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB) | 115730 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.13, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| NVIDIA DGX B300 (8x NVIDIA B300-SXM-270GB) | 112954 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| G894-SD3-AAX7 (8x NVIDIA B300-SXM-270GB) | 109343 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Dell PowerEdge XE9780L (8x NVIDIA B300-SXM-270GB) | 108669 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| HPE ProLiant Compute XD685 (8x NVIDIA B200-SXM-180GB) | 103954 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.11, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| smci355-ccs-aus-m09-17 (8x AMD Instinct MI355X 288GB HBM3e) | 103480 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · PyTorch 2.9.1+git8907517, ROCm 7.0.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| SYS-822GS-NBRT (8x NVIDIA B200-SXM-180GB) | 101595 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| QuantaGrid D74A-7U (8x AMD Instinct MI355X 288GB HBM3e) | 100754 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TODO | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| G893-ZX1-AAX4 (8x AMD Instinct MI355X 288GB HBM3e) | 100310 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · PyTorch 2.9.1+git8907517, ROCm 7.0.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Inventec P9000AG7 (8x NVIDIA B200-SXM-180GB) | 100291 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.11, CUDA 12.9 | B 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, open division, data-centre suite |
| Dell PowerEdge XE9785L (8x AMD Instinct MI355X 288GB HBM3e) | 100008 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · vLLM 0.14.0rc3.dev30+gb026cf14e.d20260212.rocm700, Pytorch 2.9.1+git8907517, ROCm 7.0.0.70000-38~22.04 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| BM.GPU.MI355X.8 (8x AMD Instinct MI355X 288GB HBM3e) | 99874 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| AS-8126GS-TNMR,vLLM (8x AMD Instinct MI350X 288GB HBM3e) | 80478 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · vLLM 0.14.0, RHEL 9.6 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Cisco UCS 8xMI350X (8x AMD Instinct MI350X 288GB HBM3e) | 78937 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · vLLM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 36711 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14.1.48, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 29908 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14.1.48, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| ESC8000A_E13P_RTX_PRO_6000_PCIE_96GBx8 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 28039 tok/s throughput_server b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| ESC8000_E12P_RTX_PRO_6000_PCIE_96GBx8 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 27730 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Lenovo ThinkSystem SR675i V3 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 27160 tok/s throughput_server b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Dell PowerEdge XE7745 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 26830 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Cisco UCS X215c M8 - X580P PCIe Node (4x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 13530 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.11, CUDA 12.9 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 13362 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.14.1.48, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 6862 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp4 · TensorRT 10.11.0.33, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
MLPerf gpt-oss-120b · fp8 (6)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Heterogeneous Four-Cluster (8x AMD Instinct MI300X 192GB HBM3 (x2x8) and AMD Instinct MI355X 288GB HBM3e (x2x8)) | 285454 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · ROCm | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| MangoBoost 2x8 MI355X (8x AMD Instinct MI355X 288GB HBM3e) | 228488 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · ROCm 7.2.0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| PowerEdge XE9785L (8x AMD Instinct MI355X 288GB HBM3e) | 116953 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · ROCm | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| PowerEdge XE9785 (8x AMD Instinct MI355X 288GB HBM3e) | 115294 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · ROCm | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| SMC Lab OpenShift GB200-NVL4 4xGB200-186GB (4x NVIDIA GB200) | 61206 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · vLLM 0.24.0, Red Hat OpenShift 4.21 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| 8xMI300X_2xEPYC_9534 (8x AMD Instinct MI300X 192GB HBM3) | 30198 tok/s throughput_offline b=unknown | MLPerf gpt-oss-120b fp8 · vLLM 0.24.0 (stock vllm/vllm-openai-rocm:v0.24.0), ROCm 7.2.1 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
MLPerf llama2-70b-99.9 · fp4 (19)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Cisco_UCS_B300x16_G200_4x2_RO (8x NVIDIA B300-SXM-270GB) | 227570 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14.1.48; TensorRT-LLM 1.3.0rc0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| CoreWeave B300 SXM (8x NVIDIA B300-SXM-270GB) | 115530 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML:mlperf-inf-mm-q3vl-v6.0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| G894-AD3-AAX7 (8x NVIDIA B300-SXM-270GB) | 115034 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| G894-ZD3-AAX7 (8x NVIDIA B300-SXM-270GB) | 114826 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| QuantaGrid D75H-10U (8x NVIDIA B300-SXM-270GB) | 113985 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| XA NB3I-E12 (8x NVIDIA B300-SXM-270GB) | 113455 tok/s throughput_server b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Dell XE9785L (8x NVIDIA B300-SXM-270GB) | 113101 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| SYS-822GS-NB3RT (8x NVIDIA B300-SXM-270GB) | 111247 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| AS-8126GS-NB3RT (8x NVIDIA B300-SXM-270GB) | 110802 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Inventec P5800G7 (8x NVIDIA B300-SXM-270GB) | 109933 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Inventec P9000G7 (8x NVIDIA B300-SXM-270GB) | 109755 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| CoreWeave B200 SXM (8x NVIDIA B200-SXM-180GB) | 102703 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| MiTAC_G4520G6_RTX_PRO_6000_PCIE_96GBx8 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 30034 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.2 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Cisco UCS C845A M8 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 29904 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Dell PowerEdge XE7740 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 29797 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| PRIMERGY_CDI_RTX_PRO_6000_PCIE_96GBx8) (8x NVIDIA RTX PRO 6000 Blackwell) | 27341 tok/s throughput_server b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.16, CUDA 13.2, cuDNN 9.21 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Cisco UCS X210c M8 - X580P PCIe Node (4x NVIDIA RTX PRO 6000 Blackwell Server Edition) | 14861 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| B300-SXM-270GBx1_TRT (1x NVIDIA B300-SXM-270GB) | 14637 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
| Dell PowerEdge XE7745 (8x NVIDIA RTX PRO 4500 Blackwell) | 4374 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp4 · TensorRT 10.14, CUDA 13.3 | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
MLPerf llama2-70b-99 · fp8 (6)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Dell_MangoBoost Heterogeneous Cluster (8x AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8) and AMD Instinct MI355X 288GB HBM3e (x8)) | 151843 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · ROCm | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| PowerEdge XE9785 (8x AMD Instinct MI355X 288GB HBM3e (Power Cap 1000 W)) | 81871 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · ROCm | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| MangoBoost Mi325X (8x AMD Instinct MI325X 256GB HBM3e) | 33880 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · ROCm 6.10.5 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Dell PowerEdge XE7740 (8x NVIDIA H200-NVL-141GB) | 32004 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · TensorRT 10.13, CUDA 13.0 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| MangoBoost Mi300X (8x AMD Instinct MI300X 192GB HBM3) | 27599 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · ROCm 6.10.5 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| Dell PowerEdge XE9680 (8x AMD Instinct MI300X 192GB HBM3) | 27073 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 fp8 · vLLM 0.9.0.2.dev108+g71faa1880.d20260213, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1 | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
MLPerf llama2-70b-99.9 · nvfp4 (1)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Orrick Optimized 6x NVIDIA B200 L2 (6x NVIDIA B200-SXM-180GB) | 130928 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 nvfp4 · TensorRT-LLM 1.3.0rc20 | B 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, open division, data-centre suite |
MLPerf llama2-70b-99.9 · fp8,fp4 (1)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| Cisco UCS H200 x8 + MI350X x8 G200 2x2 TOR (8x NVIDIA H200-SXM-141GB + AMD Instinct MI350X 288GB HBM3e) | 99684 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp8,fp4 · vLLM | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
MLPerf llama2-70b-99.9 · fp16 (1)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| HPE Cray XD670 (8x NVIDIA H200-SXM-141GB) | 25940 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 fp16 · vLLM v0.15.0-dev | B 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, open division, data-centre suite |
MLPerf llama2-70b-99 · INT8 (3)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| 1-node-4x-BMG-Pro-B60-Dual (4x MS-Intel Arc Pro B60 Dual 48G Turbo) | 3271 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 INT8 · PyTorch vLLM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| 1-node-4x-BMG-B70 (4x Intel(R) Arc Pro(R) B70) | 2459 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 INT8 · PyTorch vLLM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
| 1-node-4x-BMG-B60 (4x Intel(R) Arc Pro(R) B60) | 1698 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99 INT8 · PyTorch vLLM | A 2026-04-01 (published) | unknown no power published | unknown | MLPerf Inference v6.0, closed division, data-centre suite |
MLPerf llama2-70b-99.9 · INT8 (1)
| Platform | Result | Model · precision · stack | Grade · date | Power | Price | Source |
|---|---|---|---|---|---|---|
| 1-node-4x-BMG-B70 (4x Intel(R) Arc(TM) Pro B70) | 2465 tok/s throughput_offline b=unknown | MLPerf llama2-70b-99.9 INT8 · PyTorch | A 2026-09-16 (published) | unknown no power published | unknown | MLPerf Inference v6.1, closed division, data-centre suite |
Confidence: A audited MLPerf closed, B documented third-party, C community, D vendor-published, E estimate. Unknown is never assumed. Other like-for-like groups: LLM >40B, batch not stated (93); LLM 5–14B, batch not stated (48); unknown, batch not stated (32)