language models (NLP) benchmarks on edge AI hardware
141 published rows. Each table compares only results with the same model-size class and batch size. Search this workload · JSON.
Fastest in the best-covered like-for-like group (unknown, batch 1): Rockchip RK3588 (NPU, single core of 3) at 987 FPS C 2026-10-13 (page last edited). 2 groups, 32 platforms.
Show data — 2 like-for-like tables
unknown · batch 1 (8 platforms)
✓ Like-for-like: model-size class unknown, batch 1 (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Rockchip RK3588 (NPU, single core of 3) | 987 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3576 (NPU, single core of 2) | 981 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3562 (NPU) | 858 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1126B (NPU) | 853 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3566/RK3568 (NPU) | 386 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK1808 (NPU) | 110 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1126 (NPU) | 65.80 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1109 (NPU) | 48.00 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
unknown · batch unknown (24 platforms)
✓ Like-for-like: model-size class unknown, batch unknown (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Intel Xeon Platinum 6972P CPU-only | 7863 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8580 CPU-only | 4497 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8480+ CPU-only | 2894 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Lenovo ThinkEdge SE455i V3 (2x NVIDIA L4-PCIe-24GB) | 1521 FPS throughput_offline · b=unknown | MLPerf bert-99 int8 · TensorRT 10.8, CUDA 13.1 · unknown | A | unknown power n/a no power published | MLPerf Inference v6.0, closed division, edge suite 2026-04-01 (published) |
| Intel Xeon Platinum 8380 CPU-only | 879 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Arc B-Series Graphics | 757 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Gold 6338N CPU-only | 623 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8280 CPU-only | 588 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Data Center GPU Flex 170 | 443 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Arc A-Series Graphics | 348 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 9 288V | 302 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 7 265H CPU-only | 276 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 7 155H CPU-only | 245 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Gold 5218T CPU-only | 217 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| NVIDIA Jetson Orin NX 16GB | 195 FPS throughput_offline · b=unknown | MLPerf BERT INT8/FP16 (MLPerf closed) · TensorRT (MLPerf Inference, older round) · unknown | D | unknown power n/a no power published | NVIDIA Jetson Benchmarks (MLPerf v3.1 results table) Older data (2023) |
| NVIDIA Jetson Orin NX 16GB (MaxQ) | 137 FPS throughput_offline · b=unknown | MLPerf BERT INT8/FP16 (MLPerf closed) · TensorRT (MLPerf Inference, older round) · unknown | D | 17.04 W measured reported in result source — measured | NVIDIA Jetson Benchmarks (MLPerf v3.1 results table) Older data (2023) |
| Intel Core i5-13600K CPU-only | 117 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-12700H CPU-only | 112 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1185G7 CPU-only | 85.39 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1355U | 84.21 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1185GRE CPU-only | 50.77 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i5-1335U CPU-only | 49.09 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i5-1235U | 44.51 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Celeron 6305E CPU-only | 44.00 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
Fastest in the best-covered like-for-like group (unknown, batch 1): Rockchip RK3588 (NPU, single core of 3) at 987 FPS C 2026-10-13 (page last edited). 2 groups, 33 platforms.
Show data — 2 like-for-like tables
unknown · batch 1 (8 platforms)
✓ Like-for-like: model-size class unknown, batch 1 (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Rockchip RK3588 (NPU, single core of 3) | 987 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3576 (NPU, single core of 2) | 981 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3562 (NPU) | 858 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1126B (NPU) | 853 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK3566/RK3568 (NPU) | 386 FPS throughput · b=1 | lite-transformer-encoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RK1808 (NPU) | 110 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1126 (NPU) | 65.80 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
| Rockchip RV1109 (NPU) | 48.00 FPS throughput · b=1 | lite-transformer-decoder-16 FP16 · RKNN-Toolkit2 / RKNPU (rknn_model_zoo) · unknown | C | unknown power n/a no power published | Rockchip rknn_model_zoo README benchmark table 2026-10-13 (page last edited) |
unknown · batch unknown (25 platforms)
✓ Like-for-like: model-size class unknown, batch unknown (size or batch not stated by source — unverified). Model, precision and stack may still differ.
| Platform | Result | Model · precision · stack | Grade | Power | Source |
|---|---|---|---|---|---|
| Intel Xeon Platinum 6972P CPU-only | 7863 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8580 CPU-only | 4497 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8480+ CPU-only | 2894 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Lenovo ThinkEdge SE455i V3 (2x NVIDIA L4-PCIe-24GB) | 1521 FPS throughput_offline · b=unknown | MLPerf bert-99 int8 · TensorRT 10.8, CUDA 13.1 · unknown | A | unknown power n/a no power published | MLPerf Inference v6.0, closed division, edge suite 2026-04-01 (published) |
| Intel Xeon Platinum 8380 CPU-only | 879 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Arc B-Series Graphics | 757 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Gold 6338N CPU-only | 623 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Platinum 8280 CPU-only | 588 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| NVIDIA Jetson AGX Orin 64GB | 554 FPS throughput_offline · b=unknown | MLPerf BERT INT8/FP16 (MLPerf closed) · TensorRT (MLPerf Inference, older round) · unknown | D | 65.00 W spec platform spec envelope (not measured with this result) | NVIDIA Jetson Benchmarks (MLPerf v3.1 results table) Older data (2023) |
| Intel Data Center GPU Flex 170 | 443 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Arc A-Series Graphics | 348 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased FP16 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 9 288V | 302 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 7 265H CPU-only | 276 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core Ultra 7 155H CPU-only | 245 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Xeon Gold 5218T CPU-only | 217 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| NVIDIA Jetson Orin NX 16GB | 195 FPS throughput_offline · b=unknown | MLPerf BERT INT8/FP16 (MLPerf closed) · TensorRT (MLPerf Inference, older round) · unknown | D | unknown power n/a no power published | NVIDIA Jetson Benchmarks (MLPerf v3.1 results table) Older data (2023) |
| NVIDIA Jetson Orin NX 16GB (MaxQ) | 137 FPS throughput_offline · b=unknown | MLPerf BERT INT8/FP16 (MLPerf closed) · TensorRT (MLPerf Inference, older round) · unknown | D | 17.04 W measured reported in result source — measured | NVIDIA Jetson Benchmarks (MLPerf v3.1 results table) Older data (2023) |
| Intel Core i5-13600K CPU-only | 117 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-12700H CPU-only | 112 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1185G7 CPU-only | 85.39 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1355U | 84.21 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i7-1185GRE CPU-only | 50.77 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i5-1335U CPU-only | 49.09 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Core i5-1235U | 44.51 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
| Intel Celeron 6305E CPU-only | 44.00 FPS throughput (OpenVINO benchmark_app) · b=unknown | bert-base-cased INT8 · OpenVINO 2026.4.1 · unknown | D | unknown power n/a no power published | OpenVINO official performance benchmarks (graph-data-ov.json) 2026-10-01 (published) |
How to read these numbers
Every number is labelled. Confidence: A MLPerf closed, audited · B documented third-party method · C community / review · D vendor-published, not independently verified · E estimate. Power basis: measured with the result · spec vendor spec/envelope · power n/a not published.
Machine-readable: /find.json (see llms.txt); full tables are in “Show data”. Confidence grades: A audited MLPerf closed · B documented third-party · C community · D vendor-published · E estimate. Unknown values are shown as unknown, never estimated. Generated 2026-10-16 from public sources.