Skip to content
InfratoApp
InfratoApp / MLPerf results

MLPerf results, as submitted

Submissions published by MLCommons. Every figure here is for the whole system that produced it, shown with its accelerator count — we do not divide a rack result down to a per-GPU number, because no submitter claims one and scaling does not support the arithmetic.

v6.1 · 31 benchmark and scenario pairs

gpt-oss-120b · Offline · Tokens/s

  1. 1Crusoe · Crusoe Cloud 64x8 MI355X (512 GPU) — gpt-oss-120b / vLLM5,749,440 Tokens/s · 512× AMD Instinct MI355X 288GB HBM3esource
  2. 2CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)1,197,710 Tokens/s · 72× NVIDIA GB300source
  3. 3AMD · 9x8xMI355X_2xEPYC_9575F1,039,900 Tokens/s · 72× AMD Instinct MI355X 288GB HBM3esource
  4. 4CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)911,566 Tokens/s · 72× NVIDIA GB200source
  5. 5Dell_MangoBoost · Heterogeneous Four-Cluster (2x8x MI300X + 2x8x MI355X, LLMBoost)285,454 Tokens/s · 32× AMD Instinct MI300X 192GB HBM3 (x2x8) and AMD Instinct MI355X 288GB HBM3e (x2x8)source
  6. 6Cisco · Cisco_UCS_B300x16_G200_4x2_RO220,527 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  7. 7AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)121,818 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  8. 8Dell · XE9785L_8xMI355X117,804 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  9. 9Supermicro · AS-4126GS-NMR-LCC117,005 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  10. 10Dell_MangoBoost · PowerEdge XE9785L116,953 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource

showing 10 of 28

gpt-oss-120b · Server · Tokens/s

  1. 1Crusoe · Crusoe Cloud 64x8 MI355X (512 GPU) — gpt-oss-120b / vLLM5,392,930 Tokens/s · 512× AMD Instinct MI355X 288GB HBM3esource
  2. 2CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)1,160,490 Tokens/s · 72× NVIDIA GB300source
  3. 3AMD · 9x8xMI355X_2xEPYC_9575F964,468 Tokens/s · 72× AMD Instinct MI355X 288GB HBM3esource
  4. 4CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)901,058 Tokens/s · 72× NVIDIA GB200source
  5. 5Dell_MangoBoost · Heterogeneous Four-Cluster (2x8x MI300X + 2x8x MI355X, LLMBoost)253,501 Tokens/s · 32× AMD Instinct MI300X 192GB HBM3 (x2x8) and AMD Instinct MI355X 288GB HBM3e (x2x8)source
  6. 6Cisco · Cisco_UCS_B300x16_G200_4x2_RO214,774 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  7. 7AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)113,241 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  8. 8CoreWeave · CoreWeave B300 SXM (8x B300-SXM-270GB)111,247 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9GigaComputing · G894-AD3-AAX7111,218 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10GigaComputing · G894-ZD3-AAX7110,592 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 28

llama2-70b-99.9 · Offline · Tokens/s

  1. 1CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)1,136,100 Tokens/s · 72× NVIDIA GB300source
  2. 2CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)894,002 Tokens/s · 72× NVIDIA GB200source
  3. 3Cisco · Cisco_UCS_B300x16_G200_4x2_RO227,570 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  4. 4Orrick Industries LLC · Orrick Optimized 6x NVIDIA B200 L2130,928 Tokens/s · 6× NVIDIA B200-SXM-180GBsource
  5. 5CoreWeave · CoreWeave B300 SXM (8x B300-SXM-270GB)115,530 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6GigaComputing · G894-AD3-AAX7115,034 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7GigaComputing · G894-ZD3-AAX7114,826 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8Dell · Dell XE9785L (8x B300-SXM-270GB, TensorRT)113,101 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9ASUSTeK · XA NB3I-E12113,007 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Supermicro · SYS-822GS-NB3RT111,247 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 25

llama2-70b-99.9 · Server · Tokens/s

  1. 1CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)944,902 Tokens/s · 72× NVIDIA GB300source
  2. 2CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)910,621 Tokens/s · 72× NVIDIA GB200source
  3. 3Cisco · Cisco_UCS_B300x16_G200_4x2_RO179,597 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  4. 4GigaComputing · G894-AD3-AAX7114,832 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5GigaComputing · G894-ZD3-AAX7114,578 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6CoreWeave · CoreWeave B300 SXM (8x B300-SXM-270GB)114,576 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7ASUSTeK · XA NB3I-E12113,455 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8Supermicro · SYS-822GS-NB3RT109,848 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9Dell · Dell XE9785L (8x B300-SXM-270GB, TensorRT)106,898 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Supermicro · AS-8126GS-NB3RT106,872 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 24

deepseek-r1 · Offline · Tokens/s

  1. 1Crusoe · Crusoe Cloud 64x8 MI355X (512 GPU) — DeepSeek-R1 / SGLang2,901,950 Tokens/s · 512× AMD Instinct MI355X 288GB HBM3esource
  2. 2Azure · Azure GB200 NVL72 (288x GB200-186GB_aarch64, TensorRT)2,091,190 Tokens/s · 288× NVIDIA GB200source
  3. 3Azure · Azure GB300 NVL72 (72x GB300-288GB_aarch64 TensorRT)661,143 Tokens/s · 72× NVIDIA GB300source
  4. 4Azure · Azure GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)542,075 Tokens/s · 72× NVIDIA GB200source
  5. 5CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)529,991 Tokens/s · 72× NVIDIA GB200source
  6. 6Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)510,781 Tokens/s · 72× NVIDIA GB200source
  7. 7HPE · NVIDIA GB200 NVL72 by HPE (56x GB200-186GB_aarch64, TensorRT)336,125 Tokens/s · 56× NVIDIA GB200source
  8. 8HPE · NVIDIA GB200 NVL72 by HPE (32x GB200-186GB_aarch64, TensorRT)245,352 Tokens/s · 32× NVIDIA GB200source
  9. 9Cisco · Cisco_UCS_B300x16_G200_4x2_RO137,067 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  10. 10HPE · NVIDIA GB200 NVL72 by HPE (16x GB200-186GB_aarch64, TensorRT)126,223 Tokens/s · 16× NVIDIA GB200source

showing 10 of 20

deepseek-r1 · Server · Tokens/s

  1. 1Crusoe · Crusoe Cloud 64x8 MI355X (512 GPU) — DeepSeek-R1 / SGLang2,405,310 Tokens/s · 512× AMD Instinct MI355X 288GB HBM3esource
  2. 2Azure · Azure GB200 NVL72 (288x GB200-186GB_aarch64, TensorRT)1,598,510 Tokens/s · 288× NVIDIA GB200source
  3. 3Azure · Azure GB300 NVL72 (72x GB300-288GB_aarch64 TensorRT)589,391 Tokens/s · 72× NVIDIA GB300source
  4. 4Azure · Azure GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)426,796 Tokens/s · 72× NVIDIA GB200source
  5. 5CoreWeave · CoreWeave GB200 NVL72 (72x GB200-186GB_aarch64)419,778 Tokens/s · 72× NVIDIA GB200source
  6. 6Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)388,590 Tokens/s · 72× NVIDIA GB200source
  7. 7Cisco · Cisco_UCS_B300x16_G200_4x2_RO88,916 Tokens/s · 16× NVIDIA B300-SXM-270GBsource
  8. 8GigaComputing · G894-ZD3-AAX770,772 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9GigaComputing · G894-AD3-AAX770,496 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Supermicro · SYS-822GS-NB3RT66,754 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 16

llama3.1-8b · Offline · Tokens/s

  1. 1Orrick Industries LLC · Orrick Optimized 6x NVIDIA B200274,927 Tokens/s · 6× NVIDIA B200-SXM-180GBsource
  2. 2Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)171,114 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Cisco · Cisco UCS H200 x8 + MI350X x8 G200 2x2 TOR163,292 Tokens/s · 16× NVIDIA H200-SXM-141GB + AMD Instinct MI350X 288GB HBM3esource
  4. 4VibeHPC · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)161,767 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5Supermicro · AS-4126GS-NMR-LCC159,060 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  6. 6Dell · XE9785L_8xMI355X158,458 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  7. 7Dell_AMD · XE7745_8xMI350P_2xEPYC_945573,643 Tokens/s · 8× AMD Instinct MI350Psource
  8. 8Cisco · Cisco UCS C845A M8 (8x RTXPro6000-PCIe-96GB, TensorRT)50,454 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  9. 9Cisco · Cisco UCS X215c M8 - X580P PCIe Node (4x RTXPro6000-PCIe-96GB, TensorRT)24,914 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  10. 10TTA · ManyCoreSoft DeepGadget dg5W (4x NVIDIA RTX PRO 6000 Blackwell 96GB, TensorRT)24,512 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource

showing 10 of 15

llama3.1-8b · Server · Tokens/s

  1. 1Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)173,950 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2Cisco · Cisco UCS H200 x8 + MI350X x8 G200 2x2 TOR161,110 Tokens/s · 16× NVIDIA H200-SXM-141GB + AMD Instinct MI350X 288GB HBM3esource
  3. 3VibeHPC · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)156,867 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4Supermicro · AS-4126GS-NMR-LCC148,710 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  5. 5Dell · XE9785L_8xMI355X148,563 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  6. 6Dell_AMD · XE7745_8xMI350P_2xEPYC_945572,305 Tokens/s · 8× AMD Instinct MI350Psource
  7. 7Cisco · Cisco UCS C845A M8 (8x RTXPro6000-PCIe-96GB, TensorRT)49,175 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  8. 8TTA · ManyCoreSoft DeepGadget dg5W (4x NVIDIA RTX PRO 6000 Blackwell 96GB, TensorRT)24,511 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  9. 9Cisco · Cisco UCS X215c M8 - X580P PCIe Node (4x RTXPro6000-PCIe-96GB, TensorRT)24,499 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  10. 10VibeHPC · B300-SXM-270GBx1_TRT20,420 Tokens/s · 1× NVIDIA B300-SXM-270GBsource

showing 10 of 14

whisper · Offline · Samples/s

  1. 1VibeHPC · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)53,267 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)52,222 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Cisco · Cisco UCS C845A M8 (8x RTXPro6000-PCIe-96GB, TensorRT)18,992 Samples/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  4. 4Cisco · Cisco UCS X215c M8 - X580P PCIe Node (4x RTXPro6000-PCIe-96GB, TensorRT)9,237 Samples/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  5. 5Cisco · Cisco UCS C845A M8 (8x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)7,778 Samples/s · 8× NVIDIA RTX PRO 4500 Blackwellsource
  6. 6VibeHPC · B300-SXM-270GBx1_TRT7,059 Samples/s · 1× NVIDIA B300-SXM-270GBsource
  7. 7Supermicro · 1-node-4x-BMG-B705,855 Samples/s · 4× Intel(R) Arc(TM) Pro B70source
  8. 8Cisco · Cisco UCS C240 M8 - (5x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)4,809 Samples/s · 5× NVIDIA RTX PRO 4500 Blackwellsource
  9. 9Cisco · Cisco UCS X210 M8 - X580P PCIe Node (4x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)3,878 Samples/s · 4× NVIDIA RTX PRO 4500 Blackwellsource
  10. 10HPE · HPE ProLiant Compute DL345 Gen12 (4x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT3,686 Samples/s · 4× NVIDIA RTX PRO 4500 Blackwell Server Editionsource

llama3.1-8b · Interactive · Tokens/s

  1. 1Dell · XE9785L_8xMI355X130,540 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  2. 2Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)129,168 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Cisco · Cisco UCS H200 x8 + MI350X x8 G200 2x2 TOR121,557 Tokens/s · 16× NVIDIA H200-SXM-141GB + AMD Instinct MI350X 288GB HBM3esource
  4. 4Dell_AMD · XE7745_8xMI350P_2xEPYC_945562,724 Tokens/s · 8× AMD Instinct MI350Psource
  5. 5Cisco · Cisco UCS C845A M8 (8x RTXPro6000-PCIe-96GB, TensorRT)46,164 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  6. 6Cisco · Cisco UCS X215c M8 - X580P PCIe Node (4x RTXPro6000-PCIe-96GB, TensorRT)22,447 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  7. 7Cisco · Cisco UCS C845A M8 (8x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)14,316 Tokens/s · 8× NVIDIA RTX PRO 4500 Blackwellsource
  8. 8Cisco · Cisco UCS C240 M8 - (5x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)8,495 Tokens/s · 5× NVIDIA RTX PRO 4500 Blackwellsource
  9. 9Cisco · Cisco UCS X210 M8 - X580P PCIe Node (4x NVIDIA RTX PRO 4500 Blackwell-32GB, TensorRT)6,884 Tokens/s · 4× NVIDIA RTX PRO 4500 Blackwellsource

text_to_video · Offline · Samples/s

  1. 1Dell · Dell PowerEdge XE97120.6455254875 Samples/s · 72× NVIDIA GB300source
  2. 2VibeHPC · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)0.0783824 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)0.0731282 Samples/s · 8× AMD Instinct MI355X 288GB HBM3esource
  4. 4GigaComputing · G894-AD3-AAX70.0629152672 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5GigaComputing · G894-ZD3-AAX70.062512308 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)0.0612749683 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7AMD · Quanta Grand Teton 1.5 MI3500.0565013 Samples/s · 8× AMD Instinct MI350X 288GB HBM3esource
  8. 8Dell_AMD · XE7745_8xMI350P_2xEPYC_94550.0345875 Samples/s · 8× AMD Instinct MI350Psource

text_to_video · SingleStream · Latency (s)

  1. 1Dell_AMD · XE7745_8xMI350P_2xEPYC_945541.837923108 Latency (s) · 8× AMD Instinct MI350Psource
  2. 2VibeHPC · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)21.279754716 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  3. 3AMD · Quanta Grand Teton 1.5 MI35020.603024726 Latency (s) · 8× AMD Instinct MI350X 288GB HBM3esource
  4. 4Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)20.004214518 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  5. 5GigaComputing · G894-AD3-AAX719.0794043884 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  6. 6GigaComputing · G894-ZD3-AAX718.0517986718 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  7. 7AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)16.124484197 Latency (s) · 8× AMD Instinct MI355X 288GB HBM3esource
  8. 8Dell · Dell PowerEdge XE971210.5850005407 Latency (s) · 72× NVIDIA GB300source

VLM · Offline · Samples/s

  1. 1CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)1,135 Samples/s · 72× NVIDIA GB300source
  2. 2Crusoe · NVIDIA GB200 NVL72 (16x GB200-186GB_aarch64, Dynamo)230 Samples/s · 16× NVIDIA GB200source
  3. 3GigaComputing · G894-ZD3-AAX7132 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4GigaComputing · G894-AD3-AAX7131 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5Supermicro · SYS-822GS-NB3RT129 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6CoreWeave · CoreWeave GB300 NVL72 (4x GB300-288GB_aarch64)72.598002941 Samples/s · 4× NVIDIA GB300source
  7. 7Crusoe · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, Dynamo)58.3958752389 Samples/s · 4× NVIDIA GB200source
  8. 8Krai · HPE Cray XD670 (8x H200-SXM-141GB, Krai/Gentic)9.2216804291 Samples/s · 8× NVIDIA H200-SXM-141GBsource

VLM · Server · Queries/s

  1. 1CoreWeave · CoreWeave GB300 NVL72 (72x GB300-288GB_aarch64)1,196 Queries/s · 72× NVIDIA GB300source
  2. 2Crusoe · NVIDIA GB200 NVL72 (16x GB200-186GB_aarch64, Dynamo)199 Queries/s · 16× NVIDIA GB200source
  3. 3GigaComputing · G894-ZD3-AAX7116 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4Supermicro · SYS-822GS-NB3RT112 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5GigaComputing · G894-AD3-AAX7112 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6CoreWeave · CoreWeave GB300 NVL72 (4x GB300-288GB_aarch64)64.1320203068 Queries/s · 4× NVIDIA GB300source
  7. 7Crusoe · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, Dynamo)49.696288324 Queries/s · 4× NVIDIA GB200source

agentic edge · SingleStream · Latency (ms)

  1. 1Atlas_Inference · AMD Strix Halo desktop (Ryzen AI Max+ 395 / Radeon 8060S)7,059 Latency (ms) · 1× AMD Radeon 8060S Graphicssource
  2. 2Atlas_Inference · NVIDIA DGX Spark (GB10 Grace Blackwell Superchip)3,808 Latency (ms) · 1× NVIDIA GB10source
  3. 3HPE · HPE ProLiant DL145 Gen11 (1x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT)2,758 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwell Server Editionsource
  4. 4HPE · HPE ProLiant Compute DL345 Gen12 (4x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT2,616 Latency (ms) · 4× NVIDIA RTX PRO 4500 Blackwell Server Editionsource
  5. 5Lambda · Lambda 8x B200-SXM-180GB (SGLang, Kimi-K2.6 edge-agentic / BFCL v4)771 Latency (ms) · 8× NVIDIA B200-SXM-180GBsource

llama2-70b-99.9 · Interactive · Tokens/s

  1. 1Dell · XE9785L_8xMI355X73,238 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  2. 2AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)73,192 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  3. 3Supermicro · AS-4126GS-NMR-LCC73,182 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  4. 4Cisco · Cisco UCS H200 x8 + MI350X x8 G200 2x2 TOR70,355 Tokens/s · 16× NVIDIA H200-SXM-141GB + AMD Instinct MI350X 288GB HBM3esource
  5. 5Dell_AMD · XE7745_8xMI350P_2xEPYC_945522,861 Tokens/s · 8× AMD Instinct MI350Psource

dlrm-v3 · Server · Queries/s

  1. 1AMD · 8xMI355X_2xEPYC_9575F16,194 Queries/s · 8× AMD Instinct MI355Xsource
  2. 2AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)12,199 Queries/s · 8× AMD Instinct MI355X 288GB HBM3esource
  3. 3AMD · 8xMI350X_2xEPYC_9575F9,298 Queries/s · 8× AMD Instinct MI350X 288GB HBM3esource
  4. 4Dell_AMD · XE7745_8xMI350P_2xEPYC_94554,504 Queries/s · 8× AMD Instinct MI350Psource

dlrm-v3 · Offline · Samples/s

  1. 1AMD · 8xMI355X_2xEPYC_9575F16,363 Samples/s · 8× AMD Instinct MI355Xsource
  2. 2AMD · smci355-ccs-aus-m13-13 (AS -4126GS-NMR-LCC)13,028 Samples/s · 8× AMD Instinct MI355X 288GB HBM3esource
  3. 3AMD · 8xMI350X_2xEPYC_9575F10,891 Samples/s · 8× AMD Instinct MI350X 288GB HBM3esource
  4. 4Dell_AMD · XE7745_8xMI350P_2xEPYC_94554,838 Samples/s · 8× AMD Instinct MI350Psource

VLM · Interactive · Queries/s

  1. 1Crusoe · NVIDIA GB200 NVL72 (16x GB200-186GB_aarch64, Dynamo)39.8507913343 Queries/s · 16× NVIDIA GB200source
  2. 2GigaComputing · G894-ZD3-AAX712.8817797408 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3GigaComputing · G894-AD3-AAX712.4854321324 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4Crusoe · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, Dynamo)3.9940567448 Queries/s · 4× NVIDIA GB200source

deepseek-r1 · Interactive · Tokens/s

  1. 1Azure · Azure GB300 NVL72 (72x GB300-288GB_aarch64 TensorRT)260,098 Tokens/s · 72× NVIDIA GB300source
  2. 2Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)232,160 Tokens/s · 72× NVIDIA GB200source

gpt-oss-120b · Interactive · Tokens/s

  1. 1Dell_MangoBoost · PowerEdge XE9785L29,586 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  2. 2Dell_MangoBoost · PowerEdge XE978528,266 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource

yolo-95 · multistream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL345 Gen12 (1x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT)676 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwell Server Editionsource
  2. 2Dell · Dell Pro Max (1x GB10)105 Latency (ms) · 1× NVIDIA GB10source

yolo-95 · offline · Samples/s

  1. 1HPE · HPE ProLiant Compute DL345 Gen12 (1x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT)146 Samples/s · 1× NVIDIA RTX PRO 4500 Blackwell Server Editionsource
  2. 2Dell · Dell Pro Max (1x GB10)78.7257 Samples/s · 1× NVIDIA GB10source

yolo-95 · singlestream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL345 Gen12 (1x NVIDIA RTX PRO 4500 Blackwell Server Edition 32GB, TensorRT)76.664039 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwell Server Editionsource
  2. 2Dell · Dell Pro Max (1x GB10)13.865536 Latency (ms) · 1× NVIDIA GB10source

llama3.1-8b-edge · Offline · Tokens/s

  1. 1HPE · HPE ProLiant DL345 Gen12 (4x NVIDIA RTX PRO 4500 Blackwell-PCIe-32GB, TensorRT)9,558 Tokens/s · 4× NVIDIA RTX PRO 4500 Blackwell-PCIe-32GBsource

llama3.1-8b-edge · SingleStream · Latency (ms)

  1. 1HPE · HPE ProLiant DL345 Gen12 (4x NVIDIA RTX PRO 4500 Blackwell-PCIe-32GB, TensorRT)1,204 Latency (ms) · 4× NVIDIA RTX PRO 4500 Blackwell-PCIe-32GBsource

resnet · multistream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL380 Gen12 (1x NVIDIA RTX PRO 4500 Blackwell-PCIe-32GB)5.030642 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwell-PCIe-32GBsource

resnet · singlestream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL380 Gen12 (1x NVIDIA RTX PRO 4500 Blackwell-PCIe-32GB)1.480867 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwell-PCIe-32GBsource

yolo-99 · multistream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)105 Latency (ms) · 1× NVIDIA GB10source

yolo-99 · offline · Samples/s

  1. 1Dell · Dell Pro Max (1x GB10)78.7257 Samples/s · 1× NVIDIA GB10source

yolo-99 · singlestream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)13.865536 Latency (ms) · 1× NVIDIA GB10source

What this page will not tell you

  • A per-accelerator figure. Dividing a whole-system result by its accelerator count invents a number nobody submitted.
  • A comparison across benchmarks. Results are grouped by benchmark, scenario and unit, because only within one of those is a larger number actually better.
  • Research hardware. Submissions markedrnd are left out: they cannot be bought, and listing them beside available systems would mislead.