Skip to content
InfratoApp
InfratoApp / MLPerf results / v5.1

MLPerf v5.1, as submitted

Every figure is for the whole system that produced it, shown with its accelerator count. We do not divide a rack result down to a per-GPU number.

v5.1 · 30 benchmark and scenario pairs

llama2-70b-99 · Offline · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)124,879 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)102,131 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)101,253 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)68,895 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  5. 5Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)62,352 Tokens/s · 16× NVIDIA H100-SXM-80GBsource
  6. 6Dell_MangoBoost · Dell PowerEdge XE9680 (2x8x MI300X-192GB MangoBoost-LLMBoost)53,572 Tokens/s · 16× AMD Instinct MI300X 192GB HBM3source
  7. 7Azure · ND_GB200_v652,062 Tokens/s · 4× NVIDIA GB200source
  8. 8Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)35,317 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  9. 9ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)35,075 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  10. 10Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)34,588 Tokens/s · 8× NVIDIA H200-SXM-141GBsource

showing 10 of 20

llama2-70b-99 · Server · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)122,274 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)99,214 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)99,139 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)66,439 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  5. 5Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)61,101 Tokens/s · 16× NVIDIA H100-SXM-80GBsource
  6. 6Dell_MangoBoost · Dell PowerEdge XE9680 (2x8x MI300X-192GB MangoBoost-LLMBoost)47,562 Tokens/s · 16× AMD Instinct MI300X 192GB HBM3source
  7. 7Azure · ND_GB200_v646,017 Tokens/s · 4× NVIDIA GB200source
  8. 8ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)34,194 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  9. 9Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)33,385 Tokens/s · 8× Virtualized NVIDIA H200-SXM-141GBsource
  10. 10Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)33,244 Tokens/s · 8× NVIDIA H200-SXM-141GBsource

showing 10 of 19

llama2-70b-99.9 · Offline · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)124,879 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)102,131 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)101,253 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)68,895 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  5. 5Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)62,352 Tokens/s · 16× NVIDIA H100-SXM-80GBsource
  6. 6Dell_MangoBoost · Dell PowerEdge XE9680 (2x8x MI300X-192GB MangoBoost-LLMBoost)53,572 Tokens/s · 16× AMD Instinct MI300X 192GB HBM3source
  7. 7Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)35,317 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  8. 8ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)34,887 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  9. 9Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)34,588 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  10. 10AMD · QuantaGrid D74A-7U34,555 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource

showing 10 of 19

llama2-70b-99.9 · Server · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)122,274 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)99,214 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)99,139 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)66,439 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  5. 5Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)61,101 Tokens/s · 16× NVIDIA H100-SXM-80GBsource
  6. 6Dell_MangoBoost · Dell PowerEdge XE9680 (2x8x MI300X-192GB MangoBoost-LLMBoost)47,562 Tokens/s · 16× AMD Instinct MI300X 192GB HBM3source
  7. 7Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)33,371 Tokens/s · 8× Virtualized NVIDIA H200-SXM-141GBsource
  8. 8Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)33,244 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  9. 9Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)33,163 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  10. 10Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)33,161 Tokens/s · 8× NVIDIA H200-SXM-141GBsource

showing 10 of 18

retinanet · Offline · Samples/s

  1. 1Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)14,961 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)14,923 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)14,543 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)14,452 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  5. 5Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)13,421 Samples/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)13,320 Samples/s · 8× NVIDIA H200-NVL-141GBsource
  7. 7Cisco · Cisco UCS C845A M8 (8x H200-NVL-141GB, TensorRT)13,263 Samples/s · 8× NVIDIA H200-NVL-141GBsource
  8. 8Cisco · Cisco UCS C845A M8 (8x L40S-48GB, TensorRT)6,518 Samples/s · 8× NVIDIA L40Ssource
  9. 9Dell · Dell PowerEdge XE7740 (8x L40S, TensorRT)6,510 Samples/s · 8× NVIDIA L40Ssource
  10. 10GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)1,760 Samples/s · 2× NVIDIA GeForce RTX 4090source

showing 10 of 14

retinanet · Server · Queries/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)14,406 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)14,356 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)13,603 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)13,603 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  5. 5Cisco · Cisco UCS C845A M8 (8x H200-NVL-141GB, TensorRT)13,203 Queries/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)12,802 Queries/s · 8× NVIDIA H200-NVL-141GBsource
  7. 7Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)12,601 Queries/s · 8× NVIDIA H200-NVL-141GBsource
  8. 8Cisco · Cisco UCS C845A M8 (8x L40S-48GB, TensorRT)6,101 Queries/s · 8× NVIDIA L40Ssource
  9. 9Dell · Dell PowerEdge XE7740 (8x L40S, TensorRT)6,101 Queries/s · 8× NVIDIA L40Ssource
  10. 10GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)1,449 Queries/s · 2× NVIDIA GeForce RTX 4090source

showing 10 of 12

whisper · Offline · Samples/s

  1. 1Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)45,024 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)44,922 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)41,927 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Dell · Dell PowerEdge XE9680L (8x H200-SXM-141GB, TensorRT)34,014 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  5. 5Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)31,674 Samples/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6Dell · Dell PowerEdge XE7740 (8x L40S, TensorRT)9,272 Samples/s · 8× NVIDIA L40Ssource
  7. 7Dell · Dell PowerEdge R7715 (3x H100-NVL-94GB, TensorRT)8,302 Samples/s · 3× NVIDIA H100-NVL-94GBsource
  8. 8Dell · PowerEdge R570 (4x L4-PCIE-24GBx4, TensorRT)1,397 Samples/s · 4× NVIDIA L4-PCIe-24GBsource
  9. 9GATEOverflow · GATE Overflow Intel SPR i9 (1x RTX 4090)136 Samples/s · 1× NVIDIA GeForce RTX 4090source
  10. 10GATEOverflow · PCSPECIALIST AMD AM5 PC (1x RTX 4090)110 Samples/s · 1× NVIDIA GeForce RTX 4090source

showing 10 of 12

llama2-70b-99 · Interactive · Tokens/s

  1. 1Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)60,138 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)21,916 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)19,622 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4AMD · QuantaGrid D74A-7U18,407 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  5. 5Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)16,419 Tokens/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6ASUSTeK · ESC_A8A_MI325X_256GBx815,566 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  7. 7AMD · Supermicro AS-8125GS-TNMR28,840 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source
  8. 8Dell · Dell Poweredge XE96808,455 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source

llama2-70b-99.9 · Interactive · Tokens/s

  1. 1Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)60,138 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)21,916 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)19,622 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4AMD · QuantaGrid D74A-7U18,407 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  5. 5Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)16,419 Tokens/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6ASUSTeK · ESC_A8A_MI325X_256GBx815,566 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  7. 7AMD · Supermicro AS-8125GS-TNMR28,840 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source
  8. 8Dell · Dell Poweredge XE96808,455 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source

stable-diffusion-xl · Offline · Samples/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)32.2916 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)31.8413 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)19.1227 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)18.6415 Samples/s · 8× Virtualized NVIDIA H200-SXM-141GBsource
  5. 5AMD · Quanta S7PA18.5856 Samples/s · 8× AMD Instinct MI325X 256GB HBM3esource
  6. 6Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)17.0907 Samples/s · 8× NVIDIA H200-NVL-141GBsource
  7. 7Dell · Dell PowerEdge XE7740 (8x L40S, TensorRT)5.52041 Samples/s · 8× NVIDIA L40Ssource
  8. 8Dell · PowerEdge R570 (4x L4-PCIE-24GBx4, TensorRT)1.00634 Samples/s · 4× NVIDIA L4-PCIe-24GBsource

stable-diffusion-xl · Server · Queries/s

  1. 1Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)28.7042 Queries/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)28.4651 Queries/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)18.3718 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)17.9503 Queries/s · 8× Virtualized NVIDIA H200-SXM-141GBsource
  5. 5Dell · Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)16.2171 Queries/s · 8× NVIDIA H200-NVL-141GBsource
  6. 6AMD · Quanta S7PA16.2044 Queries/s · 8× AMD Instinct MI325X 256GB HBM3esource
  7. 7Dell · Dell PowerEdge XE7740 (8x L40S, TensorRT)5.9027 Queries/s · 8× NVIDIA L40Ssource
  8. 8Dell · PowerEdge R570 (4x L4-PCIE-24GBx4, TensorRT)0.880856 Queries/s · 4× NVIDIA L4-PCIe-24GBsource

llama3.1-405b · Offline · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)1,661 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)1,650 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)1,648 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Azure · ND_GB200_v6847 Tokens/s · 4× NVIDIA GB200source
  5. 5Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)547 Tokens/s · 8× Virtualized NVIDIA H200-SXM-141GBsource

llama3.1-405b · Server · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)1,247 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)1,244 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)1,243 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Azure · ND_GB200_v6584 Tokens/s · 4× NVIDIA GB200source
  5. 5Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)277 Tokens/s · 8× Virtualized NVIDIA H200-SXM-141GBsource

mixtral-8x7b · Offline · Tokens/s

  1. 1AMD · QuantaGrid D74A-7U68,781 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  2. 2ASUSTeK · ESC_A8A_MI325X_256GBx868,121 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  3. 3ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)61,372 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Dell · Dell Poweredge XE968053,334 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source
  5. 5AMD · Supermicro AS-8125GS-TNMR253,017 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source

mixtral-8x7b · Server · Tokens/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)59,993 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2AMD · QuantaGrid D74A-7U59,784 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  3. 3ASUSTeK · ESC_A8A_MI325X_256GBx855,800 Tokens/s · 8× AMD Instinct MI325X 256GB HBM3esource
  4. 4AMD · Supermicro AS-8125GS-TNMR247,804 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source
  5. 5Dell · Dell Poweredge XE968047,802 Tokens/s · 8× AMD Instinct MI300X 192GB HBM3source

llama3.1-8b · Offline · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)143,904 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)143,309 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)58,304 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Cisco · Cisco UCS C845A M8 (8x H200-NVL-141GB, TensorRT)50,777 Tokens/s · 8× NVIDIA H200-NVL-141GBsource

llama3.1-8b · Server · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)128,794 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)128,787 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3Cisco · Cisco UCS C885A M8 (8x H200-SXM-141GB, TensorRT)41,291 Tokens/s · 8× NVIDIA H200-SXM-141GBsource
  4. 4Cisco · Cisco UCS C845A M8 (8x H200-NVL-141GB, TensorRT)38,697 Tokens/s · 8× NVIDIA H200-NVL-141GBsource

llama3_1-405b · Offline · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)1,562 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)1,086 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  3. 3Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)783 Tokens/s · 16× NVIDIA H100-SXM-80GBsource

llama3_1-405b · Server · Tokens/s

  1. 1Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)1,127 Tokens/s · 32× NVIDIA H100-SXM-80GBsource
  2. 2Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)584 Tokens/s · 16× NVIDIA H200-SXM-141GBsource
  3. 3Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)563 Tokens/s · 16× NVIDIA H100-SXM-80GBsource

deepseek-r1 · Offline · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)30,767 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)30,493 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

deepseek-r1 · Server · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)13,963 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)13,913 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

dlrm-v2-99 · Offline · Samples/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)643,122 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)580,460 Samples/s · 8× NVIDIA H200-NVL-141GBsource

dlrm-v2-99 · Server · Queries/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)591,162 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)450,100 Queries/s · 8× NVIDIA H200-NVL-141GBsource

dlrm-v2-99.9 · Offline · Samples/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)388,058 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)348,225 Samples/s · 8× NVIDIA H200-NVL-141GBsource

dlrm-v2-99.9 · Server · Queries/s

  1. 1ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)370,066 Queries/s · 8× NVIDIA H200-SXM-141GBsource
  2. 2ASUSTeK · ESC8000A-E12 (8x H200-NVL-141GB, TensorRT)330,555 Queries/s · 8× NVIDIA H200-NVL-141GBsource

llama3.1-405b · Interactive · Tokens/s

  1. 1Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)752 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)751 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

retinanet · MultiStream · Latency (ms)

  1. 1GATEOverflow · GATE Overflow Intel SPR i9 (1x RTX 4090)226 Latency (ms) · 1× NVIDIA GeForce RTX 4090source
  2. 2GATEOverflow · PCSPECIALIST AMD AM5 PC (1x RTX 4090)112 Latency (ms) · 1× NVIDIA GeForce RTX 4090source

retinanet · SingleStream · Latency (ms)

  1. 1GATEOverflow · GATE Overflow Intel SPR i9 (1x RTX 4090)22.700909 Latency (ms) · 1× NVIDIA GeForce RTX 4090source
  2. 2GATEOverflow · PCSPECIALIST AMD AM5 PC (1x RTX 4090)13.34974 Latency (ms) · 1× NVIDIA GeForce RTX 4090source

3d-unet-99 · Offline · Samples/s

  1. 1GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)8.34576 Samples/s · 2× NVIDIA GeForce RTX 4090source

llama3.1-8b · Interactive · Queries/s

  1. 1Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)102,907 Queries/s · 8× NVIDIA B200-SXM-180GBsource