Skip to content
InfratoApp
InfratoApp / MLPerf results / v6.0

MLPerf v6.0, as submitted

Every figure is for the whole system that produced it, shown with its accelerator count. We do not divide a rack result down to a per-GPU number.

v6.0 · 51 benchmark and scenario pairs

llama2-70b-99 · Offline · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,126,850 Tokens/s · 72× NVIDIA GB300source
  2. 2AMD · AMD MI355X Custom Cluster1,042,110 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)888,054 Tokens/s · 72× NVIDIA GB200source
  4. 4Cisco · Cisco UCS 16xMI350X159,932 Tokens/s · 16× AMD Instinct MI350X 288GB HBM3esource
  5. 5Dell_MangoBoost · Dell_MangoBoost Heterogeneous Cluster (8x MI300X + 8x MI325X + 8x MI355X, LLMBoost)151,843 Tokens/s · 24× AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8) and AMD Instinct MI355X 288GB HBM3e (x8)source
  6. 6Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)115,730 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)112,954 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8ASUSTeK · XA NB3I-E12109,625 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9GigaComputing · G894-SD3-AAX7109,343 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)108,669 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 42

llama2-70b-99 · Server · Tokens/s

  1. 1AMD · AMD MI355X Custom Cluster1,016,380 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  2. 2NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)868,278 Tokens/s · 72× NVIDIA GB300source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)810,104 Tokens/s · 72× NVIDIA GB200source
  4. 4Cisco · Cisco UCS 16xMI350X153,022 Tokens/s · 16× AMD Instinct MI350X 288GB HBM3esource
  5. 5Dell_MangoBoost · Dell_MangoBoost Heterogeneous Cluster (8x MI300X + 8x MI325X + 8x MI355X, LLMBoost)141,521 Tokens/s · 24× AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8) and AMD Instinct MI355X 288GB HBM3e (x8)source
  6. 6Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)114,504 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)108,392 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)107,318 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9GigaComputing · G894-SD3-AAX7104,623 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)103,635 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

showing 10 of 41

llama2-70b-99.9 · Server · Tokens/s

  1. 1AMD · AMD MI355X Custom Cluster1,016,380 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  2. 2NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)868,278 Tokens/s · 72× NVIDIA GB300source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)810,104 Tokens/s · 72× NVIDIA GB200source
  4. 4Dell_MangoBoost · Dell_MangoBoost Heterogeneous Cluster (8x MI300X + 8x MI325X + 8x MI355X, LLMBoost)141,521 Tokens/s · 24× AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8) and AMD Instinct MI355X 288GB HBM3e (x8)source
  5. 5Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)108,392 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)107,318 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7GigaComputing · G894-SD3-AAX7104,623 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)103,627 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  9. 9AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)100,282 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  10. 10Supermicro · SYS-822GS-NBRT (8x B200-SXM-180GB, TensorRT)99,305 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

showing 10 of 32

llama2-70b-99.9 · Offline · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,126,850 Tokens/s · 72× NVIDIA GB300source
  2. 2AMD · AMD MI355X Custom Cluster1,042,110 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)888,054 Tokens/s · 72× NVIDIA GB200source
  4. 4Dell_MangoBoost · Dell_MangoBoost Heterogeneous Cluster (8x MI300X + 8x MI325X + 8x MI355X, LLMBoost)151,843 Tokens/s · 24× AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8) and AMD Instinct MI355X 288GB HBM3e (x8)source
  5. 5NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)112,954 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6GigaComputing · G894-SD3-AAX7109,343 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)108,669 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)104,572 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  9. 9AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)103,480 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  10. 10Supermicro · SYS-822GS-NBRT (8x B200-SXM-180GB, TensorRT)101,526 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

showing 10 of 32

whisper · Offline · Samples/s

  1. 1Supermicro · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)50,562 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)49,327 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3GigaComputing · G894-SD3-AAX748,117 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)47,434 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  5. 5Lenovo · ThinkSystem SR680a V4 (8x B300-SXM-270GB, TensorRT)47,269 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)46,034 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  7. 7Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)45,659 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8Netweb_technologies · Camarero PDI200A2HG-810 (8x B200-SXM-180GB, TensorRT)45,083 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  9. 9Supermicro · SYS-822GS-NBRT (8x B200-SXM-180GB, TensorRT)42,390 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  10. 10RedHat · IBM Cloud qx3d-160x1792x8h200,NVIDIA 8xH200-SXM-141GB,vLLM 0.15.136,396 Samples/s · 8× NVIDIA H200-SXM-141GBsource

showing 10 of 31

gpt-oss-120b · Server · Tokens/s

  1. 1Nebius · Nebius GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,096,770 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,072,250 Tokens/s · 72× NVIDIA GB300source
  3. 3CoreWeave · CoreWeave GB300 NVL72 (64x GB300-288GB_aarch64, TensorRT)956,523 Tokens/s · 64× NVIDIA GB300source
  4. 4AMD · AMD MI355X Custom Cluster900,054 Tokens/s · 96× AMD Instinct MI355X 288GB HBM3e (x94)source
  5. 5NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)899,218 Tokens/s · 72× NVIDIA GB200source
  6. 6CoreWeave · CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)743,550 Tokens/s · 64× NVIDIA GB200source
  7. 7Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)110,655 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8GigaComputing · G894-SD3-AAX7106,397 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9Lenovo · ThinkSystem SR680a V4 (8x B300-SXM-270GB, TensorRT)100,687 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10ASUSTeK · XA NB3I-E12100,656 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 30

gpt-oss-120b · Offline · Tokens/s

  1. 1Nebius · Nebius GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,046,150 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,042,980 Tokens/s · 72× NVIDIA GB300source
  3. 3AMD · AMD MI355X Custom Cluster1,031,070 Tokens/s · 96× AMD Instinct MI355X 288GB HBM3e (x94)source
  4. 4CoreWeave · CoreWeave GB300 NVL72 (64x GB300-288GB_aarch64, TensorRT)910,468 Tokens/s · 64× NVIDIA GB300source
  5. 5NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)879,542 Tokens/s · 72× NVIDIA GB200source
  6. 6CoreWeave · CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)774,539 Tokens/s · 64× NVIDIA GB200source
  7. 7Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)111,496 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8ASUSTeK · XA NB3I-E12110,077 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9Lenovo · ThinkSystem SR680a V4 (8x B300-SXM-270GB, TensorRT)109,141 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10GigaComputing · G894-SD3-AAX7107,696 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 29

llama3.1-8b · Offline · Tokens/s

  1. 1ASUSTeK · XA NB3I-E12166,745 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)165,432 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Lambda_SIT · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)160,403 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)152,625 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  5. 5GigaComputing · G894-SD3-AAX7150,707 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6Inventec · Inventec P9000AG7 (8x B200-SXM-180GB, TensorRT)142,508 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  7. 7HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)60,680 Tokens/s · 10× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  8. 8Dell · PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)53,194 Tokens/s · 8× NVIDIA H200-NVL-141GBsource
  9. 9HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)49,580 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  10. 10Dell · PowerEdge XE7745 (8x RTXPro6000-PCIe-96GB, TensorRT)49,036 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource

showing 10 of 20

deepseek-r1 · Offline · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)2,494,310 Tokens/s · 288× NVIDIA GB300source
  2. 2Nebius · Nebius GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)673,936 Tokens/s · 72× NVIDIA GB300source
  3. 3NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)647,076 Tokens/s · 72× NVIDIA GB300source
  4. 4CoreWeave · CoreWeave GB300 NVL72 (64x GB300-288GB_aarch64, TensorRT)627,653 Tokens/s · 64× NVIDIA GB300source
  5. 5NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)486,141 Tokens/s · 72× NVIDIA GB200source
  6. 6CoreWeave · CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)468,665 Tokens/s · 64× NVIDIA GB200source
  7. 7Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)307,874 Tokens/s · 72× NVIDIA GB200source
  8. 8CoreWeave · CoreWeave GB300 NVL72 (8x GB300-288GB_aarch64, TensorRT)78,564 Tokens/s · 8× NVIDIA GB300source
  9. 9Nebius · Nebius GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)76,347 Tokens/s · 8× NVIDIA GB300source
  10. 10NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)70,326 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 19

llama3.1-8b · Server · Tokens/s

  1. 1ASUSTeK · XA NB3I-E12148,067 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)141,565 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3GigaComputing · G894-SD3-AAX7141,484 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)131,270 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  5. 5Lambda_SIT · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)130,008 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  6. 6HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)59,633 Tokens/s · 10× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  7. 7HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)48,794 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  8. 8ASUSTeK · ESC8000A_E13P_RTX_PRO_6000_PCIE_96GBx848,531 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  9. 9ASUSTeK · ESC8000_E12P_RTX_PRO_6000_PCIE_96GBx847,581 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  10. 10Dell · PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)45,076 Tokens/s · 8× NVIDIA H200-NVL-141GBsource

showing 10 of 19

deepseek-r1 · Server · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)1,555,110 Tokens/s · 288× NVIDIA GB300source
  2. 2Nebius · Nebius GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)575,580 Tokens/s · 72× NVIDIA GB300source
  3. 3NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)400,437 Tokens/s · 72× NVIDIA GB300source
  4. 4NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)336,106 Tokens/s · 72× NVIDIA GB200source
  5. 5CoreWeave · CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)311,531 Tokens/s · 64× NVIDIA GB200source
  6. 6Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)156,781 Tokens/s · 72× NVIDIA GB200source
  7. 7Nebius · Nebius GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)64,510 Tokens/s · 8× NVIDIA GB300source
  8. 8Nebius · Nebius B300 n1 (8x B300-SXM-270GB, TensorRT)60,413 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  9. 9Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)58,553 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Nebius · Nebius B200 n1 (8x B200-SXM-180GB, TensorRT)51,693 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

showing 10 of 18

llama2-70b-99 · Interactive · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)814,128 Tokens/s · 72× NVIDIA GB300source
  2. 2AMD · AMD MI355X Custom Cluster785,522 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)754,855 Tokens/s · 72× NVIDIA GB200source
  4. 4Cisco · Cisco UCS 16xMI350X103,633 Tokens/s · 16× AMD Instinct MI350X 288GB HBM3esource
  5. 5AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)73,608 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  6. 6GigaComputing · G893-ZX1-AAX472,547 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  7. 7ORACLE · BM.GPU.MI355X.872,117 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  8. 8Supermicro · QuantaGrid D74A-7U71,296 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  9. 9HPE · HPE ProLiant Compute XD685 (8x AMD Instinct MI355X 288GB, vLLM)70,954 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  10. 10Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)70,724 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

showing 10 of 18

llama2-70b-99.9 · Interactive · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)814,128 Tokens/s · 72× NVIDIA GB300source
  2. 2AMD · AMD MI355X Custom Cluster785,522 Tokens/s · 88× AMD Instinct MI355X 288GB HBM3e (x87)source
  3. 3NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)754,855 Tokens/s · 72× NVIDIA GB200source
  4. 4AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)73,608 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  5. 5GigaComputing · G893-ZX1-AAX472,547 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  6. 6ORACLE · BM.GPU.MI355X.872,117 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  7. 7Supermicro · QuantaGrid D74A-7U71,296 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  8. 8HPE · HPE ProLiant Compute XD685 (8x AMD Instinct MI355X 288GB, vLLM)70,954 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  9. 9Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)70,724 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  10. 10Dell · PowerEdge XE9785L (8x MI355X)70,611 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource

showing 10 of 14

llama3.1-8b · Interactive · Tokens/s

  1. 1Lambda_SIT · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)128,750 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2GigaComputing · G894-SD3-AAX7128,633 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)126,143 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)53,055 Tokens/s · 10× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  5. 5HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)44,087 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  6. 6ASUSTeK · ESC8000A_E13P_RTX_PRO_6000_PCIE_96GBx834,241 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  7. 7HPE · HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition-450W, TensorRT)20,119 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  8. 8HPE · HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition-450W, TensorRT)19,991 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  9. 9HPE · HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)10,589 Tokens/s · 2× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  10. 10HPE · HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)10,460 Tokens/s · 2× NVIDIA RTX PRO 6000 Blackwell Server Editionsource

llama3.1-405b · Offline · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)19,512 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)15,462 Tokens/s · 72× NVIDIA GB200source
  3. 3AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)2,011 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  4. 4Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)1,971 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5Quanta_Cloud_Technology · QuantaGrid D75H-10U (8x B300-SXM-270GB, TensorRT)1,954 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)1,952 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7GigaComputing · G894-SD3-AAX71,948 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  8. 8Inventec · Inventec P9000AG7 (8x B200-SXM-180GB, TensorRT)1,632 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  9. 9Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)1,350 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

wan-2.2-t2v-a14b · SingleStream · Latency (s)

  1. 1Krai · HPE Cray XD670 (8x H200-SXM-141GB, KISS-V)57.401991155 Latency (s) · 8× NVIDIA H200-SXM-141GBsource
  2. 2Krai · HPE Cray XD670 (8x H200-SXM-141GB, KISS-V Fast Mode)43.936000879 Latency (s) · 8× NVIDIA H200-SXM-141GBsource
  3. 3NVIDIA · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)40.243719116 Latency (s) · 4× NVIDIA GB200source
  4. 4NVIDIA · NVIDIA GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)30.910508163 Latency (s) · 4× NVIDIA GB300source
  5. 5AMD · SuperMicro AS-4126GS-NMR-LCC27.384519297 Latency (s) · 8× AMD Instinct MI355X 288GB HBM3Esource
  6. 6NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)25.458383354 Latency (s) · 8× NVIDIA B200-SXM-180GBsource
  7. 7NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)23.778912625 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  8. 8Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)21.425919695 Latency (s) · 8× NVIDIA B300-SXM-270GBsource
  9. 9GigaComputing · G894-SD3-AAX720.897894239 Latency (s) · 8× NVIDIA B300-SXM-270GBsource

qwen3-vl-235b-a22b · Offline · Samples/s

  1. 1RedHat · Dell B200,8xB200-SXM-180GB,RHEL 10.1,vLLM CentML:mlperf-inf-mm-q3vl-v6.079.0373 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Nebius · Nebius B300 n1 (8x B300-SXM-270GB, TensorRT)78.2775 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3NVIDIA · NVIDIA GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)61.2443 Samples/s · 4× NVIDIA GB300source
  4. 4GigaComputing · DLB2-CB360.9525 Samples/s · 4× NVIDIA GB300source
  5. 5Nebius · Nebius GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)60.6154 Samples/s · 4× NVIDIA GB300source
  6. 6NVIDIA · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)43.7662 Samples/s · 4× NVIDIA GB200source
  7. 7Google · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)42.4736 Samples/s · 4× NVIDIA GB200source
  8. 8RedHat · IBM Cloud gx3d-160x1792x8h200,8xH200-SXM-141GB,RHEL 9.6,vLLM CentML:mlperf-inf-mm-q3vl-v6.018.218 Samples/s · 8× NVIDIA H200-SXM-141GBsource

qwen3-vl-235b-a22b · Server · Queries/s

  1. 1RedHat · Dell B200,8xB200-SXM-180GB,RHEL 10.1,vLLM CentML:mlperf-inf-mm-q3vl-v6.067.8642 Queries/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2Nebius · Nebius B300 n1 (8x B300-SXM-270GB, TensorRT)45.1513 Queries/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Nebius · Nebius GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)43.1538 Queries/s · 4× NVIDIA GB300source
  4. 4NVIDIA · NVIDIA GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)41.6351 Queries/s · 4× NVIDIA GB300source
  5. 5GigaComputing · DLB2-CB341.6342 Queries/s · 4× NVIDIA GB300source
  6. 6NVIDIA · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)38.0313 Queries/s · 4× NVIDIA GB200source
  7. 7Google · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)35.5625 Queries/s · 4× NVIDIA GB200source
  8. 8RedHat · IBM Cloud gx3d-160x1792x8h200,8xH200-SXM-141GB,RHEL 9.6,vLLM CentML:mlperf-inf-mm-q3vl-v6.011.0498 Queries/s · 8× NVIDIA H200-SXM-141GBsource

wan-2.2-t2v-a14b · Offline · Samples/s

  1. 1NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)0.0585234 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  2. 2GigaComputing · G894-SD3-AAX70.0582849 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  3. 3Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)0.0573123 Samples/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)0.0464402 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  5. 5NVIDIA · NVIDIA GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)0.036622 Samples/s · 4× NVIDIA GB300source
  6. 6NVIDIA · NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)0.0269817 Samples/s · 4× NVIDIA GB200source
  7. 7Krai · HPE Cray XD670 (8x H200-SXM-141GB, KISS-V Fast Mode)0.0241194 Samples/s · 8× NVIDIA H200-SXM-141GBsource
  8. 8Krai · HPE Cray XD670 (8x H200-SXM-141GB, KISS-V)0.0181337 Samples/s · 8× NVIDIA H200-SXM-141GBsource

llama3.1-405b · Server · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)18,628 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)14,134 Tokens/s · 72× NVIDIA GB200source
  3. 3Quanta_Cloud_Technology · QuantaGrid D75H-10U (8x B300-SXM-270GB, TensorRT)1,484 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)1,460 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  5. 5Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)1,446 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  6. 6GigaComputing · G894-SD3-AAX71,341 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  7. 7Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)984 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

mixtral-8x7b · Offline · Tokens/s

  1. 1HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)126,468 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)41,242 Tokens/s · 10× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  3. 3HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)33,637 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  4. 4HPE · HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition-450W, TensorRT)14,828 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  5. 5HPE · HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)8,198 Tokens/s · 2× NVIDIA RTX PRO 6000 Blackwell Server Editionsource

mixtral-8x7b · Server · Tokens/s

  1. 1HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)127,209 Tokens/s · 8× NVIDIA B200-SXM-180GBsource
  2. 2HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)39,234 Tokens/s · 10× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  3. 3HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)32,151 Tokens/s · 8× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  4. 4HPE · HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition-450W, TensorRT)13,846 Tokens/s · 4× NVIDIA RTX PRO 6000 Blackwell Server Editionsource
  5. 5HPE · HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)7,743 Tokens/s · 2× NVIDIA RTX PRO 6000 Blackwell Server Editionsource

dlrm-v3 · Offline · Samples/s

  1. 1NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)104,637 Samples/s · 72× NVIDIA GB200source
  2. 2Netweb_technologies · Camarero PDI200A2HG-810 (8x B200-SXM-180GB, TensorRT)10,737 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)10,547 Samples/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)10,447 Samples/s · 8× NVIDIA B200-SXM-180GBsource

dlrm-v3 · Server · Queries/s

  1. 1NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)99,997 Queries/s · 72× NVIDIA GB200source
  2. 2Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)10,007 Queries/s · 8× NVIDIA B200-SXM-180GBsource
  3. 3NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)10,007 Queries/s · 8× NVIDIA B200-SXM-180GBsource
  4. 4Netweb_technologies · Camarero PDI200A2HG-810 (8x B200-SXM-180GB, TensorRT)10,007 Queries/s · 8× NVIDIA B200-SXM-180GBsource

gpt-oss-120b · Interactive · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)677,199 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)624,929 Tokens/s · 72× NVIDIA GB200source
  3. 3ASUSTeK · XA NB3I-E1226,006 Tokens/s · 8× NVIDIA B300-SXM-270GBsource
  4. 4Nebius · Nebius B200 n1 (8x B200-SXM-180GB, TensorRT)13,155 Tokens/s · 8× NVIDIA B200-SXM-180GBsource

llama3_1-8b · Offline · Tokens/s

  1. 1Intel · 1-node-4x-BMG-Pro-B60-Dual6,762 Tokens/s · 4× MS-Intel Arc Pro B60 Dual 48G Turbosource
  2. 2Intel · 1-node-4x-BMG-B704,617 Tokens/s · 4× Intel(R) Arc Pro(R) B70source
  3. 3Intel · 1-node-4x-BMG-B603,347 Tokens/s · 4× Intel(R) Arc Pro(R) B60source
  4. 4Intel · 1-node-4x-BMG-B501,722 Tokens/s · 4× Intel(R) Arc Pro(R) B50source

llama3_1-8b · Server · Tokens/s

  1. 1Intel · 1-node-4x-BMG-Pro-B60-Dual6,294 Tokens/s · 4× MS-Intel Arc Pro B60 Dual 48G Turbosource
  2. 2Intel · 1-node-4x-BMG-B704,170 Tokens/s · 4× Intel(R) Arc Pro(R) B70source
  3. 3Intel · 1-node-4x-BMG-B603,145 Tokens/s · 4× Intel(R) Arc Pro(R) B60source
  4. 4Intel · 1-node-4x-BMG-B501,187 Tokens/s · 4× Intel(R) Arc Pro(R) B50source

llama3.1-405b · Interactive · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)18,365 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)14,010 Tokens/s · 72× NVIDIA GB200source
  3. 3AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)794 Tokens/s · 8× AMD Instinct MI355X 288GB HBM3esource
  4. 4GigaComputing · G894-SD3-AAX7765 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

yolo-95 · MultiStream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)4,891 Latency (ms) · 1× NVIDIA GB10source
  2. 2GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH05344 Latency (ms) · 1× NVIDIA GeForce GTX 1650 Tisource
  3. 3Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)81.093026 Latency (ms) · 16× NVIDIA L4-PCIe-24GBsource
  4. 4Dell · PowerEdge XE7745 (8x L40S, TensorRT)54.674812 Latency (ms) · 8× NVIDIA L40Ssource

yolo-95 · Offline · Samples/s

  1. 1Dell · PowerEdge XE7745 (8x L40S, TensorRT)154 Samples/s · 8× NVIDIA L40Ssource
  2. 2Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)102 Samples/s · 16× NVIDIA L4-PCIe-24GBsource
  3. 3GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH0524.187 Samples/s · 1× NVIDIA GeForce GTX 1650 Tisource
  4. 4Dell · Dell Pro Max (1x GB10)1.74887 Samples/s · 1× NVIDIA GB10source

yolo-95 · SingleStream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)612 Latency (ms) · 1× NVIDIA GB10source
  2. 2GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH0543.150031 Latency (ms) · 1× NVIDIA GeForce GTX 1650 Tisource
  3. 3Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)10.54864 Latency (ms) · 16× NVIDIA L4-PCIe-24GBsource
  4. 4Dell · PowerEdge XE7745 (8x L40S, TensorRT)7.124808 Latency (ms) · 8× NVIDIA L40Ssource

yolo-99 · MultiStream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)4,891 Latency (ms) · 1× NVIDIA GB10source
  2. 2GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH05344 Latency (ms) · 1× NVIDIA GeForce GTX 1650 Tisource
  3. 3Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)81.093026 Latency (ms) · 16× NVIDIA L4-PCIe-24GBsource
  4. 4Dell · PowerEdge XE7745 (8x L40S, TensorRT)54.674812 Latency (ms) · 8× NVIDIA L40Ssource

yolo-99 · Offline · Samples/s

  1. 1Dell · PowerEdge XE7745 (8x L40S, TensorRT)154 Samples/s · 8× NVIDIA L40Ssource
  2. 2Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)102 Samples/s · 16× NVIDIA L4-PCIe-24GBsource
  3. 3GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH0524.187 Samples/s · 1× NVIDIA GeForce GTX 1650 Tisource
  4. 4Dell · Dell Pro Max (1x GB10)1.74887 Samples/s · 1× NVIDIA GB10source

yolo-99 · SingleStream · Latency (ms)

  1. 1Dell · Dell Pro Max (1x GB10)612 Latency (ms) · 1× NVIDIA GB10source
  2. 2GATEOverflow · Lenovo IdeaPad Gaming 3 15ARH0543.150031 Latency (ms) · 1× NVIDIA GeForce GTX 1650 Tisource
  3. 3Dell · PowerEdge XE7745 (16x L4-PCIE-24GB, TensorRT)10.54864 Latency (ms) · 16× NVIDIA L4-PCIe-24GBsource
  4. 4Dell · PowerEdge XE7745 (8x L40S, TensorRT)7.124808 Latency (ms) · 8× NVIDIA L40Ssource

deepseek-r1 · Interactive · Tokens/s

  1. 1NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)250,634 Tokens/s · 72× NVIDIA GB300source
  2. 2NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)240,318 Tokens/s · 72× NVIDIA GB200source
  3. 3GigaComputing · G894-SD3-AAX74,935 Tokens/s · 8× NVIDIA B300-SXM-270GBsource

resnet · MultiStream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL380 Gen12 (5x NVIDIA RTX PRO 4500 Blackwell, TensorRT)11.077851 Latency (ms) · 5× NVIDIA RTX PRO 4500 Blackwellsource
  2. 2HPE · HPE ProLiant DL145 Gen11 (1x NVIDIA RTX PRO 4500 Blackwell, TensorRT)10.570551 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwellsource

resnet · Offline · Samples/s

  1. 1HPE · HPE ProLiant Compute DL380 Gen12 (5x NVIDIA RTX PRO 4500 Blackwell, TensorRT)94,651 Samples/s · 5× NVIDIA RTX PRO 4500 Blackwellsource
  2. 2HPE · HPE ProLiant DL145 Gen11 (1x NVIDIA RTX PRO 4500 Blackwell, TensorRT)19,211 Samples/s · 1× NVIDIA RTX PRO 4500 Blackwellsource

resnet · SingleStream · Latency (ms)

  1. 1HPE · HPE ProLiant Compute DL380 Gen12 (5x NVIDIA RTX PRO 4500 Blackwell, TensorRT)10.991706 Latency (ms) · 5× NVIDIA RTX PRO 4500 Blackwellsource
  2. 2HPE · HPE ProLiant DL145 Gen11 (1x NVIDIA RTX PRO 4500 Blackwell, TensorRT)10.561798 Latency (ms) · 1× NVIDIA RTX PRO 4500 Blackwellsource

resnet50 · MultiStream · Latency (ms)

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT0.80682 Latency (ms) · 2× NVIDIA L4-PCIe-24GBsource
  2. 2GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)0.399625 Latency (ms) · 2× NVIDIA GeForce RTX 4090source

resnet50 · Offline · Samples/s

  1. 1GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)87,946 Samples/s · 2× NVIDIA GeForce RTX 4090source
  2. 2Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT25,201 Samples/s · 2× NVIDIA L4-PCIe-24GBsource

resnet50 · SingleStream · Latency (ms)

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT0.400845 Latency (ms) · 2× NVIDIA L4-PCIe-24GBsource
  2. 2GATEOverflow · GATE Overflow Intel Sapphire Rapids (2x RTX 4090)0.303116 Latency (ms) · 2× NVIDIA GeForce RTX 4090source

3d-unet-99 · Offline · Samples/s

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT2.18794 Samples/s · 2× NVIDIA L4-PCIe-24GBsource

3d-unet-99 · SingleStream · Latency (ms)

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT1,881 Latency (ms) · 2× NVIDIA L4-PCIe-24GBsource

3d-unet-99.9 · Offline · Samples/s

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT2.18794 Samples/s · 2× NVIDIA L4-PCIe-24GBsource

3d-unet-99.9 · SingleStream · Latency (ms)

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT1,881 Latency (ms) · 2× NVIDIA L4-PCIe-24GBsource

bert-99 · Offline · Samples/s

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT1,521 Samples/s · 2× NVIDIA L4-PCIe-24GBsource

bert-99 · SingleStream · Latency (ms)

  1. 1Lenovo · ThinkEdge SE455i V3(2xL4, TensorRT2.166524 Latency (ms) · 2× NVIDIA L4-PCIe-24GBsource

llama3_1-70b-fp8_dyn · Offline · Tokens/s

  1. 1Krai · HPE Cray XD670 (8x H200-SXM-141GB, vLLM)25,940 Tokens/s · 8× NVIDIA H200-SXM-141GBsource

llama3_1-70b-fp8_dyn · Server · Tokens/s

  1. 1Krai · HPE Cray XD670 (8x H200-SXM-141GB, vLLM)24,822 Tokens/s · 8× NVIDIA H200-SXM-141GBsource

llama3.1-8b-edge · Offline · Tokens/s

  1. 1NVIDIA · NVIDIA Jetson AGX Thor Developer Kit 128G (TensorRT)351 Tokens/s · 1× NVIDIA Jetson AGX Thor 128Gsource

llama3.1-8b-edge · SingleStream · Latency (ms)

  1. 1NVIDIA · NVIDIA Jetson AGX Thor Developer Kit 128G (TensorRT)1,358 Latency (ms) · 1× NVIDIA Jetson AGX Thor 128Gsource