|0°C
Next edition scheduled
Your Personal Daily Intelligence
Edition 2026-07-04

TECHNOLOGY

AMD MI355X Achieves Higher Tokens per Dollar than NVIDIA B200 in Inference Tests

AMD's MI355X GPU delivers inference throughput that is up to 2.75 times cheaper per GPU than NVIDIA's B200 while offering comparable hardware specifications.

By Hacker News · 112d ago · Source: Hacker News

Full article

AMD's MI355X GPU delivers inference throughput that is up to 2.75 times cheaper per GPU than NVIDIA's B200 while offering comparable hardware specifications. The growing demand for tokens in frontier models such as Claude Fable, GLM5.2, and Minimax M3 has driven up NVIDIA prices and token costs. Wafer AI reported that the MI355X achieves aggregate throughput of 2626 tokens per second per node on a workload with 20,000 input tokens and a 60% cache hit rate, representing 80% of the performance measured on a B200 at a fraction of the cost. Single‑stream tests on GLM5.2 using 10,000 input tokens and 1,500 output tokens achieved 213 tokens per second on AMD hardware, according to standards set by Artificial Analysis. The study used quantization of GLM‑5.2 from bf16 to MXFP4 with AMD Quark, a method that proved lossless compared with official FP8 quantization. To improve performance, the team selected sglang as the inference engine, because it supported the quantized weights and avoided compatibility issues observed with vLLM and ATOM. Speculative decode was enabled after two code modifications: aligning the MTP head's module prefix with the decoder stack and adding a guard for deep speculative decode on the ROCm image. These changes increased single‑stream decode throughput to 213 tokens per second. For aggregate throughput, workload characteristics showed the MI355X was prefill bound, limiting performance relative to the B200, which reached 3192 tokens per second at 3.0 requests per second. The study noted that software support, rather than raw silicon, has become the primary factor affecting AMD's competitiveness, as the CUDA moat erodes. While the MI355X does not yet match NVIDIA's peak performance, the measured performance per dollar demonstrates a viable, cost‑effective alternative for inference workloads. Continued optimization of kernel selection and framework support is expected to narrow the performance gap.

Source transparency

Publisher
Hacker News
Reliability
high
Published
7/4/2026, 10:00:21 AM
Retrieved
7/4/2026, 10:00:21 AM
Relevance
80%
Confidence
85%
Read original at Hacker News

Botwin's Morning Wire publishes the full source article for reading convenience. Please visit the publisher for the original presentation and any updates.