AMD's MI355X GPU delivers inference throughput that is up to 2.75 times cheaper per GPU than NVIDIA's B200 while offering comparable hardware specifications.
AMD's MI355X GPU delivers inference throughput that is up to 2.75 times cheaper per GPU than NVIDIA's B200 while offering comparable hardware specifications. The growing demand for tokens in frontier models such as Claude Fable, GLM5.2, and Minimax M3 has driven up NVIDIA prices and token costs. Wafer AI reported that the MI355X achieves aggregate throughput of 2626 tokens per second per node on a workload with 20,000 input tokens and a 60% cache hit rate, representing 80% of the performance measured on a B200 at a fraction of the cost. Single‑stream tests on GLM5.2 using 10,000 input tokens and 1,500 output tokens achieved 213 tokens per second on AMD hardware, according to standards set by Artificial Analysis. The study used quantization of GLM‑5.2 from bf16 to MXFP4 with AMD Quark, a method that proved lossless compared with official FP8 quantization. To improve performance, the team selected sglang as the inference engine, because it supported the quantized weights and avoided compatibility issues observed with vLLM and ATOM. Speculative decode was enabled after two code modifications: aligning the MTP head's module prefix with the decoder stack and adding a guard for deep speculative decode on the ROCm image. These changes increased single‑stream decode throughput to 213 tokens per second. For aggregate throughput, workload characteristics showed the MI355X was prefill bound, limiting performance relative to the B200, which reached 3192 tokens per second at 3.0 requests per second. The study noted that software support, rather than raw silicon, has become the primary factor affecting AMD's competitiveness, as the CUDA moat erodes. While the MI355X does not yet match NVIDIA's peak performance, the measured performance per dollar demonstrates a viable, cost‑effective alternative for inference workloads. Continued optimization of kernel selection and framework support is expected to narrow the performance gap.
- Publisher
- Hacker News
- Reliability
- high
- Published
- 7/4/2026, 10:00:21 AM
- Retrieved
- 7/4/2026, 10:00:21 AM
- Relevance
- 80%
- Confidence
- 85%

