A new evaluation framework called StarSkirmish measures how effectively large language models can design autonomous StarCraft: Brood War bots under time constraints.
A new evaluation framework called StarSkirmish measures how effectively large language models can design autonomous StarCraft: Brood War bots under time constraints. The latest version, v0.2, assesses flagship models from OpenAI, Anthropic, Google and Meta using a standardized one‑hour coding challenge. Each model receives a skeleton Protoss bot written in C++ and access to practice matches, then must produce a functional bot that competes against a fixed set of human‑crafted opponents. The benchmark evaluates long‑term reasoning and iterative learning, as successful bots must incorporate insights gathered during the hour‑long session. Scores are reported as average win rates across five independent runs, ranging from 0 to 100. Gemini 4 Argon achieved the highest score, followed closely by Claude Opus 5.5 and Claude Fable 5.1, while GPT‑6 Astra ranked second. Human‑written bots such as Stardust and PurpleWave continue to outperform all tested models, with the best AI achieving only a 7 % win rate against them. Cost efficiency varied widely, with GPT‑6.1 Sol offering the most favorable balance between performance and expense. The test also revealed significant run‑to‑run variability, particularly for Gemini 4 Argon, whose results spanned an 18‑point range. These findings highlight both progress and persistent gaps in AI systems’ ability to generate complex, strategic code under real‑world constraints.
- Publisher
- Hacker News
- Reliability
- high
- Published
- 10/17/2026, 1:11:56 PM
- Retrieved
- 10/17/2026, 1:11:56 PM
- Relevance
- 80%
- Confidence
- 85%

