Guozhen AIGlobal AI field notes and model intelligence

Realtime AI News

OpenAI reports 87% average pass rate across 20 coding languages on new MultiLangBench

OpenAI has published MultiLangBench results showing its GPT-6 Astra model reaching an 87.4% average test pass rate across 20 programming languages, far above the 26.8% scored by GPT-5.6 Sol. A companion evaluation tested COBOL migration into modern languages with byte-for-byte output checks, and OpenAI is releasing the test suite on GitHub.

Published

OpenAI has published results from MultiLangBench, a new evaluation built to test whether its models can produce working software across different programming languages. According to EdTech Innovation Hub's report on October 27, its newest model, GPT-6 Astra, reached an average test pass rate of 87.4% across 20 languages, up from 26.8% for GPT-5.6 Sol.

The benchmark deliberately avoids small, exercise-style coding problems. Models must build a SQLite-compatible database engine from scratch in a specified language — organizing code, managing data, handling errors and adding functionality — and the resulting database files are then compared against OpenAI's reference implementation.

At maximum reasoning effort, GPT-6 Astra averaged 87.4% and exceeded 83% in 19 of the 20 languages, with COBOL remaining the hardest case. Gains were recorded in every language tested, and the biggest jumps came in the languages where the previous model had struggled most.

OpenAI also tested a cheaper option, GPT-6 Luna, which averaged 40.3% — still well above GPT-5.6 Sol's 26.8% — at an estimated average API cost of $0.77 per sample versus $3.48, a reduction of roughly 78%. That trade-off between capability and cost is exactly what enterprise coding teams are trying to price.

A second evaluation posed a different question: can a model translate existing COBOL programs into C++, Java, Kotlin, Python or Rust without changing behaviour? The bar is not code that runs but byte-for-byte identical observable output, and a migration counts only if every hidden test passes. GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, while cutting estimated API cost by 39%; at higher reasoning effort it reached 97.9%.

These languages warrant their own benchmark because far less public code and documentation exists for them than for Python or JavaScript. Figures cited by OpenAI put COBOL behind more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions, while Fortran is still used in weather forecasting and nuclear reactor safety analysis — often with incomplete documentation and business rules that exist only inside the code.

OpenAI says it will release the MultiLangBench setup, test suite and per-language results on GitHub so developers can inspect the tests themselves. The company also acknowledges that these are results on its own evaluation, not proof of success in production banking or other critical systems, and that real migrations still require expert review and testing against existing business rules and dependencies.

For developers, the more interesting signal is not the 87% headline but the fact that obscure languages and legacy modernization now have a public, reproducible yardstick. The next thing to watch is how other models score on the same tests, and how much of a real migration project enterprises are willing to hand over to a model.

Why it matters

If independent reruns hold up, legacy modernization becomes one of the first enterprise coding use cases with a credible public yardstick, and benchmark-setting power concentrates further among frontier labs.

OpenAICodingBenchmark
Back to AI Daily

Nearby Updates

All

10/27, 10:00

OpenAI turns its coding tools into dots, an always-on personal assistant

OpenAI has turned its coding tools into dots, an always-on personal assistant, according to TechCrunch. The report centers on the executive responsible for dots, Alexander Embiricos, whose bluntest line — that he never wants to meet with someone else's AI agent — points to where the product is heading.

10/27, 08:00

Check Point uses OpenAI Daybreak to speed up vulnerability research

OpenAI has published a customer story on how Check Point uses Daybreak to accelerate its vulnerability research. The account claims analysts can turn vulnerability intelligence into customer defenses faster, though the speed-up rests on the two companies' own description.

10/27, 10:52

Tencent Hunyuan open-sources Hy-SimulMT, a speech interpreter with about 3 seconds of lag

Tencent Hunyuan has open-sourced Hy-SimulMT, a speech-to-speech interpreter that turns Chinese speech into English and English into Chinese while the speaker is still talking, cloning the speaker's voice in real time. Tencent reports about three seconds of lag on long-form speech using the StreamLAAL metric, and the model is already live in the free Hy Translate app.

10/27, 06:55

GammaDX launches agent experience practice as its index finds only 12 of 50 Australian sites agent-ready

Australian digital experience agency GammaDX has launched a dedicated agent experience design practice alongside its first Gamma AXD Index, which scored 50 Australian websites across five sectors on a 100-point scale. Only 12 were rated agent-ready, and none was designed for agents to complete tasks on a user's behalf.