OpenAI reports 87% average pass rate across 20 coding languages on new MultiLangBench

OpenAI has published results from MultiLangBench, a new evaluation built to test whether its models can produce working software across different programming languages. According to EdTech Innovation Hub's report on October 27, its newest model, GPT-6 Astra, reached an average test pass rate of 87.4% across 20 languages, up from 26.8% for GPT-5.6 Sol.

The benchmark deliberately avoids small, exercise-style coding problems. Models must build a SQLite-compatible database engine from scratch in a specified language — organizing code, managing data, handling errors and adding functionality — and the resulting database files are then compared against OpenAI's reference implementation.

At maximum reasoning effort, GPT-6 Astra averaged 87.4% and exceeded 83% in 19 of the 20 languages, with COBOL remaining the hardest case. Gains were recorded in every language tested, and the biggest jumps came in the languages where the previous model had struggled most.

OpenAI also tested a cheaper option, GPT-6 Luna, which averaged 40.3% — still well above GPT-5.6 Sol's 26.8% — at an estimated average API cost of $0.77 per sample versus $3.48, a reduction of roughly 78%. That trade-off between capability and cost is exactly what enterprise coding teams are trying to price.

A second evaluation posed a different question: can a model translate existing COBOL programs into C++, Java, Kotlin, Python or Rust without changing behaviour? The bar is not code that runs but byte-for-byte identical observable output, and a migration counts only if every hidden test passes. GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, while cutting estimated API cost by 39%; at higher reasoning effort it reached 97.9%.

These languages warrant their own benchmark because far less public code and documentation exists for them than for Python or JavaScript. Figures cited by OpenAI put COBOL behind more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions, while Fortran is still used in weather forecasting and nuclear reactor safety analysis — often with incomplete documentation and business rules that exist only inside the code.

OpenAI says it will release the MultiLangBench setup, test suite and per-language results on GitHub so developers can inspect the tests themselves. The company also acknowledges that these are results on its own evaluation, not proof of success in production banking or other critical systems, and that real migrations still require expert review and testing against existing business rules and dependencies.

For developers, the more interesting signal is not the 87% headline but the fact that obscure languages and legacy modernization now have a public, reproducible yardstick. The next thing to watch is how other models score on the same tests, and how much of a real migration project enterprises are willing to hand over to a model.