OpenAI Publishes New Evaluations for Coding in Low-Resource Languages Like COBOL

OpenAI published a post describing how its latest models handle low-resource programming languages such as COBOL, Fortran and Visual Basic - languages with far less public code and documentation than Python or JavaScript, yet still underpinning critical infrastructure and essential services.

The company framed the work around real demand rather than benchmark prestige. COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, while Fortran is used in weather forecasting and nuclear safety analysis. OpenAI said the focus came from developers and enterprise customers who reported that coding models often struggle with these languages, in situations where documentation is incomplete, original authors have left and business rules survive only inside the code.

To measure progress, OpenAI built two evaluations. MultiLangBench asks a model to construct a SQLite-compatible database engine from scratch in 20 different programming languages, testing codebase organization, state management, error handling and feature work. The COBOL migration evaluation asks a model to move existing COBOL programs into C++, Java, Kotlin, Python or Rust while preserving behaviour, judged by hidden tests that require byte-for-byte parity on observable outputs.

The MultiLangBench results show a large jump. GPT‑6 Astra reaches an average test pass rate of 87.4% across the 20 languages, up from 26.8% for GPT‑5.6 Sol, with gains in every language tested. COBOL rises from 10.9% to 72.1%, OCaml to 85.8%, Fortran to 83.5% and Visual Basic to 89.4%. Astra clears 83% in 19 of the 20 languages, with COBOL still the hardest at roughly 72%.

Cost matters as much as accuracy here. GPT‑6 Luna, positioned as the cheaper option, averages 40.3% but at an estimated $0.77 per sample versus $3.48 for GPT‑5.6 Sol, roughly 78% lower, and it still beats the older model in every language tested.

The migration evaluation produced the sharpest contrast. GPT‑6.1 Sol scores 86.6% at low reasoning effort against 5.4% for GPT‑5.6 Sol, while costing 39% less; at higher reasoning effort accuracy climbs to 97.9%, nearly saturating the test. OpenAI explained why the task is hard: COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested, so a rewrite can run cleanly and still produce different financial results.

OpenAI also released the MultiLangBench setup, test suite and per-language results on GitHub, so developers can inspect what the evaluation actually measures and where their own language of interest stands.

The open question is verification. Whether outside teams can reproduce these numbers, and how enterprises translate byte-level parity tests into the far more expensive regression and compliance checks of a real migration, will decide how much of this shows up in production modernisation work. Coverage of further low-resource languages is the other signal to watch.