COBOL Migration at 97.9% Accuracy? Why Banks Should Not Trust OpenAI's Number Alone

OpenAI says GPT-6.1 Sol reaches 97.9% accuracy on its COBOL migration evaluation at high reasoning effort. That is a major improvement on the same company’s earlier models, but it is not permission to convert a bank’s production system and trust the generated code.

The evaluation is deliberately demanding. A model must migrate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving observable behavior. Hidden tests require byte-for-byte parity, and a migration counts as successful only when every test passes.

That makes the result more useful than a subjective coding demo. It still measures a bounded benchmark designed and reported by OpenAI. A production mainframe includes undocumented business rules, batch schedules, external files, database behavior, operational procedures and decades of exceptions that no benchmark can automatically capture.

What OpenAI measured

OpenAI published two related evaluations for low-resource programming languages. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 languages. A separate COBOL migration evaluation tests behavior-preserving translation of existing programs.

On MultiLangBench, GPT-6 Astra averaged an 87.4% test pass rate across the 20 languages, up from 26.8% for GPT-5.6 Sol. Its COBOL result was lower than the overall average at 72.1%, which is a useful reminder that the headline capability varies by language.

On the migration evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort and 97.9% at higher effort. OpenAI says the lower-effort configuration also cost less than GPT-5.6 Sol on the same task.

These results suggest that current models are better at reading unfamiliar code, preserving semantics and using tests to repair mistakes. OpenAI also released the MultiLangBench setup and results, allowing developers to inspect and reproduce that portion of the work.

Why 97.9% is not a production safety rate

Accuracy in this evaluation means the migrated benchmark program passed its hidden tests. It does not mean 97.9% of every line is correct, nor that 97.9% of real migrations will be safe.

The denominator matters. A compact benchmark can cover tricky language behavior while still omitting the scale and integration density of a bank, insurer or government system. Real applications interact with databases, message queues, job schedulers, proprietary file formats, transaction monitors and manual operating procedures.

More importantly, tests only protect behavior they express. A legacy system may contain rules known only to experienced employees. Some apparently strange behaviors are deliberate responses to past incidents. Others are bugs that downstream systems now expect. An AI can preserve the wrong behavior perfectly or modernize away an undocumented requirement.

OpenAI acknowledges this boundary. Its announcement says production use still requires expert review and testing against the system’s business rules and dependencies.

COBOL semantics can fail quietly

OpenAI highlights fixed-point arithmetic as an example. COBOL may discard excess decimal digits unless rounding is explicitly requested. A translation that rounds instead can compile, run and produce plausible financial values while changing the result.

That is the dangerous category of migration bug: not a crash, but a subtle semantic difference. Similar risks appear in date handling, record layouts, character encodings, file ordering, transaction boundaries and missing-value behavior.

A useful migration process therefore needs more than unit tests generated from the source. It needs golden outputs from production-like workloads, reconciliation against historical runs, domain-expert review and explicit tolerances for every financial field.

A safer way to use the capability

The strongest near-term use is not autonomous replacement. It is accelerated modernization with a controlled verification pipeline.

First, inventory the system and its dependencies before generating target code. Second, preserve a runnable reference environment. Third, build characterization tests from real inputs and outputs. Fourth, let the model propose migrations in bounded modules. Fifth, compare results at byte, record and business-metric levels. Finally, run both systems in parallel before any cutover.

Human experts should focus on the places where generated migration is least trustworthy: arithmetic, data conversion, transaction handling, error recovery and interfaces that are not represented in the code repository.

The model can reduce the cost of writing candidate code and tests. It cannot decide which behaviors the business is legally or operationally required to preserve.

The broader significance

Low-resource languages have less public code and documentation available for training than Python or JavaScript, yet they support critical infrastructure. Progress here expands the market for coding agents from greenfield development into maintenance and modernization budgets.

It also changes the economics of migration. Organizations may be able to test more modernization options, translate smaller modules earlier and document systems that were previously too expensive to analyze.

But the 97.9% number should start a proof of concept, not end a risk review. OpenAI has shown that benchmark COBOL migration is approaching saturation under its test conditions. Banks still need to prove that generated replacements preserve the rules their customers, regulators and accounting systems depend on.