Hugging Face Proposes Model Integration Contracts to Catch Transformers Regressions Before Merge

Hugging Face published a proposal on November 3 for Model Integration Contracts (MICs), a standard way to package regression tests for the Transformers library around deliberately shrunk models instead of full checkpoints. The post, written by Tarek Ziadé, ships alongside the transformers-mic package, a huggingface-mic collection of published contracts and a tmic command-line tool that drafts contracts and builds fixtures.

The motivation is scale. Transformers supports more than 750 model types, and its test suite runs 88,000-plus CPU tests on main with another 200,000-plus GPU tests in a daily pass, covering pipelines, training, distributed execution, quantization and attention kernels. Loading real checkpoints on real hardware for every pull request would keep a large fleet of GPUs busy all day, so integration tests are scheduled rather than blocking, and by the time a scheduled run finds a regression it may already be in main.

The post's central example is a T5/UMT5 flaw. On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention for T5; as a side effect, UMT5's decoder could attend to future tokens and, during training, peek at the answer it was supposed to predict. Nothing crashed and the bug stayed in main for almost two months, until a fix landed on September 28. Replaying five months of commits against a small UMT5 fixture running on CPU flagged the regression on every one of them, with label log-probabilities off the reviewed baseline by 1.22 and training loss by about 0.20.

One detail decided whether the check worked: the contract exercised SDPA, the default attention path. Running only the eager attention variant missed the bug entirely, which the authors use to argue that a fixture only helps if it goes through the same code as the real checkpoint. The UMT5 fixture carries 66 MB of weights against 1,227 MB for the source checkpoint, and the selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.

Fixtures are built with a four-step recipe: shrink the configuration while keeping every layer kind, including mixtures of sliding and full attention or dense and MoE layers; leave tokenizers and processors byte-for-byte unchanged; build and validate weights, falling back to brief training if seeded random weights do not respond; then freeze the result and record baselines for each supported platform and precision. The published Llama 4 Scout shrink report shows the effect at scale, taking a 108.6-billion-parameter model down to 44.7 million, with the vision stack cut from 34 layers to 2 and the language stack from 48 to 4.

A contract has three parts: a pinned fixture, a runnable Markdown usage path such as generating text or running a training step, and the outputs recorded from that example and reviewed by a human. It covers an architecture and a usage path rather than a single checkpoint, so compatible checkpoints can share a fixture while keeping their own expected results. The SmolLM contract, for instance, generates from prompts such as “The capital of France is”, compares token IDs with a recorded answer, checks that training loss matches, that gradients are finite and that an optimizer step changes the weights. A fleet dataset on the Hub pins each contract to a frozen revision along with the package versions used to record its baselines.

The choice of weights is still an open engineering question. With random weights the Whisper fixture barely responded to changes in the audio, so the team trained it on a few tones paired with words such as “red” and “green”. In a fault-injection study, random fixtures caught 18 of 24 injected faults while overfit fixtures caught all 24, but the authors note that no real regression has yet been found that only the overfit version catches, and both variants missed an assisted-decoding bug that the real Whisper checkpoint exposed.

As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking pull requests. Whether it should block depends on unmeasured questions: how many real bugs it catches on PRs rather than in replays, how often it raises false alarms, and how much work baseline review adds. Coverage across model families and APIs is still partial, and the team wants to link original Hub model pages to their contracts so a developer or an agent starting from a checkpoint can begin with an example tested on that checkpoint, its environment and its expected outputs. In effect, the proposal turns regression tests into versioned artifacts that any developer or agent can rerun, not just an internal CI job.