Hugging Face Model Integration Contracts: tiny CPU fixtures caught the UMT5 attention bug that lived in Transformers main for two months

Hugging Face has proposed a way to catch a class of bugs that slip into the Transformers library between its daily GPU test runs: tiny, pinned stand-ins for real models that run on cheap CPU machines on every pull request. In a post published on 3 November 2026, Tarek Ziadé describes the idea, called Model Integration Contracts (MICs, packaged as transformers-mic), and the regression that motivated it: a UMT5 bug that stayed in the main branch for almost two months because nothing crashed. The tooling is installable today, but the Transformers CI job that uses it reports failures without blocking pull requests, and the team says it has not yet measured how well it works on live PRs.

Key points

• On 30 July 2026 Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5; as a side effect, UMT5's decoder could attend to future tokens during training. The fix landed on 28 September (Hugging Face post).
• A "Model Integration Contract" has three parts: a shrunk fixture model, a runnable usage path in Markdown, and human-reviewed baseline outputs.
• Replaying five months of commits, a small UMT5 fixture on CPU flagged the regression on every commit from 30 July to 28 September; the experimental fixture is 66 MB against a 1,227 MB source checkpoint.
• As of 1 November 2026 the MIC job in Transformers CI reports failures but does not block PRs; false-alarm rate, review workload and PR catch rate are unmeasured.
• Install with pip install "transformers-mic[runtime]" and run tmic run org/model to draft a contract locally.

What happened

The post explains a testing gap. Transformers supports more than 750 model types, with over 88,000 CPU tests on main and over 200,000 GPU tests in a daily run, plus tests for pipelines, training, distributed execution, quantisation and attention kernels. The integration tests that load real checkpoints on real hardware are the expensive ones, and running them on every pull request would keep many GPUs busy all day. So a regression can land in main and wait for the scheduled run.

The UMT5 case shows the problem. When SDPA was switched on for T5 on 30 July, UMT5's decoder could look at tokens that come after the one it is predicting. In training, that means the model can peek at the answer. "Nothing crashed," the post says, and the bug stayed for almost two months. A forward pass would have succeeded, so a test that only loads a model and runs it would have passed.

Hugging Face's answer is to shrink the model instead of skipping the test. Each contract uses a fixture built from the real checkpoint's configuration, with fewer and narrower layers but every kind of layer the model has, and with tokenizer and processor files unchanged. The fixture is published once, pinned and reused.

Background

Library maintainers have long used tiny random models to test code paths; the post notes that Transformers already has tiny-random-* checkpoints. What the team wanted was a shared recipe and checks that go beyond load-and-forward-pass.

The attention change itself is an ordinary optimisation. PyTorch's scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, ...) takes a mask or a causal flag as arguments, and is_causal defaults to False per the PyTorch documentation. The post does not say exactly which argument was wrong in UMT5, only that its decoder could attend to future tokens, and we do not speculate beyond that.

How it works

A contract has three parts, as the post describes them.

1. Fixture. The shrunk model, published once and pinned.
2. Usage path. A runnable Markdown example of something people do with the model, such as generating text or running a training step.
3. Baselines. The outputs recorded when the usage path runs, reviewed by a human.

The contract holds if running the usage path on the fixture still reproduces the baselines. A contract covers an architecture and a usage path, not a single checkpoint: compatible checkpoints can share a fixture while keeping their own expected results. The contracts are published under huggingface-mic, a fleet dataset on the Hub pins each one to a frozen revision together with the package versions used to record its baselines, and each CI run records which fleet revision it used. On a PR, contracts are selected by the files the patch changes.

The SmolLM contract is the worked example. It generates from prompts such as "The capital of France is" and compares token IDs with a recorded answer, and it also compares training loss, checks that gradients are finite and verifies that an optimiser step changes the weights.

The shrink recipe has four steps:

1. Shrink the configuration (hidden sizes, heads, depth) but keep layer kinds, including mixes of sliding and full attention or dense and mixture-of-experts layers. Sizes can be borrowed from the model's own ModelTester in its Transformers test file.
2. Keep preprocessing unchanged: same vocabulary, special tokens and input dimensions, with tokenizer and processor files copied byte for byte.
3. Build and validate weights. Try seeded random weights and change the inputs to check the model responds; if not, train it on the contract's small batch of examples. Reload and check outputs, loss and gradients are finite, and reject failed builds.
4. Freeze and record baselines on a known-good Transformers build, per platform and precision, then publish a pinned snapshot. CI reuses it and does not rebuild per PR.

The team also keeps the original weights file format so the conversion code that runs on a real load is still exercised.

Trying it yourself. The post gives these commands:

pip install "transformers-mic[runtime]" "hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''

Replace org/model with your checkpoint. The tool drafts the contract, builds the fixture and records baselines on your machine, asking for review at each step, and stops before uploading anything. For a new model an LLM drafts the usage code through Inference Providers, billed to your account. In the post's words: "The LLM drafts; it doesn't approve." Changed expectations need human review.

The numbers

Measurement	Result (Hugging Face post)
UMT5 regression window	30 July to 28 September 2026
Label log-probability difference vs baseline	1.22
Training loss difference	about 0.20
CPU runner step for selected T5 contract	13.4 seconds, excluding setup
UMT5 fixture size	66 MB (source 1,227 MB)
Llama 4 Scout shrink	108.6 billion to 44.7 million parameters

Source checkpoint1227

Fixture66

UMT5 weights, experimental fixture versus source checkpoint (MB)

	MB
Source checkpoint	1227
Fixture	66

Source: Hugging Face, Transformers Model Integration Contracts (3 November 2026)

The Llama 4 Scout report in the post shows the vision stack going from 34 layers to 2 and the language stack from 48 to 4, keeping layer kinds and mixture-of-experts routing style. The team notes this also covers models too large for its standard GPU runners.

One detail in the UMT5 replay matters for anyone writing similar tests: the contract used SDPA, the default attention path. Running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code path as the real checkpoint. The team also reverted the fix on a proof branch in Transformers CI, and the check failed with the same differences.

Random weights versus overfit weights. A randomly initialised Whisper fixture barely responded to changes in the audio, so the team trained it on a few tones paired with words such as "red" and "green". In their weight-initialisation study, random Whisper fixtures caught 18 of 24 injected faults and overfit fixtures caught all 24. The faults were synthetic, and both versions missed an assisted-decoding bug that the real Whisper model shows. The team keeps random weights when they pass the checks and overfits when it needs more sensitivity.

What it means

For Transformers contributors. Cheap fixtures give pull requests a signal that previously only arrived after the daily GPU run. The UMT5 replay suggests that a bug that does not crash anything, which ordinary smoke tests miss, can still show up as a shift in log-probabilities and loss.

For anyone maintaining a model library. The idea transfers: keep every layer type, leave preprocessing and file formats alone, check training behaviour as well as forward passes, and pin fixtures so CI is reproducible. The authors also say they used agents to build contracts across model families, and when an edge case appeared they fixed the recipe, not the single fixture.

For model authors. The team asks for contributions: pick an uncovered model from the public coverage space, try tmic, and contribute a usage path or review a generated contract. The coverage space page returned an access error when we tried to open it without signing in, so we cannot say how much is covered.

Caution. The evidence so far is replays and a proof branch. It is not a measurement of how many real bugs this catches on live PRs.

What we don't know yet

• Whether the MIC job should block PRs. The post says that depends on three unmeasured things: real bugs caught on PRs (not just replays), false alarms, and baseline-review effort.
• How many bugs only appear with real weights, as the Whisper assisted-decoding bug did, and whether overfit fixtures catch real bugs that random ones miss. The team says it has not found a real regression caught only by the overfit version.
• Coverage. The post says it is partial across model families and APIs, and existing tests stay until their coverage is accounted for.
• The exact root cause of the UMT5 bug in code terms, which the post describes only by its effect.

What to watch next

The team plans to link original Hub model pages to their contracts, so a developer or an agent using a checkpoint can start from an example tested on that checkpoint with its environment and expected outputs. Watch the Transformers CI for a change from reporting to blocking, and for measured false-alarm and catch rates.

Sources

• Transformers Model Integration Contracts (Tarek Ziadé, Hugging Face blog, 3 November 2026), primary source
• huggingface-mic/transformers-fleet dataset (Hugging Face)
• huggingface-mic/smollm-mic model page (Hugging Face)
• transformers-mic on PyPI
• PyTorch documentation: scaled_dot_product_attention