ByteDance Seed open-sources a 94.2B-parameter cross-layer transcoder for Qwen3-8B-Base

The ByteDance-Seed organisation on Hugging Face has added a repository named Cross-Layer-TopK-CLT, shipping a cross-layer transcoder (CLT) for Qwen3-8B-Base. The card is functional rather than promotional: an Apache-2.0 licence, a table of tensor shapes, a runnable encode and decode demo, and a ByteDance copyright header.

According to the model card, the transcoder spans all 36 layers of the base model with 32,768 latents per layer. Each latent reads the residual stream immediately before that layer's MLP (resid_mid) but writes to the MLP outputs (mlp_out) of its own layer and every later layer, so a single feature can influence computation well beyond where it was detected. The full artifact is 94.2 billion parameters stored in fp32, 377 GB in total.

The mechanics are specified down to tensor shapes. Encoding is a bias-free ReLU over the encoder matrix; each latent is then scored by multiplying its activation by a decoder-column norm, and the 360 highest-scoring latents (36 layers times k=10) are kept jointly across the whole stack. Decoding sums every latent at or below the target layer. Because training-time normalisation is folded into the weights, the CLT consumes and reconstructs raw activations, and the factors file should not be applied again.

Hardware reality is spelled out too. The decoder alone accounts for 357 GB, split into per-layer files; the bundled demo expects eight GPUs under torchrun, while encode-only runs fit on a single 80 GB card if the decoder shards are excluded. The base model must be loaded in fp32, and tokens whose activation norm exceeds 20 times the layer mean are dropped as in training, which in practice means the first token.

The release matters less for a benchmark number than for access. Publishing a complete CLT for an open 8B model under a permissive licence means outside researchers no longer have to train their own transcoder before asking feature-level questions of the model.

Two caveats temper that. Cross-layer writes make attribution harder, because a latent's effect is spread across downstream layers rather than anchored to one, and 377 GB of fp32 weights keeps genuine use inside multi-GPU labs rather than on a laptop. At the time of writing the repository shows zero downloads and zero likes, so there is no community evidence yet on usability or on whether the features prove interpretable in practice.

What to watch next is adoption rather than the card itself: whether interpretability teams wire these weights into feature browsers, auto-labelling or safety-audit pipelines, whether Seed extends the approach to larger base models, and whether inference stacks add native support for weights this large.