Meta Opens Muse Spark Open, an ~816B-Parameter Multimodal MoE Model

On October 15, Hugging Face published "Welcome Muse Spark Open," announcing Meta's open-weight Muse Spark Open and documenting how to run, fine-tune and deploy it. The model ships as two checkpoints: Muse-Spark-Open-816B-A42B in BF16 and Muse-Spark-Open-816B-A42B-NVFP4.

According to the post, Muse Spark Open is a mixture-of-experts model with roughly 816 billion total parameters and about 42 billion active per token, and a context window of up to one million tokens. It accepts text, images, audio and video and returns text, giving developers a single interface for asking questions about different kinds of content.

It is the full-size model in the Muse series. Muse Glimmer, released by Meta in August, is a small dense model from the same line built for local use, and the two share a tokenizer and vocabulary. Muse Spark Open replaces each layer's dense feed-forward network with a MoE block: 256 experts per block, with a router selecting 8 per token, and the experts hold about 97% of total parameters. The language model has 62 layers, a hidden dimension of 8192, and grouped-query attention with 128 query heads and 16 key-value heads.

The post says the model was trained for agentic coding — reading a codebase, writing and debugging code, calling tools — and that it plans, acts, observes and self-corrects over long tasks, including browsing the web with images. Because it also handles screenshots, voice notes and screen recordings, one model can follow those signals while it works on code. Hugging Face calls it the most capable US open-weights model to date.

Two encoders carry the multimodal side. A perception encoder with 1.8 billion parameters, 50 layers, a hidden dimension of 1536 and 16 attention heads splits each image into 14x14 patches and produces 4096-dimensional features, with video frames taking the same path. Audio runs through an audio tokenizer and encoder that emits one token every 80 milliseconds, or 12.5 tokens per second.

Deployment is the heavy part. The BF16 checkpoint is about 1.5 TB and runs across four nodes of eight H200 GPUs with tensor parallelism at TP=32, while the NVFP4 checkpoint stores parameters in 4-bit precision for NVIDIA Blackwell GPUs and fits on a single B200 node. The post documents inference through Transformers, llama.cpp, vLLM, SGLang and MLX, plus deployment with Inference Endpoints and fine-tuning with TRL.

All 256 experts must stay in memory, so teams provision for an 816B-parameter model but run at roughly 42B speed. For groups building their own agentic coding stack, whether that trade pays off will depend on community evaluations on real repository tasks and on how quickly NVFP4 support matures across inference frameworks.