Hcompany debuts NeoMME: a multilingual multimodal encoder that cuts retrieval index storage 255x

On September 3, the Hcompany team introduced NeoMME on the Hugging Face blog: a family of multilingual multimodal encoders in 260M and 800M parameter sizes. All checkpoints are released under the Apache 2.0 license with native Hugging Face Transformers support from day one, alongside a model collection and a technical report.

The architectural bet is what sets NeoMME apart. Instead of the now-common recipe of a pretrained vision tower, a projector, and a causal language model, NeoMME runs text tokens and raw image patches through one bidirectional Transformer: images are split into 32×32 patches, projected by a small MLP, and encoded in the same stack as text. The whole model is trained from scratch with a masked discrete-diffusion objective.

A single shared computation path makes dynamic resolution natural: aspect ratio and size are preserved, so an information-dense document page receives more tokens than a smaller image. Context length reaches 16,384 tokens, enough for roughly two 4K images. The stack also adopts grouped-query attention, query-key normalization, gated attention, and 2D rotary position embeddings, plus a from-scratch 131k-token BPE tokenizer trained on multilingual text, code, math, and image transcripts.

To validate the backbone on a real task, the team fine-tuned it into NeoMME-Retriever using ColPali's page-image methodology for visual document retrieval. The model ranks screenshots of document pages directly, skipping the OCR preprocessing that text-based pipelines require, and can therefore preserve layout, charts, tables, and other visual clues that even a perfect OCR model would flatten.

NeoMME-Retriever uses a dual-head design: a dense head mean-pools the backbone's hidden states into one normalized vector for compact ANN search, while a late-interaction head projects every text token and image patch into 128-dimensional vectors to keep fine-grained local matches. A single forward pass returns both representations, giving teams flexibility across corpora sizes and infrastructure.

The numbers on ViDoRe v3 back the design: the 260M model scores 0.523 nDCG@10, the highest among all evaluated models under 800M parameters and within 0.002 of ColQwen2.5 while using roughly 14x fewer parameters; the 800M model reaches 0.556, close to the similarly sized Vultron Retriever Flash, and beats ColPali v1.3 with 3.6x fewer parameters. For visual document retrieval, this means tasks that once leaned on large generative VLMs can now come within striking distance of top-tier quality using a fraction of the parameters.

Late-interaction indexes grow with vector count, and high-resolution pages make that expensive. The team combines hierarchical token pooling with asymmetric quantization: on ViDoRe v3, storage per page drops from about 1.5 MB to 39 kB (a 39x reduction) while keeping more than 99% of retrieval quality, and an aggressive setting reaches 6 kB per page — 255x smaller — while retaining over 95% of the baseline nDCG@10.

Throughput is strong too: on one NVIDIA L40S at a matched 2048×2048 input, the 260M model encodes about 51 pages per second, roughly twice ColModernVBERT's rate, which cuts GPU time and cost when indexing large corpora. The checkpoints also plug into Sentence Transformers v6 for fine-tuning, and the team published end-to-end visual RAG examples and an online demo.

NeoMME matters beyond retrieval benchmarks: it shows that a single bidirectional Transformer natively handling text and images can be practical at small scale, and it lowers the cost floor for multilingual, high-resolution retrieval. Watch for how the 800M variant behaves on larger corpora, how the compression frontier holds up in production indexes, and whether the encoder generalizes beyond retrieval to broader multimodal tasks.