Karpathy pushes The Lord of the Rings as a new LLM benchmark

Andrej Karpathy has announced that Anthropic is testing large models in a new way — with The Lord of the Rings. The "Lord of the Rings benchmark" is replacing the once-famous "pelican riding a bicycle" SVG test as a probe of spatial understanding, long-horizon planning, and code generation.

The setup feeds the opening of The Lord of the Rings to Opus 5 and asks the model to build a Middle-earth 3D world in the browser with Three.js. The effort consumed 1 million tokens, two hours, and 5,500 lines of code, and the result looks rough — something like a late-1990s domestic 3D cartoon.

Karpathy admits that Opus 5 cannot truly "enter" the world it generates; it can only take screenshots at different points and slowly check what went wrong. That exposes a clear weakness of today's models: they can write code, build scenes, and make games, but they cannot truly watch video or play the games they built.

According to Karpathy's post, the main prompt fed to Opus 5 was the opening of chapter one, "A Long-expected Party," with Three.js, the JavaScript library for building 3D scenes in code, specified as the tool. The model had to read the prose and translate it into a live 3D world — assembling characters, buildings, and props from polygons, placing them in x, y, z coordinates, and arranging cameras, lighting, and animation. This is real-time rendering of a programmatic 3D scene, not frame-by-frame video generation.

Karpathy has open-sourced the project. In the comments, he suggested a pipeline where programmatic 3D code handles storyboarding and control, and a video-to-video model like Seedance re-renders the screen recording with higher-quality textures, lighting, and detail; he used ElevenLabs for audio.

Community tests went further: one user started an "Earth Online" project to build the entire planet with a swarm of AI agents, another built a real-time-data 3D model of New York City with Fable 5 and GPT-5.6 Sol, and one built a full Kanye West concert in the browser with Three.js — plus playable, interactive game versions.

Why make models generate The Lord of the Rings at all? Because the pelican test is nearly solved: a single SVG only measures one-shot output, not the ability to plan a complex project, work for hours, and repeatedly check and fix errors across thousands of lines of code. Karpathy swapped the "draw a picture" question for a "build a world" project.

The community is split. Some argue the demo only proves Anthropic trained well on Three.js code, not that the model truly understands space and physics; others counter that turning abstract literary prose into 3D animation requires handling spatial relationships, physical rules, and the math of computer graphics — 5,500 lines of code is not trivial.

Either way, the barrier to real-time 3D content and playable prototypes has dropped significantly. Whether the "Lord of the Rings benchmark" catches on like the pelican test, and whether programmatic 3D plus video generation becomes a new evaluation dimension, is worth watching.