Claude Code Video Animation: Six Worlds, One Brief
The whole thing started with one message typed into a terminal:
let's see what you can do with the script, the animation, the sound design and the edit. Pixel art, a first-person block world, cartoon, kawaii, stop motion and 3D.
Then I stopped touching it. Claude Opus 5.5 generated the lot as code: not a frame, a sound or a cut in the finished film came from anywhere else. What I contributed was that brief, an approval at each stage, and a running log of everything that went wrong.
The log ran to 52 entries. It is at the end.
The film itself. 6 min 53 sec, every frame generated from code. Watch on YouTube
- 0:00The brief
- 0:36How a video can be code
- 1:33World 1: pixel art
- 2:19World 2: blocks
- 2:56World 3: cartoon
- 3:34World 4: kawaii
- 4:11World 5: cut paper
- 4:49World 6: 3D
- 5:22Sound and the edit
- 6:04The scorecard
- Claude Code can produce animation, by writing each scene as a web page and having a renderer photograph it frame by frame.
- It never draws a frame. It writes the rule that positions things over time, and the machine derives all 12,390 of them.
- The music and sound effects are code as well, synthesised from note data and filtered noise rather than pulled from a library.
- The edit is a text file. Change a sentence and every scene, card and caption shifts to match.
- It caught 47 of its own mistakes. I caught 5, and all five of mine were about how the film sounded or flowed.
Can Claude Code actually make video animation?
Yes, though not in the way the question implies. It cannot draw, and it cannot watch video. It writes programs that draw, then inspects still images of what those programs produced.
A browser page can render more or less anything, from a rectangle to a lit 3D scene. So each scene here is a web page, written in HTML, Canvas, SVG or Three.js depending on what the style needed. A tool called HyperFrames, HeyGen's open-source renderer, loads that page and captures it one frame at a time at 30 frames per second.
Nothing is hand-animated. Position, rotation and colour are all functions of elapsed time, so a foot at frame 400 is wherever the maths says it should be. Claude writes the rule once and the renderer resolves every frame from it.
The checking loop is the part I found most revealing. Since it cannot play back a video, it renders a handful of stills after each change and reads them as images, roughly the way you glance at a photo before sending it. When something looks wrong in a still, it goes back into the code. That loop is the entire reason it caught most of its own errors.
The six worlds
A code-drawn version of me, orange spikes and black sweater, has to survive six visual styles that share no assets and barely share a rendering approach.
| World | Style | Built with | What broke on the first render |
|---|---|---|---|
| 1. Pixel art | 1980s console game, 320 pixels wide, scaled up with no smoothing | Canvas, plus square-wave music from a note list | The character rendered 19 pixels tall, too small to carry a scene |
| 2. Blocks | First-person, voxel terrain | Seeded noise for the landscape | The walk path went through two trees, so eight seconds were all leaves |
| 3. Cartoon | Squash, stretch and anticipation | Damped oscillation for follow-through | Speed lines sat too low and made him look like he was on stilts |
| 4. Kawaii | Round, pastel, objects with faces | Soft corners, settling bounce, no hard shadows | Softening the shapes turned his spikes into a mop and lost the likeness |
| 5. Cut paper | Torn edges, brass pins, drifting outlines | SVG with a per-point wobble on every edge | A waving arm swung behind his head and vanished |
| 6. 3D | Real lights, shadows and a moving camera | Three.js in the same web page | It crashed before drawing a frame; a later pass gave ten seconds of a black rocket against a black sky |
Three of those are worth pulling out.
The pixel world's soundtrack is arithmetic. Claude wrote a melody as note data, then a few lines of maths turned those notes into square waves, which is how consoles of that era actually produced sound. The canvas is 320 pixels across because that is what a Mega Drive put out in 1988, and it is enlarged without interpolation so every pixel stays a hard square.
The cartoon world runs on rules from the 1930s. Squash and stretch, a crouch before a jump so the jump is telegraphed, and follow-through so things settle rather than stop dead. Those principles came out of the Disney studio and were eventually written down by Thomas and Johnston in The Illusion of Life. Watch the mug after it lands: the wobble is a damped oscillation, and how quickly it dies away is one constant in the code.
The 3D world borrows from my actual desk. The rocket that flies in world six is the model that sits on the shelf behind me in every other video. It took two failed attempts to get it off the ground, including one where it was black against a black sky for ten seconds before anyone noticed.
The sound and the edit are code too
No sample library was involved anywhere. Jumping produces an upward-sliding tone. Landing is noise, clipped almost immediately. The rustle in the paper world came from running noise through filters until the character of it shifted from static toward something fibrous.
The edit surprised me more. There is no timeline anywhere in this production. The narration script is a text file; Claude reads it, calculates how long each sentence takes to speak, and positions every scene, chapter card and caption against those timings. Rewrite one line and the entire edit reflows around it, which is a property no conventional editor has.
This is the same idea as the studio setup I described in how Claude Code runs my creative studio, pushed further: the deliverable itself is generated rather than assembled.
The scorecard: 47 against 5
I logged every error from the first render onward. By the time I sat down to record the narration, the tally was 47 mistakes Claude found and fixed in its own work, against 5 that got past it to me.
Two of its catches give a sense of the range. It had scheduled the closing fade six seconds late, because its speech-timing check heard a "thank you" in the silence trailing my final line. In another pass it noticed that every chapter card was preceded by almost a full second of empty screen.
Mine were duller and more human. The clearest: at every scene change, the music from the outgoing world was still playing underneath the incoming one. Nothing in a still frame shows you that.
That pattern held across all five of my catches. Every one was about how the film sounded or how it flowed, which are precisely the two things you cannot inspect in a screenshot.
So what did "alone" actually mean?
Two elements have a human origin. The narration is my own voice, reproduced by an ElevenLabs clone. And every stage passed through my approval before it went further.
Everything else, every picture, every sound effect, every cut, was generated from code it wrote. It also found the large majority of its own errors without being told.
What it could not do was hear. Everything in my column came down to the same limitation: a system that inspects single frames will never notice that the music is smeared across a cut, because that only exists across time.
If you are wondering where to start with any of this, the model doing the work here is the one I compared against the rest of the range in Claude Opus 5.5 vs Sonnet vs Haiku.
Frequently asked questions
Can Claude Code make video animation?
Yes, indirectly. Claude Code does not draw images and cannot play back video. It writes each scene as a web page, using HTML, Canvas, SVG or Three.js depending on the style, and a renderer captures that page frame by frame into a video file. For this project every scene, every sound effect and every cut was produced that way, from a single written brief.
How does Claude Code turn code into a video file?
Through a renderer that treats a web page as a film. HyperFrames, HeyGen's open-source tool, loads the page and photographs it one frame at a time at a fixed frame rate, 30 per second in this case. Because the scene is driven by elapsed time, the renderer can ask the page what it looks like at any given moment and capture exactly that.
Does Claude Code draw each frame individually?
No, and this is the part that makes the approach practical. Position, rotation, scale and colour are written as functions of time, so the code defines a rule rather than a picture. A seven-minute film at 30 frames per second contains over twelve thousand frames, and every one is derived from those rules by the renderer rather than authored.
Can Claude see the animation it produces?
Not as video. It renders a small set of still frames after each change and reads them as images, which is how it verifies its own work. That constraint shapes what it can and cannot catch: visual errors in a single frame are found reliably, while anything that only exists across time, such as audio overlapping a scene change, tends to get through.
Did Claude write the music and sound effects too?
Yes, all of it synthesised in code with no sample library. Melodies were written as note data and converted into square waves for the retro sections, mirroring how 1980s consoles generated sound. Effects were built from first principles: a jump is a rising tone, a landing is sharply clipped noise, and the paper texture came from filtering noise until it stopped sounding like static.
How do you edit a video that has no timeline?
The edit is a text file. The narration script is read by Claude, which calculates the spoken duration of each sentence and places every scene, chapter card and caption against those timings. Rewriting a single line causes the whole edit to reflow automatically, which is behaviour a conventional non-linear editor cannot offer.
How many mistakes did Claude make?
The log recorded 52 issues. Claude identified and fixed 47 of them in its own work without prompting, including a closing fade scheduled six seconds late because its speech-timing check misread silence, and a draft where every chapter card carried nearly a second of blank screen. Five reached the human reviewer, all of them relating to sound or pacing.
What did Claude Code not do?
Two things. The narration is a human voice, reproduced through an ElevenLabs clone rather than generated. And each stage required a human approval before the project continued. Everything visual, every sound effect and the entire edit came from code Claude wrote.
Do you need to be a programmer to do this?
You need to be able to describe what you want and judge whether the result is right. The instruction that started this project was a single sentence naming four deliverables and six visual styles. The code was never opened or edited by hand, though someone did have to watch the output and say when it was wrong.
Which Claude model was used?
Claude Opus 5.5, Anthropic's leading model at the time of production, running inside Claude Code. Its ability to work unsupervised across a long task is what made a project of this shape feasible, since the build ran across many sessions with the model checking and correcting itself between them.
Built with
- Claude Code, running Claude Opus 5.5
- HyperFrames, which renders a web page to video frame by frame
- Three.js for the 3D world
Get the next post
New essays on AI, creative technology, and building at the edge. No spam, unsubscribe anytime.
Powered by Substack
