Case Study
Reelsmith is an open-source Mac video editor where your own Claude Code or Codex acts as the director. It watches the footage, commits to a concept, cuts on the beat, then renders frames to critique and fix its own edit. Everything stays on your machine and on an editable timeline.
Made with Reelsmith
Real reels cut by the app from a story, raw clips and a song, posted to Instagram.
Making a good Reel from a weekend of phone footage takes hours: scrubbing 40 clips for the three good seconds in each, cutting on the beat, timing text, and choosing transitions that don't look cheap.
Template apps are fast, but every reel looks like the template. Chat-based AI editors follow instructions like 'cut here' or 'add a caption', but they don't direct. I wanted an editor where you give intent, not instructions, and the AI makes the taste calls.
Each stage is a tool the agent calls, guided by a reels-director skill. The model makes the taste calls; deterministic code does the arithmetic and the checks.
Every clip gets a shot log: what it shows, its best moments, where it moves, where someone talks, and the soft or dark seconds to avoid. The song gets a beat grid. Shots are found by meaning ('the view from the top'), not by filename.
The agent writes two genuinely different treatments, picks the one that fits the user's story, and can show the plan for approval before it builds anything.
A compiler turns the plan into a multitrack timeline: sections back to back on the song's beat clock, a library of moves (flash cuts, holds, Ken Burns, reveals, split screens), the user's words as titles, and effects and sound landing on the music.
After the build it renders frames from every section, checks the cut 'with eyes closed' for a drop on nothing or a line between beats, names its three worst defects and fixes them. The result is applied as one undo step.
Footage never leaves the Mac. The agent is the user's own CLI, and the app never sees their credentials.
Transcription with whisper.cpp, beat tracking, shot search with embeddings in sqlite-vec, and person and depth mattes with MediaPipe all run locally. Remotion and ffmpeg render the final MP4.
Claude Code or Codex connects over MCP to a set of editor tools. Every edit goes through a draft that is reviewed and applied as a single undo step, so an agent mistake is always one ⌘Z away.
Depth lens blur and focus pulls, sky and world swaps, silhouette portals, smooth slow motion and text placed behind people, all built on the on-device mattes.
1080×1920 exports with safe-zone and loudness checks, a music-free copy for Instagram's own audio, and the same reel re-shaped to 4:5, 1:1 or 16:9.
Reelsmith started as a fork of the open-source OpenChatCut editor. It was also an experiment in how far one developer can go by directing Claude Code and Codex well.
A planning package (feature inventory, research notes, JSON schemas, architecture diagrams) came first, then a roadmap where every item has a concrete acceptance check. An item is ticked only when its check passes in the real app.
A CLAUDE.md tells agents where new code goes, which tests to run when, and what never to touch. Upstream edits stay at thin hook points and are logged, so merges from the original project stay cheap.
160+ new verify suites, including real renders compared pixel by pixel and exports probed with ffmpeg. An affected-tests runner walks the import graph so the agent gets feedback in seconds, not minutes.
Decisions came from numbers: x264 vs VideoToolbox at equal bitrate (SSIM .951 vs .945), an export bitrate change (.945 → .969 on phone footage), and a profile showing ~87% of a slow edit's output tokens were hidden reasoning, which led to an effort setting and batched tools.