inferwire
/
AI·4 min read

claude-real-video Turns Videos Into Frames Any LLM Can Read

An open-source tool picks scene-change frames, drops near-duplicates and transcribes the audio on your machine, so Claude and other image-only models can answer questions about a video.

TL;DR

  • claude-real-video, an open-source MIT-licensed tool, turns a video link or file into scene-change frames plus a transcript that any image-reading LLM can use [^1].
  • It runs locally and drops near-duplicate frames, so a model like Claude, which won't accept video files, sees fewer images and spends fewer tokens [^1].

Background

Claude's API takes images, not video [^1]. It accepts JPEG, PNG, GIF and WebP files, uses only the first frame of an animation, and can analyze several images in one request together [^2]. Each image costs visual tokens, one per 28-by-28-pixel patch, and the API caps a request at 100 or 600 images, depending on the model's context window [^2]. To show such a model a video, someone has to pick the frames first.

What happened

A developer publishing as HUANGCHIHHUNGLeo offers claude-real-video, or crv for short, on GitHub under the MIT license [^1]. Its README argues that most AI tools don't really see a video: ChatGPT reads the transcript of a YouTube link rather than the picture, Claude won't take a video file at all, and Gemini, which handles video natively, uploads it to Google and samples frames at a fixed interval, one per second by default [^1]. crv instead runs on the user's machine. It downloads a link with yt-dlp or copies a local file, then uses one ffmpeg pass to grab every scene change plus, by default, at least one frame per second [^1].

A deduplication step then compares real pixel differences against the last four kept frames, so a shot that returns after a cutaway isn't sent again [^1]. If the video has subtitles, crv uses them as the transcript; otherwise it transcribes the audio with Whisper [^1]. The result is a folder of JPEG frames, a transcript and a MANIFEST.txt that summarizes everything for the model, which the user then drops into Claude, ChatGPT or Gemini [^1]. In the README's own example, fixed sampling of a 58-second clip at one frame per second yields 58 frames, while crv keeps the 26 that differ and can pack them into three contact sheets of up to nine frames each [^1]. By the same comparison, a static 10-minute slide goes from about 600 near-identical frames to one [^1].

Further options include a local viewer.html for previewing what the model will see, an --adaptive mode for slow changes such as gradual morphs, and, since version 0.3.0, a --why flag that tells the analysis what to focus on and a --kb flag that saves the result as a dated note [^1]. The tool can be installed as a Claude Code skill, so Claude Code processes a pasted video link on its own. It needs ffmpeg and Python 3.10 or newer, and its latest release, version 0.5.3, came out on July 6 [^1]. The developer also sells a paid add-on, crv Pro, which adds camera-move labels, editing rhythm and a timeline of gestures, voice and sound events [^1].

Why it matters

The tool treats frame selection as the hard part. Fixed-interval sampling, the README argues, over-samples a static screencast and under-samples a fast-cut reel; picking frames by scene change and removing repeats is meant to give the model fewer, more meaningful images [^1]. For image-only models that is a practical constraint, not just a cost question: every frame costs visual tokens, claude.ai accepts 20 images per message, and API requests stop at 100 or 600 [^2]. crv's own default cap is 150 frames [^1].

Privacy is the other argument. Download, frame extraction and transcription happen locally, and only the frames and text the user chooses to paste into a model leave the machine [^1]. Frames and a transcript are still not video, though. The README itself concedes that a stack of keyframes can't tell a model how a video moves, its camera work and pacing, which is what the paid version claims to add [^1]. It also asks users to download only content they have the right to [^1].

Practical example

Say a colleague sends you a recorded conference talk and asks what the pricing slide said. You run crv on the file with a --why note that asks for the pricing slide [^1]. Because most of the talk is static slides, deduplication keeps each slide roughly once instead of one frame per second, and the transcript comes from the subtitles or from Whisper [^1]. You open viewer.html to check the frames, pack them into contact sheets with --grid so they fit under claude.ai's limit of 20 images per message, and upload them together with MANIFEST.txt [^1] [^2]. Claude can then answer from the slide and the transcript, and you don't have to scrub through the recording yourself.

Sources

  1. [1]GitHub — HUANGCHIHHUNGLeo/claude-real-video: Let Claude (or any LLM) actually watch a video (snapshot of July 6, 2026)
  2. [2]Claude Platform Docs — Vision (snapshot of July 6, 2026)
  3. [3]Hacker News — Claude-real-video - any LLM can watch a video