Agentic video analysis: what works and what doesn't

“Agentic video analysis” gets talked about as one capability, but published results vary widely. On LVBench, a benchmark of 1,549 questions over 103 hour-long videos, two published video agents scored 29.3% and 28.8%. GPT-4o, answering with no agent around it, scored 48.9%. A third agent, Deep Video Discovery from Microsoft Research Asia and USTC, scored 74.2%. All three are called agents, and their scores are 45 points apart. What separates them is what the loop around the model lets it do.

What an agent loop is

A plain model gets a video (or a sample of its frames) and a question, and answers in one pass. An agent works in steps. It decides what to look for, calls a tool, reads what comes back, and decides again: search somewhere else, look closer, or answer.

The Deep Video Discovery authors describe earlier video agents, these two included, as following “predefined workflows applied uniformly across different queries”: the same steps for every question. Deep Video Discovery gave the model a small set of search tools over a pre-built index of the video and let it choose which to use, in what order, based on what it had found so far.

What works

Searching an index instead of watching everything

An hour of video is too much to put in front of a model at once, and sampling a few frames misses most of it. The best-scoring agent in this comparison searches an index instead: text written about the video in advance (scene descriptions, a summary, often a transcript), where every entry points to a time range.

Deep Video Discovery built its index at three levels: the whole video (a list of the main people and objects in it), a description of each 5-second clip, and the frames themselves. Its agent had one tool for each level: browse the whole video, search the clip descriptions, and inspect the frames in a chosen time range. The authors then removed one tool at a time.

Two things stand out. Searching the clip descriptions mattered most, more than browsing the whole video. And removing clip search (12.3 points lower) cost about as much as replacing o3 with GPT-4o as the reasoning model, the model that decides each next step (13.7 points lower). The two tests start from different settings (71.9% and 76.0%, both with transcripts), so this comparison is approximate.

Looking again

Frame inspection was the second most important tool (8.4 points lower without it). A description of a clip is a summary written before anyone knew the question. When the agent can pull the actual frames for a time range and ask a narrower question of them, it can confirm or reject what the description suggested. Without that step, the agent is reasoning over someone else’s notes.

Looking again also depends on the model that does the looking. In our own test of reading chessboard images into exact piece positions, Claude Opus 4.7 got 84.72% of boards fully right in its best run, while GPT-5.4 got 100%. Claude’s misses were small and local: a piece one square over, or one empty square too many in a row. So before relying on a model to look again, test it on the specific thing your task needs.

Checking its own output against the footage

An agent can also check its own output against the footage. We saw this when Claude, using VideoDB’s agent skills, edited our launch video. It rendered a cut, uploaded that cut back into VideoDB, and indexed it again on both the spoken words and the visuals. The check showed that its cuts were landing early. The word timestamps in the transcripts were about 1.3 to 2.6 seconds ahead of where the words actually sat in each source file, and the gap was different for each file.

It fixed the problem by lining up every cut with the silences in the actual audio, re-cutting each segment on its own file’s timeline, and confirming the first and last word of each section with a separate speech-recognition pass. The same loop caught a factual slip: one take said “93,000”, which did not match the slide deck, so the agent compared the transcripts and used the take with the right number. It also found that the opening section was 7.7 dB quieter than the rest, and evened the sections out to within 0.9 dB.

In this session the agent did not stop at the render. It indexed its own output and checked it.

The loop was: take footage in, understand it, decide, render, take the render back in, check. A person still made the subjective call (which closing take to use). The agent handled everything that could be checked against the footage.

We build the same habit into retrieval. Deep Search, our open-source search loop, turns a request into a plan, runs it across several indexes, and then passes the candidate clips to a validation step that checks whether each one matches what was asked. Clips that fail are dropped. If every clip fails, the validator’s feedback revises the plan and the search runs again before anything is shown.

What doesn’t work

Cutting on a clock

Before a model sees a long video, the video is usually cut into pieces, and where those cuts fall decides what the model can know. In our NFL analysis, sending a whole game to a vision-language model gave a 68.1% hallucination rate (the share of game events where the model produced incorrect or irrelevant information). Cutting it into uniform 2, 5 and 10 second chunks made it worse, 74.2%, because the chunks split plays down the middle and the model filled in the missing half. Cutting at the real boundaries of each play, found by reading the on-screen game clock and lining it up with the official play times, dropped it to 11.4%.

The rule carries over to any footage with real events in it: a task attempt, a scene, a manipulation, a meeting agenda item. Where those events can be found, cut at their boundaries. Without an event log to align to, the fallback is detecting scene changes from the pixels, which we expect to be less reliable.

Stopping too early, or never stopping

Deep Video Discovery’s authors sorted their agent’s runs by behaviour, and two failure patterns stand out.

The first is quitting early. With GPT-4o as the reasoning model, the agent took the simplest path (one or two clip searches and frame inspections, then an answer) on 91.4% of questions, averaging 4.6 steps. That path is the most common one for every model and often works. The authors call GPT-4o’s behaviour overconfidence: it almost never tried a different strategy when the simple one was not enough.

The second is looping. With o3, a primary failure was calling clip search more than three times in a row, “often when key information is absent from the video database”. The agent kept searching for something that wasn’t there.

Our reading of that second failure: a top-k search tool, one that returns the k closest matches, always returns k results, even when nothing matches. So the agent never gets told “zero”. A tool that can return an empty set, or a count of zero, gives the agent a reason to stop, and a reason to widen the question instead of repeating it.

Deep Search handles this case with a recovery step that runs only when a search returns nothing. It either broadens the plan and searches again, or asks the user one clarifying question. An agent needs both: a tool that can report “nothing here”, and a plan for what to do next.

Claiming more than the frames show

Finding a moment and explaining it are different claims. An agent can find every episode where a robot dropped a cup and show you the two seconds before each drop. Saying why it dropped is a stronger claim, and video alone often can’t settle it.

The same care applies to footage that was not recorded from a real robot. Simulated runs and generated video (the “dreams” a world model produces) need their own checks. A generated clip can look right without matching what a real robot would do, so it should stay labelled as generated in the archive and never mix silently with real runs.

Even naming what happened is harder than it looks. On a revised WGO-Bench of 100 videos of people and robots, our annotation pipeline scored 28.93% semantic F1 (a score for finding each action at the right time and giving it a label judged correct) against 19.92% for the public baseline. That is a clear lead, and the task is still far from solved. An agent reading those labels is reading a best effort, so it should treat them as evidence to check.

A practical rule for agent output: keep what the agent observed, what it thinks, and what it couldn’t resolve in separate places, and attach the source time range to every observation so a person can watch it.

Where VideoDB fits

The agents that scored well in the results above do two things. They search an index to decide where to look, and then they look at the frames. The index points the agent in a direction, and the frames are the evidence it uses to answer, to check its own work or to take the next step. In Deep Video Discovery’s tests both mattered: removing clip search cost 12.3 points, and removing frame inspection cost 8.4. VideoDB is built to make the second step, looking at the frames, fast enough that an agent can do it as often as it needs to.

• Random access to frames. Each recording is transcoded once into a frame-addressable form, so seeking to a frame costs the same anywhere in the file. An agent can ask for one frame or one window and get exactly that, without re-encoding. Frames from a window can also be tiled into one image, so a model reads the whole window in one call. Looking again is a single call rather than a re-processing job, which keeps the loop efficient when the agent needs to look many times.
• An index that points to the frames. Recordings are annotated at archive scale (objects, hands, actions, on-screen text and speech), with any model as the analyzer, and human QC through our partners for the cases the model is unsure about. An agent can search by meaning, filter on exact facts and count, across one video or a whole archive, and gets clips and frames back within about 500 ms. The index is versioned, so a better model can re-annotate everything later.
• Output back in. A render, a clip or a new run can be ingested and indexed like any other footage, so an agent can look at the frames of its own output. That is how the launch-video agent checked its cuts.
• Results the agent can track. Every result carries episode id, timestamp, camera, source type and model version, so the agent knows which frames it is looking at and where they came from.

You can bring your own agent, with your own reasoning model and workflow, or use ours.

The quickest way to try the loop is the same route as the launch video: npx skills add video-db/skills gives Claude Code and similar agents these tools (docs). There is also a hosted MCP server at mcp.videodb.io/mcp for Claude and other MCP clients (docs).

The checklist

Before you trust an agentic video system, ask five questions:

1. Does it search an index, or re-watch the video? Look for time-ranged entries it can query.
2. Can it look again? It should be able to pull frames for a range it picks, not only read descriptions.
3. Where were the cuts made? On real events, or on a timer?
4. Can its search return an empty result, and what does it do next? Check how it decides to stop.
5. Does it check its output against the footage, and separate what it saw from what it concluded?

If a system cannot do one of these, that is the step to test most carefully on your own footage.

FAQ

What is agentic video analysis?

A setup where a language model answers questions about video in steps: it decides what to look for, calls tools (search an index, inspect frames, check a result), reads what comes back, and repeats until it can answer. A plain model answers in a single pass.

Are video agents better than just giving the video to a model?

Only when the loop is right. On LVBench, two agents with fixed workflows scored below GPT-4o answering alone, while an agent that chose its own searches over a pre-built index scored far above it.

Is a bigger model the fix?

It’s one lever among several. In Deep Video Discovery’s tests, replacing o3 with GPT-4o as the reasoning model cost about as much accuracy as taking away the clip-search tool. Where the footage is cut matters as well, as the NFL results above show.

Why do video agents loop or give up early?

Early stopping looks like overconfidence: the agent takes the first answer that fits. Looping tends to happen when the answer isn’t in the index and the search tool keeps returning something anyway. Tools that can return an empty result or a count give the agent a clear signal to stop.

Can an agent tell me why a failure happened?

It can find the failure, find similar ones and show you the moments before each. Explaining the cause is a stronger claim that needs its own evidence. Treat it as a hypothesis until something beyond the footage backs it up.

Where to read more

• Zhang et al., Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding, arXiv 2505.18079
• Wang et al., VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024
• LVBench, arXiv 2406.08035
• Claude edited our launch video with VideoDB skills
• Deep Search: finding exact moments in video
• Can a VLM annotate a robot run?
• NFL game analysis: 80% fewer VLM hallucinations
• A strong general-purpose VLM still fails on a chessboard
• VideoDB agent skills and MCP server docs

Bring the footage you want an agent to work on, or add the tools yourself with npx skills add video-db/skills.

@article{priya2026agentic,
  author = {A Priya},
  title = {Agentic video analysis: what works and what doesn't},
  journal = {VideoDB Labs},
  year = {2026},
  month = {oct},
  note = {https://videodb.io/blog/agentic-video-analysis-what-works-and-what-doesnt},
}