Why a search tool is not enough for an LLM working with video

Robotics labs record far more video than anyone can watch. The obvious next step is to hand that archive to an LLM and let it answer questions. The usual way to do that is to cut the footage into short clips, turn each clip into a vector (a list of numbers that describes what the clip looks like), and give the LLM a search tool over those vectors.

That raises a basic question we kept running into: can an LLM with only a search tool answer the questions a researcher actually asks of an archive? We break it into four smaller questions:

1. Can a top-k search tool answer “how many”?
2. Can it filter on facts that are not in the pixels, such as outcome or room?
3. Does the way footage is cut before indexing change what the model sees?
4. If search is not enough, what does the LLM need instead, and does that design hold up?

In short, we find that:

• A search tool can find a moment, but it cannot count. The number it reports follows the top_k setting, not the footage.
• It cannot filter on facts it never recorded, such as whether a run failed or which room it was in.
• Where the footage is cut matters a lot. On an NFL game, fixed-length chunks raised the hallucination rate from 68.1% to 74.2%. Cutting at the real play boundaries brought it down to 11.4%.
• A query engine that keeps recordings whole and indexes them in layers answers all six question types researchers ask. In our technical report, a retrieval pipeline built this way from general-purpose parts put the right moment first 73.09% of the time, against 65.75% for a commercial video-native search engine.

The questions a researcher asks

We start with a thought experiment. It is reasoned from how the tools work, not a recorded test. Take a lab with a few thousand recorded robot runs, which we call episodes. A typical session with an LLM asks four things in a row:

1. Find. Show me every episode where the left hand holds a glass while the right hand opens a door.
2. Count. How many do I have? Split them by environment.
3. Filter. Of those, which ones failed? Only kitchens.
4. Look closer. Show me the two seconds before each failure.

Give the LLM one tool, search(query, top_k=10), which returns the ten clips that look most like the question. The first question goes fine: ten clips of a hand, a glass and a door. Each of the next three breaks in a different way. The LLM is not being careless in any of them. It is reasoning correctly over the only tool it has.

Can a top-k search tool answer “how many”?

No. A vector search returns a ranking: the k clips that look most like the question, in order. It always returns exactly k rows. So when the LLM is asked how many episodes match, it counts the rows it got back. With top_k=10 the answer is ten. With top_k=50 it is fifty. The number comes from a setting, and nothing in the reply says so.

“How many” needs a set: every episode that matches, whether that is three or three thousand. A ranking cannot produce one. Its length is a parameter, and its edges are decided by resemblance. A look-alike (a hand holding a mug while opening a cabinet) can make the top ten, while a real match filmed from another camera misses it.

Key takeaway. A ranking is not a set. Any count an LLM reports from top-k results is a function of k.

Can search filter on facts that are not in the pixels?

Not unless those facts were put into the vector. A clip vector records roughly what a clip looks like. “Failed”, “model version 18” and “recorded in the kitchen” are facts about the episode, and often no single frame shows them. If they were not embedded, search cannot filter on them. Adding them later means processing the whole archive again.

Key takeaway. Filters need the facts stored next to the video, not inferred from how it looks.

Does where the footage is cut change what the model sees?

Yes, and we have measured it. A search pipeline cuts footage into fixed-length pieces before indexing, because each piece becomes one vector. That cut is made before anyone knows the question.

In our NFL analysis we ran three strategies on the same game and scored the first 20 minutes against the real play-by-play (running plays, catches, touchdowns and punts). The metric was the hallucination rate: how often the model reported something that did not happen.

• Whole game sent to a vision-language model: 68.1%.
• Uniform 2, 5 and 10 second chunks: 74.2%. Shorter clips made it worse, because the chunks split plays in half and the model made up the missing half.
• Cut at play boundaries, taken from the official play-by-play and matched to the video by reading the on-screen game clock: 11.4%.

The window a question needs also varies. A slip needs about five seconds. A regrasp needs ninety. Reviewing a shift needs twenty minutes. No single clip length fits all three.

Key takeaway. Keep recordings whole and let each question choose its window when it is asked.

What does the LLM need instead?

The design we tested is VideoDB, a query engine over your video: recordings go in whole, get indexed once, and answer questions from then on. Training runs, test runs and simulated runs share one archive, and every episode is labelled as real, simulated or generated. In our work with robotics teams, six kinds of question come up again and again: find, count, filter, sample, look closer, watch. Three design choices let one index answer all six.

Index once, in layers, on one timeline

When a recording comes in, analyzers read it. An analyzer is a model that watches the video and writes down what it sees. Together they write several indexes: what is said, what each scene shows, which objects appear. Metadata such as task, environment, outcome and model version sits alongside. Every entry points back to a time range in the original recording. Nothing is pre-cut and nothing is re-encoded.

Each index declares what it can do

A scene index supports search by meaning. A metadata index supports exact filters. A field used for grouping supports counts. In our technical report we call these capability-declared indexes. The LLM, or our own planner, reads what each index can do and picks the right operation for each question, instead of pushing every question through similarity search.

Answers come back as footage

Every result carries its recording and time range, plus the details attached to it (episode id, timestamp, camera, model version). On request, we build a playable evidence stream for that range, straight from the original file. It takes about half a second from the question to the moment playing, which is fast enough for an agent to look at what it found and ask the next question.

One caution before counting on counts. A count of what the camera saw is only as good as the annotations under it. Those annotations come from models, and models make mistakes. That is why every count comes with the episodes behind it, so a person or an agent can open any of them and check. Our WGO-Bench results show how far machine annotation has come, and how far it still has to go.

Key takeaway. An LLM over footage needs a set, filters on stored facts, and frames from any window. That is a database, not a search box.

Does the design hold up?

We tested the retrieval design on public data. In our technical report, a retrieval pipeline built from general-purpose parts was compared with a commercial video-native search engine on 9,800+ queries across four public datasets. The metric was Recall@k: how often the right moment is among the first k results.

• Recall@1: 73.09% against 65.75%.
• Recall@3: 83.39% against 77.13%.
• Recall@10: 91.20% against 89.10%.
• Recall@50: 96.07% against 96.42%. Here the baseline came out slightly ahead.

The gap is largest where it matters most for an agent: putting the right moment first. By k = 50 both systems find almost everything. Two caveats apply. This is our own evaluation, and the paper describes the method in full. And it measures retrieval, not counting or filtering.

This post covers retrieval: finding, counting, filtering and fetching frames. Grouping the failures an agent finds into failure modes is the next step, and it runs on the same index. Showing that a chosen set of data improved a model is a bigger claim again. That needs a controlled experiment, not a search result.

Conclusion

Before giving an LLM access to footage, check what its tools can do. That matters more than how capable the model is. The tools need to:

1. Return a set, not only a ranking, so “how many” has a real answer.
2. Filter on facts that are not in the pixels: outcome, place, version, date.
3. Keep recordings whole and let the question choose the window.
4. Hand back frames fast enough that the model can look again.

A search box gives the LLM the first question. A database gives it the rest of the chain.

Try it

If you want the LLM to plan for itself, give it search(). It reads the index schemas and decides whether a question needs semantic search, a structured query or a count. search(mode="deepsearch") remembers the session, so a follow-up narrows the last answer instead of starting over: pass session_id=response.session_id on the next call. ask() gives a written answer, with timestamped sources when you set include_sources=True.

If you would rather control each step yourself, each question maps to one call:

import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Find: a question about meaning, so semantic search over the scene index
hits = coll.semantic_search(
    query="left hand holds a glass while the right hand opens a door",
    index_names=["scene"],
)

# Count: a question about a set, so aggregate, grouped by a field you attached
by_env = coll.aggregate(index_name="episodes", group_by="environment", metric="count")

# Filter: facts that are not in the pixels, so a structured query on metadata
failed = coll.query(
    index_name="episodes",
    filter=[
        {"field": "outcome", "op": "==", "value": "failed"},
        {"field": "environment", "op": "==", "value": "kitchen"},
    ],
)

# Look: the two seconds before each failure, cut from the original recording
for shot in failed.get_shots():
    video = coll.get_video(shot.video_id)
    stream_url = video.generate_stream(timeline=[(max(0, shot.end - 2), shot.end)])

Field names like environment and outcome are whatever you attach when you index each episode. upload() does not take metadata. Instead, you add it as a custom index on each recording, with one record per episode:

video.index(
    name="episodes",
    source=[{"start": 0, "end": video.length, "environment": "kitchen", "outcome": "failed"}],
    use_for=["query", "aggregate"],
    fields={"filter": ["environment", "outcome"], "aggregate": ["environment"]},
)

Here we assume a failed episode’s recording ends at the failure, so the look step cuts the two seconds before shot.end. If you store the failure as its own timed record, cut before shot.start instead.

There are three ways to put this in front of a model:

• Python SDK. Wrap the calls above as tools in your own agent.
• MCP server. MCP (Model Context Protocol) is the standard way to give Claude and other models a set of tools. Point Claude or any MCP client at https://mcp.videodb.io/mcp, and indexing, search and streams become tools it can call in a conversation. Docs
• Agent skills. Run npx skills add video-db/skills to give Claude Code and similar agents the same abilities. Docs

For a full example, see how we built Deep Search, which is open source. It plans a search across several indexes, checks whether the clips it found really match, widens the plan when nothing comes back, and folds a follow-up into the plan instead of starting over.

FAQ

Why can’t an LLM just search my video files directly?

Language models read text and images, not hours of video. A raw video file has no transcript, no index and nothing to query, so someone has to watch the footage or write a new script for every question. An index turns the footage into something a model can ask questions of.

Isn’t a vector database over video embeddings enough?

For finding clips that look like a description, often yes. But it returns the nearest neighbours to a query, so it cannot count reliably, cannot filter on details that were not embedded, and its clip boundaries are fixed before anyone asks. Semantic search added to a log viewer has the same limit: it helps inside one run, while the questions in this post run across the whole archive.

What does “index once” mean in practice?

We analyze each recording once, when it arrives, and store the results (transcript, scene descriptions, objects, your metadata) against the original timeline. New questions run against those indexes, so the video is never processed again. When a better analyzer comes along, you can re-read the archive without collecting it again.

How do I connect VideoDB to Claude or another agent?

There are three ways: the hosted MCP server at mcp.videodb.io/mcp, the agent skills package (npx skills add video-db/skills), or the Python SDK wrapped as tools in your own agent.

Does this only apply to robot footage?

No. The same chain of questions shows up with meeting recordings, screen recordings, camera feeds and media archives. Robot episodes just make it obvious, because researchers ask “how many” and “only the failures” all day.

Where to read more

• Search over the Visual World, technical report, arXiv 2608.08075
• NFL game analysis: 80% fewer VLM hallucinations
• Search and retrieval docs: semantic_search, query, aggregate, search, ask
• MCP server docs and agent skills docs
• Deep Search on GitHub

Bring the questions you want to ask your footage, or start with the quickstart.

@article{priya2026search,
  author = {A Priya},
  title = {Why a search tool is not enough for an LLM working with video},
  journal = {VideoDB Labs},
  year = {2026},
  month = {oct},
  note = {https://videodb.io/blog/why-a-search-tool-is-not-enough-for-an-llm-working-with-video},
}