Search over the Visual World: persistent visual memory, layered indexes, and source-grounded evidence.

Agents operating over cameras, screens, streams, and archives face a search problem that cannot be reduced to ranking video files: observations arrive continuously, models interpret them at different temporal granularities, context must be selected without replaying the complete visual record, and every result must stay connected to inspectable source evidence. This report develops a formal model of search over the visual world. It shows that a pipeline of general-purpose components outperforms a commercial video-native retrieval engine on macro-averaged Recall@1/@3/@10 across 9,834 natural-language queries.

Read the paper (PDF)

Abstract

Most video-retrieval systems assume a bounded corpus and return ranked files or timestamps. Agents operating over cameras, screens, streams, and archives face a different systems problem: observations arrive continuously; different models interpret them at different temporal granularities; useful context must be selected without replaying the complete visual record; and a result must remain connected to inspectable source evidence. We argue that search over such a corpus is an infrastructure problem that cannot be reduced to ranking video files.

We develop a conceptual and formal model of search over the visual world built on analyzer-defined scenes, persistent understanding artifacts, visual memory as coexisting scene spaces over shared source time, and capability-declared indexes; we draw a strict distinction between memory (everything retained), context (what is selected for a task), and evidence (the source intervals that ground it). The VideoDB data format (VDB) realizes this conceptual model in production, and a typed search surface exposes it: planned retrieval, a bounded stateful investigation mode, direct semantic, structured, and aggregate access, and grounded synthesis.

We contrast this model-agnostic infrastructure (segmentation, sampling, model choice, embeddings, and ranking all exposed as system decisions, with live streams as first-class sources) with video-native foundation models offered as fixed APIs. In a complete-system semantic-retrieval comparison against a commercial video-native retrieval engine spanning 9,800+ natural-language queries over four public datasets, a pipeline of general-purpose components, none trained end-to-end for video retrieval, achieves higher macro-averaged Recall@1/@3/@10 (73.09/83.39/91.20 versus 65.75/77.13/89.10), while the baseline is higher at Recall@50 (96.42 versus 96.07). The results suggest that, today, retrieval quality over the visual world is governed more by system design than by video-specific pretraining, and that visual-memory infrastructure can deliver it while keeping source-grounded, playable evidence a first-class system object.

Headline results

Retrieval quality over the visual world is governed more by system design than by video-specific pretraining.

Complete-system semantic retrieval, VideoDB (general-purpose components) vs. TwelveLabs Marengo 3.0 (video-native), across 9,834 natural-language queries on four public benchmarks, over 885 videos, ~47.6 hours:

Dataset	System	R@1	R@3	R@10	R@50
MSVD	VideoDB	70.10	80.73	89.17	96.39
	TwelveLabs	67.86	78.22	89.48	97.10
YouCook2	VideoDB	65.96	80.98	93.48	97.87
	TwelveLabs	47.07	65.56	84.18	96.41
VATEX	VideoDB	83.46	90.28	95.24	97.77
	TwelveLabs	85.43	92.40	97.30	99.43
MSR-VTT	VideoDB	72.82	81.55	86.89	92.23
	TwelveLabs	62.62	72.33	85.44	92.72
Macro-average	VideoDB	73.09	83.39	91.20	96.07
	TwelveLabs	65.75	77.13	89.10	96.42

Honest reading: the general-purpose pipeline leads at the early ranks that matter for agents (R@1/@3/@10); the video-native baseline is slightly ahead at R@50, and stronger on VATEX. Full per-dataset methodology, reranking operating points, and system configurations are in the paper.

What the model contributes

The report formalizes search over continuously growing visual corpora: analyzer-defined scenes (segmentation as a system decision, not a file property), persistent understanding artifacts (model outputs retained and reusable rather than recomputed), visual memory as coexisting scene spaces over shared source time, and capability-declared indexes. The memory / context / evidence distinction keeps every answer connected to playable source intervals. Evidence becomes a first-class system object, including over live streams.

Cite

@techreport{videodb2026searchvisualworld,
  title  = {Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence},
  author = {Nagaonkar, Sankalp and Garg, Rohit and Raj, Ankit and Choithani, Ashish and Trivedi, Ashutosh},
  institution = {VideoDB},
  year   = {2026},
  month  = {July},
  url    = {https://videodb.io/research/search-over-the-visual-world}
}