Give your AI agents eyes: a builder’s guide to real-time visual perception.

Your agent can write code, book flights, search the entire web, and operate a browser like a caffeinated intern. Point it at a security camera, a YouTube video, or the screen it’s supposedly automating, and it’s helpless. It can act on the world, but it cannot watch the world.

The rest of this post is the long version: why agents are blind, why the obvious hack doesn’t scale, and three working builds. The three builds are an agent that watches YouTube, an agent that watches a live camera feed, and an agent that remembers what happened on a screen.

Agents got hands before they got eyes

Look at what the agent-infrastructure wave has actually shipped. Exa and Parallel gave agents web search and research. Ask a question, get grounded answers from the live internet.

Firecrawl gave them clean page context at scale. Browserbase and browser-use gave them hands: click, type, navigate, transact. TinyFish runs fleets of web agents against enterprise workflows.

Every one of those unlocked a product category. And every one of them operates on the same substrate: text and DOM. HTML in, tokens out.

Meanwhile, most of what actually happens in the world never touches a DOM. Meetings happen on camera. Work happens on screens. Operations happen in front of cameras: stores, hospitals, factories, streets, drones.

The largest knowledge source on the internet is a video platform, and your agent can’t watch it. AI is moving out of the chatbox. The moment it does, it hits the visual world and goes blind.

That gap is not a model problem. GPT-5, Claude, and Gemini can all describe a frame beautifully. It’s an infrastructure problem.

Why screenshots don’t add up to sight

Every builder tries the same first hack: grab a frame, base64 it, stuff it into the prompt. It demos great. Then it collapses, for five predictable reasons:

1. Moments aren’t events. A frame tells you a person is in the room. It can’t tell you they’ve been pacing near the server rack for ten minutes. Meaning in video lives in time, and single frames throw time away.
2. Nothing persists. Ask your screenshot-agent what happened an hour ago and it has no idea. Perception without memory is a party trick.
3. Tokens explode. Continuous understanding via frames-in-prompt means paying VLM prices on every tick of every source, forever, with no reuse. You re-buy the same understanding every time you ask a new question.
4. Polling misses things. Real events don’t wait for your loop interval. Real-time perception has to be push, not pull. The stream tells you when something happened.
5. You can’t search what you never indexed. “When did the delivery arrive?” “Show me every moment the user opened the settings page.” Those are retrieval queries over time. No index, no answer.

Sight, for an agent, is a pipeline: ingest → understand → remember → retrieve → act. That’s precisely the layer VideoDB provides, and here’s what it feels like to use.

Build 1: an agent that actually watches YouTube

The task: give your agent a YouTube URL and let it answer with evidence. Not a transcript dump, but the exact moments that support the answer, playable as a clip.

pip install videodb
export VIDEO_DB_API_KEY="your-api-key"

import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Ingest: any file or URL, including YouTube
video = coll.upload(url="https://www.youtube.com/watch?v=LPZh9BOjkQs")

# Understand: run analyzers over speech AND visuals
understanding = video.understand(analyzers=[
    {"type": "spoken_words", "name": "transcript"},
    {"type": "vlm", "name": "scene", "config": {"prompt": "Describe the visual content."}}
])
understanding.wait_until_complete()

# Remember: index what the analyzers produced
transcript_index = video.index(name="transcript",
                               source=understanding.get_analyzer("transcript"))
scene_index = video.index(name="scene",
                          source=understanding.get_analyzer("scene"))
transcript_index.wait_until_complete()
scene_index.wait_until_complete()

# Retrieve: semantic search across speech + visuals together
results = video.semantic_search(
    query="an explanation supported by diagrams and on-screen text",
    index_ids=[transcript_index.index_id, scene_index.index_id],
    top_k=5
)

for shot in results.get_shots():
    print(f"{shot.start:.1f}s-{shot.end:.1f}s")

# Act: compile the matching moments into a single playable evidence reel
evidence_url = results.compile()

Notice what your agent just got that a transcript scraper can’t give it. The visual channel is indexed too, and that VLM analyzer prompt is yours to shape: “extract every chart”, “note every product shown”, “flag every error message”. Search runs across both modalities, every hit is timestamped, and compile() returns a watchable clip of just the relevant moments.

Your agent doesn’t summarize a video. It cites it.

This is the pattern people are hacking together with yt-dlp + Whisper + ffmpeg + a vector DB + prayer. It’s one dependency here. Wire it behind a tool call and “go watch these 40 videos and tell me every pricing claim, with receipts” becomes a routine agent task, the same way Exa made “go read the web” routine.

Build 2: an agent with a live camera feed

Recorded video is where perception starts. The interesting part is live. RTSP is the lingua franca of cameras: security cams, drones, warehouse feeds, broadcast. This is where “give your agent eyes” stops being a metaphor.

import asyncio
import videodb

RTSP_URL = "rtsp://samples.rts.videodb.io:8554/intruder"  # public sample stream

async def main():
    conn = videodb.connect()
    coll = conn.get_collection()

    # Ingest: connect the live stream
    rtstream = coll.connect_rtstream(url=RTSP_URL, name="Lobby Cam", store=True)

    # Understand: continuous analysis in rolling windows
    understanding = rtstream.understand(
        segmentation={"type": "time", "window": "5s"},
        analyzers=[{
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe the people, movement, and activity."}
        }],
        store=True
    )

    # Remember: a live index that's queryable while the stream runs
    index = rtstream.index(name="lobby-scenes",
                           source=understanding.outputs["scene"],
                           use_for=["semantic"])

    # Act: define an event in plain language, get pushed alerts
    websocket = conn.connect_websocket(coll.id)
    await websocket.connect()

    event_id = conn.create_event(
        event_prompt="Detect when a person appears in the monitored area.",
        label="person_appears"
    )
    index.create_alert(event_id=event_id,
                       callback_url="https://your-agent.example.com/hooks/vision",
                       ws_connection_id=websocket.connection_id)

    async for message in websocket.receive():
        if message.get("channel") == "alert":
            alert = message["data"]
            print(alert["label"], alert["confidence"])
            print(alert["explanation"])   # why the event fired
            print(alert["stream_url"])    # playable clip of the moment
            break

asyncio.run(main())

Read that event definition again: it’s a sentence. Not a trained classifier, not a labeled dataset, not a CV pipeline. Just a prompt. “Detect when a person appears.” “Detect water pooling on the floor.” “Detect when the forklift enters the loading zone.”

When it fires, your agent receives the label, a confidence, an explanation, and a stream_url of the exact moment. That’s evidence again, not vibes.

The feed is continuously understood in rolling windows, whether or not anyone is asking questions. That is the difference between an agent that can look and an agent that is watching. This same loop is the backbone for the unglamorous-but-valuable stuff: intrusion detection, ICU monitoring, flood detection, quality control, drone patrol. There are runnable versions of several of these in the cookbook.

Build 3: agents that remember what happened on screen

Computer-use and browser-use agents have a peculiar disability: they act on screens they’ll never remember. Each screenshot is consumed and gone. Ask “what did the deploy dashboard look like before the incident?” and there is no answer, because there is no past.

VideoDB’s capture SDK (pip install "videodb[capture]") turns a desktop or browser session into the same kind of stream as Build 2: captured, continuously understood, indexed, searchable.

matches = screen.search(query="source code open in an editor",
                        index_id=screen_index.id)
for shot in matches.get_shots():
    print(shot.text)
    print(shot.generate_stream())  # replayable clip of that moment

That’s an agent asking questions about its own visual history. Call it episodic memory, in the concrete sense.

Your pair-programmer agent can rewind to the moment the test went red. Your QA agent can produce a replay of the exact click that broke checkout. Your meeting copilot can pull up the slide, not the paraphrase. Full walkthrough in the quickstart.

One layer, every source

The point isn’t three demos. It’s that all three run the same five verbs (ingest, understand, remember, retrieve, act) over different sources:

Your agent	Its eyes	What it can now do
Research / web agent	YouTube + any video URL	Watch, extract, cite with timestamped clips
Computer-use agent	Screen capture	Recall and replay anything it ever saw
Browser agent	Session recordings, web video	Verify visually, debug from replays
Ops / monitoring agent	RTSP cameras, drones	Watch continuously, act on plain-language events
Meeting copilot	Calls and meetings	Search what was shown, not just said
Media agent	Archives and libraries	Find any moment, compile new cuts programmatically

“Can’t I just send frames to GPT-5 or Gemini?”

For a single image, absolutely. You don’t need infrastructure to caption a photo. The honest boundary: raw VLM calls are the right tool for one-shot perception of individual moments.

You outgrow them the day your agent needs any of these: continuous sources (a stream doesn’t fit in a context window, because it never ends), search over hours of footage, events that push instead of polls that pull, memory that outlives the session, or costs that don’t scale linearly with watching.

VideoDB isn’t a competitor to the models. It’s the layer that feeds them. Analyzers are model-agnostic. The VLM does the perceiving, and VideoDB does everything that makes perceiving useful: ingestion from any source, temporal segmentation, indexing, retrieval, eventing, storage, and playback.

The playback stack (storage, encoding, delivery) was built so people could watch video. This is the equivalent stack so software can.

Plug it into the agent you already have

You likely don’t even need to write the integration:

• Agent Skills: npx skills add video-db/skills gives Claude, Cursor, and other agents video perception as tools, plus server-side video workflows they can invoke (skills repo)
• Frameworks: LlamaIndex retriever, LangChain, and REST for everything else
• No-code: n8n and Zapier nodes for the same primitives

FAQ

How do AI agents see video?

Through a perception layer. Video infrastructure ingests sources (files, URLs, live streams, screens), runs speech and vision analyzers over them continuously, indexes the results, and exposes semantic search plus real-time events via API. The agent calls tools. The infrastructure watches.

Can my agent watch a YouTube video and answer questions about it?

Yes. coll.upload(url=...) accepts YouTube URLs directly. After indexing, the agent can semantically search speech and visuals, get timestamped matches, and compile evidence clips. See Build 1 above. It’s about 20 lines.

Can agents process live streams like RTSP cameras in real time?

Yes. RTSP Connect ingests live feeds, understanding runs in rolling windows (e.g. every 5 seconds), and live indexes are queryable while streaming. Plain-language events (“detect when a person appears”) push alerts over WebSocket or webhooks, with an explanation and a clip of the moment.

What’s the difference between VideoDB and a vision model API?

A vision model perceives a frame or short clip you hand it. VideoDB is the infrastructure around the model: ingestion, temporal segmentation, continuous analysis, indexing, memory, retrieval, events, and playback. It’s model-agnostic. The VLM sees, and VideoDB makes seeing continuous, searchable, and actionable.

Does this work with Claude, Cursor, LangChain, or my agent framework?

Yes. Agent Skills give Claude, Cursor, and other agents video perception directly. There is also a LlamaIndex retriever, Python/Node SDKs, and n8n/Zapier for no-code automations.

What about privacy and deployment?

VideoDB supports Zero Data Retention configurations, SOC 2 Type II, SSO/SAML, and deployment as managed cloud or in your own cloud (AWS/GCP/Azure), with VPC and edge options for sensitive camera workloads.