Agent memory

Give your agent a memory of everything it sees

A live feed is gone as soon as it plays. VideoDB reads it as it arrives, acts on what matters now, and keeps all of it as memory: an index of what was seen, and the frames behind any moment in about half a second.

Every window of the feed gets a decision

The feed is read a few seconds at a time. Each window is described, checked against the events you wrote, and written to the index. The ones that match also go out as alerts.

Watch a camera like this live

An example, invented: four minutes of a front door camera at night, read in 48 windows of five seconds, with one event to watch for, a person checking the lock. Forty-seven windows are described and indexed without an alert. One, at 02:14:05, matches the event, and an alert goes out with its clip.

front_doorlive · windows of 5 s

Watching

  1. 02:13:55

    A car passes on the street

    indexed
  2. 02:14:00

    A person walks up the path to the door

    indexed
  3. 02:14:05

    A person stops at the front door and tries the handle

    alert sent
  4. 02:14:15

    The person walks back to the street

    indexed
An example on one camera. Every window is written to the index, and the one that matches the event also goes out as an alert, with its clip.

Long-term memory finds the moment, short-term memory shows it

In the morning someone asks what happened at the door. The index points to 02:14, and the frames from those seconds come back for a model to read.

How the frame times were measured
An example. The long-term index holds a short record of every moment at the front door this week, written as the feed arrives. A task asks what happened at the door last night; the index points to Tuesday 02:14:05 to 02:14:10, and those frames come back as a stream in 441 milliseconds, the measured median for a five-second window.

Memory that builds on itself

The frames never change. Everything above them is an index, and every task can add one.

Ask three ways
By meaning, by a filter such as camera or time, and by count
Write back
What a task works out is indexed under its own name, so the same question is one lookup next time
New indexes
Run a new prompt over frames you already have. Nothing is recorded again
Combine
One query can span several indexes, and one index name can cover every camera in a building

Why the memory sits outside the model

Four ways to give an agent a memory of video, and what each one leaves out.

Four ways to give an agent a memory of video: what each keeps, and what each leaves out
ApproachWhat it keepsWhat it leaves out
A longer context windowThe last minute or so of video, in full detailAnything older. Nothing can be searched, and the cost grows with every frame kept
Notes in a vector databaseSearchable text over a long time, cheap to storeThe picture. A note is written before the question is known, and there is no second look at the moment behind it
Video files and a clip pipelineEvery frame, in principleAn index, and time. Reaching one second means finding the file, decoding it and cutting a clip
VideoDBThe index and the frames in one system, live feeds included. Indexes that stack and combineThe model, which you bring

On a 100-hour benchmark for visual moment retrieval, VideoDB finds a matching clip for more queries than any other pipeline tested.

See the benchmark

Recall in code, or in a sentence

The same memory answers the SDK, any MCP client, and a plain question in Claude or ChatGPT.

recall.py
import videodb

coll = videodb.connect().get_collection()
cam = coll.get_rtstream("rts-...")   # the front door camera
index = cam.get_index("...")         # its long-term memory

# Ask: the index turns "last night" into a moment
found = cam.search(query="a person at the front door at night",
                   index_id=index.id)
shot = max(found.get_shots(), key=lambda s: s.start)  # the newest one

# Recall: the frames around that moment, as a stream a model reads
frames_url = shot.generate_stream()

# Remember: index what the task worked out,
# so the same question is one lookup next time
What comes back
text
What was seen, as it was written
start, end
The seconds it covers
stream_url
Those frames, as a stream
Or ask in Claude or ChatGPT
  • What happened at the front door last night?
  • How many deliveries came this week?
  • Show me the last time the gate was left open.

Questions about agent memory

What each memory is, and why the frames stay.

What is long-term memory for an agent?

A timestamped index of what its cameras and screens showed: short written records, each pointing back to the frames it came from. It is kept for days, weeks and months, and it stays small, because it holds text and pointers while the frames stay in storage.

What is short-term memory?

The frames the agent pulls back for the task in front of it: a few seconds around the moments its index points to, read now, for this question. Long-term memory says where to look. Short-term memory shows what was there.

What is real-time memory?

What is happening right now: the live feed, written into the index as it arrives, a few seconds behind the present. It keeps the long-term memory current, and it supplies the last few seconds when a task is about now.

Why keep the frames when the agent already has notes?

A note is written before the question is known. It holds what seemed important at the time, so it can miss the detail a later task needs, such as which hand was on the door or what the label said. The index gets the agent to the right second. The frames show what was in it.

Why not give the model a longer context window?

A day of video from one camera is 86,400 frames, even at one frame a second. Keeping hours of raw frames in a model’s context is slow and costly, and most of them have nothing to do with the task. A memory outside the model keeps the whole record and hands the model only the seconds it needs.

Can I build this with a vector database and files in a bucket?

You can keep notes in a vector database and recordings in a bucket. What is missing is the path from a note back to its frames while the task is still running, and an index over the recordings themselves. VideoDB keeps the index and the frames in one system, and new indexes can be built over the same frames at any time.

Does it work for screens as well as cameras?

Yes. A screen is a live feed like any other. An agent that watched a desktop for a week needs the same two things: an index of what it saw, and the exact frames back when a task refers to them.

Where do the recordings stay?

In your own storage. VideoDB runs on top of the files, in our cloud paired with your bucket or entirely inside your account.

Run in our cloud or yours

VideoDB is video infrastructure for agents and devices that see. 1,000+ developers and agents use it every week, and it has processed 300+ TB of video.

Security

Runs on any cloud

AWS, Google Cloud and Azure, or your own cloud.

Security and compliance

SOC 2 Type II, GDPR and ISO 27001 compliant.

  • SOC 2 Type II
  • GDPR
  • ISO 27001

Start with one camera or one screen

Free to start, with $20 of credits and no credit card. For a fleet of feeds, tell us about them and we will set up a pilot with you.

https://videodb.io/agent-memory.mdOpen the file

## Use VideoDB in your agent

VideoDB lets an AI agent watch, search and clip video.

- **Claude**: [Add VideoDB from the plugin directory](https://videodb.io/claude)
- **ChatGPT**: [Add VideoDB from the plugin directory](https://chatgpt.com/plugins/videodb)
- **Claude Code**: `claude mcp add --transport http videodb https://mcp.videodb.io/mcp`
- **Cursor or any MCP client**: Add the remote MCP server and log in once: `https://mcp.videodb.io/mcp`

Free to start: $20 of credits, no credit card. [Steps for every client](https://docs.videodb.io/pages/mcp/connect).