Agent memory
Give your agent a memory of everything it sees
A live feed is gone as soon as it plays. VideoDB reads it as it arrives, acts on what matters now, and keeps all of it as memory: an index of what was seen, and the frames behind any moment in about half a second.
Every window of the feed gets a decision
The feed is read a few seconds at a time. Each window is described, checked against the events you wrote, and written to the index. The ones that match also go out as alerts.
Watch a camera like this liveAn example, invented: four minutes of a front door camera at night, read in 48 windows of five seconds, with one event to watch for, a person checking the lock. Forty-seven windows are described and indexed without an alert. One, at 02:14:05, matches the event, and an alert goes out with its clip.
front_door
Watching
- 02:13:55
A car passes on the street
indexed - 02:14:00
A person walks up the path to the door
indexed - 02:14:05
A person stops at the front door and tries the handle
alert sent - 02:14:15
The person walks back to the street
indexed
Long-term memory finds the moment, short-term memory shows it
In the morning someone asks what happened at the door. The index points to 02:14, and the frames from those seconds come back for a model to read.
How the frame times were measuredMemory that builds on itself
The frames never change. Everything above them is an index, and every task can add one.
- Ask three ways
- By meaning, by a filter such as camera or time, and by count
- Write back
- What a task works out is indexed under its own name, so the same question is one lookup next time
- New indexes
- Run a new prompt over frames you already have. Nothing is recorded again
- Combine
- One query can span several indexes, and one index name can cover every camera in a building
Why the memory sits outside the model
Four ways to give an agent a memory of video, and what each one leaves out.
| Approach | What it keeps | What it leaves out |
|---|---|---|
| A longer context window | The last minute or so of video, in full detail | Anything older. Nothing can be searched, and the cost grows with every frame kept |
| Notes in a vector database | Searchable text over a long time, cheap to store | The picture. A note is written before the question is known, and there is no second look at the moment behind it |
| Video files and a clip pipeline | Every frame, in principle | An index, and time. Reaching one second means finding the file, decoding it and cutting a clip |
| VideoDB | The index and the frames in one system, live feeds included. Indexes that stack and combine | The model, which you bring |
On a 100-hour benchmark for visual moment retrieval, VideoDB finds a matching clip for more queries than any other pipeline tested.
See the benchmarkRecall in code, or in a sentence
The same memory answers the SDK, any MCP client, and a plain question in Claude or ChatGPT.
import videodb
coll = videodb.connect().get_collection()
cam = coll.get_rtstream("rts-...") # the front door camera
index = cam.get_index("...") # its long-term memory
# Ask: the index turns "last night" into a moment
found = cam.search(query="a person at the front door at night",
index_id=index.id)
shot = max(found.get_shots(), key=lambda s: s.start) # the newest one
# Recall: the frames around that moment, as a stream a model reads
frames_url = shot.generate_stream()
# Remember: index what the task worked out,
# so the same question is one lookup next time
- text
- What was seen, as it was written
- start, end
- The seconds it covers
- stream_url
- Those frames, as a stream
What happened at the front door last night?
How many deliveries came this week?
Show me the last time the gate was left open.
Questions about agent memory
What each memory is, and why the frames stay.
What is long-term memory for an agent?
A timestamped index of what its cameras and screens showed: short written records, each pointing back to the frames it came from. It is kept for days, weeks and months, and it stays small, because it holds text and pointers while the frames stay in storage.
What is short-term memory?
The frames the agent pulls back for the task in front of it: a few seconds around the moments its index points to, read now, for this question. Long-term memory says where to look. Short-term memory shows what was there.
What is real-time memory?
What is happening right now: the live feed, written into the index as it arrives, a few seconds behind the present. It keeps the long-term memory current, and it supplies the last few seconds when a task is about now.
Why keep the frames when the agent already has notes?
A note is written before the question is known. It holds what seemed important at the time, so it can miss the detail a later task needs, such as which hand was on the door or what the label said. The index gets the agent to the right second. The frames show what was in it.
Why not give the model a longer context window?
A day of video from one camera is 86,400 frames, even at one frame a second. Keeping hours of raw frames in a model’s context is slow and costly, and most of them have nothing to do with the task. A memory outside the model keeps the whole record and hands the model only the seconds it needs.
Can I build this with a vector database and files in a bucket?
You can keep notes in a vector database and recordings in a bucket. What is missing is the path from a note back to its frames while the task is still running, and an index over the recordings themselves. VideoDB keeps the index and the frames in one system, and new indexes can be built over the same frames at any time.
Does it work for screens as well as cameras?
Yes. A screen is a live feed like any other. An agent that watched a desktop for a week needs the same two things: an index of what it saw, and the exact frames back when a task refers to them.
Where do the recordings stay?
In your own storage. VideoDB runs on top of the files, in our cloud paired with your bucket or entirely inside your account.
Run in our cloud or yours
VideoDB is video infrastructure for agents and devices that see. 1,000+ developers and agents use it every week, and it has processed 300+ TB of video.
SecurityRuns on any cloud
AWS, Google Cloud and Azure, or your own cloud.
AWSGoogle CloudAzure
Security and compliance
SOC 2 Type II, GDPR and ISO 27001 compliant.
Start with one camera or one screen
Free to start, with $20 of credits and no credit card. For a fleet of feeds, tell us about them and we will set up a pilot with you.