# How a robot remembers > “Find my bottle and bring it to me.” A robot can only do this if it remembers what it has seen. This is a plain explanation of memory for robots and agents: the frames it keeps, the indexes that reach them, and how every task leaves the memory better than it found it. - Category: Essay - Published: 2026-10-27 - Authors: Ashutosh Trivedi - Canonical: https://videodb.io/blog/how-a-robot-remembers - HTML: https://videodb.io/blog/how-a-robot-remembers · Markdown: https://videodb.io/blog/how-a-robot-remembers.md - Tags: physical AI, memory, robotics, agents, retrieval --- Who this is for: anyone building or buying robots, agents or devices that live with people. No background in robotics is needed. Every term is explained where it first appears. "Find my bottle and bring it to me." A person says this to a home robot without thinking about it. Almost everything the robot needs is hidden inside the word "my". Which bottle? Where is it now? Has it moved since the morning? The robot's cameras cannot answer any of that. The answers are in what the robot saw yesterday. This post is about that problem: how a robot keeps a memory. We will keep it simple: one robot in one house, given one task. By the end you should be able to say what long-term memory, short-term memory and real-time memory mean for a machine, why that memory has to be visual, and how VideoDB provides it. Everything here applies to agents as much as robots. A computer-use agent that watched you work for a week has the same problem as a robot that watched your kitchen. ## Without memory, every task starts from zero A robot sees the world as a continuous stream of frames. Thirty frames a second, from several cameras, all day. Today's models can look at the current frame and decide what to do next, and that is a real achievement. Such a machine lives entirely in the present. Ask a robot like that to find your bottle and it has one option: search. Every room, every surface, until it sees a bottle and guesses. If there are three bottles in the house, it guesses wrong often. If the bottle is in a drawer, it may never find it. Physical Intelligence measured this directly: in their Find Object task, a policy (the model that turns what the robot sees into movement) without memory had to choose between four drawers and succeeded about 25% of the time, which is chance [1]. Most jobs in a home or a workplace refer to the past. "Put it back where it was." "Did the delivery come?" "Why did this machine stop yesterday?" "Do the kitchen like last time." Each of these needs a moment the robot already saw. Without a way to get back to that moment, none of these jobs are possible, however good the hands and the model are. A robot that lives with people has to remember. How to read Figure 1Press play. The same task runs twice in the same house: once with no memory, once with memory. _The house has four rooms and eleven surfaces. The robot without memory has to visit them until it sees a bottle, and the first bottle it sees may be the wrong one. The robot with memory asks its own record where it last saw the blue bottle, and walks there._ ## What a memory for a robot has to be Three things, in plain words. **It has to be visual.** A robot's experience is what its cameras saw. A written note like "bottle on the dining table" is useful, and it is written before anyone knows the question. The note may not record the colour of the bottle. The frames do. A memory for a robot has to keep the frames, or a way back to them. **It has to have indexes.** Nobody can rewatch yesterday to answer a question today. Fifteen hours of video from one camera at 30 frames a second is about 1.6 million frames. Even at one frame a second, which is what models typically read, it is 54,000 frames. There has to be a written record that says what happened and when, so the robot can jump to the moment instead of scanning for it. Think of the index at the back of a book, which tells you the page without containing the chapter. **It has to build on itself.** The first time the robot works something out, it should write the answer down, so the second time is one lookup. A memory that only records is a recording. A memory that also keeps its own conclusions gets cheaper to use with every task. ## Long-term, short-term and real-time memory Here is the whole design, before the walkthrough. **Long-term memory is the index.** As the robot watches, it writes a running, timestamped log of what it sees: who, what, where, in short records. Every record points back to the frames it came from. The log is kept for days, weeks and months. It is small, because it holds text and pointers while the frames stay in storage. And it can be queried: by meaning ("George drinking"), by filter (room is bedroom, after 6 pm), by count (how many times today). **Short-term memory is the frames for the task at hand.** When a task arrives, the robot pulls back a few seconds of actual frames from the moments its index points to, and processes them now, for this question. The index said "blue bottle, bedside table". The frames show which side of the lamp, whether the cap is on, whether a book is lying on top of it. The model, whether a vision-language-action model (VLA), a vision-language model or an agent, reads these frames and decides what to do. **Real-time memory is right now.** The live stream, written into the index as it arrives, a few seconds behind the present. It is what keeps the long-term memory current, and it supplies the last few seconds when the task is about now: "is the door still open?" This split is close to how Physical Intelligence built memory into their π0.6 policy, where short-term memory holds raw observations through a video encoder and long-term memory holds events in language [1]. The difference is where the memory lives. In their model it sits inside the model's context, which keeps the video part to about a minute. We put it outside the model, in a database, so it can be as long as the robot's working life, and the model only sees the part the task needs. One distinction is worth naming. Inside a policy, short-term memory means the last few seconds of observations it carries at control rate, for things like changing a grasp that failed. That buffer stays in the model, and nothing here replaces it. The short-term memory in this post is what the robot pulls back from storage for the task, and it feeds the same model. How to read Figure 2The loop runs on its own. Watch the index fill on the left, then the task arrive, then the frames come back, then the new record get written. _Long-term memory is the index, always on. Short-term memory is the frames, pulled back for the task. The first time, "my bottle" takes two lookups and a look at the frames. The robot then writes down what it worked out, and the second time takes one lookup._ ## George's bottle, step by step Let us walk through the task once, slowly. George lives with the robot. Over the day the robot watched, and as it watched it wrote records into its long-term memory. Here is a piece of that log.
TimeWhereRecord
08:02kitchenGeorge drinks from a blue bottle at the counter
08:15dining tableblue bottle next to the fruit bowl
13:40living roomGeorge drinks from a blue bottle on the sofa
19:55hallwayblue bottle carried towards the bedroom
20:12bedroomblue bottle placed on the bedside table
21:30kitchengreen bottle rinsed and left by the sink
Nothing in this log says "George's bottle". The robot never knew it would be asked. It only wrote what it saw, every few seconds, in words it would be able to search later. Now the task arrives. It is George who asks: "find my bottle and bring it to me." **Ask the index: "George drinking".** Two records come back, both from today, and in both of them the bottle is blue. "My bottle" now means "the blue bottle". This is a small piece of reasoning over two lines of text. **Ask the index again: "blue bottle", most recent.** One record: 20:12, bedroom, bedside table. The index has done its job. It turned a vague word into a place and a time. **Recall the frames.** The robot pulls the eight seconds around 20:12 into short-term memory and the model looks at them, now, for this task. The blue bottle is on the bedside table, to the right of the lamp, with the cap on. The index could never have told it all that, because nobody decided in advance that "right of the lamp" was worth writing down. **Act.** The plan is short: go to the bedroom, pick up the bottle from the right side of the bedside table, bring it to George. On the way, real-time memory keeps the index current. If the bottle has moved since 20:12, the robot asks the index once more, with "after 20:12" as a filter, and follows the newer record. **Remember.** This is the step that is easy to leave out. The robot writes a new record: "George's bottle: the blue bottle, last seen on the bedside table at 20:12", with a short visual description and a pointer to the frames that confirmed it. Tomorrow, "find my bottle" is one lookup. The memory grew because the task ran. ### When the index has nothing Suppose the robot had never written anything about drinking. The index has nothing to return for "George drinking". This is where the frames matter most. The robot asks for yesterday's frames from the kitchen and the living room, and runs a model over them with this question in mind: where does George drink, and from what. That pass is slower than a lookup, because it is real processing over real video. It also produces something the robot keeps: a new index of drinking moments. The next question about drinking is a lookup. The first answer costs real processing, and the second one is a lookup. A memory built this way keeps every piece of processing it has done, so none of it has to be repeated. The records are only as good as the analyzer that writes them. Who is drinking, what colour the bottle is and what the object is called have to come out the same way across cameras and days. That is one more reason the frames stay reachable: a record can always be checked against the picture it came from. How to read Figure 3It runs in three steps and loops. The robot watches and records are written from the frames. A new question reaches down to the frames once and leaves two layers behind. The same question again is one lookup at the top. _The frames at the bottom never change. Everything above them is an index, and every task can add one. A lookup against the top layer costs almost nothing. A question that reaches the bottom costs real processing, once, and leaves a new layer behind._ ## Why the frames have to stay A reasonable question at this point: why keep the frames at all? Why not keep good notes and throw the video away? Because a note is written before the question. Every note is a decision about what matters, made in advance: the colour of the bottle, which drawer, whether the cap was on, which side of the lamp. The note holds what seemed important at the time, and the task that arrives tomorrow can need anything. The frames hold everything, so the index gets the robot to the right second and the frames show what was in it. Keeping the frames costs less than it sounds. The recordings already exist, and VideoDB sits on top of them in your own storage. What is new is the index, and the index is small. Two results point the same way, for different parts of the system. Physical Intelligence keeps the policy's short-term memory as raw observations because dense image memory is "well-suited to resolve occlusions" and lets the robot adapt how it handles an object [1]. At Stanford, MemER gives its high-level policy a handful of saved frames (keyframes) and found that remembering with images alone did better than remembering with text, or with text and images together; their reading is that the model over-indexes on the text and misses details in the picture [2]. Both are about frames inside the policy, and the lesson carries to the archive: the detail a task needs is in the picture. ## Where VideoDB fits VideoDB is the memory outside the model. The cameras feed it, and the model asks it and gets frames back. Here is how each memory in this post maps to what runs today. - **Real-time memory.** Live streams connect directly (cameras over RTSP, the standard camera streaming protocol, and screens). The stream is kept whole, and analyzers run over it in rolling windows as it arrives, a few seconds behind the present. Alerts can fire on events described in plain language. - **Long-term memory.** The analyzers write the index: activity, location, objects, text on screen, or your own prompt that returns the same fields every time. Each record carries its timestamp and camera, and points back to the frames. Indexes are queryable three ways: by meaning, by exact filter, and by count. - **Short-term memory.** Any window around any record comes back as frames, as a stream the model reads or a person watches, with no export and no re-encoding. - **Memory that builds on itself.** A task's own processing can be indexed under a new name, so "drinking moments" or "known objects" become indexes the next task can query. The frames stay where they are, and only the indexes grow. ### Frames in half a second A robot acts in the flow of a task. When the index says "20:12, bedside table", the model needs the pictures of that moment before the robot takes its next step, and it needs them as pictures, because that is what a vision model reads. A note cannot be re-read for a detail it never held. The frames can. This is the step that is slow almost everywhere. Video sits in files, and reaching one second of one file means finding it, decoding from the nearest keyframe, cutting a clip and handing it over. That takes seconds to minutes per request, and it was designed for a person who will watch the clip later. VideoDB prepares every recording once for frame access, live streams included, and serves any window from it on request. We measured the stream step at a median of 433 to 447 ms for windows up to a minute, from one source or five, and under a second for a ten-minute window, same region ([the numbers are in the frame-access post](/blog/frame-access-that-keeps-research-moving)). The model's pass over the frames is on top of that and depends on the model. The frames arrive with their time and camera attached, so the robot's conclusion carries its provenance, and the same frames can be shown to a person who wants to check the robot's work. This is the part we built VideoDB for, and the part we have not seen anywhere else: the index and the frames in one system, about half a second apart, for recordings and live cameras alike, in the shape a model consumes. ### Indexes that stack The second thing a robot's memory needs is a way to keep getting better at finding things. In VideoDB an index is cheap to make, and there can be many of them over the same frames. - The first index is written as the robot watches: analyzers for activity, place, objects and text, or a prompt of your own with your own fields. Run them with a hosted model, a provider's model or an open-weight model you choose. - A task's processing becomes a new index. "Where does George drink" runs once over yesterday's frames and leaves "drinking moments" behind. - The model's conclusions become records too: "George's bottle = the blue bottle". - Indexes combine. One query can span several indexes by name, filter on any field (room, camera, after 20:12) and count (how many times today). One index name can cover every camera in a building and every robot in a fleet. - The frames never change, so a better model next year can re-index the same recordings without recording anything again. Each layer is built from the one below it, and a lookup at the top costs almost nothing. Every task adds to the memory instead of spending it. ### What is there, and what is not Four ways to give a robot a memory, and what each one leaves out.
ApproachWhat is thereWhat is not there
A longer context window inside the modelThe last seconds, at control rate. The right place for changing a grasp.Anything older than about a minute. Nothing can be queried, and the cost grows with every frame kept.
Notes in a vector databaseSearchable text over a long time. Cheap to store.The picture. Every note is a decision made in advance, and there is no second look at the moment behind it.
Video files in a bucket, with a clip pipelineEvery frame, in principle.An index, and time. Reaching one second means finding the file, decoding and cutting a clip: seconds to minutes per request, and nothing a robot can do mid-task.
VideoDBThe index and the frames together, about half a second apart. Indexes that stack and combine. Live cameras written in as they arrive. Recordings stay in your own storage. Runs in our cloud or yours.The model, which you bring, and the policy's own few-second buffer, which stays in the model.
### Proof We have run this at scale. VideoDB processed 100,000 hours of video for one of the largest video data providers, and more than 250 TB of video overall. On a 100-hour egocentric benchmark of 2,835 videos and 11,369 text queries, VideoDB's search led AWS Nova and Twelve Labs at finding the right moment, and when the result had to match the action's actual start and end it led by about 2x ([the benchmark post](/blog/a-faster-path-to-the-next-policy) has every number and the code). On [datasets.bot](https://datasets.bot), more than 50,000 hours of public egocentric video are searchable this way today. The George loop, in the shape the SDK uses: ```python import videodb conn = videodb.connect() coll = conn.get_collection() # 1. See: connect the robot's cameras and keep the history kitchen = coll.connect_rtstream(url=KITCHEN_RTSP, name="kitchen", store=True) # 2. Write: long-term memory is an index written every few seconds understanding = kitchen.understand( segmentation={"type": "time", "window": "5s"}, analyzers=[{ "type": "vlm", "name": "activity", "sampling": {"frame_count": 2}, "config": {"schema": {"person": "string", "action": "string", "objects": ["string"], "place": "string"}} }], store=True ) memory = kitchen.index(name="home-activity", source=understanding.outputs["activity"], use_for=["semantic", "query", "aggregate"]) # 3. Ask: two lookups turn "my bottle" into a place and a time drinking = coll.semantic_search(query="George drinking", index_names=["home-activity"], top_k=5) # both hits say "blue bottle", so "my bottle" means the blue one sightings = coll.semantic_search(query="blue bottle", index_names=["home-activity"], top_k=10) last_seen = max(sightings.get_shots(), key=lambda shot: shot.start) # the newest one # 4. Recall: the frames around that moment, for the model to read now frames_url = last_seen.generate_stream() # 20:12:04 → 20:12:12 # 5. Remember: index what the task worked out, so next time is one lookup ```

The calls follow the shapes in the VideoDB docs: a live stream connected with its history kept, an understanding run whose prompt asks the model for the same fields every time, a named index that can be searched, filtered and counted, and a stream generated from a search result. Step 5 is a second understanding run over the recalled window, indexed under a new name.

The same loop works for a fleet. Every robot writes to the same index name, and a question like "which robots saw the blue bottle this week" is one query over that shared index. It also works for an agent on a screen: the screen is the camera, the activity index is what it watched, and "the error I saw on Tuesday" is a lookup followed by eight seconds of frames. ## What the research says Memory for robots is an active field, and the results point in one direction: memory is what lets a robot finish a job that runs longer than a few seconds. **Physical Intelligence, MEM (March 2026).** Multi-Scale Embodied Memory adds a short-term video memory and a long-term language memory to the π0.6 policy. With memory, the same policy handles tasks that span up to fifteen minutes, such as making a grilled cheese sandwich and cleaning a kitchen. Across six tasks that test memory, average task progress (the share of a task completed) rises from roughly 30% without memory to roughly 73% with the video memory alone (read from Figure 8 of the paper, which compares memory designs without the language memory). After training on a small set of human corrections, the policy with memory also changes strategy after a failed attempt, where the policy without memory repeats it [1]. **Stanford, MemER (October 2025, ICLR 2026).** A high-level policy selects and keeps a few keyframes from its own experience and uses them, with the latest frames, to write short instructions for the low-level policy that moves the arm. On three real tasks that need minutes of memory, MemER passed 90% on every task, and a policy given a long raw history of 32 frames instead was on average 34% worse [2]. **NVIDIA, ReMEmbR (September 2024).** A navigation robot captions its camera feed in three-second windows and stores each caption with its time and position in a searchable database. A language-model agent then answers questions over drives of up to about 20 minutes by retrieving by text, by time or by position, in about 25 seconds a question, and the authors report that the answer time stays roughly flat as the video grows [3]. This is the long-term memory of this post, built for a robot that moves. **Ego4D, the Episodic Memory benchmark (2021).** The question "where was object X last seen in the video?" is a benchmark task over roughly 800 hours of first-person video [4]. It is the find-my-bottle problem, posed to models five years ago. **MIT and Lockheed Martin, Worth Remembering (June 2026).** This work assumes a robot cannot store everything, so it keeps only the moments a video model finds surprising, about 1.7% of the observed frames, as short episodes. Each episode stores the frames with the robot's position and the time [5]. What to keep is an open choice. This system and MemER keep the pictures themselves; ReMEmbR keeps captions. Our answer is to keep the recording where it already is and keep the index small. **Zhejiang University, EmbodiedMemory-Bench (September 2026).** A benchmark of 2,554 interactive episodes finds current agents weak at fine-grained visual memory and at tracking a changing world state, and proposes an external memory organised into spatial, event and scene memories [6]. External memory, outside the model, is the direction this post takes too, with a simpler shape: a timestamped log with the frames reachable, instead of a maintained model of the world. ## What this changes Cameras let machines see. Models let them reason. Machines that live in our world also have to remember what they have seen, and the memory has to be visual, indexed and able to grow. With that in place, the sentence at the top of this post stops being a research problem. The robot asks its own record, looks at the frames, goes to the bedside table, and comes back with the right bottle. Then it writes down what it learned, so that tomorrow it does not have to work it out again. Every task should leave the memory better than it found it. ## FAQ **What is long-term memory for a robot?** A timestamped index of what the robot saw: short written records of activity, place and objects, each pointing back to the frames it came from. It is kept for days, weeks and months, and it is small, because it holds text and pointers while the frames stay in storage. **What is short-term memory for a robot?** The frames the robot pulls back for the task at hand: a few seconds around the moments its index points to, processed now, for this question. Long-term memory says where to look. Short-term memory shows what was there. **What is real-time memory?** What is happening right now: the live camera stream, written into the index as it arrives, a few seconds behind the present. It keeps the long-term memory current and supplies the last few seconds when the task is about now. **Why does a robot's memory have to keep the frames, when it already has notes?** A note is written before the question is known. It records what seemed important at the time, so it may leave out the detail the task needs later, such as the colour of a bottle or whether a drawer was closed. The index gets the robot to the right second. The frames tell it what was in that second. **Why not give the model a longer context window instead?** Fifteen hours of video from one camera is about 1.6 million frames at 30 frames a second, and 54,000 frames even at the one frame a second models typically read. Keeping even an hour of raw frames in a model's context is slow and expensive, and Stanford's MemER found that a policy fed a long raw history did worse than one given a few selected keyframes [2]. A memory outside the model keeps the full record and lets the model see only what the task needs. **Can I build this with a vector database and video files in a bucket?** You can keep notes in a vector database and recordings in a bucket. What is missing is the path from a note back to its frames in the time a robot has: a clip pipeline takes seconds to minutes per request, and there is no index over the recordings themselves. VideoDB keeps the index and the frames in one system, about half a second apart, and lets new indexes be built over the same frames at any time. **Does this apply to software agents too?** Yes. A computer-use agent that watched a screen for a week has the same problem as a robot that watched a kitchen. It needs an index over what it saw and a way to pull the exact frames back when a task refers to them. ## References 1. Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn and Danny Driess. *MEM: Multi-Scale Embodied Memory for Vision Language Action Models.* Physical Intelligence, March 2026. [arXiv:2603.03596](https://arxiv.org/abs/2603.03596) · [research post](https://www.pi.website/research/memory) 2. Ajay Sridhar, Jennifer Pan, Satvik Sharma and Chelsea Finn. *MemER: Scaling Up Memory for Robot Control via Experience Retrieval.* Stanford University, October 2025; ICLR 2026. [arXiv:2510.20328](https://arxiv.org/abs/2510.20328) 3. Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya and Yan Chang. *ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation.* NVIDIA, September 2024. [arXiv:2409.13682](https://arxiv.org/abs/2409.13682) 4. Kristen Grauman et al. *Ego4D: Around the World in 3,000 Hours of Egocentric Video.* 2021. The Episodic Memory benchmark. [arXiv:2110.07058](https://arxiv.org/abs/2110.07058) 5. Nicolas Gorlo, Derek K. Wise, Alberto Speranzon and Luca Carlone. *Worth Remembering: Surprise-Gated Robot Episodic Memory.* MIT and Lockheed Martin, June 2026. [arXiv:2606.03787](https://arxiv.org/abs/2606.03787) 6. Lizhou Liang, Xinyu Zhong, Miao Pan, Xiaohe Zhou, Xuanyu Liu, Qinfeng Li, Peng Li, Jintao Chen, Xuhong Zhang and Wenqi Zhang. *EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks.* Zhejiang University, September 2026. [arXiv:2609.28236](https://arxiv.org/abs/2609.28236) 7. Samuel Alexander. [*Frame access that keeps research moving.*](/blog/frame-access-that-keeps-research-moving) VideoDB, October 2026. Measured stream generation latency for windows from 5 seconds to 10 minutes. 8. Ashutosh Trivedi. [*Episodic memory for agents.*](/blog/episodic-memory-for-agents) VideoDB, 2026. The earlier post on semantic and episodic memory for agents. ## FAQ **What is long-term memory for a robot?** A timestamped index of what the robot saw: short written records of activity, place and objects, each pointing back to the frames it came from. It is kept for days, weeks and months, and it is small, because it holds text and pointers while the frames stay in storage. **What is short-term memory for a robot?** The frames the robot pulls back for the task at hand: a few seconds around the moments its index points to, processed now, for this question. Long-term memory says where to look. Short-term memory shows what was there. **What is real-time memory?** What is happening right now: the live camera stream, written into the index as it arrives, a few seconds behind the present. It keeps the long-term memory current and supplies the last few seconds when the task is about now. **Why does a robot's memory have to keep the frames, when it already has notes?** A note is written before the question is known. It records what seemed important at the time, so it may leave out the detail the task needs later, such as the colour of a bottle or whether a drawer was closed. The index gets the robot to the right second. The frames tell it what was in that second. **Why not give the model a longer context window instead?** Fifteen hours of video from one camera is about 1.6 million frames at 30 frames a second, and 54,000 frames even at the one frame a second models typically read. Keeping even an hour of raw frames in a model's context is slow and expensive, and Stanford's MemER found that a policy fed a long raw history did worse than one given a few selected keyframes. A memory outside the model keeps the full record and lets the model see only what the task needs. **Can I build this with a vector database and video files in a bucket?** You can keep notes in a vector database and recordings in a bucket. What is missing is the path from a note back to its frames in the time a robot has: a clip pipeline takes seconds to minutes per request, and there is no index over the recordings themselves. VideoDB keeps the index and the frames in one system, about half a second apart, and lets new indexes be built over the same frames at any time. **Does this apply to software agents too?** Yes. A computer-use agent that watched a screen for a week has the same problem as a robot that watched a kitchen. It needs an index over what it saw and a way to pull the exact frames back when a task refers to them.