How to evaluate a video AI platform: a PoC checklist

Picture a robotics company with thousands of hours of video of its robots at work. An engineer needs to find specific moments in it, like every time a robot dropped a cup, to understand what went wrong or to pick clips for training the robot. Nobody can watch it all, so the team looks at a video AI platform: software that analyses video so you can search it by typing what you want, like “robot drops the cup”. VideoDB, Twelve Labs and AWS Nova are examples.

Before committing, the team runs an AI proof of concept (PoC): a short trial of the platform on its own recordings, to answer one question. Can it find what we need, well enough, at a price we accept? Here’s how to run one:

1. Write down the question and the pass/fail bar before you start.
2. Test on your own recordings, not the vendor’s demo.
3. Pick the right measure of “good”.
4. Test the whole system, not just the AI model.
5. Check the price for your full archive.
6. Name who will run it, then decide against the bar you set.

We’ll follow that robotics team through each step. The questions it has to answer:

1. What exactly should the trial prove?
2. How do you set the pass/fail bar before you start?
3. Why test on your own recordings?
4. How do you measure whether a search result is good?
5. Does the whole system hold up, not just the AI model?
6. What will it cost for the full archive?
7. Who runs it afterwards, and how do you decide?

In short:

• One question, written down. A trial that sets out to prove “AI is useful” can’t fail, so it can’t tell you anything.
• Your recordings, including the hard ones. A vendor’s demo shows what the platform can do on videos chosen to work, not what it will do on yours.
• “Good” depends on the task. On our 100-hour benchmark, the same platforms rank differently depending on whether a result has to show the whole moment or just part of it. We show why below.
• Price and ownership are part of the test. A platform that works on 50 hours but not on 50,000, or that nobody on the team can run, is a no.

What exactly should the trial prove?

The trial should answer one question that a decision depends on. For our robotics team: “Can we find every cup drop in last month’s robot runs without watching them?” That’s a good question. “Is video AI useful for us?” is not, because any result can be read as a yes.

Write the question so that someone outside the team could check the answer. Say what you’re looking for (cup drops), in which recordings (last month’s runs, around how many hours), and who will use the results (the engineer fixing the gripper).

Key takeaway. One question per trial. If you can’t say what a “no” would look like, the question is too broad.

How do you set the pass/fail bar before you start?

The pass/fail bar is the set of results the platform must reach for the answer to be yes. Write it before the first test. Written afterwards, it tends to slide towards whatever the platform happened to do well.

For each point, write what you’ll measure, the result you need, and how you’ll check it. For the cup-drop question it could look like this. The numbers are examples a team might choose, not measured results:

Set an end date at the same time. A trial without one tends to drag on without reaching a decision.

Key takeaway. Bar first, testing second. The same table becomes your scorecard at the end.

Why test on your own recordings, not the vendor’s demo?

A vendor’s demo uses videos picked to show the platform at its best. Your recordings have their own camera angles, lighting, robots and kinds of mistakes, and those are what the platform has to handle.

Build the test set from your real recordings:

• Include the hard ones. Dim lighting, a hand or object blocking the view, rare events and near-misses. This is where platforms tend to differ.
• Write down the answers first. Note where the drops are before you search, so you’re checking against your own list rather than looking at results and deciding they seem right.
• Add a few searches with no right answer. Ask for something that isn’t in the videos. A good platform returns nothing useful; a weak one returns confident wrong clips.

Key takeaway. Test on your recordings, with the hard cases in and the answers written down in advance.

How do you measure whether a search result is good?

There’s more than one way to score a search result, and they answer different questions. Two common ones:

• Hit@10 (the right moment appears somewhere in the first ten results). This answers “can I find it at all?”
• Temporal IoU, or tIoU (temporal intersection over union: the time the returned clip and the real moment share, divided by the total time either one covers, from 0 for no overlap to 1 for a perfect match). This answers “can I use the clip as it is?”, which matters when the clip goes into a report or into training data.

A clip can pass the first test and fail the second. Say a drop lasts ten seconds and the platform returns a four-second piece from the middle of it. That piece counts as a hit, but its tIoU is at most 0.4, because it misses most of the drop.

This difference shows up in real results. On our 100-hour action retrieval benchmark (2,835 videos filmed from the wearer’s point of view, and 11,369 typed searches), VideoDB Action Segmentation + Search found a right moment in the first ten results for 40.39% of searches, against 35.40% for AWS Nova and 32.87% for Twelve Labs Marengo 3.5. When the clip also had to line up with the real moment (tIoU ≥ 0.5), the scores were 20.49%, 10.01% and 9.61%. Under the benchmark’s looser overlap rule, Nova and Twelve Labs scored higher when the first 50 results were counted.

So which platform looks better depends on the measure. That’s the point for your own trial: choose the measure before you see the scores. The benchmark compares complete systems on videos of people, not robots, and it measures search only, not whether the clips make a robot learn better. Your own trial on your own recordings is what the decision should rest on. For how to score each search in detail, see Evaluating video VLMs on your own task.

Key takeaway. Choose the measure from the task. If you’ll reuse the clips, check that they show the whole moment, not just part of it.

Does the whole system hold up, not just the AI model?

The AI model is one part. Around it sit the steps that get your video in, make it searchable and show you the results. A platform can do well on a few uploaded clips and still struggle with the rest. Run the same path your real recordings will take:

• Getting video in. Upload recordings in the formats you actually use, from where you actually store them.
• Making it searchable. The platform analyses and labels the video (this is called indexing). Note how long it takes per hour of video.
• Searching. Type your real searches, in the words your team uses.
• Checking results. Every result should open at the right moment in the original recording, so a person can watch it and confirm.
• Re-analysing later. AI models improve. Ask what happens when you want to re-analyse the whole archive with a better one: is it a re-run, or do you have to upload everything again?

Key takeaway. Test the path from raw recording to a checked result. Surprises usually come from the steps around the model.

What will it cost for the full archive?

A trial usually runs on tens of hours of video. The real archive might be thousands. Ask for prices at the size you expect to reach, and check:

• Analysing and storing video. The cost per hour of video, and whether the platform keeps its own copy.
• Searching. What a search costs and how long it takes once the whole archive is in, not just the test set.
• Moving video. Where do your recordings live, and does the platform need them moved? If they come from sites in different countries, ask what moving them would involve, how much of it you’d actually use, and whether the analysis can run where the video is already stored.

Key takeaway. Price the archive you’ll have, not the sample you tested.

Who runs it afterwards, and how do you decide?

A platform that passes the trial still fails if nobody can run it afterwards. Before you decide, name the person or team who will own it, check who controls access to the recordings, and check that it fits your tools: an API your engineers can use, and results your other tools can open.

Then go back to the table you wrote at the start. Fill in the result for each line, mark pass or fail, and decide. If a line failed, the answer is no, or a narrower second trial with a new end date. Quietly lowering the bar doesn’t count.

Key takeaway. Decide against the bar you set first. Moving it after you see the results is how a trial stops being a test.

The same checklist works for other teams with large video archives, such as security camera footage or sports broadcasts. Swap the cup drop for the moment you need to find.

Where VideoDB fits

VideoDB is video data infrastructure for physical AI. The parts a trial like this would use are Machine annotation (segmentation and dense annotation of your footage, with one versioned index per model, so re-running a better model rewrites the index without touching the files) and Episode retrieval (plain-English search across every episode, with results returned as clips sized to the query). Video can come from S3 or GCS buckets, fleet uploads or collection partners.

If you’re planning a trial, we can run it on a sample of your own recordings against the bar you set. To try it yourself first: npx skills add video-db/skills (docs), or the hosted MCP server at mcp.videodb.io/mcp (docs).

FAQ

What is an AI proof of concept?

An AI proof of concept (PoC) is a short trial of an AI tool on your own data before you commit to it. It answers one question, such as whether the tool can find what you need in your videos, and ends in a yes or no against a pass/fail bar you set before the trial began.

What is a video AI platform?

A video AI platform is software that analyses your videos so you can search them by typing what you want to find, like “robot drops the cup”. VideoDB, Twelve Labs and AWS Nova are examples.

What is the difference between an AI proof of concept and a pilot?

A proof of concept checks whether the tool works on your data. A pilot checks whether it works inside your real workflow, with the people who will use it every day. Teams usually run the proof of concept first and the pilot after a yes.

How long should an AI proof of concept take?

Long enough to run a realistic sample of your own recordings through the whole system, and no longer. Set the end date at the same time as the pass/fail bar, so the trial cannot drift on without a decision.

Where to read more

• VideoDB leads a 100-hour action retrieval benchmark against AWS Nova and Twelve Labs: the full method and results behind the numbers above.
• Evaluating video VLMs on your own task: how to build a test set and score each search.
• Search over the Visual World: the design behind searchable video memory.
• Can a VLM annotate a robot run?: how accurately AI can mark where actions start and end.

Planning a trial of a video AI platform? Bring your own recordings and the bar you want it to meet.

@article{priya2026evaluate,
  author = {A Priya},
  title = {How to evaluate a video AI platform: a PoC checklist},
  journal = {VideoDB Labs},
  year = {2026},
  month = {oct},
  note = {https://videodb.io/blog/evaluate-video-ai-platform-poc-checklist},
}