NFL game analysis: cutting VLM hallucinations by 80%.

Vision language models shine in controlled benchmarks, then stumble on real-world, event-dense footage such as an NFL game. VideoDB bridges that gap by letting developers slice video at the right semantic boundaries, combine external stats, and run multi-tier visual and LLM pipelines that cut hallucinations by more than 80% while costing up to 70% less than a naive “1 fps into Gemini” workflow. If you need fast, cheap, accurate visual reasoning, you need more than a big VLM. You need video-native AI infrastructure.

In this writeup we explore basic to advanced approaches over the same NFL game footage. We want it to guide you through solving real-world scenarios in video understanding. To judge how each method performed, we focused on four axes:

1. The naive Gemini approach

Initially, we tried directly inputting complete NFL game footage into Gemini, expecting robust results based on benchmark promises.

Observations

The model frequently produced incorrect or imaginary events, and misclassifications were common. It also missed or overlooked critical events entirely, revealing significant weaknesses in accuracy and contextual comprehension.

Known VLM limitations (why “just send it to Gemini” falls short)

• Finite context windows. Even a 1M-token window can’t hold one NFL quarter at 30 fps.
• Image-tile token explosion. Every 1080p frame is split into roughly 4–9 tiles (about 1–4k tokens) before the model “sees” it.
• Weak event reasoning. Current VLMs reason per-frame, not per-play. They miss temporal causality, e.g. “Was the QB still behind the line when he released?”
• Cost scales linearly with frames, so 30 fps steals wallets fast.

2. Uniform-length chunks (possible with VideoDB)

We chopped footage into fixed 2s / 5s / 10s clips via VideoDB’s scene index API. The hypothesis was straightforward: shorter, consistently sized segments might simplify the VLM’s task and improve accuracy.

Code snippet for uniform chunk segmentation:

import videodb

conn = videodb.connect(api_key="YOUR_API_KEY")
collection = conn.get_collection()

video = collection.upload(url="https://www.youtube.com/watch?v=pA_xAsb5hbA")

# Analyze fixed five-second windows with eight representative frames per window
uniform_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "uniform_play_analysis",
            "sampling": {"strategy": "uniform", "frame_count": 8},
            "config": {
                "prompt": "Summarize the football action in this five-second segment.",
                "schema": {"summary": "string"},
            },
        }
    ],
)
uniform_understanding.wait_until_complete()

uniform_analyzer = uniform_understanding.get_analyzer("uniform_play_analysis")
uniform_output = uniform_analyzer.get_output()

# Make the summaries searchable
uniform_index = video.index(
    name="uniform_play_analysis",
    source=uniform_analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["summary"]},
)
uniform_index.wait_until_complete()

print(f"Understanding ID: {uniform_understanding.id}")
print(f"Index ID: {uniform_index.index_id} ({uniform_index.status})")
print(uniform_output["scenes"][0])

The goal of testing these configurations was to understand how varying segment lengths could impact model accuracy and output clarity.

Observations

Uniform chunking scored worse than sending the whole game. Cutting on a clock instead of on the action destroys the very context the model needs.

Arbitrary clip boundaries lose context

• Important actions (e.g. a QB throw) get split across two clips, so the model can’t see the full play and misjudges legality or outcome.
• The same problem shows up for catches, interceptions, and other decisive moments.

Higher hallucination rate

• The model starts “imagining” passes and catches that never happened, simply because it lacks enough temporal evidence in a single clip.
• The resulting event timeline is noisy and bloated with false positives.

Goldilocks problem with clip length

• Too long → information overload and confusion.
• Too short → not enough context.
• Neither extreme works. We need a balanced segmentation window.

3. Play-by-play segmentation (advanced pipeline with VideoDB)

During our analysis, we found that detailed statistical reports for major sports games are typically publicly available. These reports offer extensive information, ranging from basic team formations and player lineups to precise, event-specific details such as passes, touchdowns, interceptions, and catches. Most importantly for our analysis, they include exact timestamps marking the start and end of each play, making them ideal for accurate segmentation.

Reliable sources for such detailed play-by-play data include official NFL scores pages, where you can select specific seasons, weeks, and games to access comprehensive statistics. For our analysis, we referred directly to the official game summary PDF.

However, we hit a significant practical issue: the timestamps in these reports reflect the official game clock, not the video’s runtime. To segment the video correctly, we needed to align the two.

3.1 Aligning game-time with video-time

To solve this, we used the one visual feature present in every NFL broadcast: the on-screen scoreboard. It continuously displays scores, current quarter, down and yardage, and crucially, the game clock itself. By extracting that, we could map the game’s official timestamps to corresponding points in the video.

How we achieved this

• OCR-based timestamp extraction. We processed the video with an optical character recognition model to detect and extract visible game times from the scoreboard throughout the video.
• Frame sampling optimization. To optimize efficiency, we sampled just one frame per second (1 fps) for OCR. This significantly reduced computational load without compromising the accuracy of the extracted timestamps.
• Timestamp mapping creation. The OCR results gave an exact correlation between official game timestamps and actual video runtime. Using this mapping, we segmented the video accurately into individual play-by-play events.

# Analyze one representative frame for every second of video
scoreboard_understanding = video.understand(
    segmentation={"type": "time", "seconds": 1},
    analyzers=[
        {
            "type": "vlm",
            "name": "scoreboard",
            "sampling": {"strategy": "uniform", "frame_count": 1},
            "config": {
                "prompt": (
                    "Read the scorebar at the bottom of the frame. Extract both "
                    "team names and scores, the quarter number, and the game clock."
                ),
                "schema": {
                    "team_1_name": "string",
                    "team_1_score": "integer",
                    "team_2_name": "string",
                    "team_2_score": "integer",
                    "quarter_number": "integer",
                    "game_clock": "string",
                },
            },
        }
    ],
)
scoreboard_understanding.wait_until_complete()

scoreboard_output = scoreboard_understanding.get_analyzer("scoreboard").get_output()

# Map each video-runtime second to its structured scoreboard reading
scene_ocr_results = {
    float(scene["start"]): scene["data"]
    for scene in scoreboard_output["scenes"]
}

for video_time, scoreboard in list(scene_ocr_results.items())[:5]:
    print(video_time, scoreboard)

Integrating play-by-play segmentation with VideoDB

VideoDB supports indexing this customized, non-uniform timeline directly. We imported our precise timestamp mappings and created accurate, detailed index records from them:

# Step 1: Use the stats PDF to filter all play timestamps (game clock) where a
# catch occurred into `catch_play_scenes` as a list of (start, end) for plays with catches

# Step 2: Map game clocks to video timestamps using OCR outputs

# Analyze short windows once; they will be joined to official play ranges below
catch_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "catch_analysis",
            "sampling": {"strategy": "uniform", "frame_count": 8},
            "config": {
                "prompt": (
                    "This segment is part of a play containing a catch. Extract the "
                    "catch type, player position, and whether it is an interception."
                ),
                "schema": {
                    "catch_type": "string",
                    "player_position": "string",
                    "interception": "boolean",
                },
            },
        }
    ],
)
catch_understanding.wait_until_complete()
catch_output = catch_understanding.get_analyzer("catch_analysis").get_output()

def overlaps(scene, start, end):
    return float(scene["start"]) < end and start < float(scene["end"])


catch_details = []
for start_time, end_time in catch_play_scenes:
    evidence = [
        scene["data"]
        for scene in catch_output["scenes"]
        if overlaps(scene, start_time, end_time)
    ]
    if not evidence:
        continue

    catch_details.append(
        {
            "start": start_time,
            "end": end_time,
            "catch_type": ", ".join(
                dict.fromkeys(item["catch_type"] for item in evidence if item["catch_type"])
            ) or "none",
            "player_position": ", ".join(
                dict.fromkeys(
                    item["player_position"]
                    for item in evidence
                    if item["player_position"]
                )
            ) or "none",
            "interception": any(item["interception"] for item in evidence),
        }
    )

catch_index = video.index(
    name="catch_plays",
    source=catch_details,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["catch_type", "player_position"],
        "filter": ["interception"],
    },
)
catch_index.wait_until_complete()

By adopting this precise segmentation strategy, VideoDB improved accuracy, dramatically reduced misclassifications, and simplified complex visual analysis tasks, providing detailed insight into sports event analysis.

Observations

Approach comparison

Key takeaways

1. Define key sports concepts. Clearly outline each concept required for analysis. For example: catch (yes/no), running play (yes/no), scoring event (yes/no).
2. Check availability of statistical data. Determine whether these concepts can be reliably extracted from existing statistical data. If it is available, use it to isolate specific plays. If it is not, use the VLM directly for visual extraction.
3. Extract relevant plays using statistical data. Use accurate statistical information to isolate relevant video scenes using the VideoDB timeline. Record timestamps and relevant metadata for these scenes.
4. Run visual analysis with VideoDB indexing. Pass the extracted scenes into the VLM to gather detailed visual insights, e.g. identifying catch types like “overhead” and positions like “near sidelines”.
5. Structure the output data clearly. Organize the extracted visual information into structured data for clarity and ease of querying. For instance:

[
  {
    "play_start_time": 12,
    "play_end_time": 52,
    "details": {
      "catch": true,
      "type": "overhead",
      "position": "near sidelines",
      "interception": false,
      "running_play": true
    }
  }
]

6. Add a query and reasoning engine (small LLM). On a user query, feed the structured data and the query into the VideoDB search interface. The engine processes these inputs and returns relevant, accurate play-by-play results.

Pricing: VideoDB vs. Gemini at 1 fps

* Prices use Google’s published rate card: $0.10 per M input tokens, $0.40 per M output; HD frames tokenize into 1,024–4,128 tokens each.

As frame rate or resolution rises, VideoDB’s flat visual-index pricing stays predictable while pure-Gemini costs explode. Current rates are on the pricing page.

Why choose VideoDB

• Event-aligned indexing. Cut by play, scene, or any custom timeline, not crude 1-second slices.
• Hybrid reasoning pipelines. Blend stats, embeddings, and VLMs to slice hallucinations to around 11%.
• Serverless scale. Ingest petabytes or a single clip, with zero idle cost.
• Developer-first API. Python, JS, REST. One call for streams, one for scene queries.
• Transparent pricing. Pay once for storage and index, then pick Entry / Advanced / SOTA LLM pricing per query.

FAQ

Why do VLMs hallucinate on sports footage?

Three reasons compound. Context windows can’t hold a game at broadcast frame rates, every 1080p frame explodes into thousands of tokens before the model sees it, and current VLMs reason per-frame rather than per-play, so temporal causality is lost. On our NFL test the naive whole-video approach hallucinated on 68.1% of events.

Do smaller video chunks reduce hallucinations?

No. Uniform 2s / 5s / 10s chunks scored worse than sending the whole game, at 74.2%. Fixed-length cuts split plays across boundaries, so the model never sees a complete action and invents the missing half. The problem is where you cut, not how small.

What is play-by-play segmentation?

Segmenting video at real semantic boundaries instead of on a timer. For NFL footage we took official play start and end timestamps from the public game summary, mapped them onto video runtime by OCR-ing the on-screen game clock at 1 fps, and indexed each play as its own scene. Hallucinations dropped to 11.4% with perfect temporal context.

How do you align the official game clock with video timestamps?

The broadcast scoreboard is the bridge. Sample one frame per second, run OCR with a structured prompt that returns scores, quarter, and game clock as JSON, and you get a lookup table from game clock to video runtime. Stats-sheet timestamps then translate into exact video time ranges.

Is VideoDB cheaper than sending frames straight to Gemini?

At 1 fps the two are comparable. The gap opens as frame rate rises, because visual-index pricing is flat while per-frame token costs scale linearly. For a 60-minute game at 30 fps that is $12.00 of indexing against $33–$220 of Gemini tokens.

Does this only work for American football?

No. The pattern generalizes to any domain with an authoritative event log and an on-screen clock: cricket, basketball, soccer, esports, and broadcast production. Where no stats feed exists, use the VLM directly for extraction and segment on scene changes instead.

Explore the docs

Learn more about the techniques and capabilities used in this case study: visual search pipelines for multi-modal search workflows, scene indexing for segmenting and indexing video at semantic boundaries, custom annotations for domain-specific metadata, and natural-language querying for getting better results out of your prompts.

If you’re choosing a model for a job like this, our guide to evaluating multimodal VLMs covers the harness. For the retrieval architecture underneath, see Video RAG: the definitive guide, and for why scoring these systems is harder than it looks, what video benchmarks get wrong about ground truth.