### Announcing VidCap-12B: A Video Captioning Model Built for Web Scale

*[Inference.net](http://inference.net/) × Wynd Labs*

We're releasing VidCap-12B, a 12-billion parameter model that transforms video frames into structured, schema-compliant JSON designed specifically for temporal video understanding.

Unlike generic captioning models that produce varying descriptions frame-to-frame, VidCap-12B outputs consistent, structured fields that make video content programmatically accessible. Each frame generates JSON with standardized keys for objects, actions, environment, content type, and production quality. This structure enables you to track how scenes evolve, when objects appear or disappear, and how actions unfold across time, turning raw video into queryable, analyzable data.

The model was built in partnership with Wynd Labs to process over a billion videos while maintaining perfect schema adherence. Through careful distillation from larger vision-language models, VidCap-12B achieves near-identical output quality to models 10x its size while running at a fraction of the cost and latency. It's optimized for production workloads where you need consistent, valid JSON across millions to billions of frames without extensive post-processing.

---

### The Story: A Billion Videos and a Prohibitive Cost

This project began with a specific, massive-scale challenge from our partners at Wynd Labs: building a usable index of more than a billion videos captions with perfect adherence to a pre-defined JSON schema. Their initial approach using the open-source Gemma-27B was promising but the quality was not quite there, and while internal tests showed that much larger vision-language models produced superior, less hallucinatory JSON, their inference cost was prohibitive.

The goal became clear: we needed to capture the quality of a top-tier model in a package that was efficient enough to run at internet scale. This is a classic distillation problem.

### Training Process: From an Expert Teacher to a Distilled Student

Our process was methodical. We first selected a high-qualit vision-language model to act as the *teacher* and used it to generate structured JSON annotations for 1 million video frames from Wynd Labs’ dataset. We worked closely with their team, iterating on the prompt to ensure the structured output contained the exact, literal details needed for downstream tasks like retrieval.

With this high-quality dataset in hand, we began distilling that knowledge into a more compact "student." After evaluating several models, we selected Google's Gemma as the most promising student model. To find the optimal balance of performance and efficiency, we trained Gemma 12B on subsets of our data, benchmarking it at 100K and 1M samples to measure how it learned. At 1M samples, the Gemma 12B proved to nearly match the performance of Gemini 2.5 Pro*.

---

## Benchmark Results: Why 12B is the Right Choice for Production

Our benchmarks told a clear story, leading us to a definitive choice for this production workload.

### Price and Latency

That quality parity translates directly into significant savings. While these numbers are illustrative, they reflect the core trade-off we unlocked: distilling to a 12B model delivers the performance of a much larger model at a fraction of the cost and latency. For 1B requests, the price reduction becomes even more substantial: from $11M to $230K!


### ROUGE/BLEU Metrics (Accuracy of Specific Details)

Next, we measured how well the model names the specific objects and actions in a scene. ROUGE scores the overlap of exact words and phrases, while BLEU measures precision, penalizing irrelevant words. We wanted to make sure that the captions generated aligned with the captions of the teacher model. Our results showed that the more data we trained on the better the alignment between the teacher and the student outputs. To further confirm that that the distilled model was an improvement over Gemma 12B base we used LLM-as-a-judge, which showed a significant improvement in output quality and a reduction in hallucinations.


### FP8 Quantization: Efficiency Without Compromise

With a high-quality model in hand, our final step was to optimize it for production serving. We applied FP8 quantization, a technique that reduces the model’s memory footprint. We were pleased to find that, for this fine-tuned model, using FP8 had no measurable impact on the caption quality. This offers a "free" win for deployment—the same caption quality, but with significantly higher throughput and lower costs on modern hardware.


---

### Production Applications: Why Structured Output Changes Everything

Most video captioning fails because it treats each frame as an isolated image. VidCap-12B's structured JSON solves the fundamental problem: maintaining semantic consistency across time.

Consider searching for "chef cutting vegetables in professional kitchen" across a million hours of footage. Free-form captions might describe the same scene as "cooking show," "food preparation," or "kitchen scene" across consecutive frames. VidCap-12B's consistent schema means "chef" appears in the objects field every frame they're visible, "cutting vegetables" appears in actions with specific participants, and "professional kitchen" appears in environment. This isn't just better search; it's computationally tractable search at petabyte scale.

The structure enables entirely new workflows. Broadcasters detect true scene boundaries by monitoring when multiple fields change simultaneously: environment shifts from "news studio" to "outdoor location," objects change completely, and actions reset. This produces chapter markers that match how humans actually perceive scenes. Ad verification becomes deterministic: sponsored products must appear in objects, in "professional" production quality segments, for minimum durations. No more manual verification or disputed make-goods.

But the real breakthrough is temporal aggregation. Traditional captioning forces you to choose between frame-level noise and video-level averaging. Structured JSON lets you track state changes: when objects enter or leave, how actions progress, which environments persist. A car accident becomes a queryable sequence: "vehicle approaching intersection" followed by "collision with truck" followed by "airbag deployment," each with timestamped evidence. Complex queries like "find professional footage where someone picks up a product, examines it, then puts it back" become simple field comparisons across time windows. Search-first RAG systems can use hard JSON predicates for metadata filtering, slashing tokens, latency, and costs by 90%. This transforms video from opaque binary data into a structured, temporal database you can analyze, aggregate, and automate at internet scale.

## Minimal API Example

The model always takes a single frame as input and provides reliable JSON metadata directly. No external JSON schema is required.

import os
from openai import OpenAI
import base64
import requests

client = OpenAI(
    base_url="https://api.inference.net/v1",
    api_key='inference-e6c91778c252452ea5f5386e7b4d14eb'
)

SYSTEM_PROMPT = "You are an image annotation API trained to analyze YouTube video keyframes. You will be given instructions on the output format, what to caption, and how to perform your job. Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with 'This image shows' or 'This keyframe displays...', just get right into the details."

USER_PROMPT = """
You are an image annotation API trained to analyze YouTube video keyframes. You must respond with a valid JSON object matching the exact structure below.

Your job is to extract detailed **factual elements directly visible** in the image. Do not speculate or interpret artistic intent, camera focus, or composition. Do not include phrases like "this appears to be", "this looks like", or anything about the image itself. Describe what **is physically present in the frame**, and nothing more.

Return JSON in this structure:

{
    "description": "A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.",
    "objects": ["object1 with relevant visual details", "object2 with relevant visual details", ...],
    "actions": ["action1 with participants and context", "action2 with participants and context", ...],
    "environment": "Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).",
    "content_type": "The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.",
    "specific_style": "Specific genre, aesthetic, or platform style (e.e., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)",
    "production_quality": "Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.",
    "summary": "One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.",
    "logos": ["logo1 with visual description", "logo2 with visual description", ...]
}

Rules:
- Be specific and literal. Focus on what is explicitly visible.
- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.
- No artistic or cinematic analysis.
- Always include the language of any text in the image if present as an object, e.g. "English text", "Japanese text", "Russian text", etc.
- Maximum 10 objects and 5 actions.
- Return an empty array for 'logos' if none are present.
- Always output strictly valid JSON with proper escaping.
- Output **only the JSON**, no extra text or explanation.
"""

# Function to fetch image and encode into base64
def encode_image_url(image_url):
    response = requests.get(image_url)
    image_bytes = response.content
    base64_image = base64.b64encode(image_bytes).decode('utf-8')
    return base64_image

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
base64_image = encode_image_url(url)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": USER_PROMPT},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{base64_image}",
                    "detail": "high"
                },
            },
        ],
    },
]

response = client.chat.completions.create(
    model="wynd/wynd-vidcap-12b",
    messages=messages,
    temperature=0.1,
    max_tokens=2000,
    response_format={"type": "json_object"},
)

print(response.choices[0].message.content)

# Output:
{
  "actions": [],
  "content_type": "real-world footage",
  "description": "A wooden boardwalk path extends from the foreground into the distance, cutting through a field of tall, vibrant green grass. The path is flanked on both sides by the dense grass. In the background, a line of trees is visible on the horizon under a blue sky with scattered white clouds.",
  "environment": "An outdoor, natural landscape, likely a marsh or wetland, on a clear day. The scene is characterized by a wooden boardwalk, lush green vegetation, and a bright blue sky with wispy clouds.",
  "logos": [],
  "objects": [
    "Wooden boardwalk",
    "Tall green grass",
    "Blue sky",
    "White clouds",
    "Trees"
  ],
  "production_quality": "professional photography",
  "specific_style": "landscape photography",
  "summary": "A wooden boardwalk path winds through a lush green field under a bright blue sky with scattered clouds."
}

**Prompting Note:** Use the provided prompt as-is for best results. While the model may respond to other prompts, this version is extensively tested and optimized for the highest quality structured output.

---

## Availability & Integration

VidCap-12B is now available via the [**Inference.net](http://inference.net/) platform**. We offer flexible API integrations, asynchronous batch processing with webhook callbacks, and simple on-premise deployment options.

If you plan on using this model, contact us and we’ll make sure you have all the support you need to bring it to prod!

---

## Your Own Custom Model (Distillation)

The creation of VidCap-12B follows our proven **Custom LLM Distillation** pipeline. We select a powerful "teacher," generate high-quality datasets optimized to your KPIs, and distill a student model customized to your specific use case.

Customers consistently experience:

- Over **50% lower latency**
- Up to **95% lower inference costs**
- Optimized inference on dedicated instances
- Complete model ownership and transparent benchmarking

[**Schedule a 15-minute call**](https://inference.net/sales) to discuss your use case.

VidCap-12B makes structured video captioning a reality at internet scale. We can't wait to see what you build.

*We also trained Gemma 27B at 100k and 1M samples, but saw virtually no difference in quality between Gemma 27B at 1M samples and Gemma 12B at 1M samples.