← Item page
Link Triage explainer · video VLM / Hugging Face Space

Marlin-2B Video Understanding

A small open video-language model demo that turns short clips into dense scene descriptions, timestamped events, and natural-language moment search. Useful as a reference design for Dab pipelines that need to index meetings, screen recordings, CCTV-style clips, or social video snippets.

Source tweet: Victor M, 2026-05-26 Demo: Hugging Face Space Model: NemoStation/Marlin-2B License signal: Apache-2.0 model + space README

What this is

Marlin-2B is a 2B-parameter video VLM exposed through a Gradio Server demo. It has two practical workflows:

  • Caption: upload or select a video and receive a paragraph-level scene summary plus parsed timestamped event segments.
  • Find: ask for a natural-language event, then get a predicted start/end span in the clip.

The captured tweet points to the demo, while the parent tweet frames the release as “a tiny open video VLM” for understanding what happens in videos and when. Public traction at capture time was modest but meaningful for a fresh model/demo: the parent post showed 105 likes and 110 bookmarks via X API.

Workflow map

1. Video inputUpload a clip or choose one of the bundled examples.
2. Frame samplingThe app constrains video processing with env caps: FPS 2.0, max 240 frames, min 4 frames, max pixels 200704.
3. Model callNemoStation/Marlin-2B loads via Transformers with trust_remote_code, BF16, CUDA, SDPA attention.
4. Structured outputCaption returns scene text + event list; Find returns a span payload.
5. Timeline UIThe custom frontend renders a synced video timeline around the timestamped results.

Architecture / source map

README.md
Declares the Space as a ZeroGPU Gradio Server demo for Marlin-2B with two workflows: Caption and Find.
app.py
Single Python app: model loading, video duration helpers, event cleanup, GPU-decorated inference, API endpoints, and static example serving.
Server routes
/ serves custom index.html; /health reports model status; /examples.json exposes bundled examples.
Gradio APIs
analyze handles uploaded video; analyze_example runs cached/bundled examples. Both have concurrency limit 1 and 180s time limit.
Model call
MARLIN.caption(...) and MARLIN.find(...) are the key integration points. Output is JSON-serializable and frontend-friendly.

Operational profile

Model size~2.7B BF16 paramsHF API reports 2,721,801,024 BF16 parameters; model storage around 5.48 GB.
RuntimeZeroGPU A10GThe Space runtime was RUNNING on requested/current zero-a10g hardware.
Space SDKGradio 6.10Custom Gradio Server app rather than a plain default interface.
Access caveatModel gatedThe public model API showed gated access; the Space itself was public.

Example output quality

The Space includes cached example outputs. They show the intended shape: a detailed scene paragraph plus short event rows that can power a timeline or search index.

0.5–1.5s
“The man and woman walk toward each other.”
2.5–4.5s
“The man and woman begin dancing together.”
4.5–6.5s
“The man spins the woman in a circle.”

Observed timings from cached examples: ~6.9s–9.1s GPU time for short 6.5s–14s videos. That is good enough for offline triage, not necessarily live streaming.

Limitations / gotchas

  • Gated model card: the model README was inaccessible without HF model access, so this explainer relies on the public Space README, public API metadata, source files, and cached examples.
  • Hallucination risk: one cached animal example says “dogs, likely wolves,” a reminder that event text is signal, not ground truth.
  • Short-clip bias: app settings cap sampled frames; long videos need chunking, overlap, and deduplication.
  • Concurrency: the demo serializes analysis. A production pipeline needs a queue and cache.

Fit for Dab / Mission Control

Best near-term use

Use Marlin-style output as a template for video-to-index cards: source URL/file, scene summary, timestamped events, confidence/limitations, and action links. This would make videos explorable before Ananth commits to watching them.

Potential workflows

  • Link Triage expansion: for videos without transcripts, create a visual event summary and “worth watching?” recommendation.
  • Pi camera captures: summarize short clips into timestamped observations without storing sensitive raw detail on the public Notice Board.
  • Screen-recorded debugging: turn bug reproduction videos into steps, timestamps, and candidate UI states.
  • Daily podcast / briefing research: extract notable moments from product demos or conference talks before transcript work.

Implementation sketch if adopted

  1. Keep raw videos private; store only safe summaries on Notice Board.
  2. Chunk videos locally or in a queued GPU worker; cap frame budget per chunk.
  3. Persist event rows as structured JSON: {start,end,description,source_model,created_at}.
  4. Render a timeline artifact beside existing explainers, with “jump to moment” links when media is available.
  5. Run a second text model pass to normalize events into Ananth-readable takeaways and risks.

Sources inspected

Explainer written for Mission Control Link Triage on 2026-05-27T07:26:12Z.