CodeCap-Bench: Can Code Serve as a Good Video Caption?

Jiafu Tang^1,2,∗,♠ Yifan Xu^1,2,∗ Eric Tang³ Junjie Zhao⁴ Shipeng Luo⁵ Te Sun⁴ Haoxuan Hu¹ Yiyan Ji¹ Xinlong Chen⁵ Zihao Wang² Yang Shi^6,2,† Yuanxing Zhang^2,♠ Pengfei Wan² Jiaheng Liu^1,†

¹ NJU² Kling AI³ TJU ⁴ SJTU⁵ CASIA⁶ PKU

∗ Equal contribution · † Corresponding author · ♠ Project lead

Links coming soon.

Flight seat-selection reference frames compared with natural-language and code captions. Code expresses the seat layout, scrolling motion, and hover and selection states through executable rules. — **Code as Video Caption.** Executable rules specify spatial structure, compute scrolling motion, and reproduce timed hover and selection states.

Abstract

Can executable code serve as a video caption? CodeCap-Bench evaluates this idea on 323 structured 2D videos across five categories. Code captions describe appearance, layout, and motion through graphical primitives and time-dependent rules; rendering makes their fidelity directly assessable. Our experiments reveal substantial differences across models, with dynamic information consistently less faithfully preserved than static content. Under the evaluated settings, code captions also retain more question-answering information and reconstruct videos more faithfully than the tested natural language pipelines.

Benchmark Construction

Five categories cover apps, 2D games, visual design, scientific visualization, and abstract motion graphics. Each video includes static and dynamic focal points and object spatio-temporal annotations, checked through human review.

CodeCap-Bench construction pipeline: collect videos from multiple sources, filter for structured 2D content, annotate static and dynamic focal points and object locations, and review annotations manually. — **Data construction pipeline.** Collect and filter structured 2D videos, annotate visual content and timing, then review spatial and temporal accuracy.

Dataset Statistics

The benchmark contains 323 videos, with a mean duration of 19.01 seconds, an average of 16.8 annotated reference objects, and 8.63 focal points per video.

Evaluation

Rendered code captions are compared with the reference video at three levels. Visual content is generated programmatically, without reference imagery or external visual assets.

Visual similarity: Video feature cosine similarity measures overall visual consistency.
Object-level spatio-temporal IoU: Checks whether key objects appear at the correct locations and times.
Fine-grained visual information retention: Scores static appearance and layout, and dynamic motion, state changes, and timing.

Main Results

Complete results at medium thinking effort. Agentic models use tools and rendering feedback; non-agentic models generate code in one pass.

CodeCap-Bench · Medium thinking effort · All metrics ↑ higher is better
Model	Harness	Video success (%) ↑	Visual similarity ↑	Spatio-temporal IoU ↑	Fine-grained visual retention
Model	Harness	Video success (%) ↑	Visual similarity ↑	Spatio-temporal IoU ↑	Dynamic ↑	Static ↑	Overall ↑
Agentic
GPT-6-astra	Codex	99.4	0.9735	75.37	68.61	80.72	73.24
GPT-6-astra	Pi Agent	99.7	0.9771	74.45	67.66	81.35	72.87
Opus-5	Pi Agent	97.8	0.9449	61.45	55.49	65.10	58.84
Opus-5	Claude Code	99.4	0.9555	58.55	53.23	63.40	56.88
GPT-6-sol	Codex	100.0	0.9563	55.63	43.92	64.04	51.34
Qwen3.8-max	Pi Agent	93.2	0.8869	48.90	44.81	55.88	48.83
Gemini-3.8-flash	Pi Agent	98.8	0.9257	42.75	43.66	55.18	47.73
GPT-6-sol	Pi Agent	99.1	0.9423	50.42	40.33	59.88	47.53
DeepSeek-v4.1-flash	Pi Agent	91.6	0.8692	45.14	40.45	51.27	44.29
GPT-5.6-sol	Pi Agent	99.1	0.9320	46.10	38.60	53.36	44.02
GPT-5.6-sol	Codex	99.7	0.9336	42.90	37.67	51.85	42.98
Non-agentic
GPT-6-astra	-	98.5	0.9622	69.96	69.00	76.85	72.00
GPT-6-sol	-	95.7	0.9168	58.01	48.54	61.07	53.08
GPT-5.6-sol	-	96.9	0.9223	49.88	46.50	56.03	50.05
Gemini-3.8-flash	-	97.2	0.9026	35.56	43.35	49.68	45.57
Qwen3.8-max	-	82.0	0.7754	42.41	43.39	47.15	44.67

IoU and retention use a 0–100 scale. Bold marks the highest value in each column within each group. Tasks that fail to produce a video receive 0 on the fidelity metrics. On small screens, scroll the table horizontally.

73.24 overall retentionGPT-6-astra + Codex leads the agentic group, with 75.37 spatio-temporal IoU.

Dynamics remain harderDynamic retention is below static retention in every configuration.

Success and fidelity differAgentic generation improves success rates for all paired models; retention gains vary by model.

Results by Video Category

Four-model comparison across apps, games, design, science, and motion categories, measuring visual similarity, spatio-temporal IoU, and visual retention. — **Category comparison.** Pi Agent · Medium effort. GPT-6-astra leads IoU and retention in all five categories; scientific visualization has the highest IoU for each model.

Further Analysis

Thinking Effort

Generation Cost and Efficiency

Pi Agent comparison of visual retention versus generation cost, alongside input and output tokens, model turns, and mean generation time for seven models. — **Cost does not determine fidelity.** GPT-6-astra scores above Opus-5 at lower cost. Lower monetary costs for Qwen3.8-max and DeepSeek-v4.1-flash accompany higher token use and longer generation times than the GPT series.

Tool Use

Stacked bars show the share of tool calls used to view reference frames, extract and analyze frames, render and compare results, edit and test code, and perform other tasks across seven Pi Agent models. — **Different observation and feedback strategies.** Seven models with Pi Agent distribute their tool calls differently across reference inspection, frame processing, and rendered-result checks.

Evaluation Validity

Same generated outputs, scored by two evaluation models
Generator	Metric	Evaluation model
Generator	Metric	GPT-6-sol	Gemini-3.1- pro-preview
GPT-6-sol	Dynamic	40.33	35.73
	Static	59.88	59.55
	Overall	47.53	44.39
DeepSeek- v4.1-flash	Dynamic	40.45	35.30
	Static	51.27	50.18
	Overall	44.29	40.81

Generators: Pi Agent · Medium effort. Both evaluation configurations preserve the overall ranking of these two models.

Scatter plot comparing human scores with GPT-6-sol visual-retention scores for outputs from four models, with Pearson correlation r equals 0.942. — **Human agreement: Pearson r = 0.942.** Human and VLM scores are paired over four models on 50 reference videos, sampled evenly across the five categories.

Code vs. Natural Language

Compare the information retained by code and natural language captions through question answering and video reconstruction.

Question-Answering Accuracy

100 videos from DVBench, VideoMathQA, and GUI-World · Answers evaluated against ground truth · Accuracy (%) ↑
Caption generator	Code-rendered video ↑	Natural language caption ↑
GPT-5.6-sol	76.42	73.58
Opus-5	83.02	72.64
Qwen3.8-max	81.13	78.30
Original video reference	83.96

Gemini-3.8-flash answers from either the rendered video or the caption text. Code-rendered videos score higher for all three generators; Opus-5 reaches 83.02%, compared with 72.64% for natural language.

Video Reconstruction Fidelity

CodeCap-Bench · Direct code rendering vs. natural language captions reconstructed with Seedance-2.5 · Higher is better
Caption generator	Reconstruction path	AS ↑	MPS ↑	KSS ↑
GPT-5.6-sol	Code rendering	0.8907	0.5412	0.8068
GPT-5.6-sol	Natural language	0.6628	0.5156	0.5925
Opus-5	Code rendering	0.9148	0.5572	0.8165
Opus-5	Natural language	0.6932	0.5150	0.5958
Qwen3.8-max	Code rendering	0.8773	0.5488	0.7817
Qwen3.8-max	Natural language	0.6754	0.5185	0.5984

AS: Appearance Similarity · MPS: Motion Pattern Similarity · KSS: Keyframe Structural Similarity. Bold marks the better path for each model and metric. Code rendering scores higher on all three metrics for each tested generator.

Case Study

GPT-6-astra + Codex reconstructs a bubble-chart video by measuring object trajectories and timing, then uses rendering feedback to correct a slider that is one year ahead.