Small Vision-Language Models are Smart

Compressors for Long Video Understanding

Junjie Fei1,2,∗,† , Jun Chen1 , Zechun Liu1 , Yunyang Xiong1 , Chong Zhou1 , Wei Wen1 , Junlin Han1 ,
Mingchen Zhuge1,2 , Saksham Suri1 , Qi Qian1 , Shuming Liu1,2 , Lemeng Wu1 , Raghuraman
Krishnamoorthi1 , Vikas Chandra1,† , Mohamed Elhoseiny2,† , Chenchen Zhu1,†
1
Meta AI, 2 King Abdullah University of Science and Technology (KAUST)
∗
Work done at Meta, † Project lead

Adapting Multimodal Large Language Models (MLLMs) for hour-long video understanding is severely
bottlenecked by context window limits. Dense visual streams quickly saturate input token budgets and
arXiv:2604.08120v1 [[Link]] 9 Apr 2026

exacerbate the lost-in-the-middle phenomenon. Existing efficiency heuristics, such as sparse sampling
or query-agnostic uniform pooling, blindly sacrifice fidelity. They frequently discard transient decisive
moments, blur fine-grained evidence, and waste representational bandwidth on irrelevant backgrounds.
In this paper, we propose Tempo, an efficient, query-aware framework that compresses long videos for
downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) to act as a
local temporal compressor. It casts visual token reduction as an early cross-modal distillation process,
generating compact, intent-aligned video representations in a single forward pass. To enforce strict
inference budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting
the SVLM’s inherent zero-shot relevance prior and empirical semantic front-loading, ATA acts as a
training-free, O(1) dynamic router. It allocates dense bandwidth to query-critical segments while
compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive
experiments demonstrate that our compact 6B architecture achieves state-of-the-art performance
with aggressive dynamic compression (0.5–16 tokens/frame). On the extreme-long LVBench (4101s),
Tempo scores 52.3 under a strict 8K visual token budget, outperforming proprietary baselines such as
GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames pushes performance to 53.7. Crucially, empirical
profiling reveals that Tempo frequently compresses hour-long videos to token counts substantially
below theoretical computational limits, proving that true long-form video understanding relies on
intent-driven efficiency rather than greedily padded context windows.

Date: April 10, 2026

Correspondence: junjiefei@[Link], chenchenz@[Link]
Code: [Link]
Project & Demo: [Link]

1 Introduction
The advancement of Multimodal Large Language Models (MLLMs) has significantly transformed visual
understanding, empowering systems to perform complex semantic analysis over images and short video
clips Liu et al. (2023, 2024a); Zhu et al. (2023); Li et al. (2024a); Bai et al. (2025); Li et al. (2025); Zhang
et al. (2024b, 2023). However, scaling these capabilities to hour-long videos remains challenging. The core
difficulty lies in the structural mismatch between the massive, continuous visual stream of long videos and the
rigidly bounded context windows of downstream LLMs. As temporal duration expands, raw visual tokens
quickly overwhelm the input capacity, severely diluting attention mechanisms and causing models to fail at
retrieving sparse evidence buried within extensive contexts Liu et al. (2024c).
To fit long video understanding into limited contexts, existing methods typically force one of two compromises.
A common approach is sparse frame sampling Xu et al. (2024a); Li et al. (2025); Lin et al. (2024a), which
reduces compute but inevitably risks skipping the transient yet decisive moments required to answer a specific
query. Alternatively, methods retain more frames but apply query-agnostic compression, such as uniform

1
 Ours: Tempo

Existing Methods (Query-Agnostic) 55 (6B, ~6 toks/f)

50 Ours: Tempo
VideoLLaMA3
Qwen2.5-VL

(6B, ~4 toks/f) (7B, ~91toks/f)

LVBench Accuracy (%)

(7B)
45 VideoChat-Flash
Kangaroo

(7B, 16toks/f) (8B)

Sparse
40
Sampling
35 Gemini 1.5 Pro Baseline (33.1%)
Missed Transient Moment 30 GPT-4o(2024-05-13) Baseline (30.8%)
Marker Size = Params
25 LLaMA-VID

(13B)
6B 7B 8B 13B
U niform
20
1 2 4 6 8 16 64 256 2K
Pooling Average Visual Tokens per Frame (Log Scale)
(c) Efficiency vs. Performance (LVBench)
Blurred Fine-grained Evidence
16 toks

Visual Tokens / Frame

Fine-grained Details
Ours: Tempo with ATA (Query-Aware)

...
Temporal Anchors (~0.5 tok/f)

Smart
~0.5 toks

Temporal Anchors
Compressor
H ow many green Video Temporal Sequence (Frames)
mugs were on ...
Fine-grained Details (16 tok/frame)
(a) Existing Query-Agnostic Methods vs. Ours (Query-Aware) (b) Adaptive Token Allocation Visualized over Time

Figure 1 Tempo achieves SOTA long video understanding via query-aware Adaptive Token Allocation (ATA). (a) Motivation:
Query-agnostic methods either miss transient moments (sparse sampling) or blur details (uniform pooling). Tempo
instead utilizes a small vision-language model as a smart compressor for query-aware cross-modal distillation. (b)
Mechanism: ATA dynamically allocates high bandwidth (16 tokens/frame) to relevant segment for fine-grained details,
while compressing redundant contexts into minimal temporal anchors (∼0.5 tokens/frame) to maintain causality. (c)
Result: Leading performance on LVBench. Tempo-6B achieves superior accuracy at extreme compression rates (i.e., 4
or 6 tokens/frame), outperforming open-source models and proprietary baselines with a fraction of the context budget.

spatiotemporal pooling Maaz et al. (2024); Jiang et al. (2025) or token merging Bolya et al. (2022); Li
et al. (2024b); Jin et al. (2024). By compressing without knowing what the user will ask, these heuristics
often blur fine-grained evidence in query-critical segments while wasting representational bandwidth on
irrelevant backgrounds. In essence, most existing pipelines reduce visual evidence before interacting with the
language model, preventing the dynamic allocation of bandwidth to query-critical segments. Even pioneering
query-aware approaches (e.g., LongVU Shen et al. (2024)) rely on disjoint auxiliary feature-matching modules,
thereby decoupling the routing mechanism from the end-to-end multimodal pipeline.
We introduce Tempo, an efficient query-aware framework for long video understanding that natively learns to
compress videos for downstream text generation tasks. As its name suggests, Tempo acts as an intelligent
temporal compressor that dynamically distributes the rhythm of the video: it allocates high token bandwidth
to semantic beats relevant to the query while swiftly fast-forwarding through redundant contexts. Rather
than treating visual compression as a purely visual, query-agnostic operation Jiang et al. (2025); Li et al.
(2024b), Tempo casts this reduction as an early cross-modal semantic distillation process. Concretely, Tempo
leverages a Small Vision-Language Model (SVLM) as a local compressor, seamlessly bridging it with an LLM
for global understanding and response generation. By prepending the user query to the SVLM input, Tempo
performs a preliminary cross-modal distillation pass that produces compact video memory tokens aligned
with the user’s intent and is trained end-to-end with standard auto-regressive objectives.
A practical challenge is enforcing a strict token budget at inference time (e.g., representing a 1024-frame
video under an 8K visual token budget) without sacrificing either fine-grained evidence or global causal
structure. To this end, we propose Adaptive Token Allocation (ATA), a training-free inference strategy guided by
two key empirical properties of the Tempo architecture. (i) Zero-shot relevance prior and temporal anchors.
Inheriting from the base model’s extensive multimodal pre-training, the local compressor exhibits a zero-shot
ability to estimate query-video relevance without auxiliary supervision. ATA exploits this prior to allocate
budgets segment-wise, enabling an aggressive dynamic compression range (0.5–16 tokens per frame). Crucially,
instead of hard pruning, which breaks causality, ATA preserves dense representational bandwidth for relevant
segments while compressing redundant contexts into minimal temporal anchors (i.e., 4 tokens) to maintain
the global storyline. (ii) Semantic front-loading driven by causal attention. Our ablations empirically reveal

2
that under the SVLM’s causal attention, salient visual semantics natively concentrate into the earliest video
memory tokens. Consequently, a simple head truncation effectively isolates high-value evidence, avoiding lossy
spatial blurring with zero overhead.
In summary, our contributions are:
• Tempo: an end-to-end, query-aware compression framework for long video understanding. It directly
addresses the context window bottleneck by unifying an SVLM-based local compressor and an LLM-based
global decoder, performing query-conditioned cross-modal distillation in a single forward pass.
• ATA: a training-free, budget-aware inference strategy leveraging the local compressor’s inherent zero-shot
relevance prior and semantic front-loading. ATA dynamically dictates the optimal token allocation,
preserving fine-grained details for query-critical moments while compressing redundancies into minimal
temporal anchors to maintain global causal structure.
• Scaling Behaviors: an empirical analysis revealing that optimal resource allocation varies with the task
and video duration. While a 4K visual token budget acts as a sweet spot for standard long video tasks
(e.g., Video-MME Long, 30–60 mins), restrictive budgets ultimately limit performance on extreme-long
videos (e.g., LVBench, >1 hour). Scaling to larger capacities unlocks new performance peaks. Notably, in
practice we observe that Tempo allocates tokens largely based on semantic necessity, often compressing
hour-long videos far below the available token budget.
• Leading Performances: despite being a compact 6B model, Tempo sets a new state-of-the-art across long
video benchmarks. On challenging LVBench, it scores 52.3 under a 8K budget, outperforming proprietary
baseline (e.g., GPT-4o, Gemini 1.5 Pro) and open-source counterparts (e.g., VideoChat-Flash). Scaling
to 2048 frames with a 12K budget further pushes performance to 53.7, demonstrating robust hour-long
video understanding of our proposed Tempo.

2 Related Work

2.1 Multimodal Large Language Models for Videos

The rapid evolution of MLLMs has established a dominant paradigm: aligning pre-trained visual encoders with
powerful LLMs. Recent state-of-the-art models, such as VideoChat2 Li et al. (2025), VILA Lin et al. (2024b),
LLaVA-OneVision Li et al. (2024a), VITA-1.5 Fu et al. (2025b), Kimi-VL Team et al. (2025), InternVL3.5 Wang
et al. (2025b), Molmo2 Clark et al. (2026), and the Qwen-VL series Bai et al. (2025), demonstrate exceptional
capability in short video understanding. They typically map sampled video frames directly into the LLM’s
context window. While effective for short-horizon tasks, extending this dense representation to hour-long
videos results in a linear explosion of visual tokens. This quickly overwhelms the maximum context length of
the LLM, leading to prohibitive computational costs and exacerbating the lost-in-the-middle phenomenon Liu
et al. (2024c), where models fail to retrieve pivotal evidence buried in extensive multimodal contexts.

2.2 Context Extension and Token Reduction

To comprehend extended temporal horizons, recent efforts generally bifurcate into two directions. The first
direction focuses on context extension via algorithmic extrapolation, architectural innovations, or system-level
parallelization to natively support massive token sequences. For instance, LongVA Zhang et al. (2024a)
extrapolates the context window to comprehend extensive visual tokens, LongVILA Chen et al. (2024b)
introduces sequence parallelism for long-context training, and LongLLaVA Wang et al. (2024) employs a
hybrid Mamba-Transformer architecture to mitigate memory constraints. While these approaches successfully
preserve visual fidelity and push the context boundaries, they strictly rely on processing dense visual streams.
Consequently, ingesting hundreds of thousands of visual tokens per video still incurs exorbitant memory
footprints and computational overhead, rendering them highly resource-intensive for routine inference. The
second, more prevalent direction relies on query-agnostic token reduction. Drawing inspiration from image-level
token pruning and merging techniques like FastV Chen et al. (2024a) and ToMe Bolya et al. (2022), video
MLLMs typically employ spatiotemporal pooling or fixed-rate sparse sampling Maaz et al. (2024); Li et al.
(2025); Jin et al. (2024); Li et al. (2024b); Jiang et al. (2025). For instance, VideoChat-Flash Li et al. (2024b)

3
 The Local Compressor
Inference-Only Bypass
The Global Decoder

Micro View - Causal Base Training-Free ATA Macro View - Time-Aware Sequence
n t i eo Segment Si
I pu V d
Zero-shot Relevance <t=0.0s> <t=2.0s> <t=4.0s>
hirel
... ...

ATA
User Query

Controller
Global Budget
Q
Bmax
Vision Encoder

Sm all Vision-Language Model

(SVLM)
(e.g., SigLIP)
Intercept hirel
Large Language Model (e.g., Qwen3-4B)
Segment 1: High Score si
... G lobal Self-Attention
Sy stem
Visual Tokens User Query
Memory Tokens
...
Prompt Xi Q M (kmax=128)
Keep all details (ki = 128) G lobal Self-Attention
Segment 2: Low Score si
Causal Attention: M

e s to all conditions
att nd ...
Final Answer
Full Memory

... Keep temporal anchors (ki = 4)

Tokens (kmax)

Figure 2 Overview of the Tempo framework. Our unified architecture casts long video understanding as an end-to-end,
query-aware compression process. The Local Compressor (Left). For each segment, a Small Vision-Language Model
(SVLM) acts as a semantic temporal compressor. Under causal attention, learnable memory tokens M inherently
distill the preceding visual tokens Xi and user query Q. Inference-Only Bypass (Middle). During a single forward pass,
an Adaptive Token Allocation (ATA) controller intercepts the hidden state hrel i to compute a zero-shot relevance
score si . This enables an O(1) dynamic head truncation, allocating dense bandwidth to query-critical segments while
compressing redundancies into minimal temporal anchors to strictly satisfy a global budget Bmax . The Global Decoder
(Right). The compressed memory tokens are assembled into a highly sparse, time-aware sequence using explicit temporal
tags (e.g., <t=2.0s>). A global LLM synthesizes this condensed multimodal context to generate the final response.

leverages visual redundancy to hierarchically compress tokens, while Storm Jiang et al. (2025) applies temporal
and spatial pooling to fit tight token budgets. However, because these heuristics are completely agnostic
to the user’s textual query, they risk blurring semantic boundaries and discarding transient, fine-grained
segments that may be critical to the downstream question.

2.3 Hierarchical and Query-Aware Video Architectures

To overcome uniform processing limits, models like SLOWFAST-LLAVA, LLaVA-Videoslowf ast , and Keye-
VL-1.5 deploy dual pathways to balance spatial and temporal resolutions Feichtenhofer et al. (2019); Xu et al.
(2024b); Zhang et al. (2024b); Yang et al. (2025). However, whether utilizing static sampling or dynamic
inter-frame similarity, their resource allocation remains purely vision-driven and fundamentally detached
from the user’s textual intent. Recent works have begun to explore query-aware processing Li et al. (2024c);
Islam et al. (2025); Shen et al. (2024). BIMBA Islam et al. (2025) introduces an optional query-conditioned
token selection mechanism. LongVU Shen et al. (2024) leverages cross-modal attention for selective spatial
compression, yet still depends on disjoint auxiliary modules that decouple the routing mechanism from the
end-to-end multimodal decoding process. Tempo fundamentally advances this trajectory by natively employing
an SVLM as an active, query-conditioned temporal compressor in a single forward pass. Furthermore, our ATA
mechanism exploits the SVLM’s inherent zero-shot relevance prior to dynamically dictate the video’s rhythm.
This preserves dense, high-fidelity tokens for critical segments while compressing irrelevant backgrounds into
minimal temporal anchors, achieving causal-preserving sequence assembly with zero routing overhead.

3 TEMPO

3.1 Overall
We target the fundamental bottleneck in long video MLLMs: the downstream LLM can only attend to a
limited number of visual tokens, while hour-long videos produce a massive, continuous stream. Tempo resolves
this mismatch by turning visual token reduction into an early cross-modal distillation problem.

4
Problem Setup. Given a long video V and a user query Q, we uniformly partition V into N temporal segments
S = {S1 , . . . , SN }. Our goal is to convert each Si into a compact set of query-conditioned video memory
tokens, with the total sequence bounded by a global inference budget Bmax , enabling the downstream LLM to
process the entire video and generate the final answer efficiently.

Architecture. Tempo constitutes a two-level generative hierarchy (Fig. 2): (1) an SVLM-based local compressor
Cϕ , and (2) an LLM-based global decoder Dθ . Concretely, the SVLM’s native vision encoder maps segment
Si to dense visual tokens Xi . Its causal attention then performs query-conditioned distillation, integrating
Xi and query Q into learnable memory tokens M. This yields a fixed-capacity representation Hi of exactly
kmax tokens. A linear projector maps Hi into the LLM’s embedding space as H̃i . Finally, the global LLM Dθ
consumes all memory tokens {H̃i }N i=1 alongside Q to auto-regressively decode the answer.

Training vs. Inference. Tempo is trained with a fixed per-segment capacity kmax to learn a strong query-aware
local compressor Cϕ . At inference, we additionally enforce a global budget Bmax . We therefore introduce ATA,
a training-free strategy that uses a zero-shot relevance prior extracted from the same SVLM forward pass to
allocate per-segment budgets ki ∈ [kmin , kmax ], followed by a constant-time head truncation.

3.2 Query-Aware Visual Compression

We cast segment compression as a query-driven sequence-to-sequence transformation. An explicit information
bottleneck forces the Cϕ to discard visual redundancies and distill semantic evidence relevant to user intent.

SVLM Input Construction. For each segment Si , the SVLM constructs a single causal sequence comprising:
(i) a system prompt, (ii) visual tokens Xi (extracted via its native vision encoder), (iii) user query Q, and (iv)
learnable memory tokens M. Placing M last is critical: under causal attention, each memory token inherently
attends to all preceding visual and textual contexts. This conditions the SVLM to distill query-aligned evidence
into M. Extracting their final-layer hidden states yields the compressed representation Hi ∈ Rkmax ×ds .

Sequence Assembly & Temporal Grounding. To preserve temporal identity and causal order across the entire
video, we prepend an explicit textual timestamp (e.g., <t=2.0s>) to each segment when assembling the global
context. In practice, these temporal tags significantly stabilize long-range attribution (i.e., which evidence
comes from where) within the downstream global LLM.

End-to-End Learning. Let the ground-truth answer be A = {at }Tt=1 . The global Dθ receives all projected
segment memories {H̃i }N
i=1 in temporal order, optimized via standard auto-regressive next-token prediction:

T
X
log pθ at | a<t , Q, {H̃i }N (1)


LAR (θ, ϕ) = − i=1
t=1

Crucially, we do not impose auxiliary compression losses, routing networks, or heuristic token-dropping
regularizations during training. The fixed capacity of kmax memory tokens acts as a hard structural bottleneck.
The gradients back-propagated from LAR naturally compel the compressor Cϕ to discard query-irrelevant
backgrounds and pack the most predictive visual evidence into this bounded space.

3.3 Zero-Shot Relevance Prior

A core insight driving Tempo is that modern multimodal foundation models (e.g., Qwen3-VL Bai et al. (2025))
inherently possess a robust zero-shot capability to evaluate semantic alignment between a visual sequence and
a text query (Refer to Sec. 4.3 – D). We harness this foundational prior to extract a highly accurate relevance
signal without introducing or training any auxiliary routing modules.

Logit-Based Relevance Score. To explicitly elicit this prior during inference, we slightly augment our training
system prompt. Following the standard compression instruction, we append a strict binary directive: “Now,
before compressing, answer exactly ‘Yes’ or ‘No’: is this segment relevant to the Query?” Let hrel i ∈R
ds
be

5
Algorithm 1: Adaptive Token Allocation (ATA) at inference
Input: Segment memories {Hi }N
i=1 , relevance scores {si }i=1 , budget Bmax , bounds kmin , kmax
N

Output: Budgeted memories {Hi ATA N

}i=1
Normalize {si } → {ŝi } by min-max scaling;
Compute kiideal via Eq. (3);
if ≤ Bmax then
P ideal
i ki
ki ← kiideal ;
else
ki ← Eq. (4) with Bbase = N kmin ;
Discretize {ki } to integers s.t. i ki ≤ Bmax ;
P

HATA
i ← Hi [1:ki ], ∀i ∈ {1, . . . , N };
return {HATA
i i=1 ;
}N

the final hidden state immediately preceding the model’s binary response. Using the SVLM’s frozen language
modeling head weights for the vocabulary tokens Yes (wyes ) and No (wno ), we compute a continuous relevance
probability si via logit difference Li et al. (2026):
 
si = σ (wyes − wno )⊤ hrel
i ∈ (0, 1), (2)

where σ(·) is the Sigmoid function. This O(1) projection avoids auto-regressive decoding overhead while
yielding a highly stable ranking signal.

Single-Pass Design. The score si and the compressed memory tokens Hi are extracted within a single forward
pass of Cϕ . As illustrated in the Inference-Only Bypass (Fig. 2), we simply intercept the hidden state hrel
i to
compute the zero-shot score, and then seamlessly continue the forward pass to extract Hi . This architectural
elegance guarantees that both the relevance routing signal and the compressed representations are rigorously
conditioned on the exact same multimodal context, achieving adaptive evaluation with effectively zero latency.

3.4 Adaptive Token Allocation (ATA)

At inference, the total visual context provided to the global LLM must strictly satisfy a bounded capacity Bmax .
As illustrated in Fig. 2, the ATA controller translates the zero-shot scores {si } into dynamic per-segment
token budgets ki , executing the physical compression via zero-overhead head truncation.

Stage 1: Contrastive Linear Allocation. To guarantee causal continuity across the entire video sequence, we
enforce a minimal temporal anchor for every segment, regardless of its relevance. We first normalize the raw
scores via Min-Max scaling: ŝi = (si − min(s))/(max(s) − min(s) + ϵ). To maximize the contrast between
query-critical events and irrelevant backgrounds, we linearly map these normalized scores to a target capacity:

kiideal = kmin + ⌊(kmax − kmin ) · ŝi ⌋. (3)

Stage 2: Capacity-Aware Protection. Let Bbase = N ·kmin represent the foundational cost Prequired to maintain
the global temporal anchors. If the sum of ideal allocations satisfies the global limit ( i kiideal ≤ Bmax ), we
directly adopt {kiideal } to maximize sparsity. Otherwise, we distribute the residual budget Bres = Bmax − Bbase
proportionally based on the normalized scores:
$ %
ŝi
ki = kmin + Bres · PN . (4)
j=1 ŝj + ϵ

We then discretize {ki } and distribute any fractional remainders to strictly ensure ki ≤ Bmax (Alg. 1).
P
i

6
Head Truncation: Zero-Overhead Token Selection. Once the dynamic budget ki is allocated, we compress
the segment by simply slicing the memory sequence, i.e., HATA i = Hi [1:ki ]. Driven by the auto-regressive
nature of the SVLM’s causal attention, we empirically observe a semantic front-loading phenomenon: the
local compressor packs the most salient global evidence into the earliest generated memory tokens (Refer to
Sec. 4.3 – C). Consequently, this O(1) tensor slice naturally isolates high-value semantics without introducing
lossy spatiotemporal pooling. The final global sequence {H̃ATA i i=1 strictly conforms to Bmax , rendering
}N
memory footprints entirely predictable even for hour-long reasoning.

4 Experiments

4.1 Experimental Setup

Architecture & Implementation. Tempo’s local SVLM is initialized from Qwen3-VL-2B-Instruct, while the
global LLM uses Qwen3-LM-4B. A linear projector bridges the SVLM’s memory space to the LLM, yielding a
compact 6B-parameter architecture. We extract frames at 2 FPS via Decord, applying uniform subsampling if
limits are exceeded. During training, continuous videos are partitioned into 4-frame segments, each compressed
by the SVLM into kmax = 128 memory tokens. During inference, we expand the segment window to 8 frames.
ATA (Sec. 3.4) strictly enforces a global visual budget Bmax (4K or 8K) via head truncation. Models are
trained on a 64-GPU cluster with FSDP. Additional hyper-parameters are provided in the Appendix C.

Progressive Training Curriculum. We adopt a rigorous four-stage progressive training curriculum to ensure
stable optimization and context extrapolation:
• Stage 0 (Modality Alignment): We freeze both the SVLM and the LLM, exclusively optimizing the
linear projector on the standard LCS-558K dataset Liu et al. (2023). This establishes the fundamental
vision-language alignment, bridging the SVLM’s visual representations with the LLM’s text embedding.
• Stage 1 (Pre-training): We unfreeze the entire architecture and optimize it on a large-scale, curated
multimodal corpus comprising ∼2M images, ∼1.38M videos, and ∼143K pure text samples. During this
phase, videos are sparsely sampled at 8 frames, endowing the model with initial temporal perception.
• Stage 2 (Broad Supervised Fine-Tuning): To develop robust instruction-following and semantic-aware
temporal reasoning capabilities, we perform comprehensive SFT using a highly diverse data mixture
(∼0.93M images, ∼2.25M videos, and ∼71K text samples). In this stage, the temporal context is
systematically expanded, with the maximum number of sampled frames per video strictly capped at 128.
• Stage 3 (Long-Context SFT): To effectively extrapolate the context window, we freeze the SVLM and
exclusively fine-tune the global LLM on a high-quality subset of ∼384K samples from Stage 2. Here,
the maximum frame limit is extended to 384, enabling the LLM to handle long temporal sequences.
To curate our training data, we primarily follow the data mixtures established by VideoChat-Flash Li et al.
(2024b) and LLaVA-OneVision-1.5 An et al. (2025). All training datasets utilized throughout our progressive
curriculum are publicly accessible, ensuring full reproducibility.

Evaluation Benchmarks & Baselines. To evaluate Tempo’s long video understanding, we conduct comprehen-
sive experiments across four prominent benchmarks, i.e., LongVideoBench Wu et al. (2024), MLVU Zhou et al.
(2025), Video-MME Fu et al. (2025a), LVBench (extreme-long video) Wang et al. (2025a), spanning standard
long-form tasks to hour-long stress tests. We benchmark Tempo against widely adopted proprietary baselines
(e.g., GPT-4o, Gemini Pro 1.5), general open-weight MLLMs (e.g., InternVL, Qwen-VL), and specialized
long-video MLLMs (e.g., VideoChat-Flash, LongVA). All evaluations are conducted using the lmms-eval.

4.2 Long Video Understanding

Tab. 1 summarizes the evaluation of Tempo (capped at 1024 frames) against state-of-the-art MLLMs across
four major benchmarks. Despite a compact 6B-parameter architecture and aggressive token compression
(0.5–16 tokens/frame), Tempo achieves state-of-the-art performance. While larger open-weight models (e.g.,
Qwen3-VL 8B, Molmo2) yield strong absolute scores via exorbitant visual token consumption, Tempo operates

7
Table 1 Comparison with state-of-the-art MLLMs on long video benchmarks, highlighting Tempo’s superior accuracy and extreme
token efficiency. Bold and underline denote the best and second-best results among specialized long video MLLMs. “-”
indicates unavailable results. * indicates the average tokens per frame are dynamically adjusted. For our model, we
report the theoretical dynamic range (0.5–16) alongside the actual empirical average tokens per frame ( gray rows ),
demonstrating that Tempo inherently operates substantially below the maximum budget limits in practice.

Tokens LongVideoBench MLVU Video-MME (w/o sub.) LVBench

Model Size
per frame (473s) (651s) Overall (1010s) Long (2386s) (4101s)

Proprietary Models
GPT-4o Hurst et al. (2024) - - 66.7 64.6 71.9 65.3 30.8
Gemini 1.5 Pro Team et al. (2024) - - 64.0 - 75.0 67.4 33.1

General Open-Source MLLMs

VideoChat2-HD Li et al. (2025) 7B 72 - 47.9 45.3 39.8 -
LLaVA-OneVision Li et al. (2024a) 7B 196 56.4 64.7 58.2 - -
LLaVA-Video Zhang et al. (2024b) 7B 676 58.2 70.8 63.3 - -
VideoLLaMA3* Zhang et al. (2025) 7B ≤ 91 59.8 73.0 66.2 54.9 45.3
InternVL3.5 Wang et al. (2025b) 8B 256 62.1 70.2 66.0 - -
Molmo2 Clark et al. (2026) 8B 83 67.5 - 69.9 - 52.8
Qwen2.5-VL Bai et al. (2025) 7B 1924 56.0 70.2 65.1 - 45.3
Qwen3-VL* Bai et al. (2025) 2B ≤ 640 - 68.3 61.9 - 47.4
Qwen3-VL* Bai et al. (2025) 8B ≤ 640 - 78.1 71.4 - 58.0

Specialized Long Video MLLMs

LLaMA-VID Li et al. (2024c) 7B 2 - 33.2 25.9 - 23.9 (13B)
LongVA Zhang et al. (2024a) 7B 144 - 56.3 52.6 46.2 -
Kangaroo Liu et al. (2024b) 8B 256 54.8 61.0 56.0 46.7 39.4
LongLLaVA Wang et al. (2024) A13B 144 53.5 - 53.8 46.4 -
LongVILA Chen et al. (2024b) 7B 196 57.1 - 60.1 47.0 -
LongVU Shen et al. (2024) 7B 64 - 65.4 60.6 59.5 -
Storm Jiang et al. (2025) 7B 64 60.5 72.9 63.4 53.4 -
BIMBA Islam et al. (2025) 7B 36 59.5 71.4 64.7 - -
VideoChat-Flash Li et al. (2024b) 7B 16 64.7 74.7 65.3 55.4 48.2
Tempo* (4K Budget) 6B 0.5–16 64.5 75.6 67.8 57.8 52.7
,→ actual avg. toks/frame 2.8 2.8 3.6 3.4 2.9
Tempo* (8K Budget) 6B 0.5–16 65.1 75.2 67.7 57.0 52.3
,→ actual avg. toks/frame 3.1 3.3 4.3 4.1 3.5

under extreme efficiency. By routing evidence through ATA, Tempo strictly bounds visual tokens to 4K or
8K budgets. In practice, ATA dynamically distributes bandwidth so efficiently that the actual consumption
falls well below these limits (e.g., 2.9 tokens/frame on LVBench under the 4K budget). Remarkably, its
comparative advantage over specialized long video MLLMs amplifies as the temporal span extends.

Dominance in Ultra-Long Video Understanding. The most notable results emerge on the extreme-long
benchmark LVBench, a rigorous stress test for long-term memory and evidence retrieval. Operating strictly
within a 4K visual budget, Tempo achieves 52.7, outperforming the strongest 7B specialized MLLM, VideoChat-
Flash (48.2), by 4.5 points. Impressively, despite its compact capacity, Tempo eclipses proprietary systems in
this ultra-long setting, surpassing GPT-4o (30.8) and Gemini 1.5 Pro (33.1) by massive margins. This proves
that explicit query-aware compression is vastly superior to blindly feeding raw frames into expansive LLM
context windows, which often suffer from attention dilution.

Robustness Across Varied Temporal Contexts. This dominance consistently extends across other benchmarks.
On Video-MME, Tempo secures 67.8 under the 4K budget, exceeding VideoChat-Flash (65.3) and showing
massive improvement over its base model Qwen3-VL-2B (61.9). On the challenging Video-MME Long subset
(2386s), Tempo achieves 57.8. Similarly, Tempo delivers SOTA-level performances on MLVU (75.6) and
LongVideoBench (65.1 under 8K), asserting its robust generalization across diverse temporal scales and tasks.

8
Table 2 Ablation studies on Tempo’s core components. We decompose our framework across five dimensions: (A)
progressive training curriculum, (B) segment-level budget allocation, (C) intra-segment token reduction scheme, (D)
relevance scoring source, and (E) temporal continuity. Unless otherwise specified, all variants process videos uniformly
sampled at 2 FPS up to a maximum of 1024 frames, strictly bounded by an 8K visual token budget for fair comparison.
The default Tempo configuration is highlighted in gray . LongVB denotes LongVideoBench.

LongVB MLVU Video-MME (w/o sub.) LVBench

Ablation Setting
Overall Long
(473s) (651s) (4101s)
(1010s) (2386s)
A. Progressive Training Curriculum
w/o Stage 3 - Long-Context SFT (16K Budget) 61.4 67.2 66.1 56.3 47.3
w/o Adaptive Token Allocation (16K Budget) 62.8 73.5 67.0 56.2 51.1
Tempo Default (8K Budget) 65.1 75.2 67.7 57.0 52.3
B. Segment-Level Budget Allocation
Uniform Subsampling (Equal tokens per segment) 61.9 74.0 66.3 55.2 49.9
Random Drop (Uniform random segment selection) 59.3 70.9 63.6 55.2 49.8
Adversarial Routing (Keep lowest-scoring segments) 50.7 59.3 52.4 47.8 36.9
Hard Top-K Routing (Keep highest-scoring segments) 63.5 73.9 66.7 56.2 52.7
ATA (Adaptive token allocation, Alg. 1) 65.1 75.2 67.7 57.0 52.3
C. Intra-Segment Token Reduction Scheme
Uniform Tail Truncation (Fixed 64 tokens) 59.5 71.6 64.1 54.8 41.8
Uniform Head Truncation (Fixed 64 tokens) 63.2 73.4 66.9 56.2 51.5
Token Merging (Merge visual features to ki tokens) 63.6 74.9 66.3 55.4 53.0
Dynamic Tail Truncation (Keep last ki tokens) 61.9 73.4 64.8 54.2 50.5
Dynamic Head Truncation (Keep first ki tokens) 65.1 75.2 67.7 57.0 52.3
D. Relevance Scoring Source & Zero-Shot Prior
Base Model Prior (Standard prompt) 64.6 75.1 67.2 56.1 52.6
Base Model Prior (Explicit routing prompt) 65.7 76.3 67.6 57.3 52.7
External Dense Retriever (Qwen3-VL Reranker) 64.3 75.4 67.2 57.0 51.8
Tempo SVLM Prior (Standard prompt) 64.1 75.4 67.2 57.0 53.4
Tempo SVLM Prior (Explicit routing prompt) 65.1 75.2 67.7 57.0 52.3
E. Temporal Continuity (Minimum Token Guarantee)
Hard Pruning (0 tokens for irrelevant segments) 63.9 74.8 67.4 56.3 52.3
Minimal Temporal Anchors (kmin = 4) 65.1 75.2 67.7 57.0 52.3

The “Less is More” Phenomenon. Crucially, Tempo’s performance under the 4K budget frequently matches
or exceeds the 8K budget (e.g., 52.7 vs. 52.3 on LVBench; 57.8 vs. 57.0 on Video-MME Long subset).
This counter-intuitive phenomenon powerfully validates our ATA strategy. Enforcing a stricter information
bottleneck filters out background distractors, forcing the LLM to focus purely on high-value semantic beats.
This actively mitigates the lost-in-the-middle phenomenon without requiring additional inference compute.

Qualitative Analysis. We provide comprehensive qualitative results in Appendix B to further analyze

Tempo’s adaptive behavior. We contrast localized queries (requiring pinpoint accuracy) with global queries
(requiring holistic understanding). These comparisons explicitly demonstrate how Tempo dynamically shifts
its compression rhythm—allocating high-fidelity bandwidth to query-critical moments while applying extreme
sparsity to irrelevant backgrounds. For overarching queries lacking singular salient events, ATA gracefully
defaults to a smooth, low-variance token allocation, ensuring global temporal comprehension remains intact.

4.3 Ablation Studies

We systematically decompose Tempo’s core components in Tab. 2. Unless specified, all variants process videos
uniformly sampled at 2 FPS for a maximum of 1024 frames, strictly bounded by an 8K visual token budget.

A. Progressive Training Curriculum. We first evaluate our training stages (Tab. 2A). Stopping after Stage
2 (w/o Long-Context SFT) yields sub-optimal performance on extreme-long benchmarks (e.g., 47.3 on

9
 6FDOLQJ%HKDYLRURI7HPSR0D[LPXP)UDPHVYV7RNHQ%XGJHWV
7RWDO,QIHUHQFH9LVXDO7RNHQ%XGJHW
.%XGJHW .%XGJHW .%XGJHW'HIDXOW .%XGJHW
9LGHR00(/RQJ /9%HQFK
 

 

 
6FRUH

6FRUH
 

 

 

           
0D[LPXP)UDPH&DSDFLW\fmax 0D[LPXP)UDPH&DSDFLW\fmax

Figure 3 Scaling behavior of Tempo. We investigate the interplay between maximum frame capacity (fmax ) and total
visual token budgets. (Left) On Video-MME (Long), a strict 4K budget acts as an optimal sweet spot by aggressively
filtering redundancy, whereas larger budgets (8K/12K) yield marginal noise. (Right) On the extreme-long LVBench,
restrictive budgets eventually limit the achievable performance, whereas expansive capacities (e.g., 12K) monotonically
unlock new peaks at higher frame densities, proving the necessity of scaled context for hour-long video understanding.

LVBench), as the LLM fails to extrapolate its temporal window. Introducing Stage 3 without adaptive
allocation improves this to 51.1, despite a 16K budget. Crucially, combining the full curriculum with ATA
achieves peak performance (52.3 on LVBench) while consuming only half the visual budget (8K). This confirms
that extending context is not merely about scaling length, but maximizing information density.

B. Segment-Level Budget Allocation. To strictly satisfy the 8K token budget, all baselines fix the segment
capacity at kmax (128) tokens. Given our sampling rate of 2 FPS (capped at a maximum of 1024 frames),
Uniform Subsampling halves the initial input frames (processing up to 512 frames). In contrast, Random
Drop and the routing variants process all sampled frames (up to 1024) but discard 50% of the segments to
meet the budget (Tab. 2B). Notably, when we adversarially route by retaining only the lowest-scoring 50%
(Adversarial Routing), performance catastrophically collapses (e.g., 65.1 → 50.7 on LongVideoBench). This
validates that our zero-shot relevance scores accurately isolate query-critical evidence. Furthermore, while
Hard Top-K Routing (dropping the lowest-scoring half entirely) performs competitively, ATA outperforms it
by dynamically scaling budgets and preserving overarching causality in most cases.

C. Intra-Segment Token Reduction. We next evaluate the mechanism for token reduction (Tab. 2C). Whether
applying a fixed 64-token limit (Uniform) or utilizing our ATA-assigned budgets (Dynamic), Head Truncation
consistently eclipses Tail Truncation (e.g., 63.2 vs. 59.5 for Uniform, and 65.1 vs. 61.9 for Dynamic on
LongVideoBench). This corroborates our semantic front-loading hypothesis (Sec. 3.4): the causal SVLM
natively packs the most critical semantics into its earliest memory tokens. While Token Merging (ToMe) yields
a marginal gain on LVBench (53.0 vs. 52.3), it introduces O(N 2 ) spatial clustering overhead and degrades
performance on shorter benchmarks (e.g., 66.3 vs 67.7 on Video-MME). The O(1) dynamic head truncation
strikes the optimal balance, offering superior generalizability with strictly zero computational overhead.

D. Relevance Scoring Source & Zero-Shot Prior. We systematically investigate the origin of the relevance
scores (Tab. 2D). To isolate the routing effect, all variants here utilize Tempo’s compressed memory tokens
for final generation, differing only in how the ATA score si is computed. Surprisingly, we observe that
the official Qwen3-VL-2B checkpoint (Base Model Prior) possesses a latent yet strong zero-shot capability
for relevance alignment, achieving 76.3 on MLVU. However, extracting this prior from the base model, or

10
utilizing an External Dense Retriever (which natively scores segments but yields sub-optimal performance),
necessitates a redundant, isolated forward pass per segment. In contrast, our Tempo SVLM Prior extracts these
accurate routing logits simultaneously during the visual compression pass. When guided by an explicit binary
instruction (full prompt can be found in the Appendix E), it provides superior comprehensive performance
(e.g., 67.7 on Video-MME). This single-pass design capitalizes on the foundational prior with zero latency.

E. Temporal Continuity. We ablate the kmin constraint (Tab. 2E). Compared to a Hard Pruning strategy that
aggressively drops irrelevant segments to 0 tokens, enforcing Minimal Temporal Anchors (e.g., 4 tokens/segment)
consistently improves performance (e.g., 63.9 → 65.1 on LongVideoBench). This demonstrates that maintaining
a continuous, highly compressed timeline is essential for long-form video understanding, preventing the LLM
from losing its temporal orientation and causal tracking in hour-long videos.

Scaling Behavior of Inference Context. Fig. 3 details Tempo’s scaling dynamics. At low-frame regimes,
all budget curves coincide. As the uncompressed token counts remain below the budget thresholds, ATA
allocations are identical. As fmax expands, restrictive budgets (2K, 4K) naturally degrade: enforcing the
kmin = 4 temporal anchor across too many segments prematurely exhausts the capacity, starving query-critical
segments of representational bandwidth. At high frame counts, behavior diverges by video length. Standard
long-form tasks (e.g., Video-MME Long) peak at fmax = 1024 under a 4K budget. Conversely, the extreme-
long LVBench necessitates scaled capacities, where expansive 8K and 12K budgets allow performance to scale
monotonically, reaching 53.7 at fmax = 2048 (Bmax = 12K). Notably, our empirical profiling (we provide
detailed token distributions in the Appendix A) reveals that even under generous 8K or 12K ceilings, ATA’s
dynamic sparsity often compresses LVBench videos (fmax = 1024) to token counts substantially below the
allocated budget in practice. This demonstrates a profound property: Tempo allocates bandwidth driven by
semantic necessity, rather than greedily padding to fill the available context window.

5 Conclusion
We present Tempo, an efficient 6B-parameter framework that resolves the structural mismatch between
massive video streams and bounded LLM context windows. Departing from query-agnostic heuristics like
sparse sampling or spatiotemporal pooling, Tempo natively unifies a local SVLM and a global LLM. It
casts visual token reduction as an early cross-modal distillation process, generating highly compressed,
intent-aligned video representations in a single forward pass. To enforce strict visual budgets at inference
without sacrificing fine-grained evidence or overarching causality, we propose Adaptive Token Allocation
(ATA). Driven by the SVLM’s zero-shot relevance prior and empirical semantic front-loading, ATA executes
O(1) dynamic head truncation. It aggressively routes dense bandwidth to query-critical semantic beats while
compressing redundancies into minimal temporal anchors to maintain the global storyline. Tempo establishes
new state-of-the-art performance across diverse benchmarks, notably outperforming specialized long video
MLLMs and proprietary baselines on the extreme-long LVBench. Crucially, our scaling analysis reveals that
optimal resource allocation depends on video duration. While a compact 4K budget acts as a highly efficient
denoiser for standard long video tasks, mastering hour-long narratives necessitates scaled contextual capacities.
By compressing videos to token counts substantially below theoretical limits in practice, Tempo demonstrates
a profound property: true long-form multimodal understanding is best achieved not by greedily padding
expansive context windows, but through intent-driven, dynamic allocation based on semantic necessity.

6 Discussion and Future Works

While Tempo successfully demonstrates that high-efficiency, end-to-end multimodal compression can resolve
the context bottlenecks of hour-long videos, this query-aware paradigm opens several critical frontiers for the
community. We discuss the current limitations of our framework and possible future research directions.

Eliciting Inherent Relevance Priors via Post-Training. Currently, the Adaptive Token Allocation (ATA)
mechanism leverages the SVLM’s zero-shot capability to accurately assess whether a video segment is useful
for answering a given query. While our ablations demonstrate that this latent prior is already highly effective,

11
it operates entirely in a zero-shot manner. A promising future direction is to explicitly elicit and amplify this
inherent capability through post-training (standard supervised fine-tuning risks introducing inductive biases
or overfitting to heuristic labels, while reinforcement learning can directly optimize the SVLM’s routing policy
against the final downstream generation accuracy). By formally optimizing the local compressor to sharpen
its internal relevance judgments, we can further elevate this routing precision, potentially driving substantial
performance gains across the entire framework.

Autoregressive, Reasoning-Driven Compression. To ensure a highly efficient, single forward pass, Tempo
currently compresses video segments using a fixed number of learnable memory tokens. Inspired by recent
advancements in reasoning models that dynamically allocate test-time compute (e.g., generating internal
thought tokens before halting), a more intelligent paradigm would allow the SVLM to autoregressively generate
compressed tokens for a video segment, autonomously deciding when sufficient visual evidence has been
gathered to stop. However, adapting this autoregressive extraction without severely bottlenecking inference
latency remains a profound optimization challenge for future long video compressors.

Hierarchical On-Demand Distillation for Multi-Turn Dialogue. While Tempo efficiently compresses videos
via query-aware distillation, adapting to shifting user intents in multi-turn dialogues currently requires
re-extracting visual features from the entire video. A promising frontier is transitioning towards a hierarchical,
on-demand routing paradigm. By decoupling a persistent, query-agnostic global context from the intensive
query-aware extraction, the global LLM could be empowered to act as an active routing agent. Rather than
passively receiving features, the LLM could dynamically identify which specific temporal segments require
deeper inspection, invoking the SVLM to distill high-fidelity anchors exclusively for those targeted moments.

Acknowledgements
This work was conducted during a research internship at Meta. Junjie Fei, Mingchen Zhuge, Shuming Liu, and
Mohamed Elhoseiny were supported by funding from the King Abdullah University of Science and Technology
(KAUST) Center of Excellence for Generative AI.

12
References
Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui
Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv
preprint arXiv:2509.23661, 2025.
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang
Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025.
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token
merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is
worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European
Conference on Computer Vision, pages 19–35. Springer, 2024a.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang
Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint
arXiv:2408.10188, 2024b.
Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee,
Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language
models with video understanding and grounding. arXiv preprint arXiv:2601.10611, 2026.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In
Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang
Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms
in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
24108–24118, 2025a.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao,
Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957,
2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila
Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. Bimba: Selective-scan
compression for long-range video question answering. In Proceedings of the Computer Vision and Pattern Recognition
Conference, pages 29096–29107, 2025.
Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu,
Kurt Keutzer, et al. Storm: Token-efficient long video understanding for multimodal llms. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, pages 5830–5841, 2025.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation
empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 13700–13710, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li,
Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.
Videochat: Chat-centric video understanding. Science China Information Sciences, 68(10):200102, 2025.
Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang,
Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal
retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He,
Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint
arXiv:2501.00574, 2024b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In
European Conference on Computer Vision, pages 323–340. Springer, 2024c.

13
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual
representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural
language processing, pages 5971–5984, 2024a.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual
language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
26689–26699, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information
processing systems, 36:34892–34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024a.
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu.
Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542,
2024b.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost
in the middle: How language models use long contexts. Transactions of the association for computational linguistics,
12:157–173, 2024c.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video
understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 12585–12602, 2024.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi
Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long
video-language understanding. arXiv preprint arXiv:2410.17434, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent,
Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of
context. arXiv preprint arXiv:2403.05530, 2024.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang
Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang,
Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 22958–22967, 2025a.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong
Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.
arXiv preprint arXiv:2508.18265, 2025b.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to
1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889, 2(5):6, 2024.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved
video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857, 2024.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension
from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024a.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan.
Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15841,
2024b.
Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li,
Dunju Zang, et al. Kwai keye-vl 1.5 technical report. arXiv preprint arXiv:2509.01563, 2025.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang,
Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding.
arXiv preprint arXiv:2501.13106, 2025.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video
understanding. In Proceedings of the 2023 conference on empirical methods in natural language processing: system
demonstrations, pages 543–553, 2023.

14
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan,
Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction
tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong,
Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 13691–13701, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language
understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.

15
Appendix
A Statistical Analysis of Adaptive Token Allocation
In this section, we present a statistical analysis of the Adaptive Token Allocation (ATA) mechanism to further
assess its efficiency and adaptability under diverse video lengths and token budget constraints. The analysis is
based on the results reported in Tab. 1, with all evaluations conducted using lmms-eval. For all benchmarks,
the temporal context is partitioned into segments with a fixed window size of 8 frames during inference.

Unit Conversion. Unless otherwise specified, the statistics reported below are measured in tokens per segment.
The corresponding average tokens per frame can be obtained by dividing these values by the window size (8).
For example, an allocation of 64 tokens per segment corresponds to 8 tokens per frame.

A.1 Distribution of Token Allocation

As shown in Fig. A, we analyze the behavior of ATA by plotting the percentage of segments against their
allocated tokens. The results reveal two key insights:
• Heavy-Tailed Sparsity: Across both 4K (Top) and 8K (Bottom) budgets, the allocation exhibits a strongly
right-skewed, long-tailed distribution. The most frequent allocations consistently fall into the lowest
token bin (corresponding to highly compressed segments), accounting for a large fraction of all segments.
This indicates that Tempo compresses redundant content into minimal representations. Meanwhile, the
distribution gradually decays toward larger allocations, with a small but consistent fraction of segments
approaching the maximum capacity boundary. This suggests that high-fidelity bandwidth is selectively
reserved for rare yet query-related segments.
• Budget Robustness: The stability of this distribution across different global budgets demonstrates that
our SVLM-based compressor produces a consistent, query-driven ranking of visual importance. Rather
than uniformly scaling allocations when the budget increases (e.g., from 4K to 8K), Tempo preserves
extreme sparsity for background contexts while selectively allocating additional tokens to segments that
align with the user’s intent.

A.2 Dynamic Budget Utilization and Compression Efficiency

Fig. B visualizes the macro-level efficiency by comparing the actual average token consumption per segment
with the dataset-level average theoretical capacity. The green dashedPline denotes the average expected capacity
per segment across the entire dataset, computed as (Nsamples × B)/ S. Because shorter videos contain fewer
segments, their individual theoretical limits are naturally higher than this dataset-wide average. As a result,
their actual consumption may exceed the green line without violating the global video budget B.
• Query-Driven Adaptability: On datasets with diverse video lengths (e.g., LongVideoBench, MLVU,
Video-MME), although some shorter videos may peak above the average line, the statistical quartiles
(grey spines) and the dense clusters of actual consumption consistently lie well below the dataset-level
capacity. This indicates that ATA does not exhaust the available context budget merely because it is
available; instead, it preserves bandwidth when the video content is irrelevant to the user query.
• Hard-Boundary Reliability: Under extreme long context pressure (e.g., LVBench), where most videos are
extremely long and strongly constrained by the maximum segment count, the per-video limits closely
align with the dataset-level average. In this regime, the actual consumption forms a clear ceiling at the
theoretical limit, demonstrating that Tempo reliably respects the global capacity constraint.

B Qualitative Analysis of Query-Aware Allocation

To further illustrate the interpretability and adaptability of the ATA mechanism, we present qualitative
visualizations in Fig. C. These examples show how Tempo dynamically adjusts its temporal context budget

16
 (a) LongVideoBench (b) MLVU (c) Video-MME (d) LVBench

Percentage of Segments (%) 40

30

20

10

0
4 32 64 96 128 4 32 64 96 128 4 32 64 96 128 4 32 64 96 128
Allocated Tokens per Segment Allocated Tokens per Segment Allocated Tokens per Segment Allocated Tokens per Segment

(a) LongVideoBench (b) MLVU (c) Video-MME (d) LVBench

Percentage of Segments (%)

40

30

20

10

0
4 32 64 96 128 4 32 64 96 128 4 32 64 96 128 4 32 64 96 128
Allocated Tokens per Segment Allocated Tokens per Segment Allocated Tokens per Segment Allocated Tokens per Segment

Figure A Distribution of allocated tokens per segment. (Top) 4K budget (B = 4096); (Bottom) 8K budget (B = 8192).
Across four benchmarks, ATA consistently exhibits a strongly right-skewed, long-tailed allocation pattern. The majority
of segments are compressed into very low-token representations, while a small fraction receives substantially higher
allocations for query-aligned segments. Notably, this distribution pattern remains stable under different global budgets.

according to the semantic requirements of the input query. All examples are sampled from LVBench and
illustrate the outputs of Tempo 8K, as reported in Tab. 1.
• Precise Action Retrieval (Fig. C, Top): For localized tasks requiring specific visual evidence (e.g., identifying
what people with yellow ropes are doing), ATA exhibits an extremely sparse allocation pattern. Most
irrelevant background segments (e.g., unrelated food preparation) are aggressively compressed into
minimal temporal anchors. Conversely, a sharp allocation peak (approaching the maximum budget) is
concentrated exactly on the brief segment where the target action (lassoing the yak) occurs.
• Targeted Object Grounding (Fig. C, Middle): When queried about a specific object (e.g., the capacity of a
cooking machine), Tempo actively searches for semantically aligned visual cues. High token capacities
are dynamically assigned to segments featuring mechanical apparatuses and cooking molds, while manual
food preparation scenes are strongly suppressed. Interestingly, even if the localized object visually
deviates slightly from a standard Taiyaki maker, this behavior confirms that the allocation is guided by
the semantic intersection of the query (“machine”, “cook”) rather than simple low-level visual saliency.
• Global Video Summarization (Fig. C, Bottom): In contrast, queries requiring holistic understanding (e.g.,
identifying the overall category of a vlog) demand broader contextual coverage. In this case, ATA avoids
extreme sparsification and instead maintains a relatively dense, fluctuating token allocation across the
entire temporal sequence, ensuring that distributed thematic cues remain available for reasoning.
Overall, these qualitative examples suggest that the local SVLM functions as an effective smart compressor.
Rather than discarding segments uniformly to satisfy a budget constraint, it performs an interpretable,
query-aware cross-modal distillation that prioritizes semantically relevant temporal segments.

C Detailed Training Configurations

In the main paper, we introduced a progressive four-stage training curriculum to equip Tempo with long video
understanding capabilities. To ensure reproducibility, we provide the detailed hyper-parameter configurations
and optimization strategies for all stages extracted directly from our training framework.

17
 Dense Sampling Bound (ki = 128) Theoretical Upper Bound (B ≤ 4K)

(a) LongVideoBench (b) MLVU (c) Video-MME (d) LVBench

140

120
Avg. Allocated Tokens per Segment

100

80

60

40

20

Dense Sampling Bound (ki = 128) Theoretical Upper Bound (B ≤ 8K)

(a) LongVideoBench (b) MLVU (c) Video-MME (d) LVBench

140

120
Avg. Allocated Tokens per Segment

100

80

60

40

20

Figure B Macro-level budget utilization and adaptation efficiency. (Top) 4K budget; (Bottom) 8K budget. Each red
dot denotes the average token consumption per segment for a video sample. The dashed green line indicates the
dataset-level average theoretical capacity. Points above the line correspond to shorter videos whose per-segment capacity
is higher than the dataset-wide average. Adaptability: On datasets with diverse video lengths (e.g., LongVideoBench,
Video-MME), the consumption distribution remains well below the theoretical capacity, indicating query-driven
compression. Reliability: On extremely long videos (LVBench), the token consumption forms a clear ceiling at the
theoretical limit, demonstrating strict adherence to the global budget constraint.

C.1 Optimization and Hardware Setup

All models are trained on a high-performance compute cluster. Stage 0 (modality alignment) uses 32 NVIDIA
H100 (80GB) GPUs, while Stages 1–3 scale to 64 GPUs to accommodate the increased spatiotemporal
processing requirements. To support the long sequence lengths of videos and our 6B-parameter architecture,
we employ Fully Sharded Data Parallel (FSDP) together with gradient checkpointing. All training is performed
in mixed precision (bfloat16) to reduce memory consumption while maintaining numerical stability. Across
all stages, we use the AdamW optimizer with a cosine learning rate scheduler and a 3% warmup ratio. Weight
decay is set to 0.0.

C.2 Stage-wise Hyper-parameters

As training progresses from spatial modality alignment (Stage 0) to long-context temporal reasoning (Stage
3), the temporal context length and maximum number of frames are gradually increased. To stabilize this
scaling process, different subsets of model parameters are selectively frozen or unfrozen during training. The
hyper-parameter configurations for each stage are summarized in Tab. A.

18
 Question: What are the people holding the yellow ropes trying to do at the place that has trees with blooms?

(A) They are trying to lasso the dog (B) They are trying to lasso the cat

(C) They are trying to lasso the Panda (D) They are trying to lasso the yak

Prediction: (D) They are trying to lasso the yak

Question: What is the maximum number of Taiyaki can the machine cook at the same time?

(A) 8 (B) 3 (C) 4 (D) 6

Prediction: C) 4

Question: What is the category of this video?

(A) A vlog of all kinds of food around Korea (B) A vlog of travel around Korea

(C) A vlog of all kinds of food around Seoul (D) A vlog of travel around Seoul

Prediction: A

Figure C Qualitative examples of query-aware Adaptive Token Allocation. We visualize the token allocation distribution
across three distinct types of video QA. The red-bordered key frames correspond to segments with high token allocation,
while grey-bordered frames represent redundant contexts compressed to minimal tokens. (Top) Precise Action Retrieval:
ATA isolates the exact moment the target action (lassoing the yak) occurs. (Middle) Targeted Object Grounding: ATA
highlights specific scenes containing cooking machinery relevant to the query concepts (“machine”, “cook”), while
effectively suppressing general manual food preparation. (Bottom) Global Summarization: ATA maintains a dense
allocation distribution to capture the overarching theme of the diverse food vlog.

19
Table A Hyper-parameter configurations across the four-stage training curriculum of Tempo. ✓ indicates that the module is
trainable, while ✗ indicates that it is frozen during training.

Configuration Stage 0: Alignment Stage 1: Pre-training Stage 2: Broad SFT Stage 3: Long-Context SFT
Trainable Modules
Local Compressor (SVLM) ✗ Frozen ✓ Unfrozen ✓ Unfrozen ✗ Frozen
Memory Tokens ✓ Unfrozen ✓ Unfrozen ✓ Unfrozen ✗ Frozen
Linear Projector ✓ Unfrozen ✓ Unfrozen ✓ Unfrozen ✗ Frozen
Global Decoder (LLM) ✗ Frozen ✓ Unfrozen ✓ Unfrozen ✓ Unfrozen
Optimization Hyper-parameters
LR (LLM & Projector & Memory Token) 1 × 10−3 1 × 10−5 1 × 10−5 1 × 10−5
LR (SVLM) - 2 × 10−6 2 × 10−6 -
Learning Rate Schedule Cosine Cosine Cosine Cosine
Warmup Ratio 3% 3% 3% 3%
Global Batch Size 128 256 128 64
Epochs 1 1 1 1
Data & Context Scaling
Spatial Resolution (Image) Native Native Native Native
Spatial Resolution (Video) - Max 512 (Long edge) Max 512 (Long edge) Max 512 (Long edge)
Sampling Rate (FPS) - 2 2 2
Temporal Window Size - 4 frames 4 frames 4 frames
Max Sampled Frames (fmax ) 1 (Image only) 8 128 384
Segment Capacity (kmax ) 128 128 128 128
LLM Max Context Length 8192 8192 8192 16384

Dynamic Spatial Resolution. To balance spatial fidelity with the substantial memory requirements of long
temporal contexts, we adopt a dynamic resolution strategy. Static images retain their native resolution to
preserve fine-grained visual details. For video samples, frames are resized such that the longest edge does not
exceed 512 pixels. The original aspect ratio is preserved without square padding, preventing the SVLM from
allocating tokens to redundant regions.

Strategic Freezing. During Stage 3 (long-context SFT), we freeze the local SVLM compressor, the memory
tokens, and the linear projector. Since this stage focuses on temporal extrapolation, freezing these components
isolates the gradient updates to the global LLM. This allows the model to focus on learning long-range temporal
dependencies up to 12K visual tokens while avoiding degradation of the learned cross-modal alignment.

D Dataset Statistics
Tab. B summarizes the approximate number of samples per modality (Image, Video, Text) for each training
stage. This staged data mixture allows Tempo to gradually transition from spatial alignment on static images
to complex spatiotemporal understanding on videos.

Table B Statistics of training data mixtures. Approximate number of samples per modality across the four progressive
training stages.

Modality Stage 0: Alignment Stage 1: Pre-training Stage 2: Broad SFT Stage 3: Long-Context SFT
Image Samples ∼558K ∼2.00M ∼0.93M
Video Samples - ∼1.38M ∼2.25M ∼384K (Mixed)
Text Samples - ∼143K ∼71K
Total Size ∼558K ∼3.52M ∼3.25M ∼384K

E Prompt Design and Relevance Scoring Ablation

In this section, we describe the system prompts used during training and inference, and analyze their impact
on the ATA routing mechanism.

20
E.1 System Prompt Definition
To guide the local SVLM to act as a query-conditioned visual compressor, we design two system prompts:

Standard Prompt (Pstd ):

“You are a query-conditioned visual compressor. Store in the provided memory tokens the minimal
visual information needed to answer the Query. Ignore irrelevant details.”
Explicit Routing Prompt (Proute ):

“You are a query-conditioned visual compressor. Store in the provided memory tokens the minimal
visual information needed to answer the Query. Ignore irrelevant details. Now, before compressing,
answer exactly ‘Yes’ or ‘No’: is this segment relevant to the Query?”

E.2 Zero-Shot Relevance Prior

An interesting property of the Tempo SVLM is its zero-shot relevance routing capability. During training, the
SVLM is optimized exclusively with the Standard Prompt (Pstd ) to focus on cross-modal compression, and it
is never explicitly trained to produce binary relevance labels. At inference time, we instead apply the Explicit
Routing Prompt (Proute ). The SVLM is able to follow this instruction in a zero-shot manner, accurately
producing a relevance decision before generating the compressed memory tokens (Sec. 4.3B). This behavior
suggests that the SVLM possesses the semantic alignment between the query and the visual segment, which
can be readily elicited as a routing prior.

E.3 Ablation Analysis on Relevance Source (Ablation-D)

To further validate the design, we conduct an ablation study comparing different relevance scoring sources
and prompt formulations.
• Impact of Explicit Routing: Across both the Base Model Prior and the Tempo SVLM Prior, replacing the
standard prompt with the explicit routing prompt (Proute ) generally improves performance. Encouraging
the model to make an explicit relevance decision before context aggregation acts as a useful constraint
that helps filter irrelevant segments.
• Observation on Base Model Prior: Interestingly, the Base Model Prior (Standard Prompt) uses the default
system prompt of the Qwen-VL series (i.e. “You are a helpful assistant.”). Despite the absence of an
explicit routing instruction, the model still achieves strong performance. This observation suggests that
Qwen-VL models may implicitly assess the relevance between visual inputs and user queries before
generating answers.

F Calculation of Average Tokens per Frame

For a 30-minute video sampled at 2 FPS, the total number of frames is 3,600. Since the shortest videos in
LVBench exceed 30 minutes, the raw frame count of each sample is substantially larger than our maximum
sampling limit fmax . Therefore, we estimate the theoretical upper bound of the average number of tokens per
frame as:
Bmax
Tokens per Framemax = , (5)
fmax
where fmax denotes the maximum number of sampled frames provided to the model, and Bmax denotes the
global visual token budget (context length).
In Fig. 1(c), we report Tempo under two configurations: a 4K token budget with fmax = 1024, and a 12K
token budget with fmax = 2048. Although ATA dynamically compresses the visual sequence—often resulting
in substantially lower token usage in practice—the theoretical upper bound corresponds to the scenario where
the full global budget is consumed. Because the sampled frame count for LVBench is fixed at fmax , the
resulting upper bounds are 4096/1024 = 4 and 12288/2048 = 6 tokens per frame, respectively. For the baseline
VideoLLaMA3 Zhang et al. (2025), the official configuration uses fmax = 180 and Bmax = 16K, yielding a

21
theoretical upper bound of approximately 91 tokens per frame. For LLaMA-ViD Li et al. (2024c), the reported
results are taken directly from the official LVBench leaderboard. For all other baselines shown in Fig. 1(c), we
adopt the metrics reported in VideoChat-Flash Li et al. (2024b).

22