← All research projectsTaRo
PURDUE UNIVERSITY

TaRo

Temporal candidate reuse for streaming video-language models

Streaming video models repeatedly search an expanding KV history to process each new frame. TaRo reuses promising candidate indices across frames and rescores them with the current query, reducing search without discarding the history.

ResearchPurdue UniversityCVPR 2027 · Under review
View original figure ↓
Original figure

Temporal candidate reuse in streaming prefill

TaRo Figure 3: video frames pass through a vision encoder into TaRo-equipped LLM layers. Stored indices 1, 5, 11 and current causal keys 14, 15 are rescored with the current query. Top-k feeds attention; a full-history search refreshes the bank when the query changes or the bank reaches its age limit.

The original paper figure shows where TaRo runs in video prefill and how candidate reuse works. Stored candidate indices and current-frame causal keys are rescored with the current query; the selected Top-k entries feed attention. Query change or bank age triggers a full-history search and bank refresh. The illustrated bank size, key IDs, and scores are schematic.

TaRo manuscript · Figure 3 · Under review

01 / THE PROBLEM

Can a video model search less history without throwing it away?

Selecting only a few KV entries does not automatically make streaming attention cheap. Exact Top-k still scores the complete causal history for every new visual query, so the search becomes more expensive as video accumulates. Discarding old entries reduces storage, but can remove evidence needed by a later question. TaRo targets the repeated search while retaining that history.

02 / THE APPROACH

Reuse candidates, recompute the decision

01

Keep a bank for each visual-token position

Corresponding positions in consecutive video frames often produce similar queries. TaRo uses that continuity to maintain a small bank of promising historical key indices for each layer, query head, and visual-token position. The bank stores indices; the complete keys and values remain in the KV cache.

02

Rescore with the current frame

For each new query, gather keys from the bank and the current frame’s causal KV entries. Recompute their scores, select the final Top-k entries, and perform attention. The selected set can change on every frame even when the candidate bank is reused.

03

Refresh when the query changes or the bank gets old

A drop in query similarity or a maximum bank age triggers a full-history search. That search both serves the current query and rebuilds the bank. Query heads sharing a KV head coordinate refresh decisions, so the policy respects grouped-query attention.

04

Make the reduced search efficient on the GPU

Fused Triton kernels combine candidate gathering, current-query scoring, selection, and attention. Evaluation includes bank management and refresh work, so the reported latency reflects the cost of maintaining reuse as well as the work it avoids.

03 / MY CONTRIBUTION

Method design and GPU implementation

As first author, I designed the candidate-reuse method and refresh policy, implemented fused Triton kernels for gathering, rescoring, selection, and attention, and evaluated two video-language models on an NVIDIA H100.

04 / EVALUATION

Setup and results

I evaluated LLaVA-OneVision-0.5B and Qwen2.5-VL-3B on an NVIDIA H100 80 GB in BF16, using matched video prefixes from OVO-Bench and StreamingBench subsets. The main configuration keeps 512 candidates and selects 128 entries for attention. Comparisons include ReKV, Exact Top-k, and Dense attention.

LLM-prefill speedup
1.37–3.58×
Versus ReKV across the evaluated model and context-length settings.
Lower Vision+LLM latency
30.6–31.0%
For LLaVA, including vision encoding and video prefill, versus ReKV.
Early-to-late latency growth
2.7–4.9%
On fixed evaluated video sets; Exact Top-k grew by 38.9–99.1%.
05 / OUTCOME & SCOPE

Less repeated search as history grows

Accuracy changes relative to ReKV range from −4.60 to +6.59 percentage points across model–benchmark settings. The main speedups measure video prefill; question processing and answer generation are outside that timing scope. TaRo retains the full KV cache and adds candidate-bank state, so its benefit is reduced search work rather than reduced KV-cache capacity. The evaluated histories reach approximately 32K entries for LLaVA and 16K for Qwen.