← All research projectsEdgeVQA
PURDUE · POSTECH COLLABORATION

EdgeVQA

Sustaining video and answer rates on bandwidth-limited devices

Video ingestion and answer generation compete for the same model-weight bandwidth. EdgeVQA brings frame processing and speculative answer verification into one weight sweep, then adjusts frame chunks to keep answer delivery on pace.

ResearchPurdue · POSTECH collaborationDAC 2027 · Under review
View project flow ↓
DAC 2027 · Under review

One weight sweep, two ongoing tasks

Scroll to follow the full flow ↔

An adaptive scheduler combines incoming-video processing and draft-answer verification in a shared weight sweep. Answer progress guides the next video chunk size.

An adaptive scheduler combines incoming-video processing and draft-answer verification in a shared weight sweep. Answer progress guides the next video chunk size.

Scope and reading notes

The two tasks share target-model linear-layer weight reads while preserving separate attention contexts, positions, and KV state. Frames are committed even if draft tokens are rejected; only accepted answer tokens are committed. The diagram summarizes the execution policy; modeled system rates and FPGA timings are distinct evaluations. Solid arrows show data; dashed arrows show control or scheduling.

01 / THE PROBLEM

How can a streaming model keep up with video and deliver answers on time?

A streaming video-language model must ingest frames continuously while producing an answer at a usable rate. Both activities repeatedly read the same large model weights. Grouping more frames into one sweep amortizes those reads but leaves fewer opportunities to emit answer tokens; smaller chunks improve responsiveness but consume more bandwidth. On an edge device, a schedule that meets the frame rate can still stall the answer.

02 / THE APPROACH

Share a weight sweep and pace the next chunk

01

Verify draft tokens while processing frames

A smaller model proposes answer tokens. EdgeVQA places their verification rows beside the incoming frame rows in the target model’s execution, allowing one weight sweep to ingest video and verify several answer tokens. Frame and answer rows retain separate attention contexts and rotary positions.

02

Size chunks from delivered-output lead

Draft acceptance changes over time. The scheduler tracks accepted-token yield, how far delivered output is ahead of its target rate, and the estimated duration of the next sweep. It chooses the largest frame chunk that preserves an output cushion, subject to buffer, compute, and frame-freshness constraints. Verification-only execution can recover when answer delivery falls behind.

03

Preserve numerical and KV-state correctness

Sharing a sweep changes execution shape. The hardware-support design keeps the accumulation order and per-row normalization stable, streams verification through a top-1 head, and commits only the accepted answer-KV prefix. Newly processed frames remain committed even when draft answer tokens are rejected.

03 / MY CONTRIBUTION

Joint execution and hardware support

As co-first author in a collaboration with POSTECH, I co-designed joint execution and acceptance-paced scheduling, contributed to the hardware-support and KV-state design, and implemented the C++ LLaVA-OneVision execution foundation for numerical validation.

04 / EVALUATION

Setup and results

The collaborative evaluation combines a bandwidth/compute model for LLaVA-OneVision-7B, an eight-video real-model streaming study with a 25 GB/s bandwidth clock, and a ZCU104 hardware evaluation. The streaming comparison reports delivered tokens and stall-free answers separately from answer accuracy. The full 7B model was not demonstrated at 10 frames/s on the FPGA.

Required bandwidth
55.5 → 23.0 GB/s
Modeled requirement for 10 frames/s and 3 tokens/s, versus Sarathi-style piggybacking.
Delivered tokens per second
3.46 vs. 1.61
Eight-video real-model study at a modeled 25 GB/s bandwidth limit.
Stall-free answers
78% vs. 1%
Same streaming study; this measures output continuity, not answer accuracy.
05 / OUTCOME & SCOPE

Bandwidth savings with steadier answer delivery

The ZCU104 evaluation reports a 63.4% lower estimated area–time product than Sarathi-style hardware. The verification and partial-KV-commit extension adds 544 LUTs with no additional DSPs or BRAMs. The benefit is strongest when weight traffic is a limiting resource; available compute, row-buffer capacity, drafter cost, and frame freshness still constrain the operating point.