01 / THE PROBLEMHow can a streaming model keep up with video and deliver answers on time?
A streaming video-language model must ingest frames continuously while producing an answer at a usable rate. Both activities repeatedly read the same large model weights. Grouping more frames into one sweep amortizes those reads but leaves fewer opportunities to emit answer tokens; smaller chunks improve responsiveness but consume more bandwidth. On an edge device, a schedule that meets the frame rate can still stall the answer.
02 / THE APPROACHShare a weight sweep and pace the next chunk
01Verify draft tokens while processing frames
A smaller model proposes answer tokens. EdgeVQA places their verification rows beside the incoming frame rows in the target model’s execution, allowing one weight sweep to ingest video and verify several answer tokens. Frame and answer rows retain separate attention contexts and rotary positions.
02Size chunks from delivered-output lead
Draft acceptance changes over time. The scheduler tracks accepted-token yield, how far delivered output is ahead of its target rate, and the estimated duration of the next sweep. It chooses the largest frame chunk that preserves an output cushion, subject to buffer, compute, and frame-freshness constraints. Verification-only execution can recover when answer delivery falls behind.
03Preserve numerical and KV-state correctness
Sharing a sweep changes execution shape. The hardware-support design keeps the accumulation order and per-row normalization stable, streams verification through a top-1 head, and commits only the accepted answer-KV prefix. Newly processed frames remain committed even when draft answer tokens are rejected.
03 / MY CONTRIBUTIONJoint execution and hardware support
As co-first author in a collaboration with POSTECH, I co-designed joint execution and acceptance-paced scheduling, contributed to the hardware-support and KV-state design, and implemented the C++ LLaVA-OneVision execution foundation for numerical validation.
Algorithm–hardware co-designSpeculative decodingMemory bandwidth
04 / EVALUATIONSetup and results
The collaborative evaluation combines a bandwidth/compute model for LLaVA-OneVision-7B, an eight-video real-model streaming study with a 25 GB/s bandwidth clock, and a ZCU104 hardware evaluation. The streaming comparison reports delivered tokens and stall-free answers separately from answer accuracy. The full 7B model was not demonstrated at 10 frames/s on the FPGA.
- Required bandwidth
- 55.5 → 23.0 GB/s
- Modeled requirement for 10 frames/s and 3 tokens/s, versus Sarathi-style piggybacking.
- Delivered tokens per second
- 3.46 vs. 1.61
- Eight-video real-model study at a modeled 25 GB/s bandwidth limit.
- Stall-free answers
- 78% vs. 1%
- Same streaming study; this measures output continuity, not answer accuracy.
05 / OUTCOME & SCOPEBandwidth savings with steadier answer delivery
The ZCU104 evaluation reports a 63.4% lower estimated area–time product than Sarathi-style hardware. The verification and partial-KV-commit extension adds 544 LUTs with no additional DSPs or BRAMs. The benefit is strongest when weight traffic is a limiting resource; available compute, row-buffer capacity, drafter cost, and frame freshness still constrain the operating point.