Velox: Spatial Memory Makes Long-context World Models Efficient

Shen, Qiuhong; Wang, Shizun; Wang, Xinchao

Velox World/Research/VeloxPreprint · arXiv soon

Velox

Spatial memory makes long‑context world models efficient.

National University of Singapore

Interactive world models have to remember what they have already generated, but dense attention over a growing history makes every new chunk slower. Velox is a training-free attention router: camera frustum slabs from streaming geometry tell each region of the video which parts of the past it actually needs.

Watch the rollouts Paper arXiv · soon Code soon

WBench navigation · same case, two attention paths

Dense attention Velox

0:00 / 0:00

Clip 1 of 20 · 832 × 464 · 24 fps · same WBench case, same model, dense attention against Velox routing

WBench navigation clips, each rendered twice: by dense attention and by Velox routing. Both renderings use the same model and the same case definition; only the attention path differs. Scores over all 158 navigation cases are in the results table below.

4.36×

Faster attention

Per-call latency at chunk 48: 34.2 ms down to 7.8 ms.

2.43×

Faster denoising

Per chunk at chunk 48: 7.19 s down to 2.96 s, geometry built asynchronously on the same H200.

0.15pts

Gap to dense on WBench

API-free average over all 158 navigation cases, versus dense LingBot-Fast-Cam.

0steps

Training steps needed

The pretrained generator and its conditioning interface are left unchanged.

01 · The problem

A world model that remembers pays for it on every chunk.

Long-horizon generation needs persistent visual history, so chunked video world models keep it in a growing KV cache.

An infinite window lets every current block attend to every retained block, even though the camera can only see a small part of the past. Later chunks get progressively slower, which is exactly wrong for interaction. Frame memory retrieves whole past views, which is coarse.

The fix is not to forget. It is to route: keep the memory, and expose each query block to a bounded set of the blocks that matter.

Three ways to use visual history. Infinite window: attend to all past frames, with cost growing over time. FOV frame memory: retrieve past views. Velox token routing: route to relevant blocks. — **Three ways to use history.** Attend to everything, retrieve whole past views, or route each token block to the blocks that are relevant.

02 · Method

Route by where things are, guided by what the model already wants.

Velox leaves the generator and its conditioning interface untouched and changes only the inference-time visibility between current query blocks and historical KV blocks.

Velox pipeline in six stages: past chunks with camera poses, spatial memory as a colored 3D point structure, the current chunk, geometric overlap between current blocks and memory, Velox routing from query blocks to historical KV blocks plus safeguard blocks, and sparse attention producing the next chunk. — **Overview.** Past chunks are kept as a spatial memory. For the current chunk, geometric overlap decides which memory each query block is exposed to, a bounded route (plus safeguard blocks) is built, and block-sparse attention produces the next chunk.

01

Remember as frustum slabs

After each chunk, streaming geometry turns every historical KV block into slabs: a source-camera token rectangle with a robust depth interval.

02

Give the new block a depth

Several cached depth frames are warped forward into the target camera, so each query block gets a depth interval of its own.

03

Let overlap nudge, not decide

Fully covered blocks use a geometry-weighted route with a small budget (80). Blocks that are not fully covered keep the plain block-sparse route with a larger budget (150).

04

Run a bounded route

A fixed-size block list, plus the previous chunk and Gilbert-curve neighbours, feeds a block-sparse kernel that matches FlashAttention tiles.

Mechanism sketch · toy scene, top-down

Chunk—

Remembered blocks—

Dense sees—

Velox sees—

Long-range recalls—

Remembered slab Overlaps current view Current query block Geometry-guided Model-only Previous chunk, always kept

An illustration of the mechanism on a toy scene, not real model state. Drag the scene or the slider to scrub the rollout.

p̃_i,r ∝ p_i,r · ( 1 + λ · ĝ_i,r ), ĝ = 𝟙[g > τ] · min( g / g_ref, 1 )

pThe layer's own importance for historical block r, from mean query–key affinity. It is never thrown away.

gOverlap (IoU) between the query block's slab and the historical block's slabs, in world space.

λHow hard geometry may push. With λ = 2, a vetted block gains at most 3× its original importance.

Covered and uncovered blocks. Well-covered query blocks use the fused route (K = 80). Blocks without enough geometric coverage, such as sky or newly exposed regions, keep the model-only route (capped at K = 150). The previous chunk is always retained, and Gilbert-curve neighbours act as deterministic safeguards.

What the router sees

For a query block, multiwarp provides a depth map, and the best-matching historical blocks are found by slab overlap. When the camera revisits a place, the matches come from chunks far in the past.

A query block in chunk 3 on a desk scene, its multiwarp depth map, and the three best-matching historical blocks from chunks 2 and 1 with slab-overlap scores 0.486, 0.455 and 0.399. — A block in chunk 3, matched to blocks from chunks 1 and 2.

A query block in chunk 18 on the same desk scene, its multiwarp depth map, and the three best-matching historical blocks from chunks 9, 10 and 7 with slab-overlap scores 0.684, 0.673 and 0.649. — A block in chunk 18, matched to memory from chunks 7, 9 and 10.

Why the route is a block, not a token

FlashAttention streams fixed Q and K/V tiles. A token-wise mask cuts across many tiles, so most are still loaded. A block route keeps or skips whole tiles before the kernel runs.

Left: a token-wise route crosses many FlashAttention tiles, so gathers can become memory-bound. Right: Velox's block route selects or skips whole historical K/V tiles, skipping K/V reads and QK tile matmuls. — **Block routing connects geometry and efficient attention.** Routes are entire K/V tile choices, so skipped blocks remove both the K/V reads and the QKᵀ tile products.

Keeping neighbours adjacent in the block order

A Gilbert (space-filling) order keeps spatially and temporally adjacent tokens adjacent in the flattened sequence, which keeps sparse blocks local. Applied chunk by chunk it breaks temporal adjacency; flipping the mapping horizontally on alternate chunks restores it.

Toy case with latent height 3, width 4 and block size 2, over two chunks of three frames each. Top: THW-linear rank. Bottom: global Gilbert rank. — **THW-linear against global Gilbert order** on a toy latent (H = 3, W = 4, block size 2).

Applying the Gilbert order chunk by chunk with a plain repeat breaks temporal adjacency; flipping chunk 1 horizontally recovers it. — **Incremental Gilbert order.** A plain repeat per chunk breaks adjacency (red); a horizontal flip on alternate chunks recovers it (green).

03 · Results

Late chunks get much cheaper. Aggregate quality stays near dense.

Runtime is measured over a 48-chunk rollout on a single H200, with geometry construction running asynchronously on the same GPU. Quality is measured on WBench, a multi-turn benchmark for interactive world models, over its 158 navigation cases of 16 to 48 generated chunks.

Two line charts over generated chunks 1 to 48. Left: per-call attention time grows from near zero to about 34 milliseconds for dense FlashAttention and stays near 8 milliseconds for Velox-Attention. Right: chunk denoising time grows to about 7 seconds for dense and stays near 3 seconds for Velox. — **Runtime scales with generated length.** At chunk 48, per-call attention falls from about 34.2 ms to 7.8 ms (4.36×) and chunk denoising from about 7.19 s to 2.96 s (2.43×). For the first few chunks dense is the faster path; the curves cross within the first seven chunks. In the figure, Velox-Attention is the routed attention.

Bounded memory, bounded cost. Dense FlashAttention schedules every current–history tile pair. Velox schedules only the routed blocks, so a skipped block removes both the QKᵀ product and the KV load.

Geometry adds value beyond sparsity. Against a matched model-only block-sparse route, soft fusion lifts the API-free average from 73.11 to 73.51 and the overall average from 75.01 to 75.60. Dense stays marginally ahead on the API-free average, by 0.15 points, and ahead on some individual metrics; the table shows all of them.

Geometry and the model agree. On calibration scenes, blocks with positive slab overlap land in the model's own top-50 far more often than chance: R_geo = 0.73 at layer 14, against a random expectation of 0.35. Fusion at λ = 2 raises top-80 positive-overlap recall from 0.60 to 0.85 while changing 9.0% of selected blocks.

Scope. Timings are for the reported single-H200 asynchronous implementation on LingBot-Fast-Cam. The paper does not claim wall-clock superiority in every setting.

WBench navigation · 158 cases

Scores are in [0, 100]. The three LingBot-based rows share generated cases and the Gemini judge. Published baselines use the Doubao judge, so only the API-free average is comparable across the two groups. Within the LingBot group, bold marks the best value per column.

Method	Average		Task scores				Consistency
Method	API-free avg.	Overall avg.	Quality score	Setting adh.	Interaction adh.	Physical score	Back. const.	Spatial const.	Gated spatial	Persp. const.	Segment cont.	Geom. const.	Photo. const.	Subject const.
Built on LingBot-Fast (Gemini judge)
LingBot-Fast-Cam dense	73.66	75.52	77.7	79.9	79.4	39.0	87.9	73.0	73.0	81.7	90.5	80.5	77.6	87.2
Block-sparse model-only route	73.11	75.01	77.9	79.1	78.0	38.6	87.6	71.8	71.8	75.7	94.3	80.4	77.1	85.8
Velox	73.51	75.60	78.1	80.3	78.8	38.8	88.0	74.3	74.3	77.0	92.4	80.5	78.0	86.4

Show the other 13 methods from the paper's table

Method	Average		Task scores				Consistency
Method	API-free avg.	Overall avg.	Quality score	Setting adh.	Interaction adh.	Physical score	Back. const.	Spatial const.	Gated spatial	Persp. const.	Segment cont.	Geom. const.	Photo. const.	Subject const.
Text-driven video generation models (Doubao judge)
Seedance 1.5	73.03	79.68	82.1	82.9	68.0	68.4	89.6	72.7	72.4	70.5	96.2	82.4	76.8	90.1
Wan 2.7	72.35	80.76	81.5	91.4	66.0	71.8	89.4	71.0	71.0	78.2	92.4	83.7	76.4	90.7
Kling 3.0	74.03	81.53	81.4	91.0	70.3	69.3	92.3	75.2	75.1	76.8	93.0	88.9	79.9	88.5
YUME 1.5	71.87	76.51	77.7	72.4	72.0	65.2	90.3	71.5	71.4	48.0	99.4	88.0	83.3	88.8
HY-Video 1.5	74.12	81.16	77.6	85.6	71.8	67.4	92.1	79.2	75.1	86.6	99.4	94.6	80.3	91.6
LTX 2.3	69.42	76.21	77.1	85.2	67.6	64.9	88.3	70.2	70.2	69.8	75.8	76.9	79.2	87.2
LongCat-Video	71.83	78.65	75.4	72.3	63.1	68.9	95.1	83.3	66.2	81.5	99.4	95.4	82.2	93.4
Kairos 3.0	69.93	75.34	74.0	70.3	65.1	60.4	91.1	76.8	62.0	76.3	94.3	89.0	80.8	90.8
Camera-controlled world models (Doubao judge)
SANA-WM	74.65	77.86	79.3	76.1	82.1	61.9	90.5	77.1	76.4	49.0	97.5	88.7	81.2	85.4
HY-World 1.5	77.78	80.43	78.1	72.2	87.5	66.3	92.7	90.6	84.9	62.5	100.0	92.0	83.1	89.1
Fantasy-World	72.65	77.58	72.4	71.3	72.1	66.8	94.2	80.6	64.2	79.8	100.0	95.3	84.8	92.5
InSpatio-World	73.95	77.99	71.5	71.4	72.8	65.2	95.0	93.8	66.5	72.5	100.0	97.3	87.4	94.4
Astra	65.67	67.31	67.1	59.6	67.7	51.4	85.3	64.7	63.3	30.0	86.6	85.6	87.5	83.5

04 · Release

What is coming, and when it lands here.

This page will be updated in place when the preprint is public. Citation details appear here at that point.

01

Preprint

The paper, with appendix: routing configuration, geometry construction, execution path, validation scope.

arXiv · soon

02

Code and results

Public release of the router, the sparse kernel and the generated outputs for the WBench navigation split.

Planned