Velox World/Research/RPOPreprint · arXiv soon

RPO

Routing Policy Optimization for online diffusion reinforcement learning.

Reinforcement learning can align visual generators with rewards beyond pretraining, but score-function methods usually inject noise into the reverse trajectory and pay for it with large groups of complete rollouts. RPO moves exploration inside the generator: attention routing becomes a discrete policy, and inference returns to dense attention.

Compare base and RPO Paper arXiv · soon Code soon

Authors and affiliations are listed with the preprint.

Wan2.2-TI2V-5B · same prompt, same noise · drag to wipe

Base · Wan2.2 + RPO

HPSv3 · base 7.89 RPO 8.96 change +1.07

Prompts 0 to 5 of the 61-prompt comparison set, in order, with HPSv3 for each clip. Across all 61 prompts the mean HPSv3 goes from 7.03 to 8.74; RPO scores higher on 52 prompts and lower on 9. Every prompt is in the gallery below.

24.1GPU-h

To train on Wan2.2

H100 GPU-hours. Flow-GRPO took 75.4 and DanceGRPO 102.3; the runs are not compute-matched.

3.1–4.2×

Less training compute

Than Flow-GRPO and DanceGRPO respectively, with the same HPSv3 reward.

Dense-model reward gain

Reached within about 15 steps and held through step 50, measured on the dense model.

Rollouts per group

A matched pair, against G = 8 for DanceGRPO. Both share prompt and noise.

01 · The idea

Put the randomness inside attention, not on the sampler's path.

Sampler-space RL such as Flow-GRPO and DanceGRPO explores by perturbing the reverse process, then estimates an advantage from a group of complete stochastic rollouts.

RPO keeps the deterministic sampler. At one denoising level, each query block keeps a random subset of its candidate key blocks, with Bernoulli probabilities taken from the generator's own query–key affinities. The sampled masks are the actions.

Sampler-space RLRPO

ExplorationNoise injected into the reverse trajectoryBernoulli routes inside attention

Per updateGroups of complete rollouts, G = 8 in DanceGRPOMatched pairs, G = 2, 64 pairs per step

AdvantageGroup statistics over the rolloutsReward difference of the pair

SamplerStochastic during trainingOriginal, deterministic, throughout

Two rollouts share the prompt, the initial latent, the routed level and the routed layers. They differ only in their masks, so their reward difference measures the effect of the two routing choices. Below, each cell is one candidate key block. Change how sharp the policy is and watch the two masks agree or diverge.

Route lab · rows are query blocks, columns are key blocks

a₁Rollout 1

Keeps — of candidates

a₂Rollout 2

Keeps — of candidates

ΔWhere they differ

Differ on — of candidates

Temperature T 1.00 Offset β₀ +0.0

Safety block, always active Kept candidate Dropped (shade = keep probability) Δ: kept only in a₁ Δ: kept only in a₂

Illustration of the sampling rule p = σ(z / T + β₀) on toy logits, not real attention maps.

Denoise densely

Run the reverse process as usual down to a sampled routed level. The sampler is untouched.

Draw two masks

Safety blocks stay on. Each candidate block is kept with its Bernoulli probability.

Finish the pair

Same prompt, latent, level and layers. Only the sampled routes differ.

Score the difference

The reward difference becomes the advantage for the recorded routing actions.

Update shared weights

A replay likelihood gradient updates the attention parameters dense inference reuses.

Nothing changes at inference. All masks are removed: dense attention and the original deterministic sampler. The gains transfer because the routing policy reuses the generator's native attention parameters.

02 · Results

Faster to learn, and the gain survives dense inference.

Training rewards are scored on routed or SDE rollouts, so the comparison that matters is the dense model that ships: same prompts, same noise, same sampler.

Dense evaluation gain · Wan2.2, HPSv3 reward

Markers are values stated in the paper's text; lines between them are schematic. Gain is measured against each method's own step-0 score.

RPO reaches its final gain within about 15 steps and keeps it. It gains +2.17 by step 5 and about +2.9 by step 15, then stays near +3 through step 50.

Flow-GRPO follows its training reward. The dense model dips to about −3 at step 10 and only recovers to +2.7 by step 50, so it needs the whole 50-step window to approach the level RPO reaches within about 15.

DanceGRPO accumulates almost no gain, peaking at +0.29 at step 5 and staying between about 0 and +0.2.

RPO collects 64 matched pairs per step and routes a single denoising level; DanceGRPO uses groups of G = 8 and updates the full model.

Training cost on Wan2.2

GPU-hours on H100. The runs are not compute-matched.

RPO

24.1

Flow-GRPO

75.4

DanceGRPO

102.3

VBench transfer

6,220 videos from shared prompts and initial noise. Scores in percent, best per row in bold.

Dimension	Wan2.2 base	RPO	Dance- GRPO	Flow- GRPO
Aesthetic quality	61.91	62.64	61.74	60.09
Imaging quality	67.71	68.65	67.69	67.43
Subject consistency	98.37	98.69	98.38	97.59
Motion smoothness	98.76	99.06	98.75	97.88
Object class	81.01	85.41	80.81	79.10
Multiple objects	59.15	61.30	59.34	59.98
Human action	79.40	84.20	80.00	81.80
Scene	37.41	40.84	36.95	36.73
Overall score	81.61	82.00	81.46	81.19
Training GPU-hours (H100)	—	24.1	102.3	75.4

The paper's table lists the eight VBench dimensions where RPO exceeds both RL baselines, plus the overall score. Other dimensions show trade-offs, described alongside.

RPO is the only trained checkpoint above the base model overall (82.00 against 81.61; DanceGRPO 81.46, Flow-GRPO 81.19). Differences from the base are largest on semantic dimensions: human action +4.80, object class +4.40, scene +3.43 and multiple objects +2.15, and smaller on quality and consistency dimensions.

There are trade-offs. RPO lowers color and appearance style relative to the base model, although it still leads the trained baselines there. The four Wan2.2 overall scores span only 0.81 points on one seed set, so the aggregate ranking is directional.

SD3.5-Medium · DrawBench, no CFG

ImageReward−0.54 → −0.18

UnifiedReward2.10 → 2.32

DeQA-Score3.35 → 3.56

1,000 images from 200 prompts and 5 seeds. None of these metrics is the training reward.

03 · Gallery

Every prompt, side by side.

Full comparison pages, with per-prompt scores, wipe and frame-by-frame viewers. Both models render the same prompts from the same starting noise.

Wan2.2-TI2V-5B · video

61 prompts, each a 5-second 1280 × 704 clip, base against RPO, with a wipe, side-by-side and per-frame viewer.

HPSv3 7.03 → 8.74 · RPO higher on 52 of 61

SD3.5-Medium · image

64 prompts at 512 × 512, base against RPO, same prompt and same starting noise.

HPSv3 3.97 → 10.00 · RPO higher on 63 of 64

MiniMax-H3

MiniMax-H3 · video

Comparison page for a third generator.

Coming soon

04 · Release

What is coming, and when it lands here.

This page will be updated in place when the preprint is public. Citation details appear here at that point.

Preprint

The paper, with implementation settings and evaluation protocols in the appendix.

arXiv · soon

Code and results

Training code for the routing policy and the evaluation outputs.

Planned

MiniMax-H3 gallery

A third comparison page, next to Wan2.2 and SD3.5.

Planned