World-Anchored Canonicalization

WACO

Towards Pose-Robust Few-Shot Vision-Language-Action Models

A bumped tripod or a re-docked robot base makes a few-shot VLA reach for the wrong place. WACO recovers both shifts from a single RGB-D frame and maps the observation back to the pose the policy was trained at.

Paper (coming soon) Code Model Dataset Benchmark
3 min 30 s · with sound

Towards Pose-Robust Few-Shot Vision-Language-Action Models

Wenhao Li1,2 Zhongyi Han1 Yuhao Wei4,2 Liang Han3 Yilong Yin1 Liqiang Nie3

1Shandong University 2Shenzhen Loop Area Institute 3Harbin Institute of Technology, Shenzhen 4Xidian University

1Shandong University  ·  2Shenzhen Loop Area Institute  ·  3Harbin Institute of Technology, Shenzhen  ·  4Xidian University

Abstract

A bumped camera and a re-docked robot are one phenomenon.

Fine-tuning a vision-language-action (VLA) model on a few dozen demonstrations is the standard way to put a manipulation policy on a robot, yet the resulting policy silently assumes that the camera and the robot base stay exactly where they were during data collection: a bumped tripod or a base re-docked a few centimetres away makes it reach for the wrong place. We show that both failures are one phenomenon, a change of the camera-to-base extrinsic that the policy has memorised, and that the robot carries everything needed to undo it. We introduce WACO, a world-anchored canonicalization framework for VLA models: the arm, placed by forward kinematics at the measured joint angles, serves as a metric calibration target, and the static scene serves as the world anchor that tells a moved camera from a moved robot. From a single RGB-D frame, WACO estimates both shifts without ground-truth extrinsics or extra data, canonicalises the image to the calibration viewpoint and the state, action and start pose to the calibration base frame, and fine-tunes the unmodified policy under the artefacts of its own estimator. We also introduce LIBERO-PoseShift, the first benchmark for RGB VLAs with graded camera, base and compound shifts. Across LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world tasks on a UR5e robot, WACO consistently outperforms the standard few-shot recipe and prior viewpoint-robust methods, raising shifted success by 13.2 points in simulation and by 36.7 points on the real robot, recovering 90% of the simulation gap to a ground-truth oracle and 87% of the real-robot gap, and establishing a new state of the art among methods that use no perturbed training data, with no test-time calibration.

Overview: a few-shot fine-tuned VLA misses the target after a pose shift; WACO estimates the shift and restores the training view
Overview. A few-shot fine-tuned VLA memorises the camera-to-base extrinsic of its demonstrations and misses the target once the camera is bumped or the robot is re-docked (left: our UR5e setup at the calibration pose and after the table and camera were moved 10 cm away from the robot). WACO estimates both shifts from a single RGB-D frame, with the arm placed by forward kinematics as the metric anchor and the static scene as the world anchor, and canonicalises the observation back to the calibration viewpoint and base frame, so that an unmodified π0.5 sees the world it was trained in.
01 — Method

Recover the shift, then map the observation back.

Instead of asking the policy to learn to generalise across poses from data, WACO recovers the shifts from what the robot already knows and maps the observation back to the pose it was trained at. The policy architecture is untouched, and a base shift is reduced to “the object is elsewhere on the table”, a variation the demonstrations already cover.

Both shifts are one quantity.Each changes the camera-to-base extrinsic, and a base shift also moves the objects in the base frame.Tbc = ΔTb−1 · Tbc0 · ΔTc
The arm is its own calibration target.Forward kinematics at the measured joint angles places every link in the base frame, so the visible arm pins down the extrinsic from a single RGB-D frame, without a checkerboard and without training.
The arm alone is not enough.Every camera and base shift with the same extrinsic renders the arm identically yet demands different actions; the static scene provides the missing world anchor for free.
Model pipeline of WACO
Model pipeline. WACO wraps an unmodified π0.5. At reset, a training-free front end registers the FK-placed arm model to the observed depth (arm anchor) and matches the static scene to an empty-workspace reference (world anchor); their fusion yields both shifts and an out-of-plane residual that gates the estimate. Channel A warps the image with the camera shift only; channel B re-expresses state, action and start pose with the base shift only. The policy is fine-tuned once on the canonicaliser's own artefacts.
1

Two-anchor shift estimation

The arm anchor registers the FK-placed arm cloud to the observed depth and yields the extrinsic metrically, with no learned component. The world anchor matches the live frame to an empty-workspace reference and recovers the camera shift independently of the robot. Fusion gives the base shift in closed form; a base cannot leave the floor, so an out-of-plane residual above 2 cm or 2° means that an anchor has failed, and the episode then runs without canonicalisation.

2

Channel-separated canonicalization

Channel A forward-splats the third-person image to the calibration camera with the camera shift alone; disocclusions are never hallucinated and the wrist image is never warped. Channel B maps the reported end-effector pose into the nominal base frame, rotates the policy's deltas back before execution (exact, no first-order approximation), and homes the end effector to the start pose of the demonstrations.

3

Error-aware fine-tuning

A policy fine-tuned on pristine demonstrations has never seen a splatted image or a slightly-off state. With probability 0.5 each, a training frame is splatted to a random virtual camera and back with a shift perturbed by the estimator's error model, and the state receives a planar base-shift error. Action labels are unchanged, the flow-matching objective is unchanged, and no token, head or parameter is added.

Two-anchor estimation
Two-anchor estimation. (a) Arm anchor: the visible part of the FK-placed arm cloud is registered to the observed depth by trimmed ICP and ranked by a render-and-compare score. (b) World anchor: the live frame is registered to the empty-workspace reference by SIFT, 3D RANSAC and point-to-plane ICP. (c) Fusion closes the extrinsic equation; the discarded height, roll and pitch of the base shift form a ground-truth-free consistency check.
02 — Benchmark

LIBERO-PoseShift: graded camera, base and compound shifts.

LIBERO-PoseShift extends LIBERO, whose tasks come with 50 demonstrations collected from one camera pose and one base pose, with a deterministic perturbation layer on its robosuite/MuJoCo environment, so that all methods see identical scenes. State and actions follow a base-frame protocol in which a moved robot reports the pose it believes it has, exactly as hardware does.

Task
Shift kind
Level
CleanL1L2L3

Camera shift · L3

Calibration pose
Camera shift · L3

Third-person observation of the same episode at the calibration pose and under the selected shift, as sampled by the benchmark (policy: the standard few-shot recipe, no canonicalisation; rollouts in real time). The shift of every task, episode, kind and level is fixed by the benchmark's deterministic sampler.

LevelCamera trans.Camera rot. / axisBase trans.Base yawpx (t / yaw)
L1≤ 2 cm± 3°≤ 2 cm± 3°6 / 12
L2≤ 5 cm± 5°≤ 5 cm± 8°15 / 21
L3≤ 10 cm± 10°≤ 10 cm± 15°31 / 42

Perturbation levels. Camera translations are uniform in a ball and base translations in a disc, both between half and the full radius; rotations are uniform per axis. Last column: displacement of a target object in the 256² third-person view for the largest translation / yaw.

Three kinds, three levels.Camera shifts rotate and translate the third-person camera; base shifts move the robot in the floor plane and rotate it about the vertical, leaving table, objects and third-person camera fixed; both applies the two independently.
Base-frame protocol.LIBERO expresses state and actions in the world frame, which hands a moved robot the correct target for free. A real robot has no world frame, so the benchmark anchors state and actions at the calibration base pose.
The main grid.LIBERO-Spatial and LIBERO-10 × 10 tasks × 10 conditions (clean plus three kinds at three levels) × 10 episodes: 2,000 episodes per model.
LIBERO-PoseShift observations
LIBERO-PoseShift observations. Episode 0 of a LIBERO-Spatial task and of a LIBERO-10 task: third-person and wrist views at the calibration pose, and the third-person view with the wrist view inset under camera, base and compound shifts of level 1–3. Yellow: outline of the calibration view. A camera shift displaces the whole third-person image and leaves the wrist view untouched; a base shift leaves the scene of the third-person image in place, moves the arm in it and changes the wrist view.
03 — Results

The standard recipe is fragile; WACO stays near the oracle.

Across LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world tasks on a UR5e robot, WACO consistently outperforms the standard few-shot recipe, and its margin grows with the size of the shift.

+13.2pts
shifted success on LIBERO-PoseShift over the standard few-shot recipe (74.0 → 87.2%)
+36.7pts
shifted success on a real UR5e, four tasks (43.3 → 80.0%)
90%
of the simulation gap to a ground-truth oracle recovered, and 87% of the real-robot gap
93.7%
on the LIBERO-Plus camera track (92.4% over three seeds), with no perturbed training data

Success under growing pose shifts

LIBERO-PoseShift (LIBERO-Spatial and LIBERO-10, 200 episodes per point, base-frame protocol). All learned policies share checkpoint, demonstrations and budget.

WACO (W1 + runtime) WACO runtime on B1 B1 (official few-shot recipe) GT oracle on W1 (upper bound)

Success rate in %. Differences below two points lie within the Wilson 95% intervals and are not interpreted.

How much of the gap to the oracle is recovered

Average success over the shifted conditions: the standard recipe, WACO, and the ground-truth oracle that canonicalises with the true shifts.

Pose-shift robustness on LIBERO-PoseShift (LIBERO-Spatial and LIBERO-10, 200 episodes per cell, base-frame protocol). Base / camera gauge: the arm's pose change read as a camera / base shift. † Upper bound with ground-truth poses. Avg is the mean over the nine shifted conditions.

LIBERO-Plus camera track (1,599 tasks, one episode each). Released: the benchmark's prompts, which append the variant's parameters; corrected: the task's own instruction. Evaluation seed 0 / mean over three evaluation seeds. The WACO runtime alone lifts the public checkpoint to 89.4% without retraining or extrinsics.

Real-robot results (UR5e, success rate in %, 20 trials per task and condition, averaged over the four tasks). Avg is the mean over the three shifted conditions; † uses the re-calibrated ground-truth shifts and is an upper bound.

Real-robot per-task performance
Real-robot per-task performance. Success under pose shifts (mean of the camera, base and compound conditions) on the four UR5e tasks. WACO improves every task, most on Toy on Shelf (+46.7 points).

Few-shot scaling: average shifted success (nine conditions) on LIBERO-Spatial and LIBERO-10 with the first 10, 20 or all 50 demonstrations per task, same recipe and budget. WACO with 10 demonstrations (82.4%) exceeds the standard recipe with 50 (74.0%).

Few-shot scaling
Few-shot scaling. The gap between WACO and the standard recipe widens as data shrinks, because the pose variation that the recipe must absorb from data is removed analytically; both the runtime and the error-aware training contribute, and their combination is better than either.

Ablations. Each block changes one component of the full method on LIBERO-Spatial and LIBERO-10. Avg is the mean over the nine shifted conditions.

04 — Videos

In simulation and on a real UR5e.

All clips are played at 3× speed.

Simulation · LIBERO-PoseShift, level 3

3× speed

The same episode and the same shift: the standard few-shot recipe on the shifted view, and WACO on the view it canonicalises for the policy.

Real robot · UR5e, four tasks

3× speed

WACO on all four tasks: one uncut trial at the calibration pose and one under the base shift, in which the table and the tripod have been moved together 8 cm to the left and 6 cm forward, away from the arm.

Calibration pose
Base shift · 10 cm
The four real-robot tasks
Real-robot tasks. The four tabletop tasks form an increasing difficulty ladder: Lemon to Basket needs only coarse localisation; Color Matching adds an instruction-dependent referent and a cup opening only slightly wider than the block; Toy on Shelf requires carrying the toy over the shelf's wire frame and releasing it on the elevated upper tier; Toast into Toaster requires an insertion whose clearance is far below the object size.
05 — Findings

What we found.

01

The standard recipe is fragile.

The official few-shot recipe loses 17.0 points at camera L3, 21.5 at base L3 and 40.0 at compound L3, and a 2D augmentation recovers part of the camera axis only. Base shifts, which no prior benchmark measures, are the harder half of the problem even for the best policy.

02

Data alone is not enough.

A policy trained on exactly WACO's augmented samples but run without canonicalisation gains 7.1 points over the standard recipe, yet stays 6.1 behind WACO, because the policy cannot undo a shift it never observes explicitly.

03

Estimation plus error-aware training closes the loop.

The runtime alone already recovers 82% of the gap to the standard recipe's own oracle, and with error-aware training WACO reaches 87.2%, 13.2 points above the standard recipe and within 1.5 points of its own oracle, with a clean score not below the standard recipe's.

04

Both anchors are needed.

The single-anchor gauges fail as the gauge ambiguity predicts: the base gauge under base shifts (base L3: 56.0 vs. 86.5 for WACO), and both gauges at camera L3 (64.0). Only the two-anchor estimate handles the compound condition (both L3: 79.0 vs. 46.0 and 49.0).

05

The advantage grows where demonstrations are scarcest.

The gain over the standard recipe grows from 13.2 points with 50 demonstrations per task to 15.7 with 20 and 19.2 with 10, so that WACO with 10 demonstrations (82.4%) exceeds the standard recipe with 50 (74.0%).

06

From simulation to a real robot.

On the UR5e, WACO raises the average shifted success from 43% to 80%, improves every task under every condition, and comes within 5.4 points of the ground-truth oracle. The front end runs once per trial in about 0.8 s on the robot's CPU and recovers the real shifts with median errors of 2.8 mm and 0.3° for the camera and 3.8 mm and 0.4° for the base.

Citation

BibTeX

@article{li2026waco,
  title   = {Towards Pose-Robust Few-Shot Vision-Language-Action Models},
  author  = {Li, Wenhao and Han, Zhongyi and Wei, Yuhao and Han, Liang and Yin, Yilong and Nie, Liqiang},
  journal = {arXiv preprint},
  year    = {2026}
}