Cross-Modal Conflict Monitoring and Steering in VLAs

Chih-Hao Hsu
National Taiwan University
Code
Two separate passes use the current camera observation. A claim-neutral diagnostic pass produces a VLM representation that a linear probe reads to estimate visual state. The conflict monitor compares that estimate with a claim from the original instruction. A second pass sends the observation and unchanged instruction through the VLA; the monitor gates a steering site inside its action expert before robot actions are generated.
In a separate, claim-neutral diagnostic pass, a probe estimates scene state. The monitor compares it with the original instruction's claim and gates steering inside the action expert during the action pass. View full-size figure

Abstract

Vision-language-action policies can receive instructions whose factual claims disagree with the current scene. This failure can originate from a person or from an upstream planner or memory, but the downstream policy still has to decide whether and how to act. We study a bounded setting in which a task configuration declares the claim to monitor, the camera is authoritative for that claim, and an allowed recovery is specified in advance.

A calibrated linear probe reads task-relevant visual evidence from the VLA, a monitor compares it with the declared language claim, and a confident disagreement activates a task-specific intervention in the action expert's flow trajectory. In a prospectively frozen 660-rollout evaluation over five configured LIBERO tasks, monitor-triggered steering raises conflict success from 8.3% to 75.0%, with positive effects on all five tasks and no observed false triggers in 60 aligned trials. A same-information prompt correction baseline reaches 81.7%; action-flow steering is therefore a distinct non-language intervention surface, not a generally superior replacement for prompt repair.

Method

A separate diagnostic pass supplies the monitor with a visual-state estimate, which gates a task-specific intervention during action generation.

The current observation is paired with a claim-neutral question in a diagnostic VLM pass. A linear probe decodes task-relevant scene state from its representation. The monitor compares this estimate with the claim extracted from the original instruction and triggers only when they disagree with sufficient confidence.

When triggered, a task-specific correction is applied during action denoising inside the action expert. The instruction is unchanged at test time, and the monitor can release the intervention as the scene changes.

Each configured task fits its own probe and controller from small offline calibration data. The contribution is a repeatable procedure across tasks, not zero-shot transfer of one shared steering vector.

Qualitative rollouts

Exact replays of three matched cells from the frozen cross-task evaluation. Within each row, the scene, conflicting instruction, simulator state, and action-flow noise are identical.

Absent-object conflict

The instruction requests milk, but milk is absent from the observed scene. The configured recovery is to pick up the ketchup.

Base VLAFailure
The policy follows the unavailable target and does not complete the task within 220 steps.
Monitor-triggered steeringSuccess
The probe predicts absent with 0.991 confidence. Steering starts at step 0, releases at grasp, and the task succeeds in 142 steps.
Setting and exact prompts
Observed scene
Milk is absent; ketchup remains visible.
Instruction used online
pick up the milk and place it in the basket
Probe query
Is the milk visible right now?
Reference instruction, offline only
pick up the ketchup and place it in the basket

Scene 46, simulator seed 42, action-flow noise seed 10057. Replay metadata

Spatial conflict: cabinet vs. cookie box

The black bowl is on the wooden cabinet, while the instruction claims that it is on the cookie box.

Base VLAFailure
The base policy searches under the conflicting spatial premise and times out at 220 steps.
Monitor-triggered steeringSuccess
The probe predicts cabinet with 0.810 confidence. Steering starts at step 0 and the task succeeds in 193 steps.
Setting and exact prompts
Observed scene
The black bowl is on the wooden cabinet.
Instruction used online
pick up the black bowl on the cookie box and place it on the plate
Probe query
Where is the black bowl right now?
Reference instruction, offline only
pick up the black bowl on the wooden cabinet and place it on the plate

Scene 46, simulator seed 42, action-flow noise seed 10057. Replay metadata

Spatial conflict: cookie box vs. ramekin

The black bowl is on the cookie box, while the instruction claims that it is on the ramekin.

Base VLAFailure
The base policy remains conditioned on the incorrect support and times out at 220 steps.
Monitor-triggered steeringSuccess
The probe predicts cookie_box with 0.921 confidence. Steering starts at step 0 and the task succeeds in 73 steps.
Setting and exact prompts
Observed scene
The black bowl is on the cookie box.
Instruction used online
pick up the black bowl on the ramekin and place it on the plate
Probe query
Where is the black bowl right now?
Reference instruction, offline only
pick up the black bowl on the cookie box and place it on the plate

Scene 46, simulator seed 42, action-flow noise seed 10057. Replay metadata

These are outcome-selected qualitative replays of cells already present in the 660-rollout evaluation, not new trials or representative samples. The reference instruction is used only to fit each controller offline and is never passed to the policy during these rollouts. Gallery manifest

Results

SmolVLA on LIBERO. Each condition has 60 rollouts across five tasks, four unseen scenes per task, and three shared action-flow noise seeds.

+66.7 percentage-point gain under conflict
40 / 0 / 20 paired wins / losses / ties
60 / 60 conflicts detected
0 / 60 aligned false triggers
Input condition Intervention Task success
Conflicting instructionBase VLA8.3% (5/60)
Monitor-triggered action-flow steering75.0% (45/60)
Oracle-triggered action-flow steering75.0% (45/60)
Always-on action-flow steering76.7% (46/60)
Wrong-sign direction1.7% (1/60)
Norm-matched random direction13.3% (8/60)
Monitor-triggered prompt correction81.7% (49/60)
Oracle prompt correction81.7% (49/60)
Aligned instructionBase VLA76.7% (46/60)
Learned monitor and action-flow steering76.7% (46/60)
Forced action-flow steering15.0% (9/60)
Each rate is over 60 matched rollouts. The task-stratified scene-cluster bootstrap interval for the action-flow gain is [+55.0, +78.3] percentage points. All ten preregistered evidence gates pass. Oracle and always-on conditions are causal controls, not deployed variants.

Per-task success

Four unseen scenes and three action-flow noise seeds per task and condition.

Conflict setting Base VLA Monitor-triggered steering
Object presenceKetchup is present; the instruction refers to milk, which is absent 41.7% (5/12) 91.7% (11/12)
Object presenceMilk is present; the instruction refers to cream cheese, which is absent 0.0% (0/12) 83.3% (10/12)
Spatial relationThe bowl is on the cookie box; the instruction says ramekin 0.0% (0/12) 83.3% (10/12)
Spatial relationThe bowl is on the cabinet; the instruction says cookie box 0.0% (0/12) 50.0% (6/12)
Spatial relationThe bowl is on the stove; the instruction says cookie box 0.0% (0/12) 66.7% (8/12)
All tasks 8.3% (5/60) 75.0% (45/60)

Evaluation artifacts: 660-rollout release integrity manifest