Abstract
Direct action generation can commit to an interaction that is plausible, even stable, and still wrong for the task: a hammer held by its head, a bowl grasped where the fingers cannot close. AFFORD-X makes that choice explicit and inspectable. A typed intent names the target, the functional region and the next action, and a feasibility-first rule removes candidate interactions the embodiment cannot execute before functional affordance and task compatibility rank the rest. In every reported run the intent comes from the task definition and the parts from a geometric partition; a semantic proposer (Gemini 3.8 Flash) and SAM3 part grounding are implemented as alternative sources. No module is trained. Evaluation is counterfactual: every compared method ranks the same content-hashed candidate set, and every candidate is executed from the identical start state in the benchmark's own simulator, so a success difference is a difference in the selected interaction. With a 7-DoF Franka on LIBERO-PRO under the position and task perturbations, and with object geometry and the goal read from simulator state so that the selection layer is measured in isolation, the full decision layer succeeds in 0.57 and 0.38 of LIBERO-Goal scenes, 0.92 and 0.96 of LIBERO-Spatial scenes and every LIBERO-Object scene, above random choice among the same candidates in all six cells (10 initial states per task). The same layer is also evaluated on the 50 Meta-World MT50 tasks with a 4-DoF Sawyer. Every comparison comes with paired bootstrap intervals and rollout videos recorded during evaluation.
From Instruction to Interaction
Five stages; only the selection rule is ours, and nothing is trained. Pick a stage.
The Terms and the Rule
Feasibility decides eligibility. Function and task decide preference. The rule decides how they combine.
Three factors
r is 1 inside the embodiment's workspace and 0.1 outside; a scores alignment with the executor's top-down approach, with a floor of 0.3; c scores clearance from obstacle points against a 5 cm margin. None of them looks at the object's function.
The gate
A candidate is eligible when r·a·c ≥ 0.5. When no candidate passes, the rule abstains and the episode counts as a failure, rather than executing something the arm cannot do.
Where the region comes from
A category prior names the handle for handled tools and containers, and it is the region source in every reported run. The semantic proposer's intent can name any region. Objects without a named region score every candidate neutrally.
Where the points come from
grasp_region:<part> scores 1.0 within 3 cm of the part's points, falling to 0 at
8 cm. The reported runs take the part points from a geometric partition of the object cloud; the
SAM3 path takes them from text-prompted masks lifted through depth.
Compiled from the next action
The declared next action (place, insert, pour, stack, hand over) compiles into geometric constraints over the candidate set: which part must stay clear, which face needs clearance, which approach keeps the view.
What it adds
Adding the task term on top of feasibility adds +0.07 [+0.02, +0.12] success on LIBERO-PRO Spatial Pos, with the lower bound of the paired interval above zero.
grasp_regionkeep_part_clearclearanceapproach_fromavoid_occluding
Measured as a factorial
Every level L1 to L4 runs the same rank terms under gate ∈ {none, feasibility} and combination ∈ {weighted sum, product}, so the gate and the combination rule are measured separately rather than as one package.
What decides on the Franka
On LIBERO-PRO Spatial Pos, gating a weighted sum adds +0.68 [+0.59, +0.77] at L1, while the gate changes nothing once the product rule is fixed: a product already drives an infeasible candidate to zero.
How We Measure
Every method ranks the same set
Candidate sets are content-hashed and shared across methods, so a difference in success is a difference in which interaction was selected.
Every candidate is executed
From the identical start state, in the benchmark's own simulator and success check. No method sees these outcomes; sampled re-executions check determinism.
Differences on the same scenes
Every difference between methods is a scene-clustered paired bootstrap interval (10,000 resamples), and weights and thresholds are fixed in the experiment config before the run.
Runs are immutable directories with resolved configs, code hashes, package versions and a file manifest, and suites launch from frozen source snapshots. Rollout videos are recorded while the selected candidates execute, and each is checked against the method's selection. In the current LIBERO-PRO runs, none of the 1,288 recorded rollouts disagrees with its unrecorded simulator outcome.
Replay a Real Selection
The hammer scene from the figure, with its 16 logged candidates, their terms and their simulator outcomes. Change the rule and see which candidate is executed and whether it succeeds.
Part source
Combination rule
Terms
Bars are each candidate's final score, normalised to the selected one; gated candidates score zero.
What this is. Logged data, not a worked example. Terms and outcomes are read from the SAM3-grounded 11-task run; each selection was computed with the engine's decision specs and checked against the selections that run recorded. Stability and simulator outcomes never enter the gate.
What the Selection Looks Like
Candidate scores, the selected interaction and its execution, taken from rollout videos recorded during evaluation. Click to enlarge.
Browse Successful Rollouts
One recorded AFFORD-X rollout per LIBERO-PRO cell and per MT50 difficulty tier: the first scene, in sorted order, in which the full decision layer's selection succeeded. Pick a clip, or use the arrow keys.
This rollout could not be played in your browser.
LIBERO-PRO Goal, Spatial and Object, Pos and Task
A 7-DoF Franka, 10 benchmark initial states per task and every method on the same candidates. Candidate geometry and the goal come from simulator state, so these runs measure the selection layer in isolation. Rates with 95% Wilson intervals; hover a bar.
| Method | Goal Pos | Goal Task | Spatial Pos | Spatial Task | Object Pos | Object Task |
|---|
Within 0.08 of the best candidate
Stability ranking picks the point nearest the centre of mass, which on these bowls is rejected as infeasible or fails in execution, and it succeeds in no Spatial scene. The full layer reaches 0.92 under Pos and 0.96 under Task, against 0.61 and 0.71 for random choice among the same candidates.
0.57 under Pos, 0.38 under Task
The full decision layer more than doubles random choice under Pos (0.25) and adds 0.15 over it under Task (0.23), on identical candidate sets.
1.00 under Pos and Task
The full layer succeeds in all 100 scenes of each cell, against 0.48 and 0.44 for random choice among the same candidates.
The Feasibility Gate in 3D
On LIBERO-PRO the gate keeps a grasp when reach × approach × collision is at least 0.5, and reach drops to 0.1 outside the Franka workspace, a box fixed to the robot base (x −0.10 to 0.75 m, y −0.50 to 0.50 m, z 0.005 to 0.90 m). Each cell shows one scene, chosen before looking at outcomes as the first in task and initial-state order where the gate removes some grasps and keeps others.
In the five scenes the gate removes 2 to 7 of 12 grasps, and 27 of the 28 removals lie outside the workspace; the one removed inside it, in LIBERO-Spatial Pos, has a collision term of 0.497. The gate removes some grasps in 41 of 80 Goal Pos, 99 of 100 Spatial Pos, 80 of 80 Spatial Task, 30 of 100 Object Pos and 10 of 100 Object Task scenes. Open the viewer on its own page.
Rollouts in 4D
One recorded AFFORD-X rollout per benchmark, replayed from the same reset with a depth camera. SAM3 tracks the object through every frame, each mask is lifted through that frame's depth, and the gripper and arm come from the simulator's geometry map. This is a visualization: no reported result reads these frames.
Meta-World MT50, 50 Tasks
The official MT50 tasks and variations with seed 0, five episodes per task, a 4-DoF Sawyer and no perturbations. Rates over 250 episodes with 95% Wilson intervals.
Handle Points in 3D
The functional term scores a grasp by its distance to the nearest handle point: 1 within 3 cm, falling to 0 at 8 cm. Every reported result takes those points from a geometric partition of the simulator's object cloud; the earlier 11-task Meta-World run took them from SAM3 masks lifted through the rendered depth. Among its eleven objects only the hammer and the wrench have a handle region. Pick one, orbit the scene, switch the source and read each candidate's distance and score.
On the three hammer episodes the geometric partition (54 points) gives no head grasp a functional score of 0.5 or more, and L3 selects a handle grasp in all nine scenes with either source. Both functional scores are recomputed from the points and equal the run log. Open the viewer on its own page.
BibTeX
@misc{sarowar2027affordx,
title = {AFFORD-X: Affordance-Grounded Agentic Policy for Zero-Shot Robot Manipulation},
author = {Md Selim Sarowar and Sungho Kim},
year = {2027},
note = {In preparation},
url = {https://physical-agi.github.io/AFFORD-X/}
}Placeholder entry, to be replaced by the proceedings entry once the paper is accepted.