A Capability-Factored Dataset and Benchmark for Robot Learning
PRAXIS: Physical Reasoning, Adaptation, and eXecution In Situ. A standalone benchmark for General Physical Intelligence that measures what an embodied agent can do, not how often it happens to finish. Counterfactually paired episodes, mid-episode physical interventions, a first-class world-model track, and a capability vector aggregated so that imbalance is punished.
A success rate is a confounded scalar
Current physical AI benchmarks report a single success rate on fixed, memorizable episodes: a scalar that confounds perception, grounding, planning and control, and that collapses under trivial perturbation. It cannot say whether an agent failed because it could not see the object, could not parse the instruction, could not plan the sequence, or could not close the gripper, and it cannot say anything at all about the world model inside a model-based agent.
PRAXIS evaluates differently. Every evaluation seed generates a base episode and variants that differ by exactly one controlled factor, so capabilities are measured as paired deltas the way a controlled experiment measures cause. The simulator injects scripted events mid-episode: objects are displaced, actuation drops out, goal regions move and are re-announced in language. An open-loop trajectory replayer passes the base episode and fails its intervened twin, and that difference is the robustness gap. A first-class world-model track scores counterfactual state prediction and plan ranking directly, the questions a planner actually poses, with the property that copy-last-frame shortcuts provably score zero on ranking.
The result is a capability vector rather than a scalar: eleven axes, each with a Wilson interval, aggregated by geometric mean so that an agent cannot buy a headline number by being excellent at control and blind everywhere else. Unseen-object generalization is an enforced invariant rather than a curation claim, because the procedural object pack samples shape, texture, size, mass, friction and colour from train and eval ranges that are disjoint per attribute, and zero asset files ship. Across 20 families and 1,200 paired episodes per agent, every non-privileged baseline we measured scores exactly zero on spatial reasoning and on planning while reaching 0.69 to 0.83 on control. That gap, invisible to a success rate, is the challenge the instrument exists to surface.
Four commitments
The survey of fifteen existing benchmarks reduces to one sentence: they report a confounded scalar on memorizable instances with no perturbations, no statistics, no memory demands, and no way to evaluate the world models inside modern agents. These are the four answers.
The agent completes the task. Success: yes.
is the measurement
Same seed, one factor changed. An open-loop replayer misses.
Scoring the pair rather than the episode cancels instance difficulty, so a family that happens to draw an easy layout cannot inflate an axis. Recovery is scored on the intervened twins only, so it cannot be earned by being good at the undisturbed task.
What a capability vector looks like
Measured on the public full_v1 suite: 20 families, 50 paired seeds each, 1,200 episodes per
agent. Toggle agents to overlay their shapes. The privileged oracle is drawn dashed and is never ranked
against learned policies.
Read the shape, not the area. vision and bc differ by 0.014 on the
headline score and are entirely different agents: vision reaches 0.620 on memory where
bc reaches 0.060, and bc is the only baseline with a non-zero planning axis at all.
A leaderboard that reported one number would call them equivalent.
Why the headline score is a product
The PRAXIS score is the geometric mean over the axes. Drag the planning axis toward zero and watch the two aggregates separate.
This is a synthetic profile chosen to make the arithmetic visible; the axis values are illustrative and the aggregation is the one the benchmark ships. Every axis also carries a Wilson interval, and ranking claims between two submissions require a paired permutation test, not a difference of point estimates.
Twenty families, five tiers
Three orthogonal coordinates: the capability axes a family probes, the tier of difficulty, and the base versus paired-counterfactual variant structure. Filter by either coordinate.
The public leaderboard
Click any column to sort. Rankings are decided on the hidden suite, not this one; a public score is a development signal, and the public minus hidden generalization gap is published alongside it.
| Agent | PRAXIS | Control | Spatial | Memory | Language | Planning | General. | Recovery | Agentic | Safety | Efficiency |
|---|
The amber row is the privileged oracle: it reads ground-truth state, exists to certify that every family is
solvable, and is never ranked against learned policies. Sort by Safety to see why supporting
axes are diagnostics rather than ranking axes: random scores 0.993 on it, because an agent that
never goes anywhere never enters a keep-out zone.
lerobot/smolvla_base), evaluated zero-shot through the plugin path, lands at 0.005,
indistinguishable from random. This is expected and we report it as such: the base checkpoint's
6-DoF real-robot action space and training domain do not match the planar action space and top-down rendered
scenes here, so its outputs carry no signal. The row is the reference untuned-VLA data point, and the
integration rather than the number is the deliverable. Any LeRobot or HuggingFace policy now plugs into the
full capability-vector evaluation without adding its dependencies to the benchmark.
World-model track
Scored on counterfactual state prediction and plan ranking, the two queries a planner actually poses.
| Model | WM score | Ranking accuracy | What it isolates |
|---|---|---|---|
| grasp_aware | 1.00 | 1.00 | Analytic ceiling: shows the probe is solvable |
| static | 0.49 | 0.00 | Effector kinematics only, no interaction model |
| no_motion | 0.42 | 0.00 | Copy-state floor |
Both shortcut baselines score a hard zero on ranking while retaining 0.42 to 0.49 on prediction. That separation is the point of the track: a model can look accurate frame-to-frame and still be useless for the only question a planner asks it, which is which of two candidate plans ends better.
A case the scalar gets backwards
Three execution regimes of one architecture, one corpus and one feature set, on 50 paired seeds of pick_place, displace_mid and frozen_effector. Retention is recovery divided by control.
The action-repeat row is a methodological control, kept in the table to document a pitfall. Freezing a per-step policy's action collapses control as much as recovery, because greedy deltas compound, so action-repeat is not a valid surrogate for chunking. A chunking ablation has to train the multi-step head.
Every family is certified solvable
The privileged oracle's solvability certificate per backend, on seeds 0 to 2, base and intervened variants, all 20 families, 72 episodes each.
| Backend | Certification | Notes |
|---|---|---|
| mock (reference) | 72 / 72 | Normative episode semantics, deterministic, CPU-only |
| mujoco (suction) | 72 / 72 | Weld grasping, real gravity, friction and settling |
| mujoco-gripper | 72 / 72 | Contact grasping by friction alone, V-cradle fingers |
| mujoco-franka | 70 / 72 · 0.97 | Franka Panda with sub-equator cradle grasp; clears the 0.95 aggregate gate |
Two backends, one semantics: a deterministic reference backend defines episode semantics and a MuJoCo 3 contact backend runs the same specs. Both share one rules module, so outcomes agree by construction rather than by promise. The residual Franka failure is a real finding, not a bug: the largest smooth held-out spheres, 38 mm diameter and up, slip during carry, and gripping them harder ejects them. That is a genuine parallel-jaw limit, and surfacing hardware-relative unseen-object limits is what the Franka embodiment is for.
What the instrument does not yet measure
No frontier VLA has been fine-tuned on it
The one real VLA evaluated so far, SmolVLA, was run zero-shot and landed at the floor for reasons of action-space mismatch rather than capability. A meaningful VLA row requires fine-tuning on a PRAXIS demo corpus converted to LeRobot format. The corpus and the exporter exist; the fine-tuned row does not yet.
Spatial and planning are zero everywhere
That is the intended finding, but it also means those two axes are currently unvalidated as discriminators: no baseline separates on them, so we cannot yet show that the axes rank agents sensibly rather than merely rejecting all of them. The first submission that moves either off zero is the result the suite was built to surface.
Simulation only, planar actions
The action space is planar (dx, dy, grasp), and there is no real-robot cell. The Franka
backend brings a real arm model and real finger contact, which is what surfaced the sphere-slip limit,
but nothing here measures sim-to-real transfer of either a policy or the capability profile itself.
Human demonstrations do not transfer yet
Behaviour cloning on the 7,750-demo human-teleop bridge corpus scores 0.000 on the procedural suite. The corpus trains, but hindsight-labelled human trajectories carry no goal coordinates and no object features, so the symbolic featuriser has nothing to condition on. Exploiting them requires pixel-space policies, which is the motivating next step rather than a defect we are hiding.
The hidden server is self-hostable by design
Rankings are decided on fresh instances drawn from a keyed PRF over the submission id and family, in a seed band disjoint from the public development set, and each result is a tamper-evident sealed bundle. The algorithm is public and only the operator's seed key is private, so nobody has to trust us to reproduce the protocol.
Anything plugs in
The agent contract is minimal and model-agnostic: a VLA, a diffusion policy, an RL agent, a classical planner or a scripted controller are evaluated identically, and external world models plug into the world-model track the same way. PRAXIS imports your factory and runs it, so you evaluate from inside your own environment without adding its dependencies to the benchmark.
praxis demo --episodes 5 prints oracle against random with full capability vectors and a
p-value. praxis eval --agent-module you:build_agent --config configs/eval/full_v1.yaml is all it
takes to score your own policy.
BibTeX
@misc{sarowar2027praxis,
title = {A Capability-Factored Dataset and Benchmark for Robot Learning},
author = {Md Selim Sarowar and Sungho Kim},
year = {2027},
note = {In preparation},
url = {https://physical-agi.github.io/PRAXIS/}
}
Placeholder entry, to be replaced with the proceedings entry once the paper is accepted. The software itself
carries a CITATION.cff in the repository.