AGI: Agentic Intelligence and Latent Lookahead for Robust Manipulation

Anonymous Authors
Under review · IEEE International Conference on Robotics and Automation (ICRA'27)
arXiv soon Paper soon Code soon Cite

Author names, affiliations and code links are withheld while the paper is under anonymous review.

Three paradigms for robot manipulation: a VLA model mapping images and language to actions, a pixel-predictive world model, and the proposed agentic model that proposes, imagines and verifies programs
Three paradigms for robot manipulation, drawn as a conceptual contrast rather than a measured comparison. (a) VLA models map images and language directly to actions; under the Position axis of LIBERO-PRO their success collapses in a pattern consistent with reuse of layout-conditioned trajectories. (b) Pixel-predictive world models forecast future frames, enabling lookahead at high decoding cost. (c) AGI also predicts the future, but in the latent space of a program-conditioned world model rather than in pixels, and ranks complete programs before any motor command; panel (c) therefore contains a world model of the (b) family with a different conditioning interface and no decoder. The capability marks summarize each family's design, not per-cell experiments, and “Low” compute refers to the local verification stack at 15.46 ms, excluding the 10.006 s proposer call.
0.00
LIBERO-PRO average success
0%
Meta-World average success
0.0
LIBERO-X average accuracy
0%
Real robot success (67/80)
0 ms
Local verification per cycle

Abstract

Vision-language-action policies that succeed in distribution collapse under small geometric perturbation: on the Position and Task axes of LIBERO-PRO Spatial, OpenVLA and π0 score 0.00 while retaining near-ceiling success on the axes that leave geometry intact. Direct action generation also exposes no commitment that can be inspected, compared, or rejected before the arm moves. We present AGI, an agent-as-policy decomposition in which a frozen vision-language agent proposes K programs in a nine-primitive, literal-only robot DSL, a Code-to-Token Bridge conditions a JEPA-style latent world model on each complete program, and a verifier ranks the predicted latent trajectories before any motor command; the selected program executes for κ primitives and the loop replans on fresh observations. AGI reaches 0.50/0.49 Position/Task success where both baselines score 0.00, 94.0% average success on Meta-World v3, 52.9% on LIBERO-X, and 83.8% (67/80) on real tabletop pick-and-place with a Piper (WEGO) arm. To separate proposal quality from selection quality, we evaluate on byte-identical candidate sets restored from Nfork = 800 simulator states with oracle and anti-oracle bounds: latent ranking raises Top-1 success from 0.57 to 0.63 over first-valid selection and captures η = 0.40 of the oracle headroom (95% CI 0.22–0.58; exact McNemar p < 0.001). The full local stack holds 277M parameters and verifies K = 3 candidates at H = 8 in 15.46 ms, while the frozen remote proposer dominates the 10.02 s synchronous decision path. Oracle coverage of 0.75 identifies candidate generation, not ranking, as the remaining constraint.

The Planning Loop

One planning round runs four stages. Pick a stage to see what it does and what it costs.

The full framework: observation and task instruction feed a frozen vision-language agent that returns K candidate DSL programs, which are embedded by the program encoder, rolled out by a V-JEPA encoder and program dynamics model, scored by a trajectory verifier, and executed by an AST-parsed operational-space controller
The complete verify-then-execute framework. A frozen vision-language agent (① Propose) returns an ordered list of K raw program candidates from one request under fixed decoding settings. Candidates are parsed first; valid programs are embedded by the Code-to-Token Bridge and rolled out in the latent space of a JEPA-style world model frozen during planning (② Imagine). The verifier (③ Verify) scores each imagined trajectory against the task embedding and selects the winner, executed through operational-space-control primitives (④ Execute). When a commit ends without success the agent re-observes and starts another planning round, at most N rounds within the 600-step budget.

Components

Four learned or symbolic pieces sit between language and motor commands.

What it is

Nine primitives, each taking literal arguments only. Cartesian positions are metres in the robot-base frame, orientations are normalized quaternions with qw ≥ 0, joint targets are radians, and wait takes an integer number of low-level steps.

Why it matters

The executor never queries a simulator object pose, so the program must carry the geometric target literals the tracker needs. Writing coordinates rather than replaying a trajectory is what keeps Position-axis success nonzero.

move_tomove_to_posegraspreleaseplacemove_jointsopen_gripperclose_gripperwait 
# one candidate program, parsed by AST before anything learned sees it
robot.open_gripper()
robot.move_to(-0.22, 0.02, 1.02)
robot.grasp('bowl')
robot.place(0.07, 0.16, 1.00)
robot.release()

The AST interpreter whitelists the nine primitives, accepts literals only, and never invokes exec(). Invalid programs are discarded without repair.

What it does

Tokenizes DSL source with BPE under a 256-token cap, applies a 4-layer, 8-head transformer with masked mean pooling, and projects to a 512-d program embedding that the world model can consume as conditioning.

How it is trained

Contrastively, on labeled program pairs: a BCE term on scaled cosine similarity plus a symmetric InfoNCE term at temperature 0.1. The loss groups literal variants of one behavior and separates distinct skills.

Stated limitation. The loss constrains program-to-program geometry, not alignment to the training-time action surrogate the dynamics head was fit on, so substituting the bridge at inference is empirical rather than invariant by construction. Adding paired alignment raises Pair-AUC from 0.70 to 0.78 in the interface ablation below.

Dual encoder

Online and target ViT-B/14 encoders share architecture; the stop-gradient target follows an EMA with α = 0.996. Twelve transformer blocks encode each 224×224 frame, temporal attention pools per-frame CLS tokens, and projected 14-d proprioception is added to the 768-d output.

Two heads

An action-dynamics head grounds the encoder in single-step consequences during pretraining. A program-conditioned head decodes H queries from the seed latent, with AdaLN-Zero conditioning made explicitly horizon-dependent so the modulation varies with the prediction step.

Every learned loss lives in the 768-dimensional latent space; the architecture contains no pixel decoder. Removing the horizon code worsens normalized latent MSE from 0.35/0.60 to 0.42/0.80 at horizons 3 and 8, and drops selector efficiency from 0.40 to 0.25.

Latent imagination rollouts in feature space across prediction steps
Latent imagination in feature space. The program-conditioned dynamics head decodes multi-step feature trajectories from the seed observation latent without pixel decoding, predicting physical consequences directly in representation space.

Two heads, one score

A 2-layer MLP trunk over the concatenation of each latent and the frozen task embedding, with a horizon head estimating success by step h and a terminal head estimating success at the end. The planning score is half the terminal head plus half the horizon mean.

Sparse supervision only

Both heads train on the sparse task-success predicate, with the horizon head weighted at 0.5. The verifier never reads program text, so different scores require the dynamics model to separate imagined outcomes.

Stated limitation. Training consumes stop-gradient target-encoder trajectories while planning scores imagined ones, leaving a measured encoded-to-imagined AUROC gap of 0.06 (0.78 versus 0.72).

Real-World Robot Experiments

We evaluate AGI on four real-world tabletop manipulation tasks using a 6-DoF AgileX Piper (WEGO) manipulator arm equipped with a parallel-jaw gripper, with local inference running on a single NVIDIA RTX 3080 GPU. Visual feedback is provided by a wrist-mounted camera and a static third-person camera overlooking the tabletop workspace. Across 80 consecutive rollouts (20 trials per task), AGI achieves an overall success rate of 83.8% (67/80).

Hardware Platform
AgileX Piper
6-DoF arm, parallel-jaw gripper
Sensing Stack
RGB Dual View
Wrist camera + third-person camera
Inference Hardware
NVIDIA RTX 3080
15.46 ms local verification cycle
Evaluation Protocol
83.8% (67/80)
20 consecutive trials per task

Task 1: Red Cube Pick-and-Place

70.0% (14/20)
Instruction: “Pick up the red cube and place it on the table”
Wilson 95% CI: [48.1%, 85.5%] Single-object tabletop placement

Task 2: White Cube Pick-and-Place

80.0% (16/20)
Instruction: “Pick up the white cube and place it on the table”
Wilson 95% CI: [58.4%, 91.9%] Single-object tabletop placement

Task 3: Pink Cube to Green Tray

90.0% (18/20)
Instruction: “Pick up the pink cube and place it into the green tray”
Wilson 95% CI: [69.9%, 97.2%] Container placement with height clearance

Task 4: Sequential Two-Cube Placement

95.0% (19/20)
Instruction: “Place one cube in a marked circle, then place the other cube in the other circle”
Wilson 95% CI: [76.4%, 99.1%] Sequential multi-stage placement
Real-robot tabletop manipulation with the Piper (WEGO) arm: setup, success rates with Wilson 95% CI, and keyframe sequences across four tasks
Real-robot tabletop manipulation with the Piper (WEGO) arm. Top: setup and success rates over 20 consecutive rollouts per task (80 total); whiskers denote Wilson 95% confidence intervals. Bottom: red- and white-cube placement on the table, pink-cube placement into the green tray, and sequential placement of two cubes in separate marked circles. Each row shows the initial state (t1), execution (t2–t5), and end state (t6). Overall success is 83.8% (67/80), with inference on an NVIDIA RTX 3080 GPU.

LIBERO-PRO

Normalized success under five perturbation axes. Pick a suite; click a column header to sort. Cell shading tracks the value, so the Position and Task columns read at a glance.

LIBERO-PRO normalized success across Goal, Spatial, LIBERO-10, and Object suites under five perturbation axes
LIBERO-PRO normalized success across Goal, Spatial, LIBERO-10, and Object suites. Columns report Object (Obj), Position (Pos), Semantic (Sem), Task, and Environment (Env) perturbations; red outlines identify the geometry-changing Position and Task axes. Gray cells are unreported, and the baseline mean uses the complete-case rows OpenVLA, π0, and π0.5.
Obj object swapped Pos object placement changed Sem instruction rephrased Task target object or action changed Env scene appearance changed
click any header to sort Avg. SR is each row's average over its own reported cells, across all four suites
Model Obj Pos Sem Task Env Avg. SR

Dashes mark cells the source did not report, and are never imputed. On the Spatial suite, OpenVLA and π0 score 0.00 on both Position and Task, while our loop reaches 0.50 and 0.49. The gain is a redistribution rather than uniform dominance: Spatial-Semantic success is 0.73 against 0.88–0.97 for the displayed baselines, and the Object-suite Object score is 0.76 against leaders at 0.98.

Rollout frames from initial state through execution to end state for the Spatial, Goal, Object and Long-horizon LIBERO suites
Representative successful Position-shift rollouts across the Spatial, Goal, Object and Long-horizon suites. Each sequence runs from the initial state through intermediate execution to a successful terminal state, qualitatively confirming that the controller adapts its explicit geometric program to changed object placement rather than executing one fixed scene-specific trajectory.

Meta-World and LIBERO-X

Hover a bar to read its exact value.

Mamba Policy HarmonyDream π0 Ours
95.4
81.0
71.8
98.3
Easy
92.2
91.0
48.2
95.7
Mid
48.3
72.5
41.7
91.3
Hard
71.2
44.5
30.0
90.6
Very Hard

Meta-World success rate (%) across difficulty splits, 20 episodes per task over 50 tasks. The three strongest displayed baselines are shown; the full table is below. Protocol differences make this panel contextual rather than a controlled cross-paper ranking.

Meta-World v3 success (%) across difficulty splits (Table I, left)
Method Venue Params EasyMidHardV. Hard Avg.
HarmonyDreamICML'2477M81.091.072.544.572.3
DPIJRR'2580M83.631.19.026.637.6
TinyVLAICRA'251.5B77.621.511.415.831.6
Mamba PolicyIROS'25–95.492.248.371.276.8
π0RSS'253.3B71.848.241.730.047.9
AGI Ours–277M98.395.791.390.694.0

Degradation under cumulative severity

LIBERO-X grades distribution shift across five cumulative severity levels. Our accuracy decreases monotonically from 73.6 at Level 1 to 33.8 at Level 5, remaining above every displayed baseline at each level.

π0.5 GR00T N1.5 π0 Ours
65.2
43.3
29.4
73.6
L1
53.2
32.9
21.9
62.9
L2
36.0
18.7
11.0
52.4
L3
24.1
13.3
7.6
41.7
L4
18.0
9.7
5.1
33.8
L5

LIBERO-X accuracy (%) under increasing cumulative perturbation severity. L1–L3 use 600 tasks, L4–L5 use 826 tasks. Averaged over levels: ours 52.9, π0.5 39.3, GR00T N1.5 23.6, π0 15.0, X-VLA 14.6, OpenVLA-OFT 13.2.

LIBERO-X accuracy (%) across cumulative severity levels (Table I, right)
Method Venue Params L1L2L3L4L5 Avg.
GR00T N1.5arXiv'253B43.332.918.713.39.723.6
OpenVLA-OFTRSS'257B29.017.68.86.44.213.2
π0.5CoRL'253.5B65.253.236.024.118.039.3
π0RSS'253.3B29.421.911.07.65.115.0
X-VLAICLR'261B30.122.610.36.04.114.6
AGI Ours–277M73.662.952.441.733.852.9
LIBERO-X robustness under cumulative perturbations: accuracy across severity levels and retained accuracy
LIBERO-X robustness under cumulative perturbations. (a) Accuracy across severity levels. (b) L5-to-L1 retained accuracy, which distinguishes absolute performance from degradation rate: AGI retains 46% of initial accuracy at L5, versus 28% for the nearest curve.

Rollouts on both benchmarks

Frames sampled evenly from successful episodes, initial state through execution to the terminal state the success checker accepted. The green outline marks the largest region of each frame that differs from the static scene. The camera never moves, so the median of an episode is the empty scene. It is computed from pixels to trace the arm, and is not an object detection.

Rollout frames from initial state through execution to end state for one Meta-World task from each of the Easy, Medium, Hard and Very Hard difficulty splits
Meta-World rollouts, one task per difficulty split. Easy presses a top-down button, Medium drives a hammer, Hard lifts a puck out of a hole, and Very Hard disassembles a peg. Drawn from our own evaluation runs rather than the cross-paper table above, whose protocol differs.
Rollout frames from initial state through execution to end state for four LIBERO-X kitchen tasks: placing, stacking, turning on a stove and opening a microwave
LIBERO-X rollouts, one task per action. The arm moves a bowl onto a plate, stacks one bowl on another, turns a stove on until the burner glows, and swings a microwave door open. Drawn from a Level 1 sweep on our own hardware, which uses a different checkpoint and task set from the severity table above.

Those four rows are a sample. Every LIBERO-X task that produced a successful episode is browsable frame by frame, with the program statistics recorded for each rollout.

Browse all 125 rollouts

What Each Part Is Worth

Switch the decision rule to separate the program interface from replanning and ranking. Every row received byte-identical ordered candidates, and candidate forks restored the same simulator and RNG state.

Latent ranker
Position-axis success0.50
Task-axis success0.49
Planning rounds3.1
Commit κ2
Env. steps410
Zero-valid1.8%

Task-aggregated point estimates on LIBERO-PRO Spatial. All K = 3 rows fix the commit length, round budget and step ceiling. At this sample size, differences below roughly 0.10 should be read cautiously; the paired test below is the powered evidence for the ranking claim.

Same-candidate selector diagnostic

Every valid candidate ran to completion from the same restored state, so the only thing that varies is which one the rule picks. M = 800 states, mean 2.88 valid candidates, prevalence 0.52.

Selection ablation on LIBERO-PRO Spatial: (a) Policy-level gains from the program boundary with replanning and from latent ranking. (b) Identical-candidate Top-1 success, showing captured selector headroom eta and the residual oracle-coverage gap.
Selection ablation on LIBERO-PRO Spatial. (a) Policy-level gains from the program boundary with replanning and from latent ranking. (b) Identical-candidate Top-1 success over 800 restored states, showing captured selector headroom η = 0.40 (95% CI 0.22–0.58) and the residual oracle-coverage gap of 0.25.
Oracle @ K 0.75
Latent ranker 0.63
Step head only 0.61
Terminal head only 0.60
First valid 0.57
Uniform random 0.55
Anti-oracle 0.40
+0.06
Top-1 gain of the ranker over first-valid selection
n10 = 112, n01 = 64, exact McNemar p < 0.001
η = 0.40
Selector efficiency: how much of the random-to-oracle gap the ranker closes
95% CI 0.22–0.58
0.25
Coverage gap left by the proposer: the oracle itself tops out at 0.75
the larger measured bottleneck

Top-1 success on identical candidate sets. Deterministic rules use paired intervals and McNemar tests against first-valid; exact random uses a paired bootstrap. The 95% interval spans the preregistered modest and principal-effect regions, so we claim a measurable ranking contribution and report its uncertainty rather than treating its magnitude as precisely established.

Component and interface diagnostics

Variant Tp/H Pair-AUC ↑ NMSE 3/8 ↓ AUROC enc. ↑ AUROC imag. ↑ AP/prev. ↑ η
Full system3/80.700.35/0.600.780.721.400.40
No horizon code3/8=0.42/0.80=0.641.210.25
Terminal head only3/8==0.750.681.260.25
Step head only3/8==0.760.701.330.30
Paired bridge alignment3/80.780.33/0.57=0.731.420.41
Matched horizon8/8=0.36/0.48=0.741.440.42
Mixed-verifier training3/8==0.760.751.460.42

“=” marks cells identical to the full system by construction. The three interface corrections are diagnostic improvements: end-to-end Position/Task success was not measured for those rows, so they are not demonstrated policy gains. The no-horizon-code row is a single retraining run, a descriptive architectural diagnostic rather than a seeded causal estimate.

Verification Is Not the Bottleneck

Measured at K = 3, H = 8 on one RTX PRO 6000 in FP16. GPU values are means over 200 cycles after 20 warmups; the proposer round trip is the mean of 60 requests.

Local verification stack: 15.46 ms · 0.154% of the decision path Remote proposer round trip: 10 006 ms · 99.846%
Measured planning cost at K = 3, H = 8
Stage Mean ms Rate (Hz) Share Params (M) Peak GB
Encoder Eθ5.95–0.059%93.40.54†
Bridge φ4.38–0.044%38.60.17†
Dprog (batched K = 3)4.28–0.043%65.90.40†
Verifier R0.85–0.008%< 1< 0.01†
Local verification stack15.4664.70.154%198.31.51‡
Proposer (VLM, round trip)10 006.000.10099.846%remoteremote
Decision path (synchronous)10 021.460.100100%–1.51‡
Tracker / env. step§0.52–excludedn/an/a

†Module-isolated peak; these entries are not additive. ‡Joint peak with all local weights resident and peak activations. §Per low-level environment step, excluded from decision-path arithmetic because commit duration varies. The cached instruction encoder, one-time parsing and preprocessing, and remote proposer parameters and memory are excluded. These are planning-throughput figures rather than motor-control rates.

Horizontal bar chart on a log scale comparing per-cycle latency: encoder 5.95 ms, bridge 4.38 ms, program dynamics 4.28 ms, verifier 0.85 ms, and the remote VLM agent at 10006 ms
Per-cycle latency on a log scale. The four local modules sum to 15.5 ms; the remote agent call is nearly three orders of magnitude larger. Foresight is cheap relative to proposal.

An Auditable Decision Record

The explicit program boundary leaves artifacts that reactive action generation does not expose directly.

Side-by-side comparison: a low-level VLA policy emits opaque joint velocities with no explainable trace, while the program-generating approach emits inspectable source code in which the failing coordinate literal can be read off directly
Decision auditability enabled by the explicit program interface. Illustrative diagnostic trace with K = 4 and four replan records, separate from the reported K = 3, N = 4 evaluation. A reactive VLA exposes only an action vector with no inspectable trace, whereas our system records ordered candidate programs, AST validity, imagined latent consequences, verifier scores, and the selected commitment before execution. In panel (b), every program parses, yet replans retain the same z = 0.952 grasp height and y = −0.45 pull target while varying only approach depth. The record localizes a shared geometric grounding error hidden in the action-vector view, showing that explicit literals need not re-ground correctly.

This is a concrete systems property of the interface, not a claim about human interpretability. No user study was run.

Evidence Boundaries

The 0.12 gain is not all ranking

The one-shot-to-full difference combines candidate count, replanning, imagination and ranking. Only the same-candidate fork isolates selection, and it yields +0.06.

Proposal is the larger bottleneck

The remaining oracle and coverage gaps are 0.12 and 0.25. Candidate generation, not ranking, is where the most headroom sits.

Grounding error is not isolated

The proposer writes the robot-base-frame literals and no learned converter modifies them. We do not report a literal-to-ground-truth coordinate-error distribution, so Position-axis claims are made at the interface level.

Robustness is a trade-off

Spatial-Semantic success stays at 0.73, below every displayed baseline. The evidence supports a redistribution of robustness, not uniform superiority.

Train and test interfaces differ

Training supervises three latent steps while planning queries eight, and the dynamics head is trained on an action surrogate but queried on the bridge. The substitution is empirical.

Correction rows are diagnostic only

The interface corrections and the no-horizon-code row lack matched end-to-end reruns or multi-seed intervals.

BibTeX

@inproceedings{anonymous2027agentic,
title     = {AGI: Agentic Intelligence and Latent Lookahead for Robust Manipulation},
author    = {Anonymous},
booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
year      = {2027},
note      = {Under review}
}

Placeholder entry. Author names are withheld while the paper is under anonymous review.