refactor(robot): unify the robot stack - one sim process shape, batched-first bridge/agent, grouped evals

[lukass16]

Issue

The robot framework maintained two parallel implementations - single-env and vectorized - with duplicated agent loops, recorders, bridges, and eval paths. Sim execution had three separate threading mechanisms, and the vectorized eval path leaked robot-specific arguments (contract=, vectorized=) into the generic Taskset.run API.

Solution

• SimThread replaces the three SimRunner strategies: the sim always owns the main thread, serving runs on a background thread. The same mechanism covers CPU envs and Isaac.
• RobotBridge is now batched-first; VecRobotBridge and IsaacBridge are removed. result() returns per-slot scores under "slots".
• Recording moves to a shared hud.telemetry.robot module, replacing the three previous recorder classes. LeRobot dataset writing is split into agents/robot/dataset.py.
• RobotAgent drives any number of env slots with a single loop; VecRobotAgent is removed.
• Taskset.run(agent, runtime=..., num_envs=N) runs vectorized evals through a generic rollout_group; the robot-specific kwargs are removed and hud/eval no longer imports robot code.
• New surfaces: env.gym(make_env) serves a gym-style sim declaratively (contract and capability derived from the factory), and hud.wrap(env) streams traces from a loop the user owns.

Outcome / Verification

Roughly 1,500 lines of duplicated code removed; the diff against main is net negative despite the new features. Test suite and lint pass. Verified end-to-end by serving a vectorized stub env in a child process and running Taskset.run(num_envs=3) to three graded traces. Downstream consumers (assembly_bench, bench, rl) updated.

---

Note

High Risk
Large protocol and lifecycle changes (slot tokens, sim subprocess shape, multi-session grading) across robot eval, bridges, and runtimes; misconfiguration can deadlock agents or grade wrong sessions.

Overview
Unifies the robot stack so sims always run in a dedicated process (main thread owns physics; WebSocket + JSON-RPC control on a background loop), replacing configurable SimRunner strategies and in-process RobotEndpoint.serve.

Env ↔ agent contract changes: endpoint.reset() returns {prompt, token}; templates yield robot.token; agents connect with RobotClient.connect(..., token=) and fail fast without it. The bridge steps vectorized sims in lockstep with per-slot scalar wire obs/actions, barrier stepping, and result(token=) grading.

New integration paths: Environment.gym(make_env) spawns GymBridge via RobotEndpoint.spawn; wrap/TracedEnv streams traces for user-owned loops. Capability contracts are fetched from the bridge (endpoint.capability() without passing contract in).

Eval & control plane: Shared(provider, width=N) refcount-shares one substrate for concurrent rollouts; Taskset.run enforces max_concurrent >= width for Shared. Control channel tracks per-session suspended tasks with parking, blind tasks.grade when one session is parked, and hello(session_id=) resume.

Agent side: RobotAgent loop inlined with adapt_chunk/adapt_action; recording splits into TraceRecorder (telemetry) and opt-in DatasetWriter (shared process dataset); BatchedModel defaults batch width to live concurrency. hud serve uses blocking serve_blocking.

^{Reviewed by Cursor Bugbot for commit 5ae0641. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hud	🟢 Ready	View Preview	Jul 6, 2026, 9:29 PM

[cursor]

Class max_steps no longer honored

High Severity

RobotAgent.__call__ always uses the max_steps=520 parameter default and no longer reads a subclass max_steps attribute. Taskset.run / rollout call agent(run) with no kwargs, so agents that previously set max_steps on the class silently run with 520 instead.

 

^{Reviewed by Cursor Bugbot for commit d1d7e23. Configure here.}

[cursor]

Recorder cleanup skipped on errors

Medium Severity

TraceRecorder.close() and DatasetWriter.end_episode() run only on the success path after _loop. If the rollout raises, video tails are never flushed and a buffered LeRobot episode is never committed, even though robot.close() still runs in finally.

 

^{Reviewed by Cursor Bugbot for commit 9884ccc. Configure here.}

[cursor]

Gym kwargs lose native types

Medium Severity

gym_command stringifies every build default onto the argv, and main reloads unknown --key value pairs as plain strings. Boolean defaults such as False become the truthy string "False", and ints/floats are no longer numeric when the factory runs in the child process.

 

^{Reviewed by Cursor Bugbot for commit 9884ccc. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 5 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5ae0641. Configure here.}

[cursor]

Disconnect stalls vectorized barrier

High Severity

When a claimed slot's WebSocket drops, the handler clears ws and action but never marks the slot idle. The tick barrier still treats that slot as pending, so remaining live rollouts wait up to step_timeout (30s) before holding. In vectorized evals, any early-finished or disconnecting slot stalls every later lockstep step.

Additional Locations (1)

• hud/environment/robot/bridge.py#L320-L329

 

^{Reviewed by Cursor Bugbot for commit 5ae0641. Configure here.}

[cursor]

Terminal observation never recorded

Medium Severity

The loop breaks on terminated before record_observation when step > 0. The post-action terminal frame is fetched at the end of the previous iteration and then dropped, so traces and datasets miss the final observation the old harness recorded before stopping.

 

^{Reviewed by Cursor Bugbot for commit 5ae0641. Configure here.}