refactor(robot): unify the robot stack - one sim process shape, batched-first bridge/agent, grouped evals

docs/v6/advanced/robots.mdx

@@ -91,22 +91,46 @@ The agent wires observations to policy inputs purely from the manifest; there is
 
 ## Environment side
 
-You implement one class - the **bridge**. 
+For a gym-style sim you implement nothing: `env.gym(make_env)` takes any callable returning a
+gym-style env, derives the [contract](#contract) from a sample observation, serves the sim over the
+``robot`` WebSocket, and returns the handle templates drive episodes through:
+
+```python env.py
+from hud import Environment
+
+env = Environment(name="my-sim")
+sim = env.gym(make_env)   # make_env: any callable returning a gym-style env
+
+@env.template()
+async def pick_and_place(task: str = "default", seed: int = 0):
+    ep = await sim.reset(task=task, seed=seed)  # {prompt, token}
+    yield {"prompt": ep["prompt"]}
+    yield await sim.result()
+```
+
+Task args are partitioned by the factory's signature: args the factory accepts define the env (a
+change rebuilds it), everything else is episodic and flows to `env.reset(seed=..., options=...)`.
+
+A single-env sim has one slot, so the template may omit the slot token as above. A vectorized sim
+(`num_envs > 1`) must thread it through so each session grades its own slot — yield
+`{"prompt": ep["prompt"], "robot": {"token": ep["token"]}}` and call `sim.result(token=ep["token"])`;
+see [vectorized evals](#vectorized-envs-and-evals).
+For a sim that isn't gym-shaped, subclass the **bridge** instead:
 
 ```python
 from hud.environment.robot import RobotBridge
 
 class MySimBridge(RobotBridge):
     async def reset(self, task_id: str, seed: int = 0) -> str:
         ...                              # build the episode
-        await self._send_observation()   # push the first frame
         return self.task_description     # becomes the task prompt
 
     def step(self, action) -> None:
         ...  # advance one tick; set success / terminated
 
     def get_observation(self):
-        return {"agentview_image": frame, "state": vec}, self.terminated
+        # Always [N, ...] arrays + [N] terminated (N=1 is fine).
+        return {"agentview_image": frames, "state": vecs}, done_mask
 ```
 
 
@@ -181,10 +205,14 @@ async def _down():
 
 @env.template()
 async def pick_and_place(task_id: str, seed: int = 0):
-    prompt = yield {"prompt": await endpoint.reset(task_id=task_id, seed=seed)}
-    yield await endpoint.result()  # {"score", "success", "total_reward"}
+    ep = await endpoint.reset(task_id=task_id, seed=seed)  # {prompt, token}
+    yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}}
+    yield await endpoint.result(token=ep["token"])  # this slot's {score, success, total_reward}
 ```
 
+Each session grades one slot. A vectorized sim fans N concurrent sessions across its slots; see
+[vectorized evals](#vectorized-envs-and-evals).
+
 ## Agent side
 
 The harness lives in `hud.agents.robot`. 
@@ -208,7 +236,7 @@ is a complete agent in a few lines. The two bundled seams *are* the LeRobot conv
   pre/post-processors, so the checkpoint behaves exactly as it does upstream. Pass an `Ensembler` to
   reduce overlapping action chunks to one action per step.
 - **`LeRobotAdapter(model_image_keys=...)`** maps the env's cameras and state onto the policy's
-  inputs from the [contract](#the-contract) - HWC `uint8` → CHW float, state and prompt passed
+  inputs from the [contract](#contract) - HWC `uint8` → CHW float, state and prompt passed
   through.
 
 ```python
@@ -275,22 +303,64 @@ into a single stacked forward.
 ```python
 from hud.agents.robot.batching import BatchedAgent
 
-agent = BatchedAgent(MyRobotAgent(), batch_size=8)   # set batch_size to your max_concurrent
+agent = BatchedAgent(MyRobotAgent())   # sizes itself to the live concurrency
 
 job = await taskset.run(agent, runtime=DockerRuntime("hud-libero-env"), max_concurrent=8)
 ```
 
-Each tick, up to `batch_size` concurrent observations are stacked into one `[N, ...]` batch, run
-through a single forward, and scattered back to each rollout; a short `max_wait_s` (default 50 ms)
-flushes early when fewer rollouts are live, so the tail of a suite never stalls. Set `batch_size` to
-match `max_concurrent` so a full batch can form.
+Each tick, the concurrent observations queued within a short window (`max_wait_s`, default 50 ms)
+are stacked into one `[N, ...]` batch, run through a single forward, and scattered back to each
+rollout - so the batch is as wide as whatever is in flight, and the tail of a suite never stalls.
+`max_concurrent` already bounds the width; pass `batch_size=` only to cap the forward below it
+(e.g. VRAM headroom).
 
 `BatchedAgent` needs an **in-process, stateless** model whose `infer` runs the whole `[N, ...]` batch
 in one forward - `LeRobotModel` qualifies. A remote OpenPI policy server is not supported (its protocol
 has no batched-request shape); run one agent per rollout against it instead. `BatchedAgent` also takes
 ownership of the agent you pass (it swaps in the batched model in place), so give it a dedicated
 instance rather than one you also use for unbatched runs.
 
+## Vectorized envs and evals
+
+A GPU sim like Isaac amortizes its cost by stepping many env copies together in one process
+(`num_envs`). Getting N graded rollouts by booting N separate containers throws that away - each
+one pays for its own physics from scratch. Instead, N rollouts share one vectorized instance, and
+every layer keeps the plain single-env contract described above:
+
+- **Vectorization is sim config, not an eval parameter.** A factory that accepts `num_envs` makes
+  `env.gym(make_env, num_envs=8)` build an eight-way batched sim. Internally the bridge steps
+  every claimed slot together each tick, but each rollout still gets its own ordinary WebSocket -
+  one observation in, one action out, no batch dimension on the wire.
+- **The agent is unchanged.** A stock `RobotAgent` drives one connection per rollout, same as a
+  single-env run. It claims its slot with the token the environment handed back on `tasks.start`
+  (`run.started["robot"]["token"]`); concurrent rollouts still coalesce GPU forwards through
+  [`BatchedModel`](#batching).
+- **`group` picks how many rollouts; `Shared` puts them on one substrate.** Left alone, `group`
+  rollouts would each provision a fresh container. Wrapping the provider in
+  [`Shared(provider, width=N)`](/v6/reference/runtime#shared) instead provisions one and lets
+  all `N` reuse it:
+
+```python
+from hud.eval import Shared, DockerRuntime
+
+job = await taskset.run(
+    MyAgent(),
+    runtime=Shared(DockerRuntime("hud-isaac-env"), width=8),
+    group=8,
+    max_concurrent=8,
+)
+```
+
+Each rollout opens its own control-channel connection and claims a slot on it: the first
+`tasks.start` resets the sim and claims slot 0, later starts claim whatever is free, and a start
+once every slot is taken errors instead of queuing silently. Grading is the same call every
+rollout already makes - the environment resolves it to that rollout's slot and returns one
+`Grade`, exactly as a single-env run would.
+
+`group` and `max_concurrent` are the ordinary [`Taskset.run` parameters](/v6/reference/tasks#running);
+set `max_concurrent` to `width` so every slot fills and none sit idle waiting for a rollout that
+never starts.
+
 ## Contract
 
 Embodiments and policies disagree on cameras, state layout, action semantics, and control rate, so
@@ -361,21 +431,18 @@ spec - the closed symbol sets and known traps - lives outside the SDK alongside
 
 </Accordion>
 
-## Sim threading
+## How sims run
 
 The loop is lockstep - the bridge steps the sim once per received action. A simulator is usually
-**thread-affine** (every touch must run on the thread that created its GL/device context), but the
-bridge's asyncio loop can't be stalled by a blocking step. **`SimRunner`** is the one-line injection
-that decides *which thread* runs the sim; the bridge routes every sim touch through it:
-
-- **`InlineSimRunner`** - runs on the event-loop thread. The default; for cheap/CPU sims and tests.
-- **`ThreadSimRunner`** - sim on a dedicated worker thread, leaving the loop free during a blocking
-  step. For render-heavy or thread-bound sims.
-- **`MainThreadSimRunner`** - sim on the main thread, for runtimes that own *both* the main thread
-  and the loop (Isaac/Omniverse); the owner's pump loop drains queued sim touches between ticks.
+**thread-affine** (every touch must run on the thread that created its GL/device context), and some
+runtimes - Isaac/Omniverse - must own the process main thread outright. HUD therefore serves every
+sim with one process shape: **the sim owns the main thread**, serving (the control channel and the
+robot WebSocket) runs on a background loop thread, and the framework queues every sim touch back to
+the main thread. A cheap CPU sim blocks on that queue; when Omniverse is loaded, the main thread
+also pumps Kit between touches.
 
-Pass one to the bridge (`RobotBridge(sim_runner=ThreadSimRunner())`), or subclass `SimRunner` for an
-exotic topology.
+There is nothing to configure. `hud serve` and `python -m hud.environment.server` detect an
+attached sim and pick the shape; any fix to the loop applies to every sim the same way.
 
 ## Recording datasets
 
@@ -388,35 +455,35 @@ it already produces, so it runs in *your* process - not the environment containe
 sims (e.g. Isaac/RoboLab) whose dependency stack conflicts with `lerobot`; only your machine needs
 `pip install 'lerobot[dataset]'`.
 
-One dataset spans the whole run - every episode the shared agent drives appends to it - and is
-finalized at process exit. Destination and Hub push come from the environment:
+All rollouts in a process record into one shared dataset: each buffers its episode and commits
+it whole, so concurrent runs (e.g. `BatchedAgent`) interleave safely. Finalized at process exit.
+Destination and Hub push come from the environment:
 
 | Env var | Effect |
 |---------|--------|
 | `RECORD_DIR` | Dataset root (default `./data`, relative to where the rollout launched) |
 | `HF_REPO` | Also push the finalized dataset to this HF namespace (needs `HF_TOKEN`) |
 | `HF_PRIVATE` | Push the dataset private |
 
-The [contract](#the-contract) drives the schema with no extra wiring: image features become
+The [contract](#contract) drives the schema with no extra wiring: image features become
 `observation.images.<camera>` (encoded to per-episode video), the lone state vector becomes
 `observation.state`, the action becomes `action`, and the task prompt rides along as each frame's
 `task`.
 
 
 ## Running a sim in another process
 
-Some simulators must **own the process main thread** - most notably **Isaac Sim / Omniverse**, where
-Kit drives its own main-thread event loop and `env.reset()` loads USD through a nested
-`run_until_complete`. That can't run inside `hud serve`, which already owns the asyncio loop. The fix
-is to move the sim into its own process and keep the env code essentially unchanged.
+Serving in one process is the default (see [how sims run](#how-sims-run)), but the sim can also
+live in its own process - on another machine, or built against a dependency stack the env
+container can't carry. The env code stays essentially unchanged.
 
 `RobotEndpoint` is built for exactly this: the same control surface (`start` / `reset` / `result` /
 `stop`) works whether the bridge is local or remote.
 
 - **Env process** - publish a *remote* handle with `RobotEndpoint.remote(host, port)`. It dials the
   sim process and forwards every control call over JSON-RPC.
-- **Sim process** - wrap the real bridge and expose it with `RobotEndpoint(bridge).serve(host, port)`,
-  using a [`MainThreadSimRunner`](#sim-threading) so every sim touch runs on the main thread.
+- **Sim process** - wrap the real bridge and expose it with
+  `RobotEndpoint(bridge).serve_blocking(host, port)`, the same sim-owns-main shape `hud serve` uses.
 
 The two planes split cleanly, which is why the agent never knows the sim is remote:
 
@@ -447,23 +514,19 @@ async def _down():
 
 @env.template()
 async def pick_and_place(task_id: str, seed: int = 0):
-    prompt = yield {"prompt": await endpoint.reset(task_id=task_id, seed=seed)}
-    yield await endpoint.result()
+    ep = await endpoint.reset(task_id=task_id, seed=seed)
+    yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}}
+    yield await endpoint.result(token=ep["token"])
 ```
 
-**Sim process** - your Isaac program builds the bridge and serves its control surface, then runs for
-the process's lifetime:
+**Sim process** - your program builds the bridge and serves its control surface, blocking for the
+process's lifetime with the sim owning the main thread:
 
 ```python sim_main.py
-import asyncio
-from hud.environment.robot import RobotEndpoint, MainThreadSimRunner
-
-async def main():
-    bridge = MySimBridge(sim_runner=MainThreadSimRunner())   # sim touches run on main
-    server = await RobotEndpoint(bridge).serve("127.0.0.1", 9100)
-    await server.wait_closed()
+from hud.environment.robot import RobotEndpoint
 
-asyncio.run(main())   # launched on the main thread the sim owns
+bridge = MySimBridge()
+RobotEndpoint(bridge).serve_blocking("127.0.0.1", 9100)
 ```
 
 Bring the two up together - the env's `connect()` retries until the sim is listening. Everything
@@ -474,15 +537,17 @@ downstream (`hud eval`, tasksets, the agent) is unchanged; only *where the bridg
 
 | Symbol | Where | Role |
 |--------|-------|------|
-| `RobotEndpoint.capability(contract=...)` | `hud.environment.robot` | Build the `openpi/0` capability after `start()` |
+| `env.gym(make_env)` / `Gym` | `hud.environment.robot` | Declarative sim: contract, capability, and serving derived from a gym factory |
+| `wrap(env)` | `hud.environment.robot` | One-line trace streaming for any gym-style env you drive yourself |
+| `RobotBridge` / `GymBridge` | `hud.environment.robot` | Env-side serve loop (slot barrier + scalar wire); subclass `RobotBridge` for a custom sim |
+| `RobotEndpoint` | `hud.environment.robot` | Episode bookkeeping + results (`reset` → `{prompt, token}`, `result(token=...)`) |
+| `RobotEndpoint.capability(...)` | `hud.environment.robot` | Build the `openpi/0` capability after `start()` |
 | `Capability.robot(name, url, contract)` | `hud.capabilities` | Lower-level constructor (usually via `endpoint.capability`) |
-| `RobotClient` | `hud.capabilities.robot` | Agent-side wire client (`spaces`, `get_observation`, `send_action`, `send_chunk`) |
-| `RobotBridge` | `hud.environment.robot` | Env-side serve loop; subclass with your sim |
-| `RobotEndpoint` | `hud.environment.robot` | Episode bookkeeping + results (local or `.remote()`) |
-| `SimRunner` (`Inline`/`Thread`/`MainThread`) | `hud.environment.robot` | Which thread runs the sim |
-| `RobotAgent` | `hud.agents.robot` | The episode-loop harness |
+| `RobotClient` | `hud.capabilities.robot` | Agent-side wire client (`connect(..., token=)`, `spaces`, `get_observation`, `send_action`) |
+| `RobotAgent` | `hud.agents.robot` | The harness: one open-loop chunk queue per scalar connection |
 | `Model` / `LeRobotModel`, `Adapter` / `LeRobotAdapter` | `hud.agents.robot` | Policy + space-translation seams |
 | `BatchedAgent` / `BatchedModel` | `hud.agents.robot.batching` | Many concurrent rollouts against one shared, batched model |
+| `Shared(provider, width=N)` | `hud.eval` | Refcounting provider: N concurrent rollouts share one substrate |
 
 ## See also
 

docs/v6/cookbooks/robot-benchmark.mdx

@@ -37,10 +37,10 @@ async def _down():
 
 @env.template(id="libero_spatial")
 async def libero_spatial(libero_task_id: int, init_state_id: int = 0):
-    prompt = await endpoint.reset(task_suite="libero_spatial",
-                                  task_id=libero_task_id, init_state_id=init_state_id)
-    yield {"prompt": prompt}
-    yield await endpoint.result()
+    ep = await endpoint.reset(task_suite="libero_spatial",
+                              task_id=libero_task_id, init_state_id=init_state_id)
+    yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}}
+    yield await endpoint.result(token=ep["token"])
 ```
 
 The image's CMD serves it with the standard entry point (`hud serve env.py --host 0.0.0.0 --port 8765`). This env lives in HUD's `demos/` examples tree, a sibling of the `hud-python` SDK; build it from the parent directory that holds **both** `demos/` and `hud-python/` so the image can install the SDK from local source:
@@ -67,8 +67,6 @@ from hud.eval import DockerRuntime, Task, Taskset
 CHECKPOINT = "lerobot/pi05_libero_finetuned"
 
 class PI05Agent(RobotAgent):
-    max_steps = 400
-
     def __init__(self):
         device = "cuda" if torch.cuda.is_available() else "cpu"
         policy = PI05Policy.from_pretrained(CHECKPOINT).to(device).eval()

docs/v6/reference/runtime.mdx

@@ -24,6 +24,7 @@ await taskset.run(agent, runtime=LocalRuntime("env.py"))   # serve env.py locall
 | `ModalRuntime("my-env")` | A fresh [Modal](https://modal.com/) sandbox per rollout | Cloud scale, no infra to manage |
 | `DaytonaRuntime("my-env")` | A fresh [Daytona](https://www.daytona.io/) sandbox per rollout | Cloud scale on Daytona |
 | `Runtime("tcp://host:8765")` | A substrate you already started | Attaching to a long-lived container or sandbox you own |
+| `Shared(provider, width=N)` | One substrate from `provider`, shared by up to N concurrent connections | Multi-agent evals - agents that share, compete, or coordinate over one live environment |
 | `HUDRuntime()` | A HUD-hosted env, leased by name and tunneled | Local agent loop against a deployed env |
 | `HostedRuntime()` | The whole rollout on a HUD-leased box | Agent and env run together off your machine |
 
@@ -182,6 +183,23 @@ Runtime(url, params=..., config=...)
 - **`url`** - control-channel address of an already-running substrate (e.g. `tcp://host:8765`).
 - **`params`** - connection-time data a transport may need (auth token, sandbox id).
 
+### `Shared`
+
+```python
+Shared(provider, *, width)
+```
+
+- **`provider`** - the `Provider` to provision once, e.g. `DockerRuntime("my-env")`.
+- **`width`** - how many concurrent connections that one substrate accepts.
+
+The first rollout to acquire it provisions through `provider`; every later rollout is just handed
+that same address instead of provisioning its own, so up to `width` rollouts end up connected to
+one live substrate - agents that need to share, compete, or coordinate in the same environment
+(a vectorized sim's `num_envs` slots, see [robots](/v6/advanced/robots#vectorized-envs-and-evals),
+are one case). Pair it with `taskset.run(..., group=width, max_concurrent=width)` so exactly
+`width` rollouts connect at once. `Runtime(url)` needs no wrapping since every caller already
+dials the same address.
+
 ## Run on your own infra
 
 A **runtime is just a function**: given a task, start a container somewhere and yield its

hud/__init__.py

@@ -60,6 +60,7 @@
     "instrument",
 ]
 
+
 try:
     from .version import __version__
 except ImportError: