Skip to content

Hosted runs submitted via CLI lose task names on the dashboard (rollouts/submit vs rollouts/run_list) #488

Description

@MittelmanDaniel

When I run a taskset remotely from the CLI (hud eval my-taskset claude --full), the job page shows every trace as an anonymous UUID card ("Task 2aeeb924") — you can't tell which rollout belongs to which task without opening each one. The same batch launched from the website's run button comes back named and grouped ("notif-bug (1)").

CLI job:

CLI job: traces labeled by UUID

Same taskset via the website button:

Website run: traces named and grouped

I dug into why. The two entry points use different endpoints. The button sends one batch request that references tasks by version (discovered via the network tab in browser dev tools):

POST /v2/rollouts/run_list
{"taskset_id": "...", "task_version_ids": [...], "model_ids": [...], "group_size": 3, "max_steps": 20}

so its traces are created already linked to named task versions. The CLI (_submit_and_await in hud/eval/runtime.py) sends one request per rollout with the task's content but no reference:

payload = {"trace_id": ..., "job_id": ..., "env": task.env, "task": task.id, "args": task.args, "agent": spec}
await platform.apost("/rollouts/submit", json=payload)

The server never learns which taskset task the rollout came from, so the dashboard has nothing to name the card with. And the client can't fix it alone: /tasksets/{id}/export (what Taskset.from_api uses) doesn't return version ids either, so there's nothing to thread through.

Repro: hud sync tasks my-taskset tasks.py, then hud eval my-taskset claude --full, open the job → UUID cards. Same set via the website button → named cards. Seen on hud-python 0.6.8, macOS.

Fix options

  • (a) Backend (the complete fix): return each task's task_version_id in the /tasksets/{id}/export records, and accept an optional task_version_id on /rollouts/submit, linking the trace the way run_list already does. Both additive, no dashboard work needed (traces with a version id already render named — verified with job 8dc2bd52-5621-419b-b6e4-204afb264100), and it covers every run shape including --task-ids subsets and custom agent configs. Client side is then a three-line thread-through.
  • (b) Client only: use /rollouts/run_list from the CLI when possible — turns out it accepts API-key auth, and taskset_id alone works (no version ids needed). I've put this up as feat(eval): run platform tasksets via /rollouts/run_list so traces get task names #489. Covers full-taskset runs with stock gateway models; can't cover subsets or custom agent configs since the endpoint carries no agent config.
  • (c) Cosmetic: accept a display name on /rollouts/submit (the CLI knows task.slug).

(a) alone closes this completely — once submit carries the version id, every run shape gets named traces. (b) is already up and fixes the common case today without any backend change (and stays worthwhile as one request instead of N per batch).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions