You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I run a taskset remotely from the CLI (hud eval my-taskset claude --full), the job page shows every trace as an anonymous UUID card ("Task 2aeeb924") — you can't tell which rollout belongs to which task without opening each one. The same batch launched from the website's run button comes back named and grouped ("notif-bug (1)").
CLI job:
Same taskset via the website button:
I dug into why. The two entry points use different endpoints. The button sends one batch request that references tasks by version (discovered via the network tab in browser dev tools):
so its traces are created already linked to named task versions. The CLI (_submit_and_await in hud/eval/runtime.py) sends one request per rollout with the task's content but no reference:
The server never learns which taskset task the rollout came from, so the dashboard has nothing to name the card with. And the client can't fix it alone: /tasksets/{id}/export (what Taskset.from_api uses) doesn't return version ids either, so there's nothing to thread through.
Repro: hud sync tasks my-taskset tasks.py, then hud eval my-taskset claude --full, open the job → UUID cards. Same set via the website button → named cards. Seen on hud-python 0.6.8, macOS.
Fix options
(a) Backend (the complete fix): return each task's task_version_id in the /tasksets/{id}/export records, and accept an optional task_version_id on /rollouts/submit, linking the trace the way run_list already does. Both additive, no dashboard work needed (traces with a version id already render named — verified with job 8dc2bd52-5621-419b-b6e4-204afb264100), and it covers every run shape including --task-ids subsets and custom agent configs. Client side is then a three-line thread-through.
(b) Client only: use /rollouts/run_list from the CLI when possible — turns out it accepts API-key auth, and taskset_id alone works (no version ids needed). I've put this up as feat(eval): run platform tasksets via /rollouts/run_list so traces get task names #489. Covers full-taskset runs with stock gateway models; can't cover subsets or custom agent configs since the endpoint carries no agent config.
(c) Cosmetic: accept a display name on /rollouts/submit (the CLI knows task.slug).
(a) alone closes this completely — once submit carries the version id, every run shape gets named traces. (b) is already up and fixes the common case today without any backend change (and stays worthwhile as one request instead of N per batch).
When I run a taskset remotely from the CLI (
hud eval my-taskset claude --full), the job page shows every trace as an anonymous UUID card ("Task 2aeeb924") — you can't tell which rollout belongs to which task without opening each one. The same batch launched from the website's run button comes back named and grouped ("notif-bug (1)").CLI job:
Same taskset via the website button:
I dug into why. The two entry points use different endpoints. The button sends one batch request that references tasks by version (discovered via the network tab in browser dev tools):
so its traces are created already linked to named task versions. The CLI (
_submit_and_awaitinhud/eval/runtime.py) sends one request per rollout with the task's content but no reference:The server never learns which taskset task the rollout came from, so the dashboard has nothing to name the card with. And the client can't fix it alone:
/tasksets/{id}/export(whatTaskset.from_apiuses) doesn't return version ids either, so there's nothing to thread through.Repro:
hud sync tasks my-taskset tasks.py, thenhud eval my-taskset claude --full, open the job → UUID cards. Same set via the website button → named cards. Seen on hud-python 0.6.8, macOS.Fix options
task_version_idin the/tasksets/{id}/exportrecords, and accept an optionaltask_version_idon/rollouts/submit, linking the trace the wayrun_listalready does. Both additive, no dashboard work needed (traces with a version id already render named — verified with job8dc2bd52-5621-419b-b6e4-204afb264100), and it covers every run shape including--task-idssubsets and custom agent configs. Client side is then a three-line thread-through./rollouts/run_listfrom the CLI when possible — turns out it accepts API-key auth, andtaskset_idalone works (no version ids needed). I've put this up as feat(eval): run platform tasksets via /rollouts/run_list so traces get task names #489. Covers full-taskset runs with stock gateway models; can't cover subsets or custom agent configs since the endpoint carries no agent config./rollouts/submit(the CLI knowstask.slug).(a) alone closes this completely — once
submitcarries the version id, every run shape gets named traces. (b) is already up and fixes the common case today without any backend change (and stays worthwhile as one request instead of N per batch).