Skip to content

feat(eval): propagate trace source framework#492

Open
howenyap wants to merge 2 commits into
hud-evals:mainfrom
howenyap:agent/trace-source-framework
Open

feat(eval): propagate trace source framework#492
howenyap wants to merge 2 commits into
hud-evals:mainfrom
howenyap:agent/trace-source-framework

Conversation

@howenyap

@howenyap howenyap commented Jul 15, 2026

Copy link
Copy Markdown

Summary

  • resolve source_framework from local environment trees using framework fingerprints
  • propagate framework provenance through CLI, task, taskset, chat, rollout, and hosted-runtime paths
  • include the resolved framework in hosted submissions and trace-enter reporting
  • default SDK-authored traces to HUD for backward compatibility

Why

The control plane now records which framework authored a trace. The SDK must resolve and propagate that provenance consistently while preserving existing HUD callers that do not pass the new argument.

Dependent PR

The corresponding control-plane and analysis-context changes are in https://github.com/hud-evals/hud-monorepo/pull/665. That PR currently pins this branch's commit and must update the submodule pointer to this PR's merge commit after merge.

Validation

  • 130 focused SDK tests covering resolver, CLI, runtime, rollout, chat, task, and taskset paths
  • Ruff lint and format checks
  • Pyright: 0 errors

Note

Medium Risk
Changes platform telemetry and hosted rollout submission contracts; behavior is mitigated by defaulting to "hud" and explicit resolution rules for remote runs.

Overview
Adds source_framework so the platform can record which authoring framework produced each trace. A new resolve_source_framework helper maps a local env tree to "hud" when Dockerfile.hud is present.

The value threads through rollout, trace_enter, Chat, Task/Taskset.run, HUDRuntime, and HostedRuntime. Remote /rollouts/submit payloads and trace-enter reporting now include source_framework. HostedRuntime can take an explicit framework, infer it from a local source directory, or fail if provenance cannot be resolved.

hud eval resolves framework from the local source root (falling back to "hud") and pins "hud" for platform tasksets with no on-disk tree. Existing callers keep working via a default of "hud" when the argument is omitted.

Reviewed by Cursor Bugbot for commit dd89244. Bugbot is set up for automated code reviews on this repo. Configure here.

@howenyap
howenyap force-pushed the agent/trace-source-framework branch from 670136a to 7afcc32 Compare July 15, 2026 09:09
@howenyap howenyap changed the title feat(eval): report trace source framework feat(eval): propagate trace source framework Jul 15, 2026
@howenyap
howenyap marked this pull request as ready for review July 17, 2026 03:13

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7afcc32. Configure here.

Comment thread hud/cli/eval.py Outdated
Local task files without a Dockerfile.hud marker (e.g. envs served via
--runtime hud or tcp://) now use the SDK default instead of aborting.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant