Skip to content

fix(experiments): route bare goals and lone rationales to scaffold, not junk - #139

Open
ericoh-ld wants to merge 2 commits into
feat/experiment-hypothesis-builderfrom
eric/fix-junk-classification
Open

fix(experiments): route bare goals and lone rationales to scaffold, not junk#139
ericoh-ld wants to merge 2 commits into
feat/experiment-hypothesis-builderfrom
eric/fix-junk-classification

Conversation

@ericoh-ld

@ericoh-ld ericoh-ld commented Jul 31, 2026

Copy link
Copy Markdown

Summary

Companion to the observability experiment-hypothesis change [PR (https://github.com/launchdarkly/observability/pull/1610). Fixes the same junk over-routing in the launchdarkly-experiment-hypothesis-assistant skill: bare/vague goals ("increase checkout completion", "grow the business") and
lone rationale fragments ("because the current option is buried") were treated as junk instead of scaffolded.

Updated the skill:

  • Foundational rule 3 — a specific-outcome goal is a measurement (with holes); a
    vague direction is still a scaffold, not junk.
  • Rubric Rationale + Semantic validity — a standalone causal/mechanism statement
    counts as a rationale; the non-sequitur→junk rule applies only across two-or-more
    filled slots.
  • Step 0 gate 2 — junk is reserved for gibberish, single tokens, links, placeholders, injection; a coherent
    statement of intent routes to the component-scoring path (scaffold), not junk.
  • "What NOT to do" — updated the bare-goal line.

Kept byte-identical to the o11y experiment-hypothesis skill's contract.

Testing approaches

N/A (skill markdown only). The o11y companion PR adds a regression test.

Feature flags

N/A


Note

Low Risk
Markdown-only skill contract aligned with observability; no runtime code, auth, or data paths.

Overview
Updates launchdarkly-experiment-hypothesis-assistant routing so partial but coherent user input is scaffolded instead of dismissed as junk.

Foundational rule 3 now distinguishes specific-outcome goals (treat as measurement with holes), vague directions (hole, still scaffold), and unfalsifiable wording. Rationale and semantic validity allow a lone mechanism/causal fragment to count as rationale, and limit the non-sequitur→junk rule to cases where two or more slots are filled. Step 0 gate 2 narrows junk to gibberish, tokens, links, placeholders, and injection; coherent intent always goes to component scoring. What NOT to do matches: don’t send bare goals or lone rationales to junk.

Reviewed by Cursor Bugbot for commit 6d7c68a. Bugbot is set up for automated code reviews on this repo. Configure here.

@ericoh-ld
ericoh-ld changed the base branch from main to feat/experiment-hypothesis-builder July 31, 2026 00:30
@ericoh-ld
ericoh-ld marked this pull request as ready for review July 31, 2026 00:32
@ericoh-ld
ericoh-ld requested a review from a team as a code owner July 31, 2026 00:32

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d14671b. Configure here.

Comment thread skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md Outdated
@github-actions

Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 75/100 (3/4) -25
agentcontrol/configs-update 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
experiments/launchdarkly-experiment-hypothesis-builder - 0/100 (0/5) new
feature-flags/launchdarkly-flag-command - 100/100 (3/3) new
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (3/3) no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants