ci(e2e): refresh e2e timings manifest

[GiladShoham]

Automated biweekly refresh of scripts/e2e-test-timings.json from the last 12 e2e runs (111 entries drifted meaningfully). Generated by the e2e_timings_refresh scheduled workflow.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

ci(e2e): refresh e2e test timings manifest

⚙️ Configuration changes 🧪 Tests 🕐 Less than 10 minutes

AI Description

• Recompute per-spec e2e duration estimates from the last 12 CI runs.
• Update the timings manifest used for e2e sharding/balancing in CI.
• Capture meaningful drift across many specs to keep CI parallelism efficient.

Diagram

graph TD
  A{{"Recent e2e runs"}} --> B(["e2e_timings_refresh workflow"]) --> C["e2e-test-timings.json"] --> D(["CI e2e sharder"]) --> E(["Parallel e2e jobs"])
  subgraph Legend
    direction LR
    _ext{{"External/Signals"}} ~~~ _svc(["CI/Workflow"]) ~~~ _file["Data file"]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Compute timings dynamically per CI run

• ➕ Eliminates PR churn from periodic manifest updates
• ➕ Automatically adapts to transient infra changes
• ➖ Harder to reproduce/compare sharding behavior across commits
• ➖ Requires fetching/maintaining historical artifacts during the run

2. Stabilize estimates with smoothing/outlier handling

• ➕ Reduces large swings in estimates between refreshes
• ➕ More resilient to flaky specs or noisy runs
• ➖ Adds complexity/parameters to the refresh job
• ➖ May lag behind real long-term performance changes

Recommendation: Keep the current committed-manifest approach (it’s simple, auditable, and works well for deterministic sharding). If estimate volatility is causing noticeable churn or imbalance, consider enhancing the refresh workflow with smoothing/outlier rejection rather than moving to fully dynamic per-run computation.

Files changed (1) +185 / -184

Other (1) +185 / -184

e2e-test-timings.jsonRefresh e2e spec duration estimates for CI sharding +185/-184

Refresh e2e spec duration estimates for CI sharding

• Updates the per-spec timing map with newly computed estimates derived from the latest 12 e2e runs. This refresh adjusts many entries to reflect observed runtime drift and keep CI parallelization balanced.

scripts/e2e-test-timings.json

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

1. Floor-clamped e2e weights 🐞 Bug ☼ Reliability

Description

The refreshed timings manifest assigns some suites exactly 15s (the generator’s hard minimum), which
reduces the resolution of scripts/split-e2e-tests.js bin-packing and can produce less-balanced
CircleCI shards if those suites still take materially longer than 15s.

Code

scripts/e2e-test-timings.json[53]

+  "e2e/harmony/compile.e2e.ts": 15,

Evidence

The generator explicitly clamps weights at 15s, and the splitter uses these weights for greedy LPT
bin-packing; therefore, having suites at the exact floor value reduces the splitter’s ability to
balance shards based on true cost.

scripts/generate-e2e-timings.js[172-179]
scripts/split-e2e-tests.js[61-82]
scripts/e2e-test-timings.json[50-55]
e2e/harmony/compile.e2e.ts[18-86]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`scripts/e2e-test-timings.json` contains multiple entries set to exactly `15`, which is the hard floor applied by the generator. These floor-clamped values reduce sharding accuracy because the splitter uses the manifest weights directly for LPT bin-packing.

### Issue Context
- The generator clamps per-file weights with `Math.max(15, ...)` to avoid zero-weight files.
- The splitter uses the manifest weights as-is to distribute tests across parallel nodes.
- This PR updates the manifest and includes entries at the floor (e.g. `e2e/harmony/compile.e2e.ts: 15`).

### Fix Focus Areas
- scripts/e2e-test-timings.json[35-55]
- scripts/generate-e2e-timings.js[172-179]
- scripts/split-e2e-tests.js[61-82]

### What to change
1. Re-generate the manifest using a larger/more diverse observation window (or a stronger prior, e.g. `--prior=manifest`) so suites don’t collapse to the floor due to under-identification.
2. Validate (in the refresh workflow or generator) that large downward shifts to the floor are expected; if not, re-run generation.
3. (Optional hardening) Add a guardrail in `generate-e2e-timings.js` to warn/fail when an unusually large fraction of files are clamped to the floor, to prevent publishing low-signal manifests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-free-for-open-source-projects]

1. Floor-clamped e2e weights 🐞 Bug ☼ Reliability

The refreshed timings manifest assigns some suites exactly 15s (the generator’s hard minimum), which
reduces the resolution of scripts/split-e2e-tests.js bin-packing and can produce less-balanced
CircleCI shards if those suites still take materially longer than 15s.

Agent Prompt

### Issue description
`scripts/e2e-test-timings.json` contains multiple entries set to exactly `15`, which is the hard floor applied by the generator. These floor-clamped values reduce sharding accuracy because the splitter uses the manifest weights directly for LPT bin-packing.

### Issue Context
- The generator clamps per-file weights with `Math.max(15, ...)` to avoid zero-weight files.
- The splitter uses the manifest weights as-is to distribute tests across parallel nodes.
- This PR updates the manifest and includes entries at the floor (e.g. `e2e/harmony/compile.e2e.ts: 15`).

### Fix Focus Areas
- scripts/e2e-test-timings.json[35-55]
- scripts/generate-e2e-timings.js[172-179]
- scripts/split-e2e-tests.js[61-82]

### What to change
1. Re-generate the manifest using a larger/more diverse observation window (or a stronger prior, e.g. `--prior=manifest`) so suites don’t collapse to the floor due to under-identification.
2. Validate (in the refresh workflow or generator) that large downward shifts to the floor are expected; if not, re-run generation.
3. (Optional hardening) Add a guardrail in `generate-e2e-timings.js` to warn/fail when an unusually large fraction of files are clamped to the floor, to prevent publishing low-signal manifests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools