Skip to content

fix(ci): stop three timing races failing unrelated PRs - #213

Merged
wmadden-electric merged 1 commit into
mainfrom
claude/fix-flaky-timing-tests
Aug 9, 2026
Merged

fix(ci): stop three timing races failing unrelated PRs#213
wmadden-electric merged 1 commit into
mainfrom
claude/fix-flaky-timing-tests

Conversation

@wmadden-electric

Copy link
Copy Markdown
Contributor

Three tests have been failing intermittently and blocking every PR on the required Test check, including #211 (which needed four attempts) and #212. None of them relate to the changes they were failing. Each races something instead of waiting for it.

The deploy-lease heartbeat

state-api.test.ts forked a heartbeat on a 10ms interval, slept 100ms of real time, then asserted at least two had fired:

yield* Effect.sleep('100 millis');
yield* Fiber.interrupt(fiber);
expect(fake.countRequests(/PATCH .*\/lease/)).toBeGreaterThanOrEqual(2);

A starved event loop fails that, which is what CI saw (Expected: >= 2). It now waits until the second heartbeat is observed, so it cannot fail that way at any speed — and it finishes sooner than the fixed sleep on a fast machine.

Honest caveat: I could not reproduce the original failure locally, 20 runs under eight CPU-burning processes. The case for this change rests on the shape of the test, not on a local repro.

The emulator daemon restart

This one was misdiagnosed as a timeout in earlier discussion. It has a 45s budget and failed at 27.5s with:

(500): postgres emulator failed to start database "pcdev-pgtest-restart-appdb" on port 51308:
A Prisma Dev server with the name "pcdev-pgtest-restart-appdb" is already running.

The daemon hosts its @prisma/dev servers in-process, and only a graceful exit closes them and releases each server's name lock. terminate gave SIGTERM 5 seconds before SIGKILL, so a daemon slow to shut down under load got killed with its locks still held. The next daemon then had to wait out proper-lockfile's stale threshold, which exhausted postgres-main.ts's already-generous "already running" retry budget (2 retries × 12s of polling).

It also returned immediately after SIGKILL, which is a bug in its own right: SIGKILL is delivered asynchronously, so stopDaemon could report success while the old daemon still held its ports and locks.

Both are fixed. SIGKILL now waits for the process to actually be reaped, and the grace period is 20s. That budget is only ever spent on a daemon that is genuinely not exiting, because the wait ends as soon as the pid is gone — all 11 emulator tests still complete in 35s together, which they could not if teardown were paying it.

The database lifecycle test

The only genuine timeout: 30151ms against a ~8s normal run. Raised to 60s, matching the 45-120s its heavier neighbours in the same file already use. There is no race to fix here — every step is awaited and it is simply slow — so the timeout is the right mechanism and it was set too tight. The idempotency test beside it has identical setup cost and gets the same budget.

Also: one transport error no longer abandons a whole sweep

deleteProjectDeep now reports a thrown transport error as "not gone" instead of propagating it. ci-cleanup.ts sweeps projects in a loop, so a single socket error abandoned every project after it, leaking workspace slots until the next run. The post-teardown project-delete retry now also rides out a blip rather than giving up.

Two tests cover it, and both fail without the change.

Verification

  • The lease test: 20 consecutive runs under load, no failures.
  • The emulator suite: all 11 tests pass, 34.9s total, confirming the longer grace period costs nothing on a clean stop.
  • The cleanup tests: 19 pass; the two new ones fail when the change is reverted.
  • Lint clean on all five files; typecheck green for both affected packages.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@wmadden-electric, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: db33ad6b-70b5-4d57-b5ea-bccb2fd372d2

📥 Commits

Reviewing files that changed from the base of the PR and between 60dcd26 and cb419e0.

📒 Files selected for processing (5)
  • packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/postgres.test.ts
  • packages/1-prisma-cloud/0-lowering/dev-emulators/src/daemon.ts
  • packages/1-prisma-cloud/0-lowering/lowering/src/state/__tests__/state-api.test.ts
  • scripts/ci-cleanup-utils.test.ts
  • scripts/ci-cleanup-utils.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pkg-pr-new

pkg-pr-new Bot commented Aug 9, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@prisma/composer@213
npm i https://pkg.pr.new/@prisma/composer-prisma-cloud@213

commit: a0b7598

Three tests were failing intermittently and blocking every PR on the required Test check. Each raced something instead of waiting for it.

**The deploy-lease heartbeat** forked a 10ms heartbeat, slept 100ms of real time, then asserted at least two had fired — so a starved event loop failed it (`Expected: >= 2`). It now waits until the second heartbeat is observed, which cannot fail that way at any speed. I could not reproduce the failure locally even under heavy load, so the case for this rests on the shape of the test rather than on a local repro.

**The emulator daemon restart** failed with "A Prisma Dev server with the name ... is already running", not a timeout. `terminate` gave SIGTERM 5s before SIGKILL, and a SIGKILLed daemon never closes its in-process `@prisma/dev` servers, so their name locks leak and the next daemon has to wait out the stale threshold. It also returned immediately after SIGKILL, letting a replacement start while the old process still held its ports. SIGKILL now waits for the process to be reaped, and the grace period is 20s — spent only on a daemon that is genuinely not exiting, since the wait ends as soon as the pid is gone. All 11 emulator tests still finish in 35s together, so a clean stop is unaffected.

**The database lifecycle test** was the one real timeout: 30151ms against a ~8s normal run. Raised to 60s, matching the 45-120s its heavier neighbours already use. No race to fix here — every step is awaited, it is simply slow.

Also makes `deleteProjectDeep` report a thrown transport error as "not gone" rather than propagating it. The sweep in ci-cleanup.ts deletes projects in a loop, so one socket error abandoned every project after it; the post-teardown retry now also rides out a blip.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
@wmadden-electric
wmadden-electric force-pushed the claude/fix-flaky-timing-tests branch from cb419e0 to a0b7598 Compare August 9, 2026 18:50
@wmadden-electric
wmadden-electric merged commit 82090f7 into main Aug 9, 2026
17 of 18 checks passed
@wmadden-electric
wmadden-electric deleted the claude/fix-flaky-timing-tests branch August 9, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants