batchTrigger drops per-item idempotencyKey on rate-limited / "batch already sealed" retries (v4.4.6)

[brentshulman-silkline]

Summary

On self-hosted v4.4.6, batchTrigger appears to drop the per-item idempotencyKey for the items in batches that are re-driven after the batch rate limiter trips, so those runs are created un-deduplicated.

What we observed

We fan out ~6,000+ runs via a single tasks.batchTrigger(...) where every item carries a distinct idempotencyKey created with idempotencyKeys.create([...], { scope: 'global' }) and idempotencyKeyTTL.

When the fan-out exceeds the environment's batch token bucket (BATCH_RATE_LIMIT_MAX), the two-phase batch path throws:

• BatchTriggerError: Failed to create batch with N items: Rate limit exceeded - retry after Ns (isRateLimited: true)
• BatchTriggerError: Failed to stream items for batch <id> (N items): Batch <id> is already sealed and cannot accept more items

(stack points into executeBatchTwoPhase in @trigger.dev/sdk/src/v3/shared.ts).

Querying the resulting runs, the early batches have idempotency_key populated, while the batches created/re-driven after the rate-limit + "sealed" errors have an empty idempotency_key — all with the same sdk_version 4.4.6. In one run: 945 keyed runs across 3 batches, 5,066 keyless runs across 18 batches. The keyless runs bypass dedup and re-fire on every subsequent trigger cycle.

Expected

Per-item idempotencyKey (and TTL) should be preserved across batch sealing / rate-limit re-drive, so deduplication holds regardless of whether the batch rate limiter is hit.

Repro sketch

1. Set the environment's BATCH_RATE_LIMIT_MAX below your intended batch size (e.g. default 1200).
2. batchTrigger more items than the bucket, each with a distinct global-scope idempotencyKey.
3. Observe Rate limit exceeded + Batch ... is already sealed errors, then inspect the created runs: the post-seal batches have no idempotency key.

Environment

• Self-hosted, Helm chart + images v4.4.6
• SDK @trigger.dev/sdk 4.4.6

[brentshulman-silkline]

Pointers to the relevant code (tag v4.4.6):

Client side is fine — each batch item's idempotencyKey is built correctly before sending:

• packages/trigger-sdk/src/v3/shared.ts#L647-L649 (idempotencyKey: (await makeIdempotencyKey(item.options?.idempotencyKey)) ?? batchItemIdempotencyKey), and the same construction in the other batchTrigger variants (#L904-L906, #L1163-L1165, #L1425-L1427).

The bug seam is the two-phase create→stream split in executeBatchTwoPhase:

• packages/trigger-sdk/src/v3/shared.ts#L1539-L1588
  • Phase 1 apiClient.createBatch({ runCount: items.length, ... }) (#L1555) reserves the batch with a run count, but not the per-item options/idempotency keys. A 429 here throws Failed to create batch … Rate limit exceeded (#L1568).
  • Phase 2 apiClient.streamBatchItems(batch.id, items, ...) (#L1579) is what actually carries each item's idempotencyKey to the server. If the batch has already sealed, this throws Failed to stream items for batch <id> … is already sealed (#L1582).

So when rate-limiting delays/splits phase 2 and the batch seals, the reserved runs are created without the per-item idempotency keys that only phase 2 delivers — which matches what we see (early batches keyed, post-seal batches keyless).

The server-side counterpart is the v3 batch service, which reserves runCount at batch-create time separately from processing the per-item options/keys, then marks the batch sealed:

• apps/webapp/app/v3/services/batchTriggerV3.server.ts — runCount set at create (#L223, #L438, #L492), per-item item.options?.idempotencyKey read during item processing (#L379-L404, #L573), batch sealed in #processBatchTaskRunItems (#L477-L480).

Could a maintainer confirm where, in the streaming/seal path, reserved runs end up created without the streamed per-item idempotency keys? That reconciliation between phase-1 runCount and phase-2 streamed options looks like the root cause.

[matt-aitken]

This should be fixed as of @trigger.dev/* 4.5.0 — the root cause was the same as #4046, resolved in #4094.

What was actually happening

idempotencyKeys.create() returns a 64-char hash and stores the { key, scope } mapping in an in-process catalog keyed by that hash. In 4.4.x that catalog was a fixed-size LRU capped at 1000 entries. When a run creates more than 1000 keys up front (e.g. building a large batchTrigger array where each item has its own key), the earliest mappings get evicted. At trigger time the SDK looks the hash back up to attach the key/scope metadata, finds nothing, and sends it without that metadata.

The key detail: only the human-readable key/scope metadata was lost — the idempotency-key hash itself was always sent. So the dedup unique key stayed intact and deduplication kept working; what showed up empty was the idempotencyKey/scope metadata in the dashboard and analytics, not the dedup itself.

#4094 replaced the LRU with an unbounded per-run catalog (cleared at each run boundary), so every key created in a run retains its metadata regardless of count.

Reproduced + verified

Running a batch of 2000 distinct global-scope keys created up front:

version	idempotency-key hash	key/scope metadata
4.4.6	all 2000 populated	first 1000 dropped
4.5.x	all 2000 populated	all 2000 retained

On the rate-limit / "batch already sealed" angle: in the current two-phase batch path, a rate-limited or oversized batch fails cleanly at batch creation with zero runs created — it doesn't produce partially-keyed runs. Runs are only created from streamed items, each carrying its own key.

If you can still reproduce empty idempotency keys on 4.5.0 or later, please reopen with the SDK version and a repro and we'll take another look. Thanks for the detailed writeup and code pointers — they made this quick to confirm.

Closing as fixed by #4094 (same root cause as #4046).