batchTrigger drops per-item idempotencyKey on rate-limited / "batch already sealed" retries (v4.4.6)

[brentshulman-silkline]

Summary

On self-hosted v4.4.6, batchTrigger appears to drop the per-item idempotencyKey for the items in batches that are re-driven after the batch rate limiter trips, so those runs are created un-deduplicated.

What we observed

We fan out ~6,000+ runs via a single tasks.batchTrigger(...) where every item carries a distinct idempotencyKey created with idempotencyKeys.create([...], { scope: 'global' }) and idempotencyKeyTTL.

When the fan-out exceeds the environment's batch token bucket (BATCH_RATE_LIMIT_MAX), the two-phase batch path throws:

• BatchTriggerError: Failed to create batch with N items: Rate limit exceeded - retry after Ns (isRateLimited: true)
• BatchTriggerError: Failed to stream items for batch <id> (N items): Batch <id> is already sealed and cannot accept more items

(stack points into executeBatchTwoPhase in @trigger.dev/sdk/src/v3/shared.ts).

Querying the resulting runs, the early batches have idempotency_key populated, while the batches created/re-driven after the rate-limit + "sealed" errors have an empty idempotency_key — all with the same sdk_version 4.4.6. In one run: 945 keyed runs across 3 batches, 5,066 keyless runs across 18 batches. The keyless runs bypass dedup and re-fire on every subsequent trigger cycle.

Expected

Per-item idempotencyKey (and TTL) should be preserved across batch sealing / rate-limit re-drive, so deduplication holds regardless of whether the batch rate limiter is hit.

Repro sketch

1. Set the environment's BATCH_RATE_LIMIT_MAX below your intended batch size (e.g. default 1200).
2. batchTrigger more items than the bucket, each with a distinct global-scope idempotencyKey.
3. Observe Rate limit exceeded + Batch ... is already sealed errors, then inspect the created runs: the post-seal batches have no idempotency key.

Environment

• Self-hosted, Helm chart + images v4.4.6
• SDK @trigger.dev/sdk 4.4.6

[brentshulman-silkline]

Pointers to the relevant code (tag v4.4.6):

Client side is fine — each batch item's idempotencyKey is built correctly before sending:

• packages/trigger-sdk/src/v3/shared.ts#L647-L649 (idempotencyKey: (await makeIdempotencyKey(item.options?.idempotencyKey)) ?? batchItemIdempotencyKey), and the same construction in the other batchTrigger variants (#L904-L906, #L1163-L1165, #L1425-L1427).

The bug seam is the two-phase create→stream split in executeBatchTwoPhase:

• packages/trigger-sdk/src/v3/shared.ts#L1539-L1588
  • Phase 1 apiClient.createBatch({ runCount: items.length, ... }) (#L1555) reserves the batch with a run count, but not the per-item options/idempotency keys. A 429 here throws Failed to create batch … Rate limit exceeded (#L1568).
  • Phase 2 apiClient.streamBatchItems(batch.id, items, ...) (#L1579) is what actually carries each item's idempotencyKey to the server. If the batch has already sealed, this throws Failed to stream items for batch <id> … is already sealed (#L1582).

So when rate-limiting delays/splits phase 2 and the batch seals, the reserved runs are created without the per-item idempotency keys that only phase 2 delivers — which matches what we see (early batches keyed, post-seal batches keyless).

The server-side counterpart is the v3 batch service, which reserves runCount at batch-create time separately from processing the per-item options/keys, then marks the batch sealed:

• apps/webapp/app/v3/services/batchTriggerV3.server.ts — runCount set at create (#L223, #L438, #L492), per-item item.options?.idempotencyKey read during item processing (#L379-L404, #L573), batch sealed in #processBatchTaskRunItems (#L477-L480).

Could a maintainer confirm where, in the streaming/seal path, reserved runs end up created without the streamed per-item idempotency keys? That reconciliation between phase-1 runCount and phase-2 streamed options looks like the root cause.