Split workflowexecution into hot/cold live+archive; fix duplicate-_id and multi-generation alias bugs for stateful indexes (org_statistics, notifications)

[P4sca1]

Summary

Two related classes of bugs come from the same root cause: OpenSearch alias writes/reads route to whichever generation currently owns the alias, with no cross-generation _id awareness. Any index that is (a) rollover-enabled and (b) written to by re-writing the same _id in place (rather than pure append) will eventually split a single logical record across two physical generations, or — worse — leave an alias pointing at more than one index, which OpenSearch flatly refuses to run single-document ops against:

status: 400, type: illegal_argument_exception, reason: alias [shuffle_org_statistics] has more than one index associated with it [shuffle_org_statistics-000001, shuffle_org_statistics-000002], can't execute a single index op

workflowexecution, notifications, and org_statistics were all rollover + keyed-write indexes and all exposed to this. This PR fixes both the write path (going forward) and retroactively heals any deployment that has already hit this (like the error above), with no manual intervention required.

Why

• workflowexecution is Shuffle's highest-volume index. SetWorkflowExecution re-writes the same execution_id repeatedly as a run progresses. Once the index rolled over, an in-place update after the boundary left a stale copy in the older generation — the same execution split across two documents, with list/search/count results depending on which generation happened to be searched.
• org_statistics (and notifications) are stateful keyed stores, not append logs, but were previously left on the rollover list. Once they roll over to a second generation, the alias itself becomes multi-index, and any single-doc read/write against the alias — e.g. GetOrgStatistics, SetOrgStatistics — starts failing outright with the illegal_argument_exception above. This is a real, observed production error, not a hypothetical.

What changed

workflowexecution: hot/cold live + archive split (execution_lifecycle.go, new)

• workflowexecution_live — a new, single (non-rolling), mutable index. All in-flight execution writes go here, so in-place _id updates are always safe regardless of rollover state.
• workflowexecution (unchanged name) — becomes the archive: an append-only rollover index. After a 1-hour grace window past terminal status (to tolerate late async decision-fixups), a background sweep moves executions from live to archive. Archive writes are resolve-before-write (findExecutionInArchive), so re-archiving the same id after a rollover updates the existing copy instead of duplicating it. Deletes always target the concrete backing index (deleteExecutionFromArchiveIndex), never alias-by-id.
• Reads check live first, then fall back to archive (resolveExecutionWriteTarget / ErrExecutionArchived: a write to an already-archived, terminal execution is a duplicate/late re-affirmation; a write to an already-archived, non-terminal execution is a legitimate reopen and unarchives the doc back to live).
• All execution readers — GetWorkflowExecution, GetAllWorkflowExecutions[V2], GetUnfinishedExecutions, GetWorkflowRunCount, GetWorkflowRunsBySearch — converted to search both indices with AllowNoIndices/IgnoreUnavailable and dedupe by execution_id.
• GetWorkflowRunCount uses a cardinality aggregation on execution_id instead of track_total_hits, so a duplicate _id briefly spanning live+archive during a crash/sweep-race window is counted once, not twice.
• Existing deployments migrate with zero downtime and no bulk reindex: non-terminal docs sitting in the legacy workflowexecution index move to workflowexecution_live on startup; the legacy index keeps rolling as the archive going forward.

org_statistics / notifications: stop rolling stateful stores, retroactively collapse existing multi-generation aliases (db-connector.go)

• GetOpensearchRolloverIndexes() (line 124) now only lists genuinely append-only, high-volume stores. org_statistics and notifications are still in GetOpensearchBaseIndexes() (line 94, still managed/created) but are no longer rollover-enabled — each stays a single backing index, so in-place _id writes can never split across generations again.
• collapseSingleIndexAliases (new): for every base index that is not on the rollover list, on every InitOpensearchIndexes() startup run (unless SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT=true), this lists all physical generations currently under the alias; if more than one exists (e.g. a customer who already hit the org_statistics alias error, or upgraded from a version where these indexes rolled), it reindexes every older generation into the newest (op_type: create, so the newest generation's copy always wins on _id collision — no data loss, no duplication); removes the alias from the older generation(s) and deletes them; and detaches rollover/ISM from the sole remaining generation so it can't split again.
• Idempotent and safe to run on every startup; a single-generation index just gets its rollover detached (no-op if already detached).
• workflowexecution_live and the workflowexecution archive share their index mapping via a single Go map reference (opensearchCoreMappings), so they can never drift apart.
• created/edited/started_at mapped as long, not date. OpenSearch date fields store doc values internally as epoch milliseconds regardless of the epoch_second format annotation, while long stores the raw value as written. Every pre-existing customer index has these fields dynamically inferred as long (the app always wrote raw epoch-second ints). Mapping them date on new generations would silently rank cross-generation sorts by magnitude (ms vs. seconds) instead of true time — no query error, just wrong order. Verified live against a real cluster. Using long everywhere avoids this with no reindex needed.
• Generation-aware create/attach loop so upgrades whose surviving generation isn't -000001 still attach ISM/rollover correctly instead of failing on a write-index conflict.

notification_retention.go (new)

• Opt-in (default-off) retention sweep that hard-deletes read/ignored notifications older than OPENSEARCH_NOTIFICATION_RETENTION_DAYS days.
• Mapping-compatibility check that logs an explicit [WARNING] instead of silently no-op'ing when an older generation's read/ignored/updated_at fields can't support the retention query.

Opt-in / opt-out

All new background behavior can be disabled independently:

• SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT — skips index/mapping management entirely (pre-existing flag), including the alias-collapse migration.
• SHUFFLE_SKIP_EXECUTION_LIVE_MIGRATION — skips the one-time startup migration of non-terminal legacy docs into workflowexecution_live.
• SHUFFLE_SKIP_EXECUTION_ARCHIVAL_SWEEP — skips the recurring live→archive background sweep.
• OPENSEARCH_NOTIFICATION_RETENTION_DAYS — unset/0 keeps notification retention fully off (default).

[frikky]

@yashsinghcodes 🔥