feat(builder): opt-in parallel build pipeline (concurrency across envs)

[davidfirst]

What

The build pipeline runs every task serially (build-pipe.ts → mapSeries), using ~1 core to orchestrate on a multi-core machine. For a large bit ci pr build (100 components, 5 envs) that's ~19m wall where tests are only ~3m — the rest is TS compile + preview webpack + schema, none overlapping.

This adds an opt-in scheduler that runs environments concurrently. Default is unchanged (concurrency = 1 → the original mapSeries path, byte-for-byte). Enable with BIT_BUILD_CONCURRENCY=<n> or the build.concurrency config. It's turned on for the bit_pr CI job (BIT_BUILD_CONCURRENCY=3) to measure in CircleCI.

Correctness model (readiness-based scheduler)

Derived from the actual task model — most tasks don't declare dependencies (they rely on getBuildPipe() array order + the start/middle/end location mechanism); all envs share one capsule root; the TS compiler resolves cross-component types from source. The scheduler (tasks-parallel-scheduler.ts) blocks each task until:

1. its per-env predecessor is done — each env stays a sequential chain, preserving array order and never overlapping two tasks of the same env (they share a capsule);
2. every env's instance of its declared dependency task types is done — honors the documented "complete for all envs" rule, e.g. TesterTask.dependencies = [compiler] means tests wait for every env's compile;
3. the previous location group is done — start → middle → end barrier (so preview/pkg run after all compiles even though they don't declare a compiler dep).

Only different envs run concurrently. Safe against the shared capsule root because every capsule file has a single writer env, and compile output is order-independent (source resolution).

Tests

tasks-parallel-scheduler.spec.ts (9 passing) covers: location split, blocker computation, every-entry-once, intra-env ordering, cross-env concurrency, the cross-env declared-dependency barrier (a tester waits for all envs' compile), the location barrier, and the concurrency bound.

Notes

• Conservative concurrency=3 to start (memory: in-process webpack/tsc on the 2xlarge can OOM). core-aspect-env is the long pole, so 3 already captures most of the win; tune up once proven stable.
• Does not parallelize tasks within an env (e.g. Schema ∥ Preview) — that's a riskier follow-up (same-capsule writes).

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Opt-in parallel build execution across envs with location barriers
✨ Enhancement 🧪 Tests ⚙️ Configuration changes 🕐 40+ Minutes

Description

• Add opt-in env-level concurrency while keeping per-env task order unchanged.
• Preserve correctness via start/middle/end location barriers across all envs.
• Support config/env-var concurrency and add unit tests for the scheduler.

Diagram

flowchart TD
  C["Config/Env var"] --> B["Builder svc"] --> P["Build pipe"] --> D{"Concurrency > 1?"}
  D -->|no| S["Serial map"] --> X["Run task"]
  D -->|yes| T["Parallel sched"] --> E["Env chains"] --> X["Run task"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use a concurrency helper library (e.g., p-map) for env chains

• ➕ Less custom concurrency code to maintain
• ➕ Potentially clearer semantics via well-known primitives
• ➖ May add or expand dependency surface (or complicate dependency policy)
• ➖ Still requires careful ordering + location barrier logic around the helper

2. Model tasks as a DAG and schedule by explicit dependencies

• ➕ Could unlock more parallelism (including within an env) safely
• ➕ Makes ordering constraints explicit rather than relying on queue order
• ➖ Much larger refactor (most tasks currently rely on implicit ordering)
• ➖ Higher risk of subtle behavior changes and harder migration

Recommendation: The chosen approach (parallelize only across envs, keep per-env sequencing, and enforce a location barrier) is the right incremental step: it preserves the existing implicit-ordering model while providing meaningful wall-time gains on multi-core machines. A DAG-based scheduler could yield more parallelism but is a substantially riskier redesign; a small helper library could replace the worker-pool implementation later if maintainability becomes a concern.

Files changed (4) +232 / -1

Enhancement (2) +101 / -1

build-pipe.tsAdd opt-in parallel execution path to BuildPipe +13/-1

Add opt-in parallel execution path to BuildPipe

• Introduces a new PipeOptions.concurrency setting and routes execution through the parallel scheduler when concurrency > 1. Keeps the original fully-serial mapSeries behavior as the default (concurrency=1).

scopes/pipelines/builder/build-pipe.ts

tasks-parallel-scheduler.tsNew location-barrier scheduler with bounded cross-env concurrency +88/-0

New location-barrier scheduler with bounded cross-env concurrency

• Adds a pure scheduler that splits the queue by task location, groups entries into per-env chains, and runs chains concurrently up to a configured limit. Ensures a barrier between location segments while keeping each env’s task order sequential.

scopes/pipelines/builder/tasks-parallel-scheduler.ts

Tests (1) +118 / -0

tasks-parallel-scheduler.spec.tsUnit tests for parallel scheduler semantics and bounds +118/-0

Unit tests for parallel scheduler semantics and bounds

• Adds scheduler-focused tests covering location segmentation, env grouping/order preservation, exactly-once execution, cross-env concurrency, location barriers, and concurrency limits.

scopes/pipelines/builder/tasks-parallel-scheduler.spec.ts

Other (1) +13 / -0

builder.service.tsAdd build concurrency configuration (env var + config key) +13/-0

Add build concurrency configuration (env var + config key)

• Adds getBuildConcurrency() to read BIT_BUILD_CONCURRENCY or build.concurrency, validate it, and default to 1. Wires the value into BuildPipe options for opt-in parallel builds.

scopes/pipelines/builder/builder.service.ts

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (0) 📜 Skill insights (0)

1. Cross-env deps not enforced ✓ Resolved 🐞 Bug ≡ Correctness

Description

executeTasksByLocationAndEnv() runs independent per-env chains concurrently within a location, so
a task that declares dependencies may start in envA as soon as envA’s dependency finishes even if
the dependency is still running in envB. This violates the documented BuildTask.dependencies
contract (“completed for *all* envs”) and can start real dependent tasks (e.g. TesterTask) too
early when concurrency > 1.

Code

scopes/pipelines/builder/tasks-parallel-scheduler.ts[R82-87]

+  const segments = splitByLocation(queue);
+  // sequential `for` (not Promise.all) is the location barrier — each segment fully resolves first.
+  for (const segment of segments) {
+    const envChains = groupByEnv(segment);
+    await runChainsWithConcurrency(envChains, concurrency, executor);
+  }

Evidence

BuildTask.dependencies is explicitly defined as a cross-environment barrier (“completed for *all*
envs before this task starts”), and the pipeline-ordering code emits a queue that relies on global
execution order to enforce that barrier. The new scheduler discards that global ordering by running
per-env chains concurrently within the same location segment, allowing envA to advance to dependent
tasks before envB finishes the dependency task.

scopes/pipelines/builder/build-task.ts[107-117]
scopes/pipelines/builder/build-pipeline-order.ts[109-127]
scopes/defender/tester/tester.task.ts[16-19]
scopes/pipelines/builder/tasks-parallel-scheduler.ts[82-87]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The parallel scheduler groups a location segment by env (`groupByEnv`) and runs each env-chain sequentially but concurrently across envs. This breaks the cross-env dependency guarantee documented by `BuildTask.dependencies` (dependency must complete for *all* envs before the dependent task starts), because envA can advance to the dependent task while envB is still executing the dependency.
### Issue Context
`calculatePipelineOrder()` intentionally orders tasks such that dependency-ordered task nodes are queued as “all envs run dependency task” before “any env runs dependent task”. The original fully-serial executor preserved this guarantee; the new scheduler must preserve it too.
### Fix Focus Areas
- scopes/pipelines/builder/tasks-parallel-scheduler.ts[12-88]
- scopes/pipelines/builder/tasks-parallel-scheduler.spec.ts[49-117]
### Suggested fix
1. Within each location segment, introduce an additional **barrier stage** that preserves the queue’s cross-env ordering at least at the “task node” level:
- Split `segment` into consecutive batches where the *serialized task id* changes (e.g. `${task.aspectId}:${task.name}`), **preserving the original queue order**.
- For each batch, run concurrency across envs (bounded) but do **not** start the next batch until the current batch fully completes.
- This preserves the “all envs for dependency task” barrier when the queue is constructed that way (which is exactly how `calculatePipelineOrder()` emits dependency-ordered tasks).
2. Add a unit test proving the dependency barrier:
- Queue: `envA:compiler`, `envB:compiler`, `envA:test(depends-on-compiler)`, `envB:test`.
- Make `envB:compiler` slow; assert no `test` starts before `envB:compiler` ends.
3. Document in the scheduler comment that it preserves `BuildTask.dependencies` semantics, and ensure the implementation matches.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Fail-fast no longer strict 🐞 Bug ☼ Reliability ⭐ New

Description

When concurrency > 1, multiple tasks can start before a first failure is recorded, so
exitOnFirstFailedTask (and dependency-based skipping) can’t prevent already-started tasks from
running. This changes the behavior vs the serial mapSeries path and can execute tasks that would
have been skipped in the fully-serial model.

Code

scopes/pipelines/builder/build-pipe.ts[R90-95]

+    const concurrency = this.options?.concurrency ?? 1;
+    if (concurrency > 1) {
+      await executeTasksByLocationAndEnv(this.tasksQueue, concurrency, ({ task, env }) => this.executeTask(task, env));
+    } else {
+      await mapSeries(this.tasksQueue, async ({ task, env }) => this.executeTask(task, env));
+    }

Evidence

The PR enables a concurrent scheduler for concurrency > 1. Skipping is only evaluated at the start
of executeTask(), while failures are recorded later (after task.execute resolves), so tasks that
started earlier can’t be retroactively prevented from running. The scheduler itself only aborts on
promise rejection, not on recorded component-level failures.

scopes/pipelines/builder/build-pipe.ts[87-96]
scopes/pipelines/builder/build-pipe.ts[114-152]
scopes/pipelines/builder/build-pipe.ts[154-165]
scopes/pipelines/builder/build-pipe.ts[227-237]
scopes/pipelines/builder/tasks-parallel-scheduler.ts[74-112]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
With parallel scheduling enabled (`concurrency > 1`), the pipeline can start multiple tasks before a failure is known/recorded. Because skipping is decided at task start (`shouldSkipTask()`), tasks that are already in-flight will continue even when `exitOnFirstFailedTask` is enabled, deviating from the fully-serial behavior.

## Issue Context
- Fail-fast is currently implemented by checking `failedTasks` at the start of each task, and recording failures only after task completion.
- The parallel scheduler starts up to `concurrency` tasks and only aborts on executor promise rejection, not on “task finished with component errors”.

## Fix Focus Areas
- scopes/pipelines/builder/build-pipe.ts[87-96]
- scopes/pipelines/builder/build-pipe.ts[114-191]
- scopes/pipelines/builder/build-pipe.ts[227-237]
- scopes/pipelines/builder/tasks-parallel-scheduler.ts[74-116]

## Implementation directions
Choose one:
1) **Force serial when fail-fast is enabled**: if `options.exitOnFirstFailedTask` is true, override `concurrency` to 1 (keep semantics identical).
2) **Add scheduler-level stop condition**: extend `executeTasksByLocationAndEnv/runLocation` to stop launching new tasks once BuildPipe has observed a failure (e.g., via a shared `shouldStop()` predicate), and ensure remaining not-started tasks are recorded as skipped/pending in a way consistent with the existing UI expectations.
3) **Convert “task failed (component errors)” into abort signal when fail-fast is enabled**: when `compsWithErrors.length` and fail-fast is on, throw a specific error to stop scheduling quickly (and ensure the caller formats/handles it appropriately).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Nondeterministic results ordering 🐞 Bug ☼ Reliability

Description

With parallel execution enabled, BuildPipe appends task results as tasks finish, so tasksResults
becomes completion-ordered instead of queue-ordered. Downstream aggregation/reporting iterates
tasksResults in array order, so pipeline reports and metadata merges can vary across runs for the
same input.

Code

scopes/pipelines/builder/build-pipe.ts[R90-95]

+    const concurrency = this.options?.concurrency ?? 1;
+    if (concurrency > 1) {
+      await executeTasksByLocationAndEnv(this.tasksQueue, concurrency, ({ task, env }) => this.executeTask(task, env));
+    } else {
+      await mapSeries(this.tasksQueue, async ({ task, env }) => this.executeTask(task, env));
+    }

Evidence

Parallel workers complete in variable order, and results are recorded by push().
Reporting/aggregation consumes results in array order (e.g. pipeline report generation uses .map()
over tasksResults, and metadata aggregation reduces over tasksResults), so nondeterministic
insertion order leads to nondeterministic outputs.

scopes/pipelines/builder/build-pipe.ts[181-191]
scopes/pipelines/builder/tasks-parallel-scheduler.ts[51-61]
scopes/pipelines/builder/build-pipeline-result-list.ts[50-60]
scopes/pipelines/builder/build-pipeline-result-list.ts[62-79]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
When `concurrency > 1`, tasks complete in nondeterministic order. `BuildPipe.executeTask()` currently records results via `this.taskResults.push(taskResults)`, so the final `tasksResults` array order depends on timing.
Downstream code (e.g. pipeline report generation and metadata reduction) iterates `tasksResults` in the given order, which can make output unstable between runs.
### Issue Context
This is a regression only in the parallel path; in the serial path, insertion order matched queue order.
### Fix Focus Areas
- scopes/pipelines/builder/build-pipe.ts[87-102]
- scopes/pipelines/builder/build-pipe.ts[181-191]
### Suggested fix
1. Before executing tasks, build an index map from queue order:
- key example: `${env.id}::${task.aspectId}:${task.name}` -> index.
2. After parallel execution completes (and before creating `TaskResultsList`), sort `this.taskResults` using that index map so it matches `this.tasksQueue` order.
3. (Optional but safer) Assert every queue entry produced exactly one result; throw a clear error if a result is missing.
This keeps the final results deterministic without changing task execution semantics.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Misleading progress in parallel 🐞 Bug ◔ Observability ⭐ New

Description

In parallel mode, progress counters are incremented when tasks start, but completion logs also print
getProgress(), so completion messages can show a progress value that reflects “started” rather
than “completed”. This makes CI output misleading and harder to interpret under concurrency.

Code

scopes/pipelines/builder/build-pipe.ts[R90-95]

+    const concurrency = this.options?.concurrency ?? 1;
+    if (concurrency > 1) {
+      await executeTasksByLocationAndEnv(this.tasksQueue, concurrency, ({ task, env }) => this.executeTask(task, env));
+    } else {
+      await mapSeries(this.tasksQueue, async ({ task, env }) => this.executeTask(task, env));
+    }

Evidence

logProgress() increments the internal counter, and BuildPipe calls it before the task runs;
completion logs later read getProgress() without incrementing on completion. With concurrent
starts, the progress displayed in completion logs no longer corresponds to completed work.

scopes/pipelines/builder/build-pipe.ts[114-170]
scopes/harmony/logger/long-process-logger.ts[15-51]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`LongProcessLogger` increments its counter on `logProgress()` calls. `BuildPipe.executeTask()` calls `logProgress()` at task start, and later logs completion messages using `getProgress()`. Under parallel execution, many tasks can start before any completes, so completion logs show inflated progress.

## Issue Context
This becomes noticeable only when `concurrency > 1`, because the serial path keeps “started” and “completed” nearly aligned.

## Fix Focus Areas
- scopes/pipelines/builder/build-pipe.ts[114-170]
- scopes/harmony/logger/long-process-logger.ts[15-51]

## Implementation directions
- Option A: Add a separate “completed” counter and use it for completion logs.
- Option B: In parallel mode, avoid calling `logProgress()` at start; instead set a status line without incrementing, and call `logProgress()` only on completion (so the counter reflects completed tasks).
- Option C: Extend `LongProcessLogger` with explicit `logStart()`/`logComplete()` APIs so progress semantics are unambiguous.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Wrong method name in docs 🐞 Bug ⚙ Maintainability ⭐ New

Description

The new getBuildConcurrency() doc comment references BuildPipe.executeTasksInParallel, but the
actual implementation uses executeTasksByLocationAndEnv. This misdirects maintainers trying to
trace the parallel execution behavior.

Code

scopes/pipelines/builder/builder.service.ts[R173-178]

+  /**
+   * Number of environments whose build task-chains may run concurrently. Defaults to 1 (fully
+   * serial — the original behavior). Opt into parallelism with the `BIT_BUILD_CONCURRENCY` env var
+   * (handy for CI) or the `build.concurrency` config. In parallel mode each env's tasks still run
+   * sequentially; only different envs run concurrently — see `BuildPipe.executeTasksInParallel`.
+   */

Evidence

The comment points to BuildPipe.executeTasksInParallel, but BuildPipe’s parallel branch calls
executeTasksByLocationAndEnv from tasks-parallel-scheduler.ts.

scopes/pipelines/builder/builder.service.ts[173-183]
scopes/pipelines/builder/build-pipe.ts[87-95]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A doc comment references a non-existent/incorrect symbol (`BuildPipe.executeTasksInParallel`), while the code path actually uses `executeTasksByLocationAndEnv`.

## Issue Context
This is a comment-only issue but it directly affects code navigation and maintainability.

## Fix Focus Areas
- scopes/pipelines/builder/builder.service.ts[173-178]
- scopes/pipelines/builder/build-pipe.ts[87-95]

## Implementation directions
Update the comment to reference the correct function (`executeTasksByLocationAndEnv` or `tasks-parallel-scheduler.ts`) or the correct `BuildPipe.execute()` code path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Previous review results

Review updated until commit 60350a1

Results up to commit N/A

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Cross-env deps not enforced ✓ Resolved 🐞 Bug ≡ Correctness

Description

executeTasksByLocationAndEnv() runs independent per-env chains concurrently within a location, so
a task that declares dependencies may start in envA as soon as envA’s dependency finishes even if
the dependency is still running in envB. This violates the documented BuildTask.dependencies
contract (“completed for *all* envs”) and can start real dependent tasks (e.g. TesterTask) too
early when concurrency > 1.

Code

scopes/pipelines/builder/tasks-parallel-scheduler.ts[R82-87]

+  const segments = splitByLocation(queue);
+  // sequential `for` (not Promise.all) is the location barrier — each segment fully resolves first.
+  for (const segment of segments) {
+    const envChains = groupByEnv(segment);
+    await runChainsWithConcurrency(envChains, concurrency, executor);
+  }

Evidence

BuildTask.dependencies is explicitly defined as a cross-environment barrier (“completed for *all*
envs before this task starts”), and the pipeline-ordering code emits a queue that relies on global
execution order to enforce that barrier. The new scheduler discards that global ordering by running
per-env chains concurrently within the same location segment, allowing envA to advance to dependent
tasks before envB finishes the dependency task.

scopes/pipelines/builder/build-task.ts[107-117]
scopes/pipelines/builder/build-pipeline-order.ts[109-127]
scopes/defender/tester/tester.task.ts[16-19]
scopes/pipelines/builder/tasks-parallel-scheduler.ts[82-87]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The parallel scheduler groups a location segment by env (`groupByEnv`) and runs each env-chain sequentially but concurrently across envs. This breaks the cross-env dependency guarantee documented by `BuildTask.dependencies` (dependency must complete for *all* envs before the dependent task starts), because envA can advance to the dependent task while envB is still executing the dependency.
### Issue Context
`calculatePipelineOrder()` intentionally orders tasks such that dependency-ordered task nodes are queued as “all envs run dependency task” before “any env runs dependent task”. The original fully-serial executor preserved this guarantee; the new scheduler must preserve it too.
### Fix Focus Areas
- scopes/pipelines/builder/tasks-parallel-scheduler.ts[12-88]
- scopes/pipelines/builder/tasks-parallel-scheduler.spec.ts[49-117]
### Suggested fix
1. Within each location segment, introduce an additional **barrier stage** that preserves the queue’s cross-env ordering at least at the “task node” level:
 - Split `segment` into consecutive batches where the *serialized task id* changes (e.g. `${task.aspectId}:${task.name}`), **preserving the original queue order**.
 - For each batch, run concurrency across envs (bounded) but do **not** start the next batch until the current batch fully completes.
 - This preserves the “all envs for dependency task” barrier when the queue is constructed that way (which is exactly how `calculatePipelineOrder()` emits dependency-ordered tasks).
2. Add a unit test proving the dependency barrier:
 - Queue: `envA:compiler`, `envB:compiler`, `envA:test(depends-on-compiler)`, `envB:test`.
 - Make `envB:compiler` slow; assert no `test` starts before `envB:compiler` ends.
3. Document in the scheduler comment that it preserves `BuildTask.dependencies` semantics, and ensure the implementation matches.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Nondeterministic results ordering 🐞 Bug ☼ Reliability

Description

With parallel execution enabled, BuildPipe appends task results as tasks finish, so tasksResults
becomes completion-ordered instead of queue-ordered. Downstream aggregation/reporting iterates
tasksResults in array order, so pipeline reports and metadata merges can vary across runs for the
same input.

Code

scopes/pipelines/builder/build-pipe.ts[R90-95]

+    const concurrency = this.options?.concurrency ?? 1;
+    if (concurrency > 1) {
+      await executeTasksByLocationAndEnv(this.tasksQueue, concurrency, ({ task, env }) => this.executeTask(task, env));
+    } else {
+      await mapSeries(this.tasksQueue, async ({ task, env }) => this.executeTask(task, env));
+    }

Evidence

Parallel workers complete in variable order, and results are recorded by push().
Reporting/aggregation consumes results in array order (e.g. pipeline report generation uses .map()
over tasksResults, and metadata aggregation reduces over tasksResults), so nondeterministic
insertion order leads to nondeterministic outputs.

scopes/pipelines/builder/build-pipe.ts[181-191]
scopes/pipelines/builder/tasks-parallel-scheduler.ts[51-61]
scopes/pipelines/builder/build-pipeline-result-list.ts[50-60]
scopes/pipelines/builder/build-pipeline-result-list.ts[62-79]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
When `concurrency > 1`, tasks complete in nondeterministic order. `BuildPipe.executeTask()` currently records results via `this.taskResults.push(taskResults)`, so the final `tasksResults` array order depends on timing.
Downstream code (e.g. pipeline report generation and metadata reduction) iterates `tasksResults` in the given order, which can make output unstable between runs.
### Issue Context
This is a regression only in the parallel path; in the serial path, insertion order matched queue order.
### Fix Focus Areas
- scopes/pipelines/builder/build-pipe.ts[87-102]
- scopes/pipelines/builder/build-pipe.ts[181-191]
### Suggested fix
1. Before executing tasks, build an index map from queue order:
 - key example: `${env.id}::${task.aspectId}:${task.name}` -> index.
2. After parallel execution completes (and before creating `TaskResultsList`), sort `this.taskResults` using that index map so it matches `this.tasksQueue` order.
3. (Optional but safer) Assert every queue entry produced exactly one result; throw a clear error if a result is missing.
This keeps the final results deterministic without changing task execution semantics.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 60350a1