Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 20 additions & 20 deletions .claude/skills/incremental-build/architecture.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions .claude/skills/incremental-build/performance-investigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ Findings:

The one-stage-per-step design was reviewed as the suspected structural cause of the branch building slower than `main`. Measured, it is not. Count stages from the perf log (`importStages ... with N stages`, or the unique `task/...::step/...` ids) and compare `main` to this branch for a `ui5 build` and a `ui5 build --all`: a library builds with the same stage count on both. Every shipped step-based task (`minify`, `buildThemes`, the three `replace*`, `escapeNonAsciiCharacters`, `enhanceManifest`) is single-step, and `generateThemeDesignerResources` (the only multi-step task, three steps) is not in the default library task set, so step count equals task count. A single-step task already is exactly one stage, so collapsing it to `task/{taskName}` removes no stage, leaves `StepRunner`'s per-key machinery in place, and was measured and rejected as a pure stage-id rename. Per-stage `updateProjectIndices` cost is the same on both revisions.

The branch's warm and stale deltas against `main` are real but are not a stage-count effect. On a stale build they are the per-key map-step machinery (`#keyId` content hashing, item 12; the delta fold, item 13; per-unit allocation). On a warm build the project is served from cache and tasks are skipped, so the delta is startup and module loading, not the step pipeline; the plan-time module-loading part is item 18. Precise numbers live in the benchmark results repository alongside the config that produced them, not here, since they drift with the code.
The branch's warm and stale deltas against `main` are real but are not a stage-count effect. On a stale build they are the per-key map-step machinery (`#keyId` content hashing, item 12; the delta fold, item 13; per-invocation allocation). On a warm build the project is served from cache and tasks are skipped, so the delta is startup and module loading, not the step pipeline; the plan-time module-loading part is item 18. Precise numbers live in the benchmark results repository alongside the config that produced them, not here, since they drift with the code.

### 15. The step invocation sidecar is re-serialized only for stages that re-recorded

Expand All @@ -413,7 +413,7 @@ Largest single row ~94 KB (`replaceVersion`), ~515 KB across the four projects p

The fix tracks which stages re-recorded (a dirty `Set` added to by `setStepInvocationData`, the only mutation path; `getStepInvocationData` loads without marking) and emits only those. On a one-file delta build (`Button.js` edited), `sap.m` re-records 3 of its 7 loaded step stages; the other 4 (and every step stage of the three dependencies, all full hits on this delta) are no longer rewritten. `writeCache` for `sap.m` was ~150 ms on this scenario; the sidecar serialization it now skips is a small fraction of that, so the direct wall-clock win is minor in CLI mode where the write is deferred off the critical path. As with the request-graph dirty-flag fix (§13), it matters more in BuildServer mode (Phase 4), where cache writes are awaited and each skipped row shortens the awaited write.

Alongside this, `#storeStepReturns` previously opened one SQLite transaction per returning unit; it now buffers compressed rows and the driver flushes one transaction per step (`StepRunner.#runGroup` calls the store's `flush` after a step's units run). No shipped builder task returns resources from `each`, so this path is latent, but the first task that does would otherwise pay a transaction per key. The return descriptors are also built from the metadata `#prepareStageResources` already computed rather than re-reading each resource's `getIntegrity()`/`getSize()`.
Alongside this, `#storeStepReturns` previously opened one SQLite transaction per returning invocation; it now buffers compressed rows and the driver flushes one transaction per step (`StepRunner.#runGroup` calls the store's `flush` after a step's invocations run). No shipped builder task returns resources from `each`, so this path is latent, but the first task that does would otherwise pay a transaction per key. The return descriptors are also built from the metadata `#prepareStageResources` already computed rather than re-reading each resource's `getIntegrity()`/`getSize()`.

### 16. The stage prefetch never paid off and was removed

Expand Down Expand Up @@ -444,7 +444,7 @@ Left as-is after analysis:

Skipped after measurement (no win in any scenario):
- **Write-buffer flush.** `StepRunner.#flushWriteBuffer` drains a map step's buffered writes with a sequential `await` loop. The duplicate-path rejection is enforced at buffer insertion, not flush, so the flush is a pure replay of distinct-path writes to the in-memory stage workspace, and the buffer already holds every write before the flush, so parallelizing does not cut peak memory. Instrumented totals: on a **stale** one-file build no flush batch exceeds 50 writes (sub-millisecond); on a **cold** build the flush totals ~147 ms across the whole ~21 s build (largest single batch 2565 writes in ~40 ms, ~15 µs/write, consistent with memory-adapter writes). Parallelizing would save a fraction of that against a real write-ordering risk, so the sequential loop stays.
- **Per-unit `MonitoredTaskUtil` proxy.** The proxy allocates a wrapper closure per tracked-method access. Counting `get`-trap invocations across a whole build: ~4,973 on a stale one-file build, ~183,261 on a cold build (every unit runs). At tens of nanoseconds per short-lived closure this is a few milliseconds on a ~21 s cold build and negligible on stale, so caching bound methods on the per-unit proxy (against the deliberate per-unit isolation) was not worth it.
- **Per-invocation `MonitoredTaskUtil` proxy.** The proxy allocates a wrapper closure per tracked-method access. Counting `get`-trap invocations across a whole build: ~4,973 on a stale one-file build, ~183,261 on a cold build (every invocation runs). At tens of nanoseconds per short-lived closure this is a few milliseconds on a ~21 s cold build and negligible on stale, so caching bound methods on the per-invocation proxy (against the deliberate per-invocation isolation) was not worth it.

**Aggregate before/after (hyperfine, isolated `UI5_DATA_DIR`):**

Expand All @@ -453,7 +453,7 @@ Skipped after measurement (no win in any scenario):
| Stale (one file appended to `Button.js`, warmup 2 / runs 8) | 4.630 s ± 0.067 | 4.723 s ± 0.086 |
| Cold (empty cache, warmup 1 / runs 3) | 21.438 s ± 0.165 | 20.598 s ± 0.195 |

Stale is within noise (the ranges overlap; these edits barely execute when few units run). Cold is ~0.8 s faster (~4%), consistent with the O(n squared) accumulation removal being largest where ~12k resources are written. The headline is correctness and hygiene, with a measurable cold-build improvement and no stale regression.
Stale is within noise (the ranges overlap; these edits barely execute when few invocations run). Cold is ~0.8 s faster (~4%), consistent with the O(n squared) accumulation removal being largest where ~12k resources are written. The headline is correctness and hygiene, with a measurable cold-build improvement and no stale regression.

### 18. Step-based task modules must not import their processors at plan time

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ There are two kinds of steps, run in array order:
* **Scalar step** `{name, needs?, run}` — runs once. `run: async ({needs, workspace, dependencies, taskUtil, options}) => value?`
* **Map step** `{name, needs?, keys, each}` — runs once per key. `keys: async ({needs, workspace, dependencies, taskUtil, options}) => keySet` enumerates the keys, and `each: async (key, {needs, workspace, dependencies, taskUtil, options}) => value?` processes one key. A key is a resource or a stable string.

Each step gets its own `name` (a non-empty string, unique within the task). A map step is the usual cache-aware shape: the build cache treats every key as its own cached unit, so a delta build re-processes only the keys whose inputs changed.
Each step gets its own `name` (a non-empty string, unique within the task). A map step is the usual cache-aware shape: the build cache treats every key as its own cached invocation, so a delta build re-processes only the keys whose inputs changed.

A step may list earlier step names in `needs`; those steps' return values then arrive as `needs.<name>` in the step's callbacks. A step may reference only steps that come before it in the array, so there can be no cycle. A step's return value (resources, or any JSON-serializable value) folds into the cache signature of every step that consumes it, so a changed producer re-runs its consumers.

Expand Down Expand Up @@ -355,7 +355,7 @@ export const stepBased = true;
export default function build(options) {
return [{
name: "render",
// One cached unit per Markdown file, so a delta build re-renders only the files that changed.
// One cached invocation per Markdown file, so a delta build re-renders only the files that changed.
keys: async ({workspace}) => workspace.byGlob("**/*.md"),
each: async (resource, {workspace, taskUtil}) => {
// Import the processor lazily so a fully cached build does not load it
Expand Down Expand Up @@ -391,7 +391,7 @@ const log = getLogger("builder:tasks:renderMarkdownFiles");
module.exports = function build(options) {
return [{
name: "render",
// One cached unit per Markdown file, so a delta build re-renders only the files that changed.
// One cached invocation per Markdown file, so a delta build re-renders only the files that changed.
keys: async ({workspace}) => workspace.byGlob("**/*.md"),
each: async (resource, {workspace, taskUtil}) => {
// Import the processor lazily so a fully cached build does not load it
Expand Down
2 changes: 1 addition & 1 deletion packages/project/lib/build/TaskRunner.js
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ function mergeInputRecordings(base, extra) {
* Folds a stage's per-key reads (the {@link StepRunner} fold) into the stage-level monitored requests,
* deduping against what the base recording already requests.
*
* The recorder stores resolved paths and the glob patterns a unit issued, and a path or pattern is commonly
* The recorder stores resolved paths and the glob patterns an invocation issued, and a path or pattern is commonly
* read by more than one key (a shared marker probe, a dependency each key resolves, a shared glob) and also
* recorded at the stage level, so a plain concatenation carried duplicates that only collapse later in the
* request graph (which keys on a Set). Deduping here keeps the recording the request graph rebuilds minimal;
Expand Down
2 changes: 1 addition & 1 deletion packages/project/lib/build/cache/BuildStageCache.js
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ export default class BuildStageCache {
// future use case, bundling many files from outside the UI5 dirs (e.g. node_modules) into the
// build result, would want per-file delta re-runs like project/dependency; that needs a
// changed-root-path signal, list-valued root signatures in the stage delta candidates, and root
// reads threaded into step unit selection, and is left as a separate change.
// reads threaded into step invocation selection, and is left as a separate change.
this.#rootRequestManagers = rootRequestManagers ?? {
gitignore: new ResourceRequestManager(projectName, `${stageId}#root`, false),
noGitignore: new ResourceRequestManager(projectName, `${stageId}#root-no-gitignore`, false),
Expand Down
10 changes: 5 additions & 5 deletions packages/project/lib/build/cache/ProjectBuildCache.js
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ export default class ProjectBuildCache {
// setStepInvocationData.
#stepInvocationData = new Map();

// Compressed CAS rows for resources returned by the step runner, buffered per unit and flushed in
// Compressed CAS rows for resources returned by the step runner, buffered per invocation and flushed in
// one transaction per step via the return value store's flush() (see #flushStepReturns).
#pendingStepReturnCasRows = [];

Expand Down Expand Up @@ -1065,8 +1065,8 @@ export default class ProjectBuildCache {
* Persists the content of resources a step returned and describes them for later
* reconstruction. Content goes through the same compression and dedup pipeline as stage resources;
* the compressed rows are buffered in {@link #pendingStepReturnCasRows} and written by
* {@link #flushStepReturns}, which the driver calls once per step so a step returning many units
* costs one transaction rather than one per unit. The descriptors are recorded in the step's
* {@link #flushStepReturns}, which the driver calls once per step so a step returning many invocations
* costs one transaction rather than one per invocation. The descriptors are recorded in the step's
* invocation data. The CAS write uses INSERT OR IGNORE, so content shared with a stage output is
* stored once and a build that later fails before the flush leaves nothing behind.
*
Expand Down Expand Up @@ -1094,8 +1094,8 @@ export default class ProjectBuildCache {

/**
* Writes the step-return CAS rows buffered since the last flush in a single transaction. The driver
* calls this once per step (after the step's units have stored their returns), so one step costs one
* transaction regardless of how many units returned resources. A no-op when nothing was buffered.
* calls this once per step (after the step's invocations have stored their returns), so one step costs one
* transaction regardless of how many invocations returned resources. A no-op when nothing was buffered.
*/
#flushStepReturns() {
if (!this.#pendingStepReturnCasRows.length) {
Expand Down
Loading
Loading