Skip to content

perf(pmtiles): compress the leaf directories on all cores - #990

Open
dktue wants to merge 1 commit into
systemed:masterfrom
dktue:perf/pmtiles-close-parallel
Open

dktue wants to merge 1 commit into
systemed:masterfrom
dktue:perf/pmtiles-close-parallel

Conversation

@dktue

@dktue dktue commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Gain: planet (in memory, 96 threads) "Closing pmtiles file" 9 s → 3 s (−6 s wall). Output byte-identical.

PMTiles::close built the directories and, every time a leaf directory reached 10M entries, serialized it, gzipped it and wrote it. All of that ran on one thread, about 9 s for the planet's ~15 leaves.

Now close first collects the entries and cuts them into leaves exactly as before: same run-length encoding, same cut points, same root entries for z0–z5. Then it serializes and compresses the leaves on all cores, and writes them in order. appendTileEntry and flushEntries are no longer needed.

Measurements

"Closing pmtiles file" (planet, 96 threads)
before 9 s at 1 thread
after 3 s

.pmtiles output byte-identical (Germany at 32 and 8 threads, Chilean fjords extract; with #981/#942/#983 for deterministic output).

🤖 Generated with Claude Code

PMTiles::close serialized and gzipped the leaf directories (up to 10M
entries each) one after the other while building them: ~9 s for the
planet on one thread. Now it first collects the entries and cuts them into
leaves exactly as before, then serializes and compresses the leaves in
parallel, and writes them in order. The file is byte-identical.

Planet (in memory, 96 threads): "Closing pmtiles file" 9 s -> 3 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cldellow added a commit to cldellow/tilemaker that referenced this pull request Nov 8, 2026
relslow-patched: port upstream systemed#989 and systemed#990 (dktue)
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 15, 2026
…+ systemed#990)

systemed#989 and systemed#990 adapted to perf/shared-pieces, as the base of this series.

- SortedNodeStore/SortedWayStore::finalize publish the leftover groups
  (groups split between worker batches) from all threads instead of one
  after the other. On the planet that was ~10 s after the ways pass with
  every other thread idle. Same store content. (systemed#989, unchanged)
- PMTiles::close collects the directory entries and cuts them into leaves
  exactly as before, then serializes and compresses the leaves on all
  cores and writes them in order: ~9 s -> 3 s on the planet, byte-identical
  file. (systemed#990; on perf/shared-pieces the two-step writer already holds the
  file, so the leaf writes need no lock and fileMutex goes.)

Planet (in memory, 96 threads): -9 s and -6 s. Output byte-identical to
perf/shared-pieces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 15, 2026
…+ systemed#990)

systemed#989 and systemed#990 adapted to perf/shared-pieces, as the base of this series.

- SortedNodeStore/SortedWayStore::finalize publish the leftover groups
  (groups split between worker batches) from all threads instead of one
  after the other. On the planet that was ~10 s after the ways pass with
  every other thread idle. Same store content. (systemed#989, unchanged)
- PMTiles::close collects the directory entries and cuts them into leaves
  exactly as before, then serializes and compresses the leaves on all
  cores and writes them in order: ~9 s -> 3 s on the planet, byte-identical
  file. (systemed#990; on perf/shared-pieces the two-step writer already holds the
  file, so the leaf writes need no lock and fileMutex goes.)

Planet (in memory, 96 threads): -9 s and -6 s. Output byte-identical to
perf/shared-pieces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dktue
dktue force-pushed the perf/pmtiles-close-parallel branch from 9555939 to 30d5447 Compare November 16, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant