Repository navigation
Conversation
PMTiles::close serialized and gzipped the leaf directories (up to 10M entries each) one after the other while building them: ~9 s for the planet on one thread. Now it first collects the entries and cuts them into leaves exactly as before, then serializes and compresses the leaves in parallel, and writes them in order. The file is byte-identical. Planet (in memory, 96 threads): "Closing pmtiles file" 9 s -> 3 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cldellow
added a commit
to cldellow/tilemaker
that referenced
this pull request
Nov 8, 2026
relslow-patched: port upstream systemed#989 and systemed#990 (dktue)
This was referenced Nov 15, 2026
Open
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 15, 2026
…+ systemed#990) systemed#989 and systemed#990 adapted to perf/shared-pieces, as the base of this series. - SortedNodeStore/SortedWayStore::finalize publish the leftover groups (groups split between worker batches) from all threads instead of one after the other. On the planet that was ~10 s after the ways pass with every other thread idle. Same store content. (systemed#989, unchanged) - PMTiles::close collects the directory entries and cuts them into leaves exactly as before, then serializes and compresses the leaves on all cores and writes them in order: ~9 s -> 3 s on the planet, byte-identical file. (systemed#990; on perf/shared-pieces the two-step writer already holds the file, so the leaf writes need no lock and fileMutex goes.) Planet (in memory, 96 threads): -9 s and -6 s. Output byte-identical to perf/shared-pieces. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 15, 2026
…+ systemed#990) systemed#989 and systemed#990 adapted to perf/shared-pieces, as the base of this series. - SortedNodeStore/SortedWayStore::finalize publish the leftover groups (groups split between worker batches) from all threads instead of one after the other. On the planet that was ~10 s after the ways pass with every other thread idle. Same store content. (systemed#989, unchanged) - PMTiles::close collects the directory entries and cuts them into leaves exactly as before, then serializes and compresses the leaves on all cores and writes them in order: ~9 s -> 3 s on the planet, byte-identical file. (systemed#990; on perf/shared-pieces the two-step writer already holds the file, so the leaf writes need no lock and fileMutex goes.) Planet (in memory, 96 threads): -9 s and -6 s. Output byte-identical to perf/shared-pieces. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
force-pushed
the
perf/pmtiles-close-parallel
branch
from
November 16, 2026 10:58
9555939 to
30d5447
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gain: planet (in memory, 96 threads) "Closing pmtiles file" 9 s → 3 s (−6 s wall). Output byte-identical.
PMTiles::closebuilt the directories and, every time a leaf directory reached 10M entries, serialized it, gzipped it and wrote it. All of that ran on one thread, about 9 s for the planet's ~15 leaves.Now
closefirst collects the entries and cuts them into leaves exactly as before: same run-length encoding, same cut points, same root entries for z0–z5. Then it serializes and compresses the leaves on all cores, and writes them in order.appendTileEntryandflushEntriesare no longer needed.Measurements
.pmtilesoutput byte-identical (Germany at 32 and 8 threads, Chilean fjords extract; with #981/#942/#983 for deterministic output).🤖 Generated with Claude Code