Repository navigation
Conversation
boost::sort::block_indirect_sort gets very slow with many threads: sorting the planet's tile list took 200-300 s at 96 threads (the threads spinning on its internal lock) and about 7 s at 16. Cap the threads for this sort. Planet, in memory, 96 threads: 29:51 -> 24:43 / 23:50, user CPU -28%. Output unchanged: same comparator, total order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 7, 2026
Same change as systemed#986. With it, the sort before the tile phase takes ~13 s instead of 2-5 minutes at 96 threads, so planet times no longer vary by minutes between runs (14:12 / 14:08 in memory at 96 threads). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Nov 7, 2026
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 15, 2026
Fewer idle cores before and during the tile phase; output byte-identical (the batches, their order and the clip cache scope are unchanged). - Tile list: filtered in parallel column stripes, sorted as 64-bit keys with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp, as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its storage touched in parallel and resized on a thread of its own while the keys are sorted; low zooms collected in the counting pass; batch starts computed in parallel. - finalizeObjects: big shards run concurrently in the pool with threads in proportion to their share, sort buffers reused per thread, budget raised to half the objects. - Hand-out: batches of z<=8 first; consecutive batches handed out as a group (OrderedBatches begin/end per group, group size adapted to the batch count); the groups expected to be slowest (z12 object counts gathered while collecting) handed out up to 5% ahead; idle threads take over the units of a running group (tile stealing). - No deadlock with few threads: groups handed out ahead of their place (z0-z8, slow groups) could fill the held-tile limit while the group to be written next had not been handed out, so every thread waited for room (seen on the planet at 16 threads). OrderedBatches now knows the first group not handed out yet (setFirstUnhanded), and when that group holds the next unit to write, waitForRoom lets one thread go on and take it, even over the limit. - Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed from them. Planet (96 threads): about -11% wall over the rounds (e.g. batch groups -12 s, parallel tile list -19 s). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 15, 2026
Fewer idle cores before and during the tile phase; output byte-identical (the batches, their order and the clip cache scope are unchanged). - Tile list: filtered in parallel column stripes (set bits found with a small portable helper, _BitScanForward64 on MSVC), sorted as 64-bit keys with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp, as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its storage touched in parallel and resized on a thread of its own while the keys are sorted; low zooms collected in the counting pass; batch starts computed in parallel. - finalizeObjects: big shards run concurrently in the pool with threads in proportion to their share, sort buffers reused per thread, budget raised to half the objects. - Hand-out: batches of z<=8 first; consecutive batches handed out as a group (OrderedBatches begin/end per group, group size adapted to the batch count); the groups expected to be slowest (z12 object counts gathered while collecting) handed out up to 5% ahead; idle threads take over the units of a running group (tile stealing). - No deadlock with few threads: groups handed out ahead of their place (z0-z8, slow groups) could fill the held-tile limit while the group to be written next had not been handed out, so every thread waited for room (seen on the planet at 16 threads). OrderedBatches now knows the first group not handed out yet (setFirstUnhanded), and when that group holds the next unit to write, waitForRoom lets one thread go on and take it, even over the limit. - Held-tile limit by memory: threads may run ahead by 1/16 of the memory (physical RAM, or the cgroup v2/v1 limit if lower), at least 64 MB per thread as before and at most 4 GB; 64 MB per thread where the memory isn't known (e.g. Windows). With 64 MB per thread only, a 16-thread planet with --store on 64 GB spent minutes with one thread on a slow multipolygon and the others waiting for room: 54 min -> 25 min. - Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed from them. Planet (96 threads): about -11% wall over the rounds (e.g. batch groups -12 s, parallel tile list -19 s). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
pushed a commit
to dktue/tilemaker
that referenced
this pull request
Nov 16, 2026
Fewer idle cores before and during the tile phase; output byte-identical (the batches, their order and the clip cache scope are unchanged). - Tile list: filtered in parallel column stripes (set bits found with a small portable helper, _BitScanForward64 on MSVC), sorted as 64-bit keys with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp, as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its storage touched in parallel and resized on a thread of its own while the keys are sorted; low zooms collected in the counting pass; batch starts computed in parallel. - finalizeObjects: big shards run concurrently in the pool with threads in proportion to their share, sort buffers reused per thread, budget raised to half the objects. - Hand-out: batches of z<=8 first; consecutive batches handed out as a group (OrderedBatches begin/end per group, group size adapted to the batch count); the groups expected to be slowest (z12 object counts gathered while collecting) handed out up to 5% ahead; idle threads take over the units of a running group (tile stealing). - No deadlock with few threads: groups handed out ahead of their place (z0-z8, slow groups) could fill the held-tile limit while the group to be written next had not been handed out, so every thread waited for room (seen on the planet at 16 threads). OrderedBatches now knows the first group not handed out yet (setFirstUnhanded), and when that group holds the next unit to write, waitForRoom lets one thread go on and take it, even over the limit. - Held-tile limit by memory: threads may run ahead by 1/16 of the memory (physical RAM, or the cgroup v2/v1 limit if lower), at least 64 MB per thread as before and at most 4 GB; 64 MB per thread where the memory isn't known (e.g. Windows). With 64 MB per thread only, a 16-thread planet with --store on 64 GB spent minutes with one thread on a slow multipolygon and the others waiting for room: 54 min -> 25 min. - Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed from them. - compress_string returns a string of exactly the compressed size (it kept the compression bound as capacity), so held tiles take what is counted. Planet (96 threads): about -11% wall over the rounds (e.g. batch groups -12 s, parallel tile list -19 s). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
force-pushed
the
perf/sort-threads
branch
from
November 16, 2026 10:58
4dd482d to
e13033f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gain (planet, in memory, 96 threads): sorting the tile list takes about 7 s instead of 200–300 s; the whole build 29:51 → 24:43 / 23:50 (−17% / −20%), user CPU −28%. A one-line change; the output is unchanged.
Problem
Before writing tiles,
sortTileCoordinatessorts the list of all tiles to write (346M on the planet) withboost::sort::block_indirect_sort, using every thread. At high thread counts that sort gets very slow: the threads spend their time spinning on its internal lock (in a CPU profile at 96 threads, of a build with the open performance PRs, 36% of all samples before the tile phase were inboost::sort::common::spinlock_t::lock, plus 10% insched_yield).Sorting the planet's 358M possible z0–z14 tiles with the comparator from
sortTileCoordinates, outside tilemaker, on the same server (48 cores, 96 hardware threads):The slowdown isn't specific to this comparator: plain 64-bit keys take 242 s at 96 threads and 2.5 s at 16. It's also erratic: 32 threads were slower than 24 and 48.
Change
Use at most 16 threads for this sort. The comparator and the result are unchanged (it's a total order), so the output is the same.
Planet, in memory, default threads (96),
.pmtiles, OpenMapTiles profileAlternating runs, master 2789651 and master with this PR:
Most of the remaining 37 s is collecting tiles, which #972 brings down to under 1 s. The time the old sort takes also varies a lot between runs: master took 25:17 in an earlier run of the same command, so part of the run-to-run variation on many-core machines comes from this sort.
With all open performance PRs (see #977), the same sort phase took about 170–210 s of a 14–17 minute planet build at 96 threads, so the share is even larger there.
block_indirect_sortis also used inBinarySearchNodeStore::finalizeandBinarySearchWayStore::finalize, which only run for unsorted.osm.pbfinput. They may slow down the same way with many threads; I haven't measured them, so this PR leaves them alone.🤖 Generated with Claude Code