Skip to content

perf: sort the tile list with at most 16 threads (planet -17% at 96 threads) - #986

Open
dktue wants to merge 1 commit into
systemed:masterfrom
dktue:perf/sort-threads
Open

dktue wants to merge 1 commit into
systemed:masterfrom
dktue:perf/sort-threads

Conversation

@dktue

@dktue dktue commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Gain (planet, in memory, 96 threads): sorting the tile list takes about 7 s instead of 200–300 s; the whole build 29:51 → 24:43 / 23:50 (−17% / −20%), user CPU −28%. A one-line change; the output is unchanged.

Problem

Before writing tiles, sortTileCoordinates sorts the list of all tiles to write (346M on the planet) with boost::sort::block_indirect_sort, using every thread. At high thread counts that sort gets very slow: the threads spend their time spinning on its internal lock (in a CPU profile at 96 threads, of a build with the open performance PRs, 36% of all samples before the tile phase were in boost::sort::common::spinlock_t::lock, plus 10% in sched_yield).

Sorting the planet's 358M possible z0–z14 tiles with the comparator from sortTileCoordinates, outside tilemaker, on the same server (48 cores, 96 hardware threads):

threads time
8 11.5 s
12 7.8 / 7.9 / 7.9 s
16 6.6 / 6.6 / 6.7 s
24 5.8 s
32 24.5 s
48 5.0 s
96 200–300 s (in the planet runs below)

The slowdown isn't specific to this comparator: plain 64-bit keys take 242 s at 96 threads and 2.5 s at 16. It's also erratic: 32 threads were slower than 24 and 48.

Change

Use at most 16 threads for this sort. The comparator and the result are unchanged (it's a total order), so the output is the same.

Planet, in memory, default threads (96), .pmtiles, OpenMapTiles profile

Alternating runs, master 2789651 and master with this PR:

wall user CPU collecting tiles + sort
this PR 24:43 57,357 s 38 s
master 29:51 79,602 s 318 s
this PR 23:50 57,676 s 37 s

Most of the remaining 37 s is collecting tiles, which #972 brings down to under 1 s. The time the old sort takes also varies a lot between runs: master took 25:17 in an earlier run of the same command, so part of the run-to-run variation on many-core machines comes from this sort.

With all open performance PRs (see #977), the same sort phase took about 170–210 s of a 14–17 minute planet build at 96 threads, so the share is even larger there.

block_indirect_sort is also used in BinarySearchNodeStore::finalize and BinarySearchWayStore::finalize, which only run for unsorted .osm.pbf input. They may slow down the same way with many threads; I haven't measured them, so this PR leaves them alone.

🤖 Generated with Claude Code

boost::sort::block_indirect_sort gets very slow with many threads: sorting
the planet's tile list took 200-300 s at 96 threads (the threads spinning on
its internal lock) and about 7 s at 16. Cap the threads for this sort.

Planet, in memory, 96 threads: 29:51 -> 24:43 / 23:50, user CPU -28%.
Output unchanged: same comparator, total order.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 7, 2026
Same change as systemed#986. With it, the sort before the tile phase takes ~13 s
instead of 2-5 minutes at 96 threads, so planet times no longer vary by
minutes between runs (14:12 / 14:08 in memory at 96 threads).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 15, 2026
Fewer idle cores before and during the tile phase; output byte-identical
(the batches, their order and the clip cache scope are unchanged).

- Tile list: filtered in parallel column stripes, sorted as 64-bit keys
  with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp,
  as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its
  storage touched in parallel and resized on a thread of its own while the
  keys are sorted;
  low zooms collected in the counting pass; batch starts computed in
  parallel.
- finalizeObjects: big shards run concurrently in the pool with threads in
  proportion to their share, sort buffers reused per thread, budget raised
  to half the objects.
- Hand-out: batches of z<=8 first; consecutive batches handed out as a
  group (OrderedBatches begin/end per group, group size adapted to the
  batch count); the groups expected to be slowest (z12 object counts
  gathered while collecting) handed out up to 5% ahead; idle threads take
  over the units of a running group (tile stealing).
- No deadlock with few threads: groups handed out ahead of their place
  (z0-z8, slow groups) could fill the held-tile limit while the group to
  be written next had not been handed out, so every thread waited for room
  (seen on the planet at 16 threads). OrderedBatches now knows the first
  group not handed out yet (setFirstUnhanded), and when that group holds
  the next unit to write, waitForRoom lets one thread go on and take it,
  even over the limit.
- Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed
  from them.

Planet (96 threads): about -11% wall over the rounds (e.g. batch groups
-12 s, parallel tile list -19 s).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 15, 2026
Fewer idle cores before and during the tile phase; output byte-identical
(the batches, their order and the clip cache scope are unchanged).

- Tile list: filtered in parallel column stripes (set bits found with a
  small portable helper, _BitScanForward64 on MSVC), sorted as 64-bit keys
  with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp,
  as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its
  storage touched in parallel and resized on a thread of its own while the
  keys are sorted; low zooms collected in the counting pass; batch starts
  computed in parallel.
- finalizeObjects: big shards run concurrently in the pool with threads in
  proportion to their share, sort buffers reused per thread, budget raised
  to half the objects.
- Hand-out: batches of z<=8 first; consecutive batches handed out as a
  group (OrderedBatches begin/end per group, group size adapted to the
  batch count); the groups expected to be slowest (z12 object counts
  gathered while collecting) handed out up to 5% ahead; idle threads take
  over the units of a running group (tile stealing).
- No deadlock with few threads: groups handed out ahead of their place
  (z0-z8, slow groups) could fill the held-tile limit while the group to
  be written next had not been handed out, so every thread waited for room
  (seen on the planet at 16 threads). OrderedBatches now knows the first
  group not handed out yet (setFirstUnhanded), and when that group holds
  the next unit to write, waitForRoom lets one thread go on and take it,
  even over the limit.
- Held-tile limit by memory: threads may run ahead by 1/16 of the memory
  (physical RAM, or the cgroup v2/v1 limit if lower), at least 64 MB per
  thread as before and at most 4 GB; 64 MB per thread where the memory
  isn't known (e.g. Windows). With 64 MB per thread only, a 16-thread
  planet with --store on 64 GB spent minutes with one thread on a slow
  multipolygon and the others waiting for room: 54 min -> 25 min.
- Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed
  from them.

Planet (96 threads): about -11% wall over the rounds (e.g. batch groups
-12 s, parallel tile list -19 s).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue pushed a commit to dktue/tilemaker that referenced this pull request Nov 16, 2026
Fewer idle cores before and during the tile phase; output byte-identical
(the batches, their order and the clip cache scope are unchanged).

- Tile list: filtered in parallel column stripes (set bits found with a
  small portable helper, _BitScanForward64 on MSVC), sorted as 64-bit keys
  with sample_sort on all threads (systemed#986 v2; called from tile_sorting.cpp,
  as boost/sort in tilemaker.cpp clashes with Boost.Range on MSVC), its
  storage touched in parallel and resized on a thread of its own while the
  keys are sorted; low zooms collected in the counting pass; batch starts
  computed in parallel.
- finalizeObjects: big shards run concurrently in the pool with threads in
  proportion to their share, sort buffers reused per thread, budget raised
  to half the objects.
- Hand-out: batches of z<=8 first; consecutive batches handed out as a
  group (OrderedBatches begin/end per group, group size adapted to the
  batch count); the groups expected to be slowest (z12 object counts
  gathered while collecting) handed out up to 5% ahead; idle threads take
  over the units of a running group (tile stealing).
- No deadlock with few threads: groups handed out ahead of their place
  (z0-z8, slow groups) could fill the held-tile limit while the group to
  be written next had not been handed out, so every thread waited for room
  (seen on the planet at 16 threads). OrderedBatches now knows the first
  group not handed out yet (setFirstUnhanded), and when that group holds
  the next unit to write, waitForRoom lets one thread go on and take it,
  even over the limit.
- Held-tile limit by memory: threads may run ahead by 1/16 of the memory
  (physical RAM, or the cgroup v2/v1 limit if lower), at least 64 MB per
  thread as before and at most 4 GB; 64 MB per thread where the memory
  isn't known (e.g. Windows). With 64 MB per thread only, a 16-thread
  planet with --store on 64 GB spent minutes with one thread on a slow
  multipolygon and the others waiting for room: 54 min -> 25 min.
- Shapefiles/GeoJSON are read alongside the .pbf when no layer is indexed
  from them.
- compress_string returns a string of exactly the compressed size (it kept
  the compression bound as capacity), so held tiles take what is counted.

Planet (96 threads): about -11% wall over the rounds (e.g. batch groups
-12 s, parallel tile list -19 s).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dktue
dktue force-pushed the perf/sort-threads branch from 4dd482d to e13033f Compare November 16, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant