Skip to content

perf: clip large polygons once per z8 tile and share the piece across tiles - #1001

Open
dktue wants to merge 8 commits into
systemed:masterfrom
dktue:perf/shared-pieces
Open

dktue wants to merge 8 commits into
systemed:masterfrom
dktue:perf/shared-pieces

Conversation

@dktue

@dktue dktue commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Gain: planet (in memory, 96 threads) 12:45 → 10:55 (−14.4%) on top of all open performance PRs + the #983 wake-up fix; user CPU −27%. From master (25:17): −57%. Peak RSS +7.5 GB.

Stacked on perf/all-open-prs-full (master + all open performance PRs, #933, #981, #942, #983, #986), because it uses the batch hand-out of #966 and the ordered writer of #983, so the diff on GitHub also shows those; review the last two commits (632dbe2, fc85235). The other commits are the open PRs: 3ca0fbe (a one-commit snapshot of them, not for merging), 083cd6f (#986), b923bb5 and 3660c20 (#983's fourth and fifth commits). It can be rebased onto master once those are in.

The series #993–#1000 is built on this branch, which is why it is a PR now (see #963 for the order).

Problem

Every tile batch that needs a large polygon copies it from the store and clips it from scratch, at every zoom. The batch clip cache (#939) only helps within one batch. On the planet, polygons with more than 10,000 points (about 9,300 of them) cost about 40% of the tile phase.

Change

Polygons with at least 10,000 points are clipped once per z8 tile, and every tile below that z8 tile clips from this piece instead of the whole polygon.

  • Computed up front. All pieces are computed before the tile phase, with all threads, largest polygon first. Candidates are the polygons in the large-object index. Only z8 tiles that have a written tile below them, and that the polygon's envelope reaches, get a piece. Computed on demand instead, a slow piece (up to ~15 s on the planet) held up every thread whose tiles needed it. On a 96-thread planet run that was a third of the tile phase. The pre-pass also takes over the validity the clip has just determined, instead of checking huge pieces again (is_valid takes seconds on a fjord coast).
  • Computed by the tile threads (second commit). The pre-pass ended with one thread on the largest piece (~6 s on the planet) while all others waited. Now the tile threads compute the pieces before their first tile, largest polygon first, and go on to tiles once no piece is left to start; a tile that needs a piece still being computed waits for just that one. The set of pieces is known before the tiles start, so tiles still look them up without a lock.
  • No locks while tiles are written. After the pre-pass the pieces are only read. Once the batch hand-out has passed a z6 tile and none of its batches is still running, its pieces are reset to free the memory.
  • Deterministic. A piece depends only on the polygon and the z8 tile, never on the thread or the order. Output is byte-identical for any thread count (Germany 32 vs 8 threads). It is not byte-identical to clipping the whole polygon, though; see Correctness.
  • Pieces that needed repair are not used. If clipping a piece went down the repair path (Boost intersection / repair), the piece isn't used and the tiles below clip the whole polygon as before. In an earlier version one such piece came out empty and a lake disappeared from all its child tiles.

Correctness

  • Germany: byte-identical at 32 and 8 threads.
  • Planet sample (all tiles z0–z8 + 30,000 random tiles per zoom z9–z14), compared with the same stack without pieces:
    • same feature counts in every layer;
    • area: landcover −0.003%, water +0.0005%, from clipping the piece instead of the whole polygon;
    • invalid polygons: landcover 44 → 46, water 17 → 16.
  • A third commit fixes the build of the combined base on Windows and macOS: it renames a near parameter in polylabel.h (a macro in windows.h, which boost/asio now pulls in) and includes shp_mem_tiles.h before minunit.h in its test (minunit sets _POSIX_C_SOURCE, which hides ip_mreq from boost/asio on macOS). A fourth commit fixes a crash at exit on macOS, where the main thread's thread-locals are destroyed after the statics: the allocator's region lists now live until the end, and a store lease goes back only to a TileDataSource that is still alive. CI passes on Linux, macOS and Windows.

Measurements

Planet in memory, 96 threads, tm-perf resources, same day:

wall user CPU sys peak RSS
all open PRs + wake-up fix 12:45 58,179 s 2,868 s 160.9 GB
+ shared pieces 10:55 42,527 s 2,854 s 168.4 GB

🤖 Generated with Claude Code

dktue and others added 6 commits November 7, 2026 14:48
systemed#983 (combined snapshot)

For testing only, not for merging. One commit with the combined tree that
built the planet in 14 min 03 s (in memory, 96 threads) for systemed#977:

master 2789651 + systemed#933, systemed#959, systemed#966, systemed#968, systemed#970, systemed#971, systemed#972, systemed#973, systemed#974, systemed#975,
systemed#976, systemed#978, systemed#979 (open performance PRs), systemed#981 (deterministic tile content),
systemed#942 (deterministic POI order) and systemed#983 (ordered .pmtiles/.mbtiles writer,
64 MB per thread). Each change is reviewed in its own PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Same change as systemed#986. With it, the sort before the tile phase takes ~13 s
instead of 2-5 minutes at 96 threads, so planet times no longer vary by
minutes between runs (14:12 / 14:08 in memory at 96 threads).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
end() woke every thread waiting in waitForRoom() after each batch it wrote,
although most of the time the held tiles were still over the limit. At 96
threads that meant dozens of threads waking, finding no room and sleeping
again for every batch: 125M context switches and 7,700 s of system time on
the planet. Now they are only woken once the held tiles are back under the
limit. Planet in memory, 96 threads, all open performance PRs:
14:05 -> 12:45 (-9.5%), system time 7,708 -> 2,868 s, context switches
125M -> 35M. Output unchanged (same batches, same order).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every tile batch that needs a large polygon copied it from the store and
clipped it from scratch, at every zoom. Now polygons with at least 10,000
points are clipped once per z8 tile, before the tiles are written (all
threads, largest polygon first), and tiles below clip from that piece.
After that the pieces are only read, so there is no lock while tiles are
written; a z6 tile's pieces are freed once no tile in it is still to come.

A piece only depends on the polygon and the z8 tile, so the output stays
the same for any thread count. A piece whose clip needed the repair path is
not used: once one came out empty and a lake vanished from all tiles below.

Planet in memory, 96 threads, all open performance PRs + systemed#983's wake-up
fix: 12:45 -> 10:55 (-14.4%), user CPU -27%, peak RSS +7.5 GB.
Germany byte-identical at 32 and 8 threads; planet sample: same feature
counts, area within 0.003%.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
At 96 threads the one thread that writes batches in order became the
bottleneck of the tile phase: busy 260 s of a 323 s tile phase, with the
others waiting for room for 5,100 thread-seconds. Writing the bytes was
more than half of its work (ofstream::write, and a tellp() per tile).

- The writer keeps its own tile data offset instead of calling tellp() for
  every tile, and the stream gets a 4 MB buffer.
- OrderedBatches can take a second step, flushItem: also in batch order and
  one thread at a time, but run by whichever thread is free, so it doesn't
  hold up the first step. PMTiles places tiles (offset, repeated tiny tiles,
  index) in the first step and writes their bytes in the second.

The file is byte-identical. Planet in memory, 96 threads, all open
performance PRs: 10:41 -> 9:54 (-7.3%); tile phase 323 -> 245 s, threads
busy 76 -> 94 of 96 on average. New test: the second step runs in order,
never twice at once, and always after the first (also clean under TSan).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pieces used to be computed in a pass of their own before the tile phase,
which ended with one thread on the largest piece (~6 s on the planet) while
all others waited. Now the tile threads compute them before their first
tile, largest polygon first, and go on to tiles once no piece is left to
start; a tile that needs a piece still being computed waits for just that
one (shared_future). A piece is known before the tiles start, so the map
doesn't change and tiles look pieces up without a lock as before.

Output byte-identical (Germany at 32 and 8 threads).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- polylabel.h: rename edgeDistBound's parameter `near` to `nearCell`.
  tile_data.h includes boost/asio since systemed#972, which pulls in windows.h on
  Windows, and windows.h defines `near` as an empty macro.
- test/shp_mem_tiles.test.cpp (systemed#979): include shp_mem_tiles.h before
  minunit.h. minunit.h sets _POSIX_C_SOURCE, which on macOS hides ip_mreq,
  and boost/asio (through tile_data.h) needs it.

Renames and include order only; output unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…uctors have run (macOS)

On macOS the main thread's thread-local destructors run after the static
destructors. Two of them touched objects that were already gone, and the
unit tests aborted at exit with "mutex lock failed: Invalid argument":

- mmap_allocator.cpp: thread_regions_t's destructor locks
  mmap_allocator_mutex and appends to free_mmap_files /
  free_mmap_shm_regions. These three are now allocated once and never
  destroyed. ~mmap_dir_t clears free_mmap_files first, so --store files
  are still released and deleted as before (that list used to be destroyed
  just before mmap_dir).
- LeasedStore (tile_data.cpp): a thread gives its store leases back to
  the TileDataSource when it ends; the main thread does that at exit, after
  the sources (locals of main(), or of a unit test such as
  test_shp_mem_tiles) are destroyed, locking a destroyed storeMutex. The
  sources are now registered while alive (an immortal registry), and a
  lease goes back only to a live source. Sources are also numbered: a new
  source at the address of a destroyed one no longer gets that one's
  lease.

No change in output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant