Repository navigation
Description
There are quite a few open PRs right now, many of them mine, and several touch the same code. This is a map of where they overlap, with a suggested order. No rush. Close this whenever it stops being useful.
Current state (15 November, master 2789651)
Merged since this issue was opened: #929, #943, #945, #949, #951, #953, #956, #961, #964. Thank you! #931 was closed by its author.
Suggested order
- Fix scaleGeometry dropping polygons whose ring collapses #933 (
scaleRingfix). Needed since perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949: without it, master drops some polygons at z14 (details). Fix dropped polygons in scaleRing; skip distant shapefile records... #941 contains the same fix and conflicts with master; @TimSC is fine with Fix scaleGeometry dropping polygons whose ring collapses #933 going in instead. - Fixes and build, independent of each other: fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944, fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946, fix: clean error instead of std::terminate on missing nodes and Lua errors (#843) #947, fix: report JSON config syntax errors as line/column instead of byte offset (#867) #948, fix(mbtiles): keep widest zoom range and merge vector_layers when using --merge #950, fix(build): only use stat64 on glibc, so musl and BSD builds work #952, fix: clean error instead of std::terminate when writing tiles fails (#824) #954, fix: keep small closed ways (roundabouts) that collapse when simplified (#315) #955, build: rename libpopcnt's popcount64 to avoid NetBSD libc clash (#821) #957, fix: write settings.metadata.attribution to directory metadata.json (#781) #958, build(cmake): link LuaJIT when it is used instead of Lua #962, fix: relation_postscan_function erases tags of non-accepted child relations #965, fix: keep clipped slices of large polygons when filtering by area (#763) #967, and Fix _minzoom handling in attribute_function #803 (by @pdrhlik; a rebased version of its shapefile part is linked there).
- Performance PRs, each applies to master on its own: perf: don't scan every tile coordinate when filtering z15+ tiles #959, perf: cap the mbtiles write buffer and hand out tile batches on demand #966, perf: store generated points in an AppendVector #968, perf(precheck): locate holes with a band index, validate shapefile polygons once #970, perf: set malloc top pad to 16 MB and mmap threshold to 8 MB instead of the trim threshold #971, perf: parallelise z6 finalizing and tile collection #972, perf: release the --store files from several threads at exit #973, perf(correct): find the closing point of a traced ring by hash lookup #974, perf: index big outer rings when assigning inner rings of multipolygons #975, perf: skip intersects() for rings with disjoint envelopes in correct() #976, perf(rect_clip): skip segments beyond one edge, don't load large polygons twice #978, perf(mayIntersect): check a large query box once instead of tile by tile #979 (@cldellow), perf: sort the tile list with at most 16 threads (planet -17% at 96 threads) #986, perf: publish the node/way stores' leftover groups from all threads #989, perf(pmtiles): compress the leaf directories on all cores #990, then perf: write .pmtiles and .mbtiles in batch order (byte-identical files, planet -11%) #983 (built on perf: cap the mbtiles write buffer and hand out tile batches on demand #966).
- For byte-identical output between runs: Make tile content independent of thread scheduling #981 (@geoneutrino), Make POI tag key processing deterministic #942 (@TimSC) and perf: write .pmtiles and .mbtiles in batch order (byte-identical files, planet -11%) #983.
- perf: clip large polygons once per z8 tile and share the piece across tiles #1001 (shared z8 clip pieces), on top of everything in 3.
- The series perf(lua): replay way_function and node_function from a memo; native MercatorArea and MinZoomByArea #993–perf: return as soon as the output is written; huge pages for the stores #1000, on top of perf: clip large polygons once per z8 tile and share the piece across tiles #1001, in that order. Of all the performance PRs, perf(precheck): locate holes with a band index, validate shapefile polygons once #970, perf: clip large polygons once per z8 tile and share the piece across tiles #1001 and perf(lua): replay way_function and node_function from a memo; native MercatorArea and MinZoomByArea #993 change the output slightly (validity precheck, clipping from a shared piece, the bundled profile's area function); details in the comment below and in tilemaker on the whole planet: from 2 h 10 min to 3 minutes, and how you can help land it #977.
- Docs: docs: preloading jemalloc makes large builds 10-17% faster #987 (jemalloc). perf: return as soon as the output is written; huge pages for the stores #1000 rewrites the same paragraph of
RUNNING.md, so whichever lands second needs a small rebase.
Not mine and not in the list above yet: #980 (--store relations, @geoneutrino), #984 (coastline mirror, @laem), #985 (combine_polygons_below, @geoneutrino), #988 (sparse multipolygon index, @cldellow), #991 (MinGW), #992 (pooled_string). They all apply to master; #985 and #988 conflict with #970 in test/geom.test.cpp (new tests at the same spot), and #988 also with the series in tile_data.cpp.
Where a later PR extends or replaces an earlier one
Merging in the order above keeps every step consistent: each later PR was built on top of the earlier ones.
| Earlier PR | Later PR | What the later one changes |
|---|---|---|
| #986 (tile list sort with at most 16 threads) | #998 | replaced: 64-bit keys sorted with sample_sort on all threads |
| #966 (batches handed out on demand) | #998 | batches handed out in groups, slow groups first, idle threads take over units |
| #972 (parallel finalize and tile collection) | #998 | tile list filtered in parallel stripes; big shards finalized concurrently |
| #983 (ordered writer) | #998, #999 | OrderedBatches per group (#998); .pmtiles tile data written with direct I/O by writer threads (#999) |
| #989 (store finalize on all threads) | #994 | leftover way groups moved instead of copied |
#990 (parallel .pmtiles close) |
#999 | directory entries and leaves built in parallel too |
| #970 (precheck hole index) | #995 | further precheck rounds in the same functions |
| #978 (rectangle clip for huge polygons) | #1001 | the shared clip cache #978 mentions as "a separate, larger change" |
| #987 (jemalloc docs) | #1000 | same paragraph, now with MALLOC_CONF=thp:always,metadata_thp:always |
Known conflicts
- fix: clean error instead of std::terminate when writing tiles fails (#824) #954 with perf: cap the mbtiles write buffer and hand out tile batches on demand #966 (and so perf: write .pmtiles and .mbtiles in batch order (byte-identical files, planet -11%) #983, perf: clip large polygons once per z8 tile and share the piece across tiles #1001 and the series):
mbtiles.cpp,tilemaker.cpp. - perf(pmtiles): compress the leaf directories on all cores #990 with perf: write .pmtiles and .mbtiles in batch order (byte-identical files, planet -11%) #983:
pmtiles.cpp, one small spot next to the removedflushEntries; keep perf(pmtiles): compress the leaf directories on all cores #990's side. - fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946 with perf: faster MVT encoding and per-tile sorting #997: perf: faster MVT encoding and per-tile sorting #997 moves
writeRingtoinclude/tile_encode.h; fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946's zero-area check has to move with it. - docs: preloading jemalloc makes large builds 10-17% faster #987 with perf: return as soon as the output is written; huge pages for the stores #1000:
docs/RUNNING.md, same paragraph. - Fix dropped polygons in scaleRing; skip distant shapefile records... #941 with master.
Build status
Apart from #972, the PRs in 1–3 pass CI (the macos-14 job fails everywhere because no runner picks it up). The series (#993–#1000) builds and passes the tests with CMake, Lua 5.3/5.4 and Boost up to 1.86; fixes for the Makefile before #995, LuaJIT, Boost 1.87+, Windows and macOS are being pushed. #972 fails on Windows on its own (near is a macro in windows.h, which boost/asio now pulls into polylabel.h); a fix is coming.
For trying everything at once: perf/planet-3min (the PRs in 1 and 3, #1001 and the series; not the fixes in 2).
Original map (29 October, against master 4e0f03d)
Polygon scaling and repair
These change the same code paths, so the order matters:
- Fix scaleGeometry dropping polygons whose ring collapses #933
scaleRingduplicate/collapsing vertex fix. Point 1 of Fix dropped polygons in scaleRing; skip distant shapefile records... #941 is the same fix (both authors agree in Fix dropped polygons in scaleRing; skip distant shapefile records... #941). - Fix dropped polygons in scaleRing; skip distant shapefile records... #941 what remains after Fix scaleGeometry dropping polygons whose ring collapses #933 and perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939 (according to @TimSC, point 2 is mostly in perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939 now). Conflicts with master.
- fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946 drops zero-area rings on the tile grid (tilemaker generates degenerate polygons not present in source #864). Cheap, always on.
- Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929 optional repair of clipped and quantised polygons. Conflicts with master. With
invalid_polygon_repair_scope: "all"it fixes most of self intersecting polygons reported when processing shapefile #403, Invalid polygons after clipping at tile boundaries #697 and the z11–12 part of Incorrect render for a large multipolygon lake #191 (numbers in my comment there). - perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 exact rectangle clip (speed). Needs the
scaleRingfix from Fix scaleGeometry dropping polygons whose ring collapses #933 or Fix dropped polygons in scaleRing; skip distant shapefile records... #941 first: without it, changing the clip's vertex order exposes the dropped polygons that Fix scaleGeometry dropping polygons whose ring collapses #933/Fix dropped polygons in scaleRing; skip distant shapefile records... #941 fix. - perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) #961 wider validity precheck (speed). Built on top of perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949, so it contains perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949's commits.
Same fix twice
- fix build on non-glibc Linux #808 / fix(build): only use stat64 on glibc, so musl and BSD builds work #952 stat64 on non-glibc. fix build on non-glibc Linux #808 as written also redefines
stat64on MSVC (details in fix build on non-glibc Linux #808). fix(build): only use stat64 on glibc, so musl and BSD builds work #952 is the same fix plus an MSVC guard. - Make POI tag ranking deterministic and rank-aware #889 / Make POI tag key processing deterministic #942 deterministic POI class. Both fix the non-determinism.
- Make POI tag key processing deterministic #942 derives the key order from
poiTagsand applies cleanly. - Make POI tag ranking deterministic and rank-aware #889 picks the best OpenMapTiles rank, which is nicer, but it is based on an older profile: its key list lacks
office, and it would revert thepoiTagsAsClasslookup. - Combining Make POI tag key processing deterministic #942's key order with Make POI tag ranking deterministic and rank-aware #889's lowest-rank choice would give both.
- Make POI tag key processing deterministic #942 derives the key order from
Complementary
- feat: add standard attribution to config-openmaptiles #782 + fix: write settings.metadata.attribution to directory metadata.json (#781) #958 attribution: feat: add standard attribution to config-openmaptiles #782 adds it to the shipped config, fix: write settings.metadata.attribution to directory metadata.json (#781) #958 writes it to directory-output
metadata.json.
The rest, grouped by the files they share (each is independent of the others)
- Error handling,
tilemaker.cpp: fix: clean error instead of std::terminate on missing nodes and Lua errors (#843) #947, fix: clean error instead of std::terminate when writing tiles fails (#824) #954, fix: report JSON config syntax errors as line/column instead of byte offset (#867) #948 - Metadata,
shared_data.cpp/mbtiles.cpp: fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944, fix(mbtiles): keep widest zoom range and merge vector_layers when using --merge #950, fix: write settings.metadata.attribution to directory metadata.json (#781) #958 - Store and allocator,
mmap_allocator.cpp: fix/perf(store): output matches in-memory, 12x faster when RAM is tight, 67x less disk writing, 3.2% less CPU #943, fix: reuse allocator regions of exited threads; error on more than 256 layers (#714) #956 - Tile output,
tile_worker.cpp: Avoid redundant work when writing features #931 (small, verified byte-identical by its author), fix: keep small closed ways (roundabouts) that collapse when simplified (#315) #955 (fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946 and Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929 above touch it too) geom.cpp: perf: much faster combine_polygons_below (#868) #945 (Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929 and perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 above touch it too)- Lua processing,
osm_lua_processing.cpp: fix: relation_postscan_function erases tags of non-accepted child relations #965 (fix: clean error instead of std::terminate on missing nodes and Lua errors (#843) #947, perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 and perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) #961 touch it too) - Build and portability: fix(build): only use stat64 on glibc, so musl and BSD builds work #952, build(cmake): use C++17 unless MLT is enabled; document C++ standards #953, build: rename libpopcnt's popcount64 to avoid NetBSD libc clash (#821) #957, build(cmake): link LuaJIT when it is used instead of Lua #962
kaguya.hpp: fix: hang when attribute_function returns an integer (C++20 regression from #938) #964 (hang in C++20 builds whenattribute_functionreturns an integer)- Standalone: fix(pbf): clear error instead of crash on negative or too-large node IDs #951, perf: don't scan every tile coordinate when filtering z15+ tiles #959
Checked against master 4e0f03d
- Apart from Fix dropped polygons in scaleRing; skip distant shapefile records... #941 and Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929, which conflict with master, every pair of the PRs above merges without conflicts. The only exceptions are the alternatives fix build on non-glibc Linux #808 / fix(build): only use stat64 on glibc, so musl and BSD builds work #952 and Make POI tag ranking deterministic and rank-aware #889 / Make POI tag key processing deterministic #942.
- They also merge cleanly all together, in this order: Fix scaleGeometry dropping polygons whose ring collapses #933 fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946 perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) #961 perf: much faster combine_polygons_below (#868) #945 fix: keep small closed ways (roundabouts) that collapse when simplified (#315) #955 Avoid redundant work when writing features #931 fix/perf(store): output matches in-memory, 12x faster when RAM is tight, 67x less disk writing, 3.2% less CPU #943 fix: reuse allocator regions of exited threads; error on more than 256 layers (#714) #956 fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944 fix(mbtiles): keep widest zoom range and merge vector_layers when using --merge #950 fix: write settings.metadata.attribution to directory metadata.json (#781) #958 feat: add standard attribution to config-openmaptiles #782 fix: clean error instead of std::terminate on missing nodes and Lua errors (#843) #947 fix: report JSON config syntax errors as line/column instead of byte offset (#867) #948 fix: clean error instead of std::terminate when writing tiles fails (#824) #954 fix(pbf): clear error instead of crash on negative or too-large node IDs #951 fix(build): only use stat64 on glibc, so musl and BSD builds work #952 build(cmake): use C++17 unless MLT is enabled; document C++ standards #953 build: rename libpopcnt's popcount64 to avoid NetBSD libc clash (#821) #957 build(cmake): link LuaJIT when it is used instead of Lua #962 fix: hang when attribute_function returns an integer (C++20 regression from #938) #964 perf: don't scan every tile coordinate when filtering z15+ tiles #959 fix: relation_postscan_function erases tags of non-accepted child relations #965 Make POI tag key processing deterministic #942 Fix residential area filtering by size #928.
- The combined tree builds without compiler warnings, passes the unit tests, and renders Canary Islands with the OpenMapTiles profile at the same speed and memory as master.
- It writes 15k fewer tiles than master. All of them are outside the extract: master reads the PBF bounding box's east edge as 0°, which is what fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944 fixes.
- Other feature counts differ only slightly, as expected from Fix residential area filtering by size #928 and fix: keep small closed ways (roundabouts) that collapse when simplified (#315) #955.
- Update: perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) #961, build(cmake): link LuaJIT when it is used instead of Lua #962, fix: hang when attribute_function returns an integer (C++20 regression from #938) #964 and fix: relation_postscan_function erases tags of non-accepted child relations #965 added to all of the checks above. With them, the Canary Islands output is identical to the combination without them (same tiles, same features per layer).
So merging in any order shouldn't cause conflicts between these. Happy to help rebase #941 and #929 if their authors would like that.
Activity
dktue commentedon Oct 29, 2026
Two additions:
attribute_functionreturning an integer recursed forever in kaguya'soperator==. It touches onlykaguya.hppand merges cleanly with everything above._minzoom, Setting _minzoom in attribute_function() doesn't work per feature #774) is still needed. A rebased version of its shapefile fix, tested with fix: hang when attribute_function returns an integer (C++20 regression from #938) #964, is linked in Fix _minzoom handling in attribute_function #803.dktue commentedon Oct 29, 2026
One addition:
filter_areakeeps clipped slices of large polygons at tile edges (Better area filtering #763, option 3 as agreed there). It touches onlyRemovePartsBelowSizeand its call intile_worker.cpp.tile_worker.cppdon't overlap with it.TimSC commentedon Oct 29, 2026
I'm not fussed if mine don't make it. I contributed a while back.
geoneutrino commentedon Oct 30, 2026
just to have it here too:
#929 was rebased to master no more conflicts
#931 closed (no longer measurable difference)
geoneutrino commentedon Oct 30, 2026
Checked #929 against all open PRs now that it merges cleanly with master (4e0f03d):
mbtiles.cpp,tilemaker.cpp), unrelated to Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929.repair_multi_polygon(). The resolution keeps thebool *wholeValidthird and puts Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929's parameter last:repair_multi_polygon(MultiPolygon &mp, bool knownInvalid = false, bool *wholeValid = nullptr, AreaBound areaBound = AreaBound::Lenient), with Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929's call becomingrepair_multi_polygon(current, true, nullptr, simplifyLevel == 0 ? AreaBound::Strict : AreaBound::Lenient). I've checked that this builds. I also changed Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929's parameter from aboolto a scoped enumAreaBound, so the other order fails to compile (cannot convert 'bool*' to 'AreaBound') instead of silently passing the pointer as a bool.Happy to rebase whichever of the two lands second.
dktue commentedon Oct 31, 2026
Update after today's merges (master is now
bfe7a2f):Merged
#943, #945, #949, #951, #953, #956, #964, later #961, and on 3 Nov #929. Thank you, @systemed! #931 was closed by its author.
scaleRingfix still needed#949 went in before #933 / #941, so master currently drops some polygons at z14 (details in #949 (comment)). #933 is the same fix, applies cleanly to current master, and is the simplest way to close the gap. #941 conflicts, and @TimSC said above he doesn't mind if his doesn't go in. So #933 next would be my suggestion.
Needs a rebase
Rebased
#970 and #976 are rebased onto the new master; after #961 was merged, #970 again, so it is now a single commit.
New since the map above
mbtiles.cpp,tilemaker.cpp: conflicts with #954 (as noted above)AppendVectortile_data.h(#972)geom.cpp(#961)tilemaker.cpptile_data.*,tile_coordinates_set.*(#959)--storefiles from several threads at exitmmap_allocator.cppcorrect()correct.hpp(#976; both apply together)osm_store.cppintersects()for disjoint ring envelopes incorrect()correct.hpp(#974),MakefileFindIntersecting()/Intersects(): check a large query box once instead of tile by tile, by cldellowMakefile(#957, #970, #976); applies together with all of themgeom.cpp,tile_data.cpp--store: relations were read several times when relation blocks were subdivided, by geoneutrinopbf_processor.cpp; applies together with all of themtilemaker.cpp's tile loop overlaps with #966 (and so #983) and #954.pmtiles/.mbtilesin batch order: byte-identical files, planet −11%; since 2026-11-08 also wakes waiting threads only once there is room (96 threads: another −9.5%)get-coastline.shfalls back to a mirror while osmdata.openstreetmap.de is down (currently the case), by laemget-coastline.shonly; applies together with all of themunion_drops incombine_polygons_below, by geoneutrinogeom.cpp(#970, #978): only the new tests intest/geom.test.cppoverlap with #970, at the same spot; the code applies together with bothblock_indirect_sorttook 200–300 s at 96 threads, ~7 s at 16 (planet −17%)tile_sorting.cpp(#981, at different places); applies together with all of themsorted_node_store.cpp,sorted_way_store.cpponly; applies together with all of them.pmtilesleaf directories compressed on all cores inclose()(9 s → 3 s, byte-identical)pmtiles.cpp/.h: one small conflict with #983 (the removedflushEntriessits next to lines #983 changes); keep #990's sideByte-identical output needs four of these together: #981 (tile content), #942 (POI key order in the OpenMapTiles profile), #983 (file layout) and #966, which #983 builds on. With all four, two planet runs give the same
.pmtilesfile (details in #969, #981 and #983).Checked against
2789651(master with #929)Every open PR in this map applies cleanly to the new master, except #941 above. (This time I applied each PR's diff as a patch; #803 and #889 can't be judged that way, because their diffs didn't apply as plain patches against the old master either, while the merge check above passed.)
For testing everything at once,
dktue/tilemaker→perf/all-open-prshas all open performance PRs merged together (still on the old master4e0f03d; numbers in #977).dktue commentedon Nov 15, 2026
New: a series on top of all open PRs (#993–#1000)
These eight PRs are one stacked series. Each is one commit on top of the previous one, and the first sits on
perf/shared-pieces, which is master plus all open performance PRs above (and #981, #942, #983, #986) plus the shared z8 clip pieces (#1001). #989 and #990 come on top as a plain branch (perf/planet-3min-00-store-finalize-pmtiles-close), adapted to that base. So they come after everything in this list, in this order:MercatorArea()/MinZoomByArea(); bundled profilemake_validcompression_levelEach one builds with CMake and passes the unit tests on its own on Linux (GCC, Boost 1.83, Lua 5.4). CI shows that some don't build yet with the Makefile at #993/#994 (still C++14 there), with LuaJIT, with Boost 1.87+, on macOS and on Windows; fixes are being prepared and will be pushed into the same commits. #997 also moves
writeRing, so it conflicts with #946: whichever lands second has to carry #946's zero-area check. Planet: 9:54 → 3:08 for the whole series (details in #977). Everything combined is onperf/planet-3min.