Skip to content

Open PRs: overlaps and a suggested merge order #963

Open

Description

@dktue

There are quite a few open PRs right now, many of them mine, and several touch the same code. This is a map of where they overlap, with a suggested order. No rush. Close this whenever it stops being useful.

Current state (15 November, master 2789651)

Merged since this issue was opened: #929, #943, #945, #949, #951, #953, #956, #961, #964. Thank you! #931 was closed by its author.

Suggested order

  1. Fix scaleGeometry dropping polygons whose ring collapses #933 (scaleRing fix). Needed since perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949: without it, master drops some polygons at z14 (details). Fix dropped polygons in scaleRing; skip distant shapefile records... #941 contains the same fix and conflicts with master; @TimSC is fine with Fix scaleGeometry dropping polygons whose ring collapses #933 going in instead.
  2. Fixes and build, independent of each other: fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944, fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946, fix: clean error instead of std::terminate on missing nodes and Lua errors (#843) #947, fix: report JSON config syntax errors as line/column instead of byte offset (#867) #948, fix(mbtiles): keep widest zoom range and merge vector_layers when using --merge #950, fix(build): only use stat64 on glibc, so musl and BSD builds work #952, fix: clean error instead of std::terminate when writing tiles fails (#824) #954, fix: keep small closed ways (roundabouts) that collapse when simplified (#315) #955, build: rename libpopcnt's popcount64 to avoid NetBSD libc clash (#821) #957, fix: write settings.metadata.attribution to directory metadata.json (#781) #958, build(cmake): link LuaJIT when it is used instead of Lua #962, fix: relation_postscan_function erases tags of non-accepted child relations #965, fix: keep clipped slices of large polygons when filtering by area (#763) #967, and Fix _minzoom handling in attribute_function #803 (by @pdrhlik; a rebased version of its shapefile part is linked there).
  3. Performance PRs, each applies to master on its own: perf: don't scan every tile coordinate when filtering z15+ tiles #959, perf: cap the mbtiles write buffer and hand out tile batches on demand #966, perf: store generated points in an AppendVector #968, perf(precheck): locate holes with a band index, validate shapefile polygons once #970, perf: set malloc top pad to 16 MB and mmap threshold to 8 MB instead of the trim threshold #971, perf: parallelise z6 finalizing and tile collection #972, perf: release the --store files from several threads at exit #973, perf(correct): find the closing point of a traced ring by hash lookup #974, perf: index big outer rings when assigning inner rings of multipolygons #975, perf: skip intersects() for rings with disjoint envelopes in correct() #976, perf(rect_clip): skip segments beyond one edge, don't load large polygons twice #978, perf(mayIntersect): check a large query box once instead of tile by tile #979 (@cldellow), perf: sort the tile list with at most 16 threads (planet -17% at 96 threads) #986, perf: publish the node/way stores' leftover groups from all threads #989, perf(pmtiles): compress the leaf directories on all cores #990, then perf: write .pmtiles and .mbtiles in batch order (byte-identical files, planet -11%) #983 (built on perf: cap the mbtiles write buffer and hand out tile batches on demand #966).
  4. perf: clip large polygons once per z8 tile and share the piece across tiles #1001 (shared z8 clip pieces), on top of everything in 3.
  5. The series perf(lua): replay way_function and node_function from a memo; native MercatorArea and MinZoomByArea #993–perf: return as soon as the output is written; huge pages for the stores #1000, on top of perf: clip large polygons once per z8 tile and share the piece across tiles #1001, in that order. Of all the performance PRs, perf(precheck): locate holes with a band index, validate shapefile polygons once #970, perf: clip large polygons once per z8 tile and share the piece across tiles #1001 and perf(lua): replay way_function and node_function from a memo; native MercatorArea and MinZoomByArea #993 change the output slightly (validity precheck, clipping from a shared piece, the bundled profile's area function); details in the comment below and in tilemaker on the whole planet: from 2 h 10 min to 3 minutes, and how you can help land it #977.
  6. Docs: docs: preloading jemalloc makes large builds 10-17% faster #987 (jemalloc). perf: return as soon as the output is written; huge pages for the stores #1000 rewrites the same paragraph of RUNNING.md, so whichever lands second needs a small rebase.

Not mine and not in the list above yet: #980 (--store relations, @geoneutrino), #984 (coastline mirror, @laem), #985 (combine_polygons_below, @geoneutrino), #988 (sparse multipolygon index, @cldellow), #991 (MinGW), #992 (pooled_string). They all apply to master; #985 and #988 conflict with #970 in test/geom.test.cpp (new tests at the same spot), and #988 also with the series in tile_data.cpp.

Where a later PR extends or replaces an earlier one

Merging in the order above keeps every step consistent: each later PR was built on top of the earlier ones.

Earlier PR Later PR What the later one changes
#986 (tile list sort with at most 16 threads) #998 replaced: 64-bit keys sorted with sample_sort on all threads
#966 (batches handed out on demand) #998 batches handed out in groups, slow groups first, idle threads take over units
#972 (parallel finalize and tile collection) #998 tile list filtered in parallel stripes; big shards finalized concurrently
#983 (ordered writer) #998, #999 OrderedBatches per group (#998); .pmtiles tile data written with direct I/O by writer threads (#999)
#989 (store finalize on all threads) #994 leftover way groups moved instead of copied
#990 (parallel .pmtiles close) #999 directory entries and leaves built in parallel too
#970 (precheck hole index) #995 further precheck rounds in the same functions
#978 (rectangle clip for huge polygons) #1001 the shared clip cache #978 mentions as "a separate, larger change"
#987 (jemalloc docs) #1000 same paragraph, now with MALLOC_CONF=thp:always,metadata_thp:always

Known conflicts

Build status

Apart from #972, the PRs in 1–3 pass CI (the macos-14 job fails everywhere because no runner picks it up). The series (#993–#1000) builds and passes the tests with CMake, Lua 5.3/5.4 and Boost up to 1.86; fixes for the Makefile before #995, LuaJIT, Boost 1.87+, Windows and macOS are being pushed. #972 fails on Windows on its own (near is a macro in windows.h, which boost/asio now pulls into polylabel.h); a fix is coming.

For trying everything at once: perf/planet-3min (the PRs in 1 and 3, #1001 and the series; not the fixes in 2).

Original map (29 October, against master 4e0f03d)

Polygon scaling and repair

These change the same code paths, so the order matters:

  1. Fix scaleGeometry dropping polygons whose ring collapses #933 scaleRing duplicate/collapsing vertex fix. Point 1 of Fix dropped polygons in scaleRing; skip distant shapefile records... #941 is the same fix (both authors agree in Fix dropped polygons in scaleRing; skip distant shapefile records... #941).
  2. Fix dropped polygons in scaleRing; skip distant shapefile records... #941 what remains after Fix scaleGeometry dropping polygons whose ring collapses #933 and perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939 (according to @TimSC, point 2 is mostly in perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939 now). Conflicts with master.
  3. fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946 drops zero-area rings on the tile grid (tilemaker generates degenerate polygons not present in source #864). Cheap, always on.
  4. Add per-layer invalid_polygon_repair_scope option to repair clipped-and-quantised polygons #929 optional repair of clipped and quantised polygons. Conflicts with master. With invalid_polygon_repair_scope: "all" it fixes most of self intersecting polygons reported when processing shapefile #403, Invalid polygons after clipping at tile boundaries #697 and the z11–12 part of Incorrect render for a large multipolygon lake #191 (numbers in my comment there).
  5. perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 exact rectangle clip (speed). Needs the scaleRing fix from Fix scaleGeometry dropping polygons whose ring collapses #933 or Fix dropped polygons in scaleRing; skip distant shapefile records... #941 first: without it, changing the clip's vertex order exposes the dropped polygons that Fix scaleGeometry dropping polygons whose ring collapses #933/Fix dropped polygons in scaleRing; skip distant shapefile records... #941 fix.
  6. perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) #961 wider validity precheck (speed). Built on top of perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949, so it contains perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949's commits.

Same fix twice

Complementary

The rest, grouped by the files they share (each is independent of the others)

Checked against master 4e0f03d

So merging in any order shouldn't cause conflicts between these. Happy to help rebase #941 and #929 if their authors would like that.

Activity

  1. dktue commented on Oct 29, 2026

    @dktue
    ContributorAuthor

    Two additions:

  2. dktue commented on Oct 29, 2026

    @dktue
    ContributorAuthor

    One addition:

  3. TimSC commented on Oct 29, 2026

    @TimSC
    Contributor

    I'm not fussed if mine don't make it. I contributed a while back.

  4. geoneutrino commented on Oct 30, 2026

    @geoneutrino
    Contributor

    just to have it here too:
    #929 was rebased to master no more conflicts
    #931 closed (no longer measurable difference)

  5. geoneutrino commented on Oct 30, 2026

    @geoneutrino
    Contributor

    Checked #929 against all open PRs now that it merges cleanly with master (4e0f03d):

    Happy to rebase whichever of the two lands second.

  6. dktue commented on Oct 31, 2026

    @dktue
    ContributorAuthor

    Update after today's merges (master is now bfe7a2f):

    Merged

    #943, #945, #949, #951, #953, #956, #964, later #961, and on 3 Nov #929. Thank you, @systemed! #931 was closed by its author.

    scaleRing fix still needed

    #949 went in before #933 / #941, so master currently drops some polygons at z14 (details in #949 (comment)). #933 is the same fix, applies cleanly to current master, and is the simplest way to close the gap. #941 conflicts, and @TimSC said above he doesn't mind if his doesn't go in. So #933 next would be my suggestion.

    Needs a rebase

    Rebased

    #970 and #976 are rebased onto the new master; after #961 was merged, #970 again, so it is now a single commit.

    New since the map above

    PR What Files it shares with other open PRs
    #966 tile batches handed out on demand, capped mbtiles write buffer mbtiles.cpp, tilemaker.cpp: conflicts with #954 (as noted above)
    #968 generated points in an AppendVector tile_data.h (#972)
    #970 hole index for the validity precheck (on top of #961) geom.cpp (#961)
    #971 malloc top pad instead of trim threshold tilemaker.cpp
    #972 parallel z6 finalizing and tile collection tile_data.*, tile_coordinates_set.* (#959)
    #973 release --store files from several threads at exit mmap_allocator.cpp
    #974 hash lookup for the ring closing point in correct() correct.hpp (#976; both apply together)
    #975 indexed outer rings when assigning inner rings osm_store.cpp
    #976 skip intersects() for disjoint ring envelopes in correct() correct.hpp (#974), Makefile
    #979 FindIntersecting() / Intersects(): check a large query box once instead of tile by tile, by cldellow Makefile (#957, #970, #976); applies together with all of them
    #978 fixes a regression from #949 (in master): profiles with huge polygons 24% faster again geom.cpp, tile_data.cpp
    #980 --store: relations were read several times when relation blocks were subdivided, by geoneutrino pbf_processor.cpp; applies together with all of them
    #981 tile content independent of thread scheduling, by geoneutrino one line in tilemaker.cpp's tile loop overlaps with #966 (and so #983) and #954
    #983 write .pmtiles/.mbtiles in batch order: byte-identical files, planet −11%; since 2026-11-08 also wakes waiting threads only once there is room (96 threads: another −9.5%) built on #966 (current master with #966 merged in), so it overlaps where #966 does
    #984 get-coastline.sh falls back to a mirror while osmdata.openstreetmap.de is down (currently the case), by laem get-coastline.sh only; applies together with all of them
    #985 keep polygons that Boost's union_ drops in combine_polygons_below, by geoneutrino geom.cpp (#970, #978): only the new tests in test/geom.test.cpp overlap with #970, at the same spot; the code applies together with both
    #986 sort the tile list with at most 16 threads: block_indirect_sort took 200–300 s at 96 threads, ~7 s at 16 (planet −17%) tile_sorting.cpp (#981, at different places); applies together with all of them
    #989 node/way stores publish their leftover groups from all threads at the end of a pass (planet −9 s) sorted_node_store.cpp, sorted_way_store.cpp only; applies together with all of them
    #990 .pmtiles leaf directories compressed on all cores in close() (9 s → 3 s, byte-identical) pmtiles.cpp/.h: one small conflict with #983 (the removed flushEntries sits next to lines #983 changes); keep #990's side

    Byte-identical output needs four of these together: #981 (tile content), #942 (POI key order in the OpenMapTiles profile), #983 (file layout) and #966, which #983 builds on. With all four, two planet runs give the same .pmtiles file (details in #969, #981 and #983).

    Checked against 2789651 (master with #929)

    Every open PR in this map applies cleanly to the new master, except #941 above. (This time I applied each PR's diff as a patch; #803 and #889 can't be judged that way, because their diffs didn't apply as plain patches against the old master either, while the merge check above passed.)

    For testing everything at once, dktue/tilemaker → perf/all-open-prs has all open performance PRs merged together (still on the old master 4e0f03d; numbers in #977).

  7. dktue commented on Nov 15, 2026

    @dktue
    ContributorAuthor

    New: a series on top of all open PRs (#993–#1000)

    These eight PRs are one stacked series. Each is one commit on top of the previous one, and the first sits on perf/shared-pieces, which is master plus all open performance PRs above (and #981, #942, #983, #986) plus the shared z8 clip pieces (#1001). #989 and #990 come on top as a plain branch (perf/planet-3min-00-store-finalize-pmtiles-close), adapted to that base. So they come after everything in this list, in this order:

    PR What Output
    #993 Lua memo; native MercatorArea()/MinZoomByArea(); bundled profile changes (borderline minimum zooms)
    #994 reading phase identical
    #995 polygon validity precheck, make_valid identical
    #996 clipping, simplification, labels identical
    #997 MVT encoding identical
    #998 tile-phase scheduling identical
    #999 PMTiles writer, compression_level identical
    #1000 early exit, huge pages identical

    Each one builds with CMake and passes the unit tests on its own on Linux (GCC, Boost 1.83, Lua 5.4). CI shows that some don't build yet with the Makefile at #993/#994 (still C++14 there), with LuaJIT, with Boost 1.87+, on macOS and on Windows; fixes are being prepared and will be pushed into the same commits. #997 also moves writeRing, so it conflicts with #946: whichever lands second has to carry #946's zero-area check. Planet: 9:54 → 3:08 for the whole series (details in #977). Everything combined is on perf/planet-3min.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions