Show installs down to city level on the /stats globe - #144
Merged
Merged
Conversation
Region labels stopped at state level, and at 55 located installs no state
reached the k>=5 floor, so every install fell into "Other" and the /stats
globe had nothing to plot. Labels now carry the city ('US-CA/San Jose'),
giving the publication floor a finer level to try first.
City-level tables are ~3.4 M IPv4 and ~3.8 M IPv6 ranges -- about 110 MB
as typed arrays, which does not fit beside the server on the 256 MB VM.
ip-geo.js therefore no longer loads the dataset: it binary-searches the
sorted CSVs in place with positioned 4 KiB reads (~35 us per lookup once
cached) and holds one 64 KiB buffer per file. The server also sheds the
~66 MB the state-level tables used and the first-request load pause.
The refresh script emits a city column and a places file mapping every
country, state and city label to an approximate centroid (spherical
mean, 0.1 degree), keyed by the same regionLabel() the ingest route
stores, now shared from utils/region-label.js. Four-column state-level
rows still parse. On the 2026-09 release, 60,000 sampled addresses
resolved to the same country and state as the old in-memory lookup.
fly.toml points at the new files under /data/dbip-city/.
The floor used to fold every sub-5 state straight into "Other". It now walks city -> state -> country: a label with at least five installs is published, and the installs of one below the floor move up to its parent, so two Californian cities of 3 and 2 publish as 'US-CA' 5. Countries still short pool into "Other", which is dropped under five. Each install lands in exactly one bucket, so no published label ever stands for fewer than five installs. On today's production data this turns [Other 55] into US 14, JP 6, AU 5, IN 5 and Other 25, and cities appear as installs report from them. Public entries also carry the place's centroid from the places file so the globe can plot cities it has no built-in table for; 'Other' and 'unknown' never do.
The globe plots each published place at the server's centroid, falling back to its own country/state tables, with a tighter scatter for a city than for a state or country. City labels read "San Jose, US-CA" in the accessible data list, and the caption and aria label say the globe is by city, state, or country, crediting DB-IP for the place data. The privacy policy now says the location label goes down to city, how the k>=5 floor applies level by level, and that the globe marks the place's centre rather than any install's location. The September 28 text is archived under Policy History, and all six locales are updated.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the globe was empty
Prod's
/api/public-stats/globalreturnedusers_by_region_365d: [{label: "Other", uniq: 55}]. The 55 located installs are spread over 38 states, and the largest has only 4, so the k≥5 floor sent every install to "Other", which has no coordinates to plot.What changes
US-CA/San Jose). At city level the tables would take ~110 MB in RAM, which doesn't fit on the 256 MB VM, soip-geo.jsbinary-searches the sorted CSVs in place instead. That's ~35 µs per lookup with one 64 KiB buffer per file, and the server also drops the ~66 MB the state-level tables used. The refresh script adds a city column and a places file (a centroid for every country, state, and city label).Deploy notes
/data/dbip-city/, checksums verified. The running server ignores it;fly.tomlin this PR switches to it on deploy./datais at 80% (186 MB free). After the deploy looks good, the old/data/dbip-city-lite*.csvfiles anddbip-city-lite.csv.2026-07.bak(~250 MB) can be deleted.Test plan
server-ip-geo(city, 4-column legacy rows, CRLF/long header/mid-file junk, centroids),-scale(lookup RSS 51 MB for 1M ranges),-merge(city merge, places file, antimeridian),server-client-ip-geo,server-region-aggregation(hierarchy, disjointness, centroids), housekeeper, public-stats,server-no-ip-leak,showcase-privacy-pagetest:ci67/67, prod build for all locales,lint:i18n, translation drift/quality, CI extraction diff clean