feat(pg): PostgreSQL datastore compatibility layer - #6
Open
dnplkndll wants to merge 78 commits into
Open
Conversation
dnplkndll
changed the title
dnplkndll
force-pushed
the
feat/pg-compat-clean
branch
3 times, most recently
from
May 14, 2026 12:57
dbcec59 to
6062962
Compare
dnplkndll
force-pushed
the
feat/pg-compat-clean
branch
2 times, most recently
from
May 23, 2026 15:53
a3b9ccf to
f9dab02
Compare
dnplkndll
force-pushed
the
feat/pg-compat-clean
branch
from
July 27, 2026 12:24
10cd4d0 to
1e6ac95
Compare
dnplkndll
added a commit
that referenced
this pull request
Jul 29, 2026
The PR #6 merge gate: the PG CI job now runs the entire datastore package unfiltered — 121 top-level tests pass on PG, 59 MySQL-only suites skip visibly (CreateMySQLDS now gates on MYSQL_TEST like CreateDS), zero failures. Test-infra parity: - No builtin-label pre-seeding on PG (MySQL test DBs are schema-only; tests create their own labels — the seeds collided on idx_label_unique_name). - PG TruncateTables clears the in-process software-title cache like the MySQL path (stale cache made UpdateHostSoftware skip recreating titles). - users_test/jobs_test converted to CreateDS; the users timestamp assertion gets an explicit 1s ceiling on PG (MySQL's whole-second TIMESTAMP gave the same slack implicitly). Cross-dialect datastore fixes: - MySQL-only 'DELETE alias FROM ... JOIN' → keyed subqueries (apple device names ×2, software title display names). - Raw REGEXP → dialect.RegexpMatch with dialect-correct word boundary (MySQL \b, PG \y) in the device-name secret scan. - Windows profile upsert bound command_uuid as an arg instead of a MySQL-only bare column reference in VALUES. - Jobs: not_before defaults to an app-clock whole second and unpinned queue reads compare against DB NOW() — immune to container clock skew and NOW()-precision differences (MySQL rounds, PG keeps micros). - query_results cleanup used LIMIT inside IN (MySQL error 1235 — a fork conversion bug on the MySQL path); wrapped in a derived table. Driver: - SUBSTRING_INDEX(x,d,1) → split_part; JSON_TYPE(x) → upper(jsonb_typeof(x)) with balanced-paren scan; unique violations gain MySQL-style 'table.constraint' phrasing via TranslateError (message-assertion parity). - default_query_exec_mode=describe_exec: pgx's statement cache breaks under live DDL ('cached plan must not change result type') — our deploy shape. Migration 20260729120000 ports the four remaining orphaned generated columns (windows profile checksum, apple declaration token, software_titles additional_identifier, calendar_events uuid) as triggers + backfill; baseline regenerated, marker 20260729120000. operating_systems_test's raw ODKU became dialect-neutral insert-if-missing. MySQL parity verified on every touched suite; both dialects green. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Adds a Postgres backend to Fleet's datastore alongside the existing
MySQL. Non-breaking: MySQL remains the default and is unaffected.
Core pieces:
- DialectHelper interface (server/datastore/mysql/dialect.go) abstracts
SQL dialect differences for upserts, aggregates, JSON ops, error
classification, and atomic swap-table DDL. mysqlDialect + postgresDialect
implementations, dialect.IsPostgres() routes runtime branches.
- pgx-rebind driver (server/platform/postgres/rebind_driver.go)
transparently translates MySQL SQL to Postgres at query execution time
via 50+ regex-based rewrites compiled once at startup. Per-table-name
regexes cached in sync.Map. knownPrimaryKeys map drives ON DUPLICATE
KEY → ON CONFLICT (<pk>) DO UPDATE rewriting.
- Embedded PG baseline (server/datastore/mysql/pg_baseline_schema.sql,
pg_baseline_post.sql) seeded from production pg_dump. Carries a
pg-baseline-up-to-migration: <ts> marker; fresh-apply seeds
migration_status_tables from code and logs a loud warning whenever
code carries migrations newer than the baseline. Object-ownership is
reasserted on every startup so atomic table swaps work even when the
baseline was loaded as the postgres superuser.
- server/goose/migration.go gains UpFnPG / DownFnPG / UpFnMySQL /
DownFnMySQL fields so individual migrations can target one dialect.
First user: 20260513210000_AddMissingPGIndexes (this commit).
- 349 missing PG indexes added via the AddMissingPGIndexes migration
(UpFnPG-only), bringing PG to index parity with MySQL on hot paths
like host_software_installed_paths (host_id, software_id).
Wiring:
- FLEET_MYSQL_DRIVER=postgres selects the new driver; standard
FLEET_MYSQL_ADDRESS / USERNAME / PASSWORD / DATABASE env vars route to
the PG cluster unchanged.
- server/config/config.go validates the new driver value.
- cmd/fleet/prepare.go threads dialect into the migration apply path.
- docker-compose.yml gains a postgres service for local dev.
Tests:
- 39 PG smoke tests (hosts, software, vulnerabilities, policies,
host-counts) and B1/B2/B3 tiers running on both backends via the new
CreateDS(t) helper.
- Driver-rewrite unit tests cover every regex (UPDATE...JOIN,
DELETE USING, GROUP_CONCAT, ON CONFLICT ambiguity resolution,
smallint-bool encoding, MAX(bool), INTERVAL placeholder, CAST NULL
AS SIGNED, FIND_IN_SET, COALESCE token, null-byte stripping, ...).
- Dialect unit tests for both dialects (LAST_INSERT_ID stripping,
ReturningID, AtomicTableSwap, CreateTableLike).
- List-options helper has new coverage for single-aggregate ORDER BY
skip and text-column cursor binding.
- Benchmarks for UpdateHostSoftware / ListSoftware / ListHosts in
server/datastore/mysql/benchmarks_test.go.
Squashed from 70+ incremental commits on feat/pg-compat-clean; full
provenance preserved on feat/pg-compat-clean-backup-2026-05-13.
…p on dep-review
CI infrastructure that gates the PG backend:
- test-go-postgres.yaml: spins up Postgres in a service container, runs
the full datastore + service test suites against the PG driver. Mirrors
the existing MySQL test workflow.
- validate-pg-compat.yml: invokes the tools/pgcompat validators on every
PR/push — check_primary_keys, check_schema_drift, check_column_drift.
Empty-allowlist gate-of-the-gate test ensures the validators themselves
can never become a no-op.
- build-ledo.yml: ledoent-specific image build that refuses to publish to
ghcr.io unless both test-go-postgres and validate-pg-compat succeeded
on the build SHA.
- sync-upstream.yml: paranoia check that refuses to force-push ledoent/main
if any non-bot commits exist outside upstream/main.
- weekly-aggregate.yml: gitaggregate cron + workflow_dispatch, pinned to
git-aggregator==4.1.
- dependency-review.yml: skip on private repos (the action requires
GitHub Advanced Security which isn't available on free private mirrors).
Upstream public fleetdm/fleet still runs it.
- test-website.yml: npm audit step added so frontend dep regressions
block PRs.
- tools/ci/apiparamcheck: custom golangci-lint plugin that flags REST
handler params not registered in the request struct, catching the
'missing query param decode' class of bug.
…rift
Three small static-analysis tools that prevent silent PG-compat regressions.
None require a running Postgres; they read Go source and SQL schema files.
- check_primary_keys: scans non-test Go for raw 'ON DUPLICATE KEY UPDATE'
SQL and verifies every targeted table has an entry in knownPrimaryKeys
(the map in server/platform/postgres/rebind_driver.go that drives the
ON CONFLICT (<pk>) DO UPDATE rewrite). Missing entries produce invalid
PG SQL at runtime.
- check_schema_drift: diffs CREATE TABLE identifier sets between
server/datastore/mysql/schema.sql (MySQL canonical) and
pg_baseline_schema.sql (PG baseline). known_schema_diff.txt records
intentional divergence and is itself validated — stale entries fail.
- check_column_drift: diffs column lists per shared table. Optional
allowlist via known_column_drift.txt.
- gen_identity_cols / gen_bool_cols: code generators that produce the
Postgres dialect's static knowledge of IDENTITY columns and bool
columns so the rebind driver can rewrite INSERTs correctly.
- validators_test.go is a gate-of-the-gate: an empty schema-diff
allowlist must produce a non-zero exit.
Designed to be extractable as a standalone PR to fleetdm/fleet — they're
useful to any Fleet operator building PG support, with or without the
larger driver/baseline layer.
Playwright API-mode test matrix that exercises every URL filter Fleet's
frontend can construct against a live server, asserting each response is
not a Postgres-driver or Postgres-syntax failure (SQLSTATE, 'must appear
in the GROUP BY', 'operator does not exist', 'cannot find encode plan',
'syntax error', etc.).
Read-only (HTTP GET only). ~220 probes in ~15s with 8 workers.
Coverage:
- /hosts + /hosts/count: status, low_disk_space, mdm_enrollment_status,
os_settings/apple_settings/disk_encryption/bootstrap_package, populate_*,
every ORDER BY allowlist key × direction, cursor pagination (after=),
vulnerability filter, search.
- /software/versions, /software/titles, /software (deprecated):
vulnerable, exploit, cvss range, self_service, available_for_install,
packages_only, team filtering, ordering.
- /vulnerabilities, /host_summary, /labels/:id/hosts, /hosts/:id/*,
sanity endpoints (/config, /version, /me, /labels, /teams, ...).
Run:
cd tools/pg-compat-harness
yarn install
export FLEET_URL=https://your-fleet
export FLEET_TOKEN=$(awk '/token:/ {print $2}' ~/.fleet/config)
yarn test
This harness found and gated the GROUP BY and cursor-encoding regressions
fixed elsewhere in this branch (selectSoftwareSQL GroupByAppend,
AppendListOptionsWithParamsSecure textOrderKeys hint).
Small Go program that parses server/datastore/mysql/schema.sql and emits
one CREATE INDEX IF NOT EXISTS statement per MySQL KEY / UNIQUE KEY clause,
suitable for embedding into a PG-only migration.
Handles:
- balanced parens in column lists (expression bodies)
- USING BTREE / USING HASH suffix (MySQL hint, PG ignores)
- DESC column ordering (PG supports natively)
- identifier quoting where required
- stable per-table grouping for reviewable diffs
Deliberately skips with explicit reasons:
- PRIMARY KEY (the CREATE TABLE handles it)
- FULLTEXT KEY, SPATIAL KEY (need pg_trgm / GiST equivalents)
- prefix-length indexes col(N) (need PG expression indexes)
- expression indexes using MySQL-specific functions (ifnull, cast as
signed) that need PG translation (COALESCE, CAST AS integer)
main_test.go drives translate() from inline schema fixtures — no file I/O
required. Covers plain/unique keys, DESC, USING BTREE, every skip reason,
balanced-paren edge cases, multi-table, PRIMARY ignored, plus unit tests
for extractParenBody and quoteIdent helpers.
Usage:
go run ./tools/pg-index-translate \
-in server/datastore/mysql/schema.sql \
-out server/datastore/mysql/migrations/tables/{ts}_AddMissingPGIndexes.sql
- docs/Deploy/postgresql.md: end-to-end guide for running Fleet against
Postgres — connection env vars, baseline schema apply, migration
apply, ownership reassertion, troubleshooting (drift warning, must
be owner of table, schema/column drift validator output).
- docs/Deploy/README.md: links the new guide from the deployment index
alongside the MySQL guide.
GetDBVersion returned a too-old current version on production PG because the baseline-seed path (and goose's own run-and-record loop for newly introduced migrations) inserted rows into migration_status_tables out of version_id order. Concretely, id 523 carried version 20260422181702 while id 521 carried 20260506171058. Plain 'ORDER BY id DESC' picked the older version, so 'fleet prepare db' tried to re-run every migration from 20260423161823 onward and failed on json_merge_patch — a MySQL-only function that PG never had, with the migration body long since folded into the embedded baseline. Switching to 'ORDER BY version_id DESC, id DESC' makes the query immune to insertion order while preserving up/down semantics: the tie-break by id DESC keeps the most recent applied/rolled-back state for the same version. MySQL is unaffected — its migration runner always applies in monotonic version order so id and version_id stay aligned. We do not change the MySQL dialect to keep blast radius minimal; that path has years of behavior to preserve. Test pins the exact ORDER BY clause via sqlmock so any future change back to the buggy form fails CI loudly.
…ces/views pg_baseline_post.sql already loops over public tables, sequences, and views and reasserts ownership to current_user, but it skipped functions. On baselines that were loaded by the postgres superuser (typical on self-hosted PG), CREATE OR REPLACE FUNCTION later in the same file errored with 'must be owner of function fleet_set_updated_at' — the application user can't replace something it doesn't own. Add a fourth loop using pg_proc / pg_namespace to enumerate public functions whose owner is not current_user, and ALTER FUNCTION ... OWNER TO current_user with the standard insufficient_privilege fallback. pg_get_function_identity_arguments() disambiguates overloaded signatures. Hit in production tonight on the AddMissingPGIndexes deploy. With this fix every future fleet prepare db on a postgres-superuser-loaded baseline succeeds without manual ALTER FUNCTION.
The existing implementation already sorts the seeded versions ascending (via versionsAtOrBelow → partitionMigrationVersions → slices.Sort), so PG assigns auto-increment ids in the same order as version_id. That property is load-bearing for any downstream consumer that infers 'current version' from MAX(id), even with the dialect query now correctly ordered by version_id DESC. No functional change — just document the invariant so a future refactor doesn't quietly drop the sort.
Required by TestVersionsAbove_EmbeddedBaselineCoversAllCode now that
AddMissingPGIndexes (20260513210000) ships in code. Dump source is
fleet.hz.ledoweb.com fleet-db-1, which has all 532 indexes applied
(11 from the original baseline + 521 added by AddMissingPGIndexes
either via the SQL we ran manually tonight or via the migration on
future fresh applies). check-pg-compat validators pass:
schema-drift: 202 MySQL tables / 205 PG tables in sync (after allowlist)
primary-keys: every ON DUPLICATE KEY UPDATE site covered
column-drift: no drift between schema.sql and pg_baseline_schema.sql
Generated via the documented procedure in the file's header:
kubectl exec -n fleet --context hetzner-ledo fleet-db-1 -- \
pg_dump -U postgres -d fleet --schema-only --no-owner --no-privileges
Stripped the pg_dump-17 \restrict/\unrestrict meta-commands and the
SET search_path='' line per the same header comment. Header preserved
with the regen recipe and verification commands.
POST /api/fleet/orbit/setup_experience/init returned 500 with: ERROR: COALESCE types integer and text cannot be matched (SQLSTATE 42804) The setup-experience init query UNION-ALLs two SELECTs — one for software_installers, one for vpp_apps — projecting a NULL placeholder in each leg for the column the other leg owns. The outer ORDER BY then COALESCEs across both columns plus a literal 0: ORDER BY sort_name ASC, COALESCE(software_installer_id, vpp_app_team_id, 0) In MySQL the untyped NULL silently coerces. In Postgres the untyped NULL resolves to text, then COALESCE sees one int leg, one text leg, one int literal — strict-type rejects. Fix: replace the bare 'NULL AS ...' projections with 'CAST(NULL AS UNSIGNED) AS ...'. MySQL keeps it as unsigned-int NULL; the PG rebind driver already rewrites 'AS UNSIGNED' → 'AS integer' (see server/platform/postgres/rebind_driver.go reAsUnsigned*). Both dialects now compose a proper-typed NULL for the outer COALESCE. Surfaced by Windows host enrollment — the iOS/Android/Apple-setup experience flow runs the same code path; this fix covers them all. A broad audit of the codebase's 487 COALESCE call sites found no other sites with the same untyped-NULL-in-UNION shape.
Extends the harness with 19 new probes covering every orbit and osquery
agent POST endpoint listed in server/service/handler.go:1006-1099 +
osquery enroll/config/distributed/carve/log.
Each probe sends a fake orbit_node_key (or node_key) and asserts the
auth-middleware SQL — typically SELECT … FROM hosts WHERE
orbit_node_key = ? — runs without an SQLSTATE crash. A 401 response is
the success case; a 500 with PG error markers is the failure case.
Required infrastructure additions:
- Probe gains optional method ('GET'|'POST'), body, and expectAuthFail
- check() does request.post(...) when method is POST
- check() permits 4xx when expectAuthFail is true (still fails on
body containing PG error markers, still fails on 5xx)
Limitation documented in the orbitProbes comment: fake-key probes
reject at the auth gate, so per-endpoint handler SQL (e.g. the
setup_experience/init COALESCE bug fixed in df17814bc7) is NOT
exercised by these probes. Catching post-auth SQL needs a fixture
host with a real orbit_node_key — tracked as a future expansion.
Current count: 242 passing (was 223; +19 orbit/osquery probes).
Two raw ON DUPLICATE KEY UPDATE sites in microsoft_mdm.go
(lines 364 and ~551) upsert into windows_mdm_commands, which has
PRIMARY KEY (command_uuid) per schema.sql. Without the
knownPrimaryKeys entry, the PG rebind driver emits
'ON CONFLICT DO UPDATE SET …' without a target, which PG rejects.
Caught by check_primary_keys in the pg-compat validator suite — same
gate that blocked the last build. Aggregated CI now passes locally:
$ go run ./tools/pgcompat/check_primary_keys
OK: every raw ON DUPLICATE KEY UPDATE site is covered by
knownPrimaryKeys.
…ploy) Upstream PR fleetdm#45367 (aggregated commit bee5eda, 2026-05-14) added the orbit_debug_until column to the hosts table in MySQL schema.sql. The PG baseline is regenerated from a production pg_dump; that migration hasn't landed on prod yet (it lands when this very deploy rolls out), so the baseline lags by one column. Adding the entry to known_column_drift.txt with a deferred-regen comment, per the file header's prescribed workflow. Once the next aggregation runs after this deploy lands, the prod baseline will include orbit_debug_until and this allowlist entry can be removed. Without this entry, the validate-pg-compat CI gate fails check_column_drift on every aggregated build, blocking the build-ledo image publish that deploys this fix in the first place. Classic chicken-and-egg — break with a documented intentional drift.
… leftovers Review finding 21 + activities audit: - common_mysql.IsReadOnlyError recognizes PostgreSQL's SQLSTATE 25006 (matched structurally via SQLState(), no pgx dependency), so every fail-fast site — sessions, withRetryTxx, common tx helpers — now trips on a PG primary→replica failover exactly as on Aurora demotion. - Prod audit: activities/host_activities (pre-rename generation the MySQL-only RENAME TABLE never converted on PG) are empty; activity_past holds all history. Migration 20260727210000 drops them with an emptiness guard that fails loudly if any deployment has stranded rows. known_schema_diff.txt comments corrected (they were mislabeled 'no MySQL equivalent'). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Review nits: - FullTextMatch concatenates multi-column input into one tsvector instead of silently dropping cols[1:]. - docs/Deploy/postgresql.md: knownBooleanColumns → generated schema_bool_cols_gen.go reality; prepared-statement LastInsertId rows marked resolved (fixed in 3.1); stale claims corrected. - Dead code removed: goose AddDualDialectMigration (20260513210000 sets UpFnPG directly), unreachable pgx branch in tables.SetDialect; indexExists documented as MySQL-test-only (migration code must use the dialect-aware indexExistsTx). - validate-pg-compat skip ledger counts isPG(ds) sites — the TODO-string grep matched nothing and reported Total: 0 forever. - pg-compat-harness playwright config requires FLEET_URL (bare yarn test no longer targets prod). - docker-compose dev postgres uses the same glibc postgres:16 image as postgres_test (musl collation differs; collation_test.go exists) and binds to 127.0.0.1 like every other service. - CLAUDE.md: migration batching convention (finding 25's residue). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Baseline drops the pre-rename activities/host_activities tables; schema-diff allowlist pruned accordingly. All six validators green; fresh + idempotent prepare db verified on scratch. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…marker The first Phase-3 prod deploy aborted on the backstop's own false positive: right after a baseline regen the marker IS the newest (not-yet-applied) migration, which goose is about to run — that is the normal pending state, not drift. Unreachable migrations are exactly those below the database's max applied version (goose only runs versions above it). Regression-covered with the exact prod scenario in TestPostgresBelowMarkerDriftCheck. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…ed_at triggers Adversarial re-review (2026-07-28) found 3 must-fix issues; all fixed: M1 — UpdateVulnerabilityHostCounts failed every hourly run since the swap change: its old-table cleanup was the one bare DROP TABLE among the four swap callers (the PG swap now drops the table itself). IF EXISTS like its siblings; confirmed live in prod cron_stats before fixing. New TestPostgresHostCountCrons runs all four swap crons end-to-end, twice. M2 — Windows ESP state machine: mdm_windows_enrollments.awaiting_configuration is a tri-state uint but sat in smallintBoolColumns, whose bool-CASE rewrite collapsed Active=2 to 0. Split-typed names are now excluded from ALL generic bool machinery: removed from smallintBoolColumns, gen_bool_cols skips names in known_bool_col_splits.txt (regenerated), both sides map natively. Driver unit test asserts pure passthrough; TestPostgresWindowsESPStateMachine walks the real 0→1→2→0 transitions. M3 — updated_at was frozen at insert time on ~125 tables: MySQL's ON UPDATE CURRENT_TIMESTAMP had a PG trigger on only the 12 driver-created tables. New gen_updated_at_triggers generates the full trigger set from schema.sql (138 triggers/137 tables incl. the three non-updated_at auto-touch columns via a generic fleet_touch_column), embedded and applied on every prepare db (CREATE OR REPLACE, converges driver-created names). fleet_set_updated_at upgraded to exact MySQL semantics: touch only when the row changed and the statement didn't assign updated_at itself. CI staleness check added. TestPostgresUpdatedAtTriggers covers touch/no-op/explicit-wins plus a coverage-breadth assertion (>=130 tables). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Its host_disks scenario is covered by TestPostgresUpdatedAtTriggers. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…opy frozen DRY pass from the branch quality review: - tools/pgcompat/internal/allowlist: single implementation of the tagged (mysql-only:/pg-only:) and plain-lines allowlist formats, replacing three near-identical copies across check_column_drift, check_constraint_drift, check_bool_col_split and the inline loop in gen_bool_cols. - 20260727170400's DO block is applied history — replace the stale 'keep in sync' instruction with a frozen-snapshot note (the dialect re-canonicalizes on every swap cycle, so drift is harmless by design). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…umn coverage Coverage pass from the branch quality review: - newPGTestHost collapses five identical fleet.Host literals in the PG smoke tests (the two tests that deliberately exercise raw NewHost keep theirs). - TestMySQLUpsertDidUpdateParity: the promised MySQL twin of TestPostgresUpsertDidUpdate — OnDuplicateKeyGuarded must yield identical did-update semantics on both dialects (insert/changed → uploaded_at moves, identical re-upsert → preserved). - TestPostgresUpdatedAtTriggers now also covers the fleet_touch_column path via sessions.accessed_at, one of the three non-updated_at auto-touch columns. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
tools/pgcompat/README.md documents all five validators, the three CI-staleness-checked generators, and the shared internal/allowlist package; docs/Deploy/postgresql.md's CI-gates section lists the full current step sequence including constraint-drift, bool-split, and the updated_at trigger generator. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
The PR #6 merge gate: the PG CI job now runs the entire datastore package unfiltered — 121 top-level tests pass on PG, 59 MySQL-only suites skip visibly (CreateMySQLDS now gates on MYSQL_TEST like CreateDS), zero failures. Test-infra parity: - No builtin-label pre-seeding on PG (MySQL test DBs are schema-only; tests create their own labels — the seeds collided on idx_label_unique_name). - PG TruncateTables clears the in-process software-title cache like the MySQL path (stale cache made UpdateHostSoftware skip recreating titles). - users_test/jobs_test converted to CreateDS; the users timestamp assertion gets an explicit 1s ceiling on PG (MySQL's whole-second TIMESTAMP gave the same slack implicitly). Cross-dialect datastore fixes: - MySQL-only 'DELETE alias FROM ... JOIN' → keyed subqueries (apple device names ×2, software title display names). - Raw REGEXP → dialect.RegexpMatch with dialect-correct word boundary (MySQL \b, PG \y) in the device-name secret scan. - Windows profile upsert bound command_uuid as an arg instead of a MySQL-only bare column reference in VALUES. - Jobs: not_before defaults to an app-clock whole second and unpinned queue reads compare against DB NOW() — immune to container clock skew and NOW()-precision differences (MySQL rounds, PG keeps micros). - query_results cleanup used LIMIT inside IN (MySQL error 1235 — a fork conversion bug on the MySQL path); wrapped in a derived table. Driver: - SUBSTRING_INDEX(x,d,1) → split_part; JSON_TYPE(x) → upper(jsonb_typeof(x)) with balanced-paren scan; unique violations gain MySQL-style 'table.constraint' phrasing via TranslateError (message-assertion parity). - default_query_exec_mode=describe_exec: pgx's statement cache breaks under live DDL ('cached plan must not change result type') — our deploy shape. Migration 20260729120000 ports the four remaining orphaned generated columns (windows profile checksum, apple declaration token, software_titles additional_identifier, calendar_events uuid) as triggers + backfill; baseline regenerated, marker 20260729120000. operating_systems_test's raw ODKU became dialect-neutral insert-if-missing. MySQL parity verified on every touched suite; both dialects green. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…ace, migration-test gating The first unfiltered CI run (correctly) caught what my local environment masked: - EnqueueDeviceLockCommand's FOR UPDATE guard relied on InnoDB gap locking when no host_mdm_actions row exists yet — PG has no gap locks, so all 20 concurrent requests in the race test passed the existence check and 4 commands landed. A transaction-scoped pg_advisory_xact_lock keyed on host_id serializes PG; MySQL keeps its gap-lock behavior. This is the first concrete instance of the review's finding-8 gap-lock class; the sibling sites (apple_psso, host_certificate_templates) remain under that finding's accepted-risk disposition pending the same treatment. - migrations/tables tests and migrations_test.go dialed MySQL unconditionally (my MYSQL_TEST shell export hid it locally); both now gate with a visible skip, and the recursive ./server/datastore/mysql/... CI target is clean under a PG-only environment. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
newDSWithConfig and createTestDatabase (mysql_test.go) open MySQL directly; four TLS/password-path tests failed on dial in the PG CI job while passing locally against the dev container. Verified by running the full recursive suite with the local MySQL container STOPPED — the exact CI shape — which now passes clean. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…ations Rebase was conflict-free. Upstream added two migrations timestamped BELOW this fork's baseline marker — the exact scenario the below-marker backstop exists for (it refuses deploys until their DDL lands). Migration 20260729190000 ports both on PG: re-runs 20260727083533's DDL through the rebind driver (with a post-condition assert so a silent no-op can never record as applied), and inlines 20260727084359's fleet-variable insert with a boolean literal (the original's integer 0 for is_prefix is a 42804 on PG). The generated updated_at trigger set now installs AFTER goose Up (in both prepare and the test harness): it references tables that post-marker migrations create. Baseline regenerated through the port (marker 20260729190000, 223 tables); gen files regenerated (128 identity tables, 140 touch triggers); all five validators green; full unfiltered PG suite, driver, goose, migrations-on- MySQL, and fresh+idempotent prepare all pass. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
dnplkndll
force-pushed
the
feat/pg-compat-clean
branch
from
July 29, 2026 15:39
aa253da to
3faeee0
Compare
The fork's root go.mod adds pgx; the new upstream tool module's go.sum needs the matching hashes (CI 'Test tools' gate). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
The first Phase-4 prod migration was refused by checkPGBelowMarkerDrift: prod's history has no rows for the two back-dated upstream migrations, and the check runs before the goose Up that would execute their porting wrapper. Local runs never hit this — fresh DBs seed history through the marker. PortedBelowMarker (declared next to the wrapper) maps each ported version to its wrapper; the check now exempts a version whose wrapper is either already applied or registered above the DB max (so the imminent goose Up runs it). Anything else below the max still fails. The wrapper also records the ported versions in goose history so steady state needs no exemption. New TestPostgresPortBackdatedWrapper executes the wrapper's PG path against a prod-shaped DB (DDL absent, history rows missing) — without it that path would first run in production — plus exemption cases in the drift test. Verified end-to-end: a DB reset to prod's exact pre-port state now runs prepare db to completion (tables, vars, and history all land). Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Prod's second Phase-4 migration attempt failed differently: the software_titles backfill touch fires the baseline-post unique_identifier trigger, and recomputing stale stored values (the formula gained application_id/upgrade_code terms after those rows were written) collided on idx_unique_sw_titles. 95 duplicate titles had accumulated on prod that MySQL's generated column + unique keys make impossible. Up_20260729120000 now merges each duplicate group (grouped by the trigger's current formula) into its lowest id before the touch: per-title aggregates are deleted (crons rebuild), the 14 referencing tables are repointed — deleting the loser's row first where the target's unique key includes the title id — and losers removed. Verified against prod data in a rolled-back transaction: 95 losers merged, 102 software rows repointed, full recompute collision-free, zero orphans. TestPostgresGeneratedColumnDedup fabricates the prod shape (stale unique_identifier inserted with triggers disabled, references attached) and executes the migration's PG path directly. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
…eview Review of the Phase-4 delta (three parallel reviewers) found real gaps in the two new migrations; all are closed here and the dedup coverage now exercises every guarded branch. - 20260727084359: is_prefix literal 0 -> false (valid on both dialects). On a PG database still below that version (older baseline markers) the ORIGINAL runs — the wrapper only covers databases already past it — and the integer literal is a 42804 deploy wedge there. - Dedup migration: policies.patch_software_title_id was missed entirely (the reference scan required the column to be named exactly title_id / software_title_id) — a merged group would have orphaned patch policies silently, since PG carries no FK. software_title_team_pins repointed with no guard against its (team_id, title_id) PK. Both now use the guarded delete-before-repoint. The guard itself also handled only keeper-vs-loser collisions; two losers of one group landing on the same slot collided with each other. All five guarded tables now rank slot occupants (keeper's row first, then lowest loser id) and delete the rest. Prod audited: zero patch policies, zero orphans — the misses did no damage on the one database that already ran this migration. - The backfill touch activates idx_software_titles_bundle_identifier for the first time (NULLs kept it vacuous), which the dedup key does not imply uniqueness under; a pre-flight DO block now fails with the colliding bundle list instead of an opaque 23505. - Wrapper: guards and post-checks BOTH ported tables (a partial manual restore could previously record the port done with one table missing forever), and the fleet-variable insert is per-row idempotent instead of keyed on one variable's presence. - checkPGBelowMarkerDrift: dropped the tautological registered-map lookup (PortedBelowMarker lives in the wrapper's own file). - Tests: dedup test now covers keeper-collision, intra-loser collision, team-pin drop, and patch-policy repoint; stale test comments corrected. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
TranslateError had no direct tests. Table-driven coverage for the wrap (message phrasing, Unwrap, SQLState passthrough and fallback) and the pass-through paths; the wrap now also requires a non-empty ConstraintName so it can never emit a malformed "table." key. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Five copy-pasted env gates collapse into skipUnlessMySQLTest; the two new PG tests share pgMustExec. GetFilteredQueuedJobs' comment claimed not_before is written with NOW() — NewJob writes the truncated app clock; the comment now states the real invariant and its skew window. Claude-Session: https://claude.ai/code/session_01NrWVk8oMToieU6y3yDtYpY
Merge the upstream mirror into the PG-compat branch. Conflict resolutions keep the branch's dialect-helper conventions while adopting upstream's new logic: OnDuplicateKey/OnDuplicateKeyGuarded wrappers regain upstream's new columns (policies patch_when_closed/resend_*, mdm_windows_enrollments ztd_registration_id, software_installers script-edited flags), Go-side policyChecksum stays over the SQL computed column, aggregate truthiness stays explicit (!= 0) in the Apple status CASE, boolean columns keep true/false literals, and upstream's new label-scope/FileVault fragments are adopted with those conversions. New upstream helpers that build upserts (setVariableAssociationsForColumnDB, setMDMAppleDDMActivationDB, batchSetDeclarationActivationsDB, upsertMDMAppleHostMDMInfoDB source enum) are threaded with DialectHelper. mysqlDialect.AtomicTableSwap gains upstream's stale-_old drop. GROUP BY in ListIOSAndIPadOSToRefetch extended for PG strictness; REGEXP sites routed through dialect.RegexpMatch.
Migration ports: - Guard the two timestamp-bumped migrations (AddTokenInvalidToABMTokens, AddHostMDMWindowsProfilesStatus) so they no-op where the old IDs already applied. - Wrapper 20260831120000 ports the five below-marker migrations (Apple builtin-label backfill, queued-VPP dedupe, upcoming_activities indexes, password-reset collation no-op, DDM custom activations) to existing PG databases and records them in goose history. - DedupeQueuedVPPAppInstalls: PG variants for SUM-over-boolean, HAVING-on- alias, jsonb = number, and DELETE...USING. - DedupeSoftwareChecksums: PG path computes checksums in Go (NUL join byte can't live in PG text), maps duplicates in-process, and rewrites the installed-paths repoint as UPDATE...FROM. - PerPlatformDiskEncryptionSettings: jsonb_set/jsonb_build_object fan-out with MySQL's JSON-boolean-true comparison semantics. - FixDockerDesktopBundleIdentifier: UPDATE IGNORE emulated with a NOT EXISTS guard on each table's (team_id, title) unique key. - CreateHostMDMAppleDeviceVitals: UpFnPG with native boolean vitals columns (sidesteps the awaiting_configuration name split). - AddDDMCustomActivations: PG-native table + token trigger (the STORED generated column has no PG translation), CASE-based CHECK swap. - ManagedLocalAccountWindowsEscrowColumns / AddManagedLocalAccountDeleted: DROP NOT NULL port and native boolean flags. - Collation-only MODIFY migrations no-op on PG (already case-sensitive). Driver: split ALTER TABLE DROP INDEX into standalone DROP INDEX, strip ADD COLUMN ... AFTER positioning, add the seven new upsert targets to knownPrimaryKeys, and pre-drop a stale _old table in the MySQL AtomicTableSwap. Validators: allowlist the pre-baseline-regen schema/column/constraint drift (remove after the post-deploy regen); regenerate touch triggers and identity columns for the new tables. Verified: fresh `prepare db` twice on scratch PG 16 (run 1 migrates, run 2 idempotent), all new tables/triggers/indexes present, make check-pg-compat green up to the committed-tree diff check.
…ite fixes - Driver now translates TINYINT(1) → native BOOLEAN (with FALSE/TRUE defaults) so new columns written from Go bools encode without rewrite machinery; wider TINYINTs stay smallint. abm_tokens.token_invalid (baked as smallint under its pre-rename ID) joins smallintBoolColumns, and the smallint-bool rewrite now covers != / <> comparisons. - MySQL null-safe equality `<=>` is rewritten to IS NOT DISTINCT FROM by the driver; mdm.go's certificate-renewal query returns to `<=>` so the MySQL side parses again (it had been converted to the PG-only spelling). - New PG-only migration 20260831130000 adds the five FK-backing indexes MySQL creates implicitly for the Aug 2026 foreign keys. - pg_baseline_schema.sql regenerated from a freshly-migrated scratch PG 16, marker bumped to 20260831130000; bool/identity/trigger artifacts regenerated; the temporary drift allowlist entries removed. - query_labels test inserts use boolean literals (portable to PG); select-software-titles SQL fixture regenerated for the dialect-helper SQL.
… LIKE escapes - Rewrite every label-scope HAVING that referenced SELECT aliases (count_installer_labels & co.) to aggregate expressions or a wrapping SELECT: PG rejects output aliases in HAVING; the pattern spans software.go, software_installers.go, vpp.go and the new upstream FMA paths. - Preserve MySQL's `updated_at = updated_at` semantics on PG: the driver sets a fleet.preserve_updated_at GUC around statements carrying the idiom and fleet_set_updated_at skips touching while it is set. - Driver: `ESCAPE '\\'` → E-string form; software_titles. additional_identifier integer comparisons quoted (text on PG). - maintained_apps: PG emulation of the UPDATE IGNORE title-merge repoints via (team_id, title) NOT EXISTS guards. - Below-marker drift test updated for stacked porting wrappers. PG suite: 0 failures. MySQL migrations suite: 0 failures.
- GetMDMAppleProfilesSummary / GetMDMAndroidProfilesSummary: filter the NULL status group in HAVING with a dialect-forked operand (MySQL only_full_group_by wants the alias, PG the repeated CASE). The previous subselect-with-outer-WHERE form hit a MySQL 8.0.44 derived-condition- pushdown bug that leaked the NULL group into the scan. - androidApplicableProfilesQuery rebuilt from upstream with each leg's HAVING repeating its own aggregate expressions: the merge had kept the branch's pre-rebase repeated aggregates, which no longer matched the SELECT lists' new unknown-label-preservation logic (bool literals are the driver's job). - GetAndroidAppsInScopeForHost passes the ninth host-id arg introduced by the exclude-leg HAVING expansion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue: internal PG-compat fork work (no upstream issue)
Checklist for submitter
SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.Testing
Database migrations
COLLATE utf8mb4_unicode_ci). (N/A on PG; MySQL paths unchanged.)What changed
Full PostgreSQL compatibility for Fleet's datastore layer, enabling the production deployment at fleet.hz.ledoweb.com to run against PostgreSQL 16 instead of MySQL.
July 2026 rebase onto upstream 5cd8edb (1,114 upstream commits)
isPostgres()branches:ALTER … CHANGE/MODIFY,DROP CHECK, inlineADD [UNIQUE] KEY, combinedDROP/ADD INDEX,VIRTUALgenerated ENUM columns (plain text on PG),IF()in CHECK expressions,REGEXP→~, MySQLinformation_schemaFK/index lookups →pg_constraint/pg_indexes. Idempotency guards added for schema this fork shipped ahead of upstream (pending-delete table,policy_gated, BYOD/ADUE).fleet prepare dbdoes (seed history ≤ marker, goose Up through the rebind driver), so migration incompatibilities surface inTestPostgres*instead of at deploy.fleet prepare db --mysql_driver=postgres; marker bumped to20260724134801.schema_identity_cols_gen.goregenerated; schema/column drift validators clean.ON UPDATE NOW(n)likeON UPDATE CURRENT_TIMESTAMP(emitting the updated_at trigger); ten newknownPrimaryKeysentries for new upstream upsert sites; the twomdm_configuration_profile_variablesupserts (per-kind conflict targets) converted to the dialect helper.connected_to_fleetCASE mixedEXISTS(...)with1/0integer literals — PG rejects mixed CASE branch types (SQLSTATE 42804), which broke/api/fleet/orbit/configon first deploy. Converted totrue/falseliterals (valid in both dialects); regression-covered byTestPostgresGetHostMDM.InsertCVEMetaon PG adds aWHERE … IS DISTINCT FROMguard so re-upserting the ~300k-row NVD set doesn't rewrite unchanged rows (dead tuples + index churn), andLoadCVEMeta's bulk-load deadline is raised 1m → 10m. This unblocks the hourlyinsert cve scores: context deadline exceededfailure observed in prod since at least Jul 12.Core dialect abstraction
DialectHelperinterface (dialect.go) abstracting all MySQL vs PostgreSQL SQL differencesmysqlDialectandpostgresDialectimplementations covering all methods:InsertIgnoreInto,ReplaceInto,FromDual,OnDuplicateKey,OnConflictDoNothing,GroupConcat,JsonQuote,JSONAgg,JSONExtract,JSONUnquoteExtract,JSONBuildObject,JSONObjectFunc,FindInSet,FullTextMatch,RegexpMatch,GoquDialect,ReturningID,IsPostgres,CreateTableLike,AtomicTableSwapRuntime SQL translation (
server/platform/postgres/rebind_driver.go)The pgx-rebind driver wraps pgx/stdlib and rewrites MySQL syntax to PG at the driver layer:
?→$Ncol = 1/col = 0→col = true/col = false(for the ~60 boolean cols listed inschema_bool_cols_gen.go)UPDATE … JOIN→UPDATE … FROM; multi-table DELETE → DELETE … USINGJSON_EXTRACT(col, '$.path')→col->'path';JSON_OBJECT(…)→jsonb_build_object(…);JSON_QUOTE(…),JSON_UNQUOTE(…),JSON_ARRAYAGG(…)→ PG equivalentsIF()→CASE WHEN,IFNULL→COALESCE,MD5()→md5(),UUID()→gen_random_uuid()::text,UTC_TIMESTAMP()→TO_CHAR(NOW() AT TIME ZONE 'UTC', …),CURDATE()→CURRENT_DATE,DATABASE()→current_schema(),HEX()/UNHEX()→encode/decode,FIND_IN_SET→array_position(string_to_array), …CAST AS UNSIGNED→CAST AS integer,CAST AS SIGNED INT→CAST AS integer,TIMESTAMP(?)→(?)::timestampINSERT IGNORE INTO→INSERT INTO … ON CONFLICT DO NOTHING;REPLACE INTO→INSERT … ON CONFLICT DO UPDATE;ON DUPLICATE KEY UPDATE VALUES(col)→ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.colBLOB→bytea,MEDIUMTEXT/LONGTEXT→TEXT,TINYINT(1)→smallint,DATETIME[(N)]→timestamp[(N)],INT UNSIGNED NOT NULL AUTO_INCREMENT→INTEGER NOT NULL GENERATED BY DEFAULT AS IDENTITY,INT UNSIGNED→INTEGER, inlineUNIQUE KEY name (cols)→CONSTRAINT name UNIQUE (cols),enum('a','b','c')→VARCHAR(255) CHECK (col IN ('a','b','c')),ENGINE=InnoDB/DEFAULT CHARSET=…/ALGORITHM=INSTANTstrippedALTER TABLE … ADD COLUMN …, ADD KEY <name> (<cols>)produces an ALTER followed by a separateCREATE INDEX.ADD UNIQUE KEY→CREATE UNIQUE INDEX. Multiple ADD KEY clauses each become their own CREATE INDEX.fleet_set_updated_at()is appended. The trigger function is installed bypg_baseline_post.sql.Migration runner
MigrateTableson PG: applies the embedded baseline on fresh DBs, seedsmigration_status_tables/migration_status_data≤ the baseline marker, then runs gooseUpso newer upstream migrations execute through the rebind driver.pg_baseline_post.sql: idempotent ownership fixups +fleet_set_updated_at()trigger function; re-applied on everyprepare db.pgcompat tooling (
tools/pgcompat/)Validators that gate every push:
check_primary_keys— every rawON DUPLICATE KEY UPDATEsite is covered byknownPrimaryKeys, and every entry's columns match a real PK/UNIQUE constraint in the baseline (also run with--include-migrations)check_schema_drift/check_column_drift—schema.sqlvspg_baseline_schema.sqltable and column sets match (allowlists for intentional drift)check_constraint_drift— PK/UNIQUE/index/FK parity by (table, kind, column-set); the deferred ~160-FK set is allowlisted with rationalecheck_bool_col_split— no column name typed boolean and smallint in different tables (split names are excluded from all name-keyed rewrites)gen_updated_at_triggers— generates theON UPDATE CURRENT_TIMESTAMPtrigger mirror (138 triggers) applied on everyprepare db; CI-staleness-checked likegen_bool_cols/gen_identity_colsgen_bool_cols/gen_identity_cols— regenerate the schema-derived driver tables; CI fails if staleprepare db(baseline + post-marker migrations), asserted idempotent on second runTest plan
Validate PG CompatibilityCI — green onfeat/pg-compat-rebased@ dd8ae2b and onaggregatedGo Tests (PostgreSQL)CI — green onaggregatedBuild & Push Ledo Fleet ImageCI — green; imagesha256:6680b5b9eefd…in registry.hz.ledoweb.comTestPostgres*suite against real PG 16, including post-baseline migration replay and the newTestPostgresInsertCVEMetaTestMigrationspasses (MySQL paths of ported migrations unchanged)fleet prepare db --mysql_driver=postgresmigrates a scratch PG cleanly through all July migrations (used to regenerate the baseline)fleet prepare dbjob, 2026-07-27) and rebased image rolled outcve_metatimeout gone, CVE/EPSS data refreshing (verified 2026-07-27)Go Tests (PostgreSQL)CI runs the datastore package unfiltered — 121 pass / 59 MySQL-only skips / 0 fail; driver + goose + pgcompat suites green; fresh + idempotentprepare dbverified post-rebase onto54b32ddf88policies.patch_software_title_idand unguardedsoftware_title_team_pinsrepoints, intra-loser slot collisions in the guarded repoint pattern, a pre-flight guard for the bundle-identifier unique index the backfill newly activates, both-table guards + per-row idempotent variable inserts in the porting wrapper,TranslateErrorunit coverage, and trigger-formula parity tests for all four generated columns. Prod audited for the missed-table gap: zero orphans (no patch policies exist there)PortedBelowMarker), and the generated-column backfill dedups the 95 duplicate software_titles rows that staleunique_identifiervalues had let accumulate (rehearsed against prod data in a rolled-back transaction first). Post-deploy: max version 20260729190000, 0 dup groups, 0 orphans, 138 updated_at-trigger tables, crons completing, hosts checking in