Make the ad network niche sorter recover from bad batches and run daily

content_docs/ad-networks/maintenance.md

@@ -15,7 +15,7 @@ The jobs run as plain cron on the OVH server and write straight to the productio
 | Discovery | Tuesday 04:00 | Active recurring sources; paused sources stay paused |
 | Publisher/business names | Wednesday 04:00 | Refresh permitted official seller feeds |
 | Traffic | Day 5, 04:00 | Latest complete dated CrUX snapshot |
-| Niche sorting, homelab | Sunday 06:00 | Up to 3,000 new unsorted sites, 100 per model batch |
+| Niche sorting, homelab | Daily 06:00 (11:30 IST) | Up to 3,000 unsorted sites, 100 per model batch; failed batches shrink to 50, then 25 |
 
 The crontab and its wrapper are [`scripts/ad-networks/ovh/`](../../scripts/ad-networks/ovh/). One session-level PostgreSQL advisory lock lets only one heavy job run at a time. A scheduled job that finds it taken retries every minute for up to 6 hours, and verification steps aside while one waits. `flock` stops the same job from starting twice. Site-based enrichment stops admitting work after 30 minutes or 5,000 sites. Source-feed jobs need a complete feed and have an outer timeout. Existing host limits remain authoritative.
 
@@ -31,7 +31,11 @@ The OVH working store started on October 19 from a consistent homelab snapshot.
 
 Postgres owns existing public niche values. A full SQLite sync may seed a newly inserted publisher's niche; it never updates an existing publisher's niche. This prevents the next OVH sync from overwriting homelab classification. The first OVH run used `sync --seed-niches`: only non-null source labels can fill target rows whose niche is still null. The target predicate is enforced in the database UPDATE, so newer homelab labels survive. `--niches-only` is reserved for a separately reviewed correction and may replace or clear labels; no automatic job uses it.
 
-The full sync now includes measured homepage title, description, h1, language, and source under `last_check_evidence.enrichment.homepage`. Homelab reads unsorted confirmed publishers and these facts directly from production. It never copies or pushes a second working store. `niche-sort.mjs` checks the exact input domain set, rejects duplicate/missing/extra domains, unsupported labels and fields, and invalid confidence before each atomic batch import. Low confidence suggestions remain private in homelab state; accepted results update only still-unsorted confirmed publishers. A changed OVH cohort cannot cause a stale full-row overwrite. Classified null results are retried after changed evidence or 180 days. Sites without usable homepage facts wait for a later homepage pass.
+The full sync now includes measured homepage title, description, h1, language, and source under `last_check_evidence.enrichment.homepage`. Homelab reads unsorted confirmed publishers and these facts directly from production. It never copies or pushes a second working store. `niche-sort.mjs` keeps only valid, unique answers for the requested domains before each atomic batch import.
+
+Malformed, duplicate, off-list, or missing answers do not discard valid rows. Missing answers and transient model failures get one retry, then failed batches shrink from 100 to 50 to 25 sites. Unresolved sites have no successful checkpoint and remain eligible for the next run. The result reports their count and the run status is `partial`, with its failure alert kept open. Operator stops, database failures, login failures, and Claude usage limits stop the run without model retries. Resume after the subscription limit resets; do not enable paid extra usage. CLI diagnostics are bounded and sanitized.
+
+Low confidence suggestions remain private in homelab state; accepted results update only still-unsorted confirmed publishers. A changed OVH cohort cannot cause a stale full-row overwrite. Classified null results are retried after changed evidence or 180 days. Sites without usable homepage facts wait for a later homepage pass.
 
 The model runs in a neutral temporary directory with tools disabled. Database credentials, tokens, passwords, secrets, and API keys are removed from its environment. Batch checkpoints live in `~/.local/state/stack127/ad-networks-niche/niche-results.json`, and the last run's result in `niche-status.json` beside it, outside Git. The import waits in the sync lock queue for up to 60 minutes, retrying after each 100-second lock timeout, because an OVH sync holds it for most of each cycle. Only the niche timer remains in `scripts/ad-networks/systemd/`. Keep the installed homelab backup timer until the original RDAP pass and gather supervisor finish: their already-loaded code still calls it at midnight. Do not stop the one-time RDAP pass.
 
@@ -287,7 +291,7 @@ A binary or misencoded preview containing NUL bytes stays null in new summaries
 
 The six mockup contracts and current development availability are in [the data contract](../../dev-space/ad-networks-data-contract.md). Pages consume `getAdNetworkDirectoryData`, `getAdNetworkPageData`, `getAdNetworkNichesPageData`, `getAdNetworkNichePageData`, and `getAdNetworkStatsPageData`. Both APIs use the same library gates. Table responses include `availableColumns`, `availableSorts`, `availableFilters`, actual filter values, and `effectiveSort`; the default page size is 10. Explicit page sizes from 1 to 100 remain supported. A stale shared URL asking for a missing traffic/age sort falls back to domain order.
 
-Broad niche comparisons open once 100 sites are classified. Since Oct 23, 2026 there's no coverage share rule, so a slow or failed weekly sort can't hide published niche pages. Before that, niche controls are empty and niche-page loaders return null. A network’s own Top niches block is available with one sorted site and uses the sorted sample as its denominator. Site-level fields use null individually; traffic, age, platform and publisher aggregates stay empty until populated. Manager charts hide when every manager is unknown. Niche indexing still needs the separate 500-site publication decision.
+Broad niche comparisons open once 100 sites are classified. Since Oct 23, 2026 there's no coverage share rule, so a slow or failed daily sort can't hide published niche pages. Before that, niche controls are empty and niche-page loaders return null. A network’s own Top niches block is available with one sorted site and uses the sorted sample as its denominator. Site-level fields use null individually; traffic, age, platform and publisher aggregates stay empty until populated. Manager charts hide when every manager is unknown. Niche indexing still needs the separate 500-site publication decision.
 
 History charts need two exact month-end snapshots. Partial months are excluded and no observations are interpolated. The annual headline needs a snapshot on the exact year-ago date (February 28 for a leap-day anniversary). A sparse older snapshot never becomes a mislabeled annual change. A 90-day net-change number needs a snapshot at least 90 days old and then counts real joins minus departures. Imported baseline events never enter joins, new-site totals, or newest-site lists. Weekly scan/checker stacks require stored source provenance for every site in the week. Publisher leaderboards admit conservative business-name suffixes and omit personal or ambiguous names. These gates carry `// data-gate:` comments in lib for the January 2027 follow-up.
 

content_docs/ad-networks/production-runbook.md

@@ -1,6 +1,6 @@
 # Ad-network production runbook
 
-Ad-network collectors, syncs, snapshots, and backups run as cron jobs on OVH. Weekly niche sorting runs on homelab. The directory, stats, 21 niche pages, 44 network pages, and checker were published on October 21, 2026. Snigel stays retired and returns 404. Automation updates existing pages; it does not publish new ones.
+Ad-network collectors, syncs, snapshots, and backups run as cron jobs on OVH. Daily niche sorting runs on homelab. The directory, stats, 21 niche pages, 44 network pages, and checker were published on October 21, 2026. Snigel stays retired and returns 404. Automation updates existing pages; it does not publish new ones.
 
 Cron is running again since October 20, 2026, 15:00 UTC, on self-hosted Postgres 17. It was paused from October 19, 12:15 UTC, until the sync sent only changed rows. History: [OVH cron plan](../../dev-space/ad-networks-ovh-cron-plan.md).
 
@@ -18,7 +18,7 @@ On OVH (`ssh ovh`, user `ubuntu`, UTC):
 
 The env file also sets `GITHUB_ALERT_TOKEN`, `AD_NETWORK_DB_PATH`, and `AD_NETWORKS_STATE_DIRECTORY`. Never print it. Homelab owns env values; Mac copies must not be edited.
 
-The wrapper is [`run-job.sh`](../../scripts/ad-networks/ovh/run-job.sh). The full schedule is [`crontab.txt`](../../scripts/ad-networks/ovh/crontab.txt). Homelab's `stack127-ad-network-niche.timer` runs Sunday at 06:00 UTC.
+The wrapper is [`run-job.sh`](../../scripts/ad-networks/ovh/run-job.sh). The full schedule is [`crontab.txt`](../../scripts/ad-networks/ovh/crontab.txt). Homelab's `stack127-ad-network-niche.timer` runs daily at 06:00 UTC (11:30 IST), with up to 3,000 sites per run.
 
 ## Counts in titles and headings
 
@@ -41,7 +41,9 @@ Sync captures daily counts and arrival observations, then closes its connection.
 - Exit 75 means the sync lock is held. Scheduled jobs retry every minute for up to six hours, skipping the collector on retries.
 - Verification yields while a scheduled job waits. Its local checkpoints will be picked up by the next sync.
 - Exit 76 means the same job is already running under `flock`.
-- Niche imports use the reviewed-sync lock and preserve existing labels during collector syncs.
+- Niche imports use the reviewed-sync lock and preserve existing labels during collector syncs. Model calls retry once, then split failed batches from 100 to 50 to 25 sites. Valid partial answers are saved before retrying the missing sites.
+- Niche sorting stops on operator signals, database failures, login failures, or Claude usage limits. Wait for a subscription limit to reset; do not enable paid extra usage. Keep the existing checkpoint files when resuming.
+- Niche status reports `candidates`, `classified`, `imported`, `withheld`, and `unresolved`. A run with unresolved sites reports `partial` and keeps its failure alert open; those sites remain eligible for the next run. CLI diagnostics are bounded and sanitized.
 - Failures open one GitHub issue per job. The next successful run closes it.
 
 Verification runs at minute 40 each hour, with four workers, a 1,000-site limit, and a ten-minute budget. Daily jobs start at 00:10, 01:00, 02:00, 03:00, and 05:00 UTC. Weekly discovery and publisher-name jobs, and monthly traffic, start at 04:00 UTC.