Skip to content

Extend reviewed CLI fact corrections with resumable chunks #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RaviTejaKNTS merged 2 commits into from
Nov 2, 2026
Merged

Extend reviewed CLI fact corrections with resumable chunks #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ed0c04
Extend reviewed CLI corrections with resumable chunks and repository …
RaviTejaKNTS Nov 1, 2026
5391315
Fix case-insensitive repository relinking and preflight validation
RaviTejaKNTS Nov 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions scripts/cli-apps/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Reviewed CLI app corrections

Use Node.js 22.18 or newer (the guard imports the directory's TypeScript option lists).

```bash
node scripts/cli-apps/apply-reviewed-corrections.mjs \
--env .env.local --file /path/to/reviewed-batch.json
```

This runs every chunk and rolls it back. Add `--apply` to commit. The default
chunk size is 25 apps; `--chunk-size` accepts integers from 1 to 25. Each chunk
commits app changes, field sources, public dates, and cache jobs together. A
failure stops the run and rolls back that chunk. Earlier committed chunks stay
applied. Re-run the same batch to resume: identical fields skip without touching
app timestamps, provenance, public dates, or cache jobs. This is chunk-level
atomicity, not a transaction across the entire file.

Every field needs its reviewed old value and an HTTPS evidence URL:

```json
{
"apps": [{
"slug": "example",
"reviewed_at": "2026-11-02",
"fields": { "platforms": ["linux", "macos"], "install_methods": ["brew", "binary"] },
"expected": { "platforms": ["linux"], "install_methods": ["binary"] },
"sources": {
"platforms": "https://example.org/docs/platforms",
"install_methods": "https://example.org/docs/install"
}
}]
}
```

Allowed fields: `name`, `summary`, `website_url`, `repository_url`, `docs_url`,
`license`, `language`, `platforms`, `categories`, `primary_type`, `additional_types`,
`pricing_model`, `source_model`, `aliases`, and `install_methods`.

Lists replace the entire field. Values must be unique. Categories need 1 to 3
entries. Platforms, categories, types, prices, source models, and install methods
use the site's allowed values. Aliases accept nonempty text. Empty arrays are
allowed except for categories. Only `website_url`, `docs_url`, `license`, and
`language` accept null. URLs and sources must be valid HTTPS URLs. Summary rules
still apply. All input is checked before the first transaction; prior values
are checked again under row locks. Current values must match either the expected
old value or the requested value. Any third value stops the chunk.

A changed `repository_url` also changes `github_project_id` in the same update.
GitHub owner/repo parsing and normalization match the refresh script. Existing
projects are matched case-insensitively and reused without updating their facts.
If older imports left case variants, the most recently checked row wins, with
a stable ID tie-breaker. Before inserting a missing project, a transaction-scoped
table lock serializes writers and the lookup runs again. This prevents duplicate
creation without changing existing rows or the database schema. Missing projects get a minimal
row with `checked_at = null`. GitHub owner/repo parsing also runs during whole-batch
validation, so malformed URLs stop the run before any chunk commits.
Non-GitHub repositories clear the link. This derived
link cannot be supplied as a reviewed field. Evidence is recorded for the
`repository_url` correction.

Published app changes use the existing `/cli-apps` public-date and cache triggers.
The tool never publishes records or changes `last_verified_at`. Dry runs roll
back trigger effects and any newly created GitHub projects too. A dry run does
not reserve values; a later apply checks them again.

For slow links, each query has a 180-second client timeout, a 120-second database
statement timeout, and a 15-second lock timeout. A connection or commit error
can leave the commit outcome uncertain; re-run the same file to check and resume.
38 changes: 9 additions & 29 deletions scripts/cli-apps/apply-reviewed-corrections.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,43 +1,23 @@
#!/usr/bin/env node
// Partial corrections only. No publishing, deletion, or whole-record date changes.
import { isDeepStrictEqual } from 'node:util';
import { readFile } from "node:fs/promises";
import { parseArgs, parseEnv } from "node:util";
import pg from "pg";
import { guardCorrection } from "./lib/reviewed-corrections.mjs";
import { applyCorrections, validateBatch } from "./lib/reviewed-corrections.mjs";

const { values } = parseArgs({ options: { env: { type: "string" }, file: { type: "string" }, apply: { type: "boolean", default: false } } });
if (!values.env || !values.file) throw new Error("Use --env <env file> --file <reviewed batch> [--apply]. Default is a rollback-only dry run.");
const { values } = parseArgs({ options: { env: { type: "string" }, file: { type: "string" }, apply: { type: "boolean", default: false }, "chunk-size": { type: "string", default: "25" } } });
if (!values.env || !values.file) throw new Error("Use --env <env file> --file <reviewed batch> [--apply] [--chunk-size 25]. Default is a rollback-only dry run.");
const environment = parseEnv(await readFile(values.env, "utf8"));
if (!environment.DATABASE_URL) throw new Error("Selected env file has no DATABASE_URL");
const batch = JSON.parse(await readFile(values.file, "utf8"));
if (!Array.isArray(batch.apps) || batch.apps.length === 0) throw new Error("Batch must contain reviewed apps");
if (new Set(batch.apps.map(app => app.slug)).size !== batch.apps.length) throw new Error("Duplicate slug in correction batch");
const pool = new pg.Pool({ connectionString: environment.DATABASE_URL, max: 1, connectionTimeoutMillis: 15000, query_timeout: 15000 });
const chunkSize = Number(values["chunk-size"]);
validateBatch(batch, chunkSize);
// Small transactions bound upload size; allow time for slow uplinks on each query.
const pool = new pg.Pool({ connectionString: environment.DATABASE_URL, max: 1, connectionTimeoutMillis: 30000, query_timeout: 180000, statement_timeout: 120000, lock_timeout: 15000 });
try {
const client = await pool.connect();
try {
await client.query("begin");
for (const record of batch.apps) {
const current = (await client.query("select * from public.terminal_apps where slug = $1 for update", [record.slug])).rows[0];
if (!current) throw new Error(`Missing CLI app: ${record.slug}`);
guardCorrection(record, current);
const fields = Object.keys(record.fields).filter(field=>!isDeepStrictEqual(current[field],record.fields[field]));
// Identifiers come exclusively from the guard's fixed allowlist.
if (fields.length) await client.query(`update public.terminal_apps set ${fields.map((field, index) => `${field} = $${index + 2}`).join(", ")}, updated_at = now() where id = $1`, [current.id, ...fields.map(field => record.fields[field])]);
for (const field of fields) {
await client.query(`insert into public.terminal_app_field_sources (app_id, field_name, source, source_url, confidence, checked_at, note)
values ($1, $2, 'stack127_primary_source_review', $3, 'high', $4::timestamptz, $5)
on conflict (app_id, field_name) do update set source = excluded.source, source_url = excluded.source_url,
confidence = excluded.confidence, checked_at = excluded.checked_at, note = excluded.note`,
[current.id, field, record.sources[field], record.reviewed_at, record.notes?.[field] ?? "Partial field review; other app facts remain unverified by this batch."]);
}
console.log(`${values.apply ? "Applied" : "Checked"} CLI correction: ${record.slug}`);
}
await client.query(values.apply ? "commit" : "rollback");
console.log(`${values.apply ? "Committed" : "Dry run passed for"} ${batch.apps.length} CLI correction(s).`);
} catch (error) {
await client.query("rollback");
throw error;
const totals = await applyCorrections(client, batch, { apply: values.apply, chunkSize });
console.log(`${values.apply ? "Committed" : "Dry run passed for"} ${totals.apps} CLI correction(s); ${totals.changedApps} changed, ${totals.skippedApps} identical apps skipped.`);
} finally { client.release(); }
} finally { await pool.end(); }
136 changes: 125 additions & 11 deletions scripts/cli-apps/lib/reviewed-corrections.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,141 @@
import { parseRepo } from "../../self-host-apps/lib/github.mjs";
import { isDeepStrictEqual } from "node:util";
import { validateSummary } from "../../self-host-apps/lib/summary-rules.mjs";
import { platformOptions, pricingOptions, sourceModelOptions, installLabels } from "../../../app/(site)/(directories)/cli-apps/cli-app-options.ts";
import { terminalAppTypes, terminalAppCategories } from "../../../lib/terminal-app-taxonomy.ts";

export const correctionFields = new Set(["name", "summary", "website_url", "repository_url", "docs_url", "license", "language", "install_methods"]);
const arrayValues = {
platforms: new Set(platformOptions.map(option => option.value)),
categories: new Set(terminalAppCategories.map(category => category.slug)),
additional_types: new Set(terminalAppTypes),
install_methods: new Set(installLabels.keys()),
aliases: null,
};
const scalarValues = {
primary_type: new Set(terminalAppTypes),
pricing_model: new Set(pricingOptions.map(option => option.value)),
source_model: new Set(sourceModelOptions.map(option => option.value)),
};
// These columns are nullable in terminal_apps. Repository removal is outside this workflow.
const nullableFields = new Set(["website_url", "docs_url", "license", "language"]);
export const correctionFields = new Set(["name", "summary", "website_url", "repository_url", "docs_url", "license", "language", ...Object.keys(arrayValues), ...Object.keys(scalarValues)]);

function isHttps(value) {
if (typeof value !== "string" || !value.startsWith("https://") || /\s/.test(value)) return false;
try { return new URL(value).protocol === "https:" && Boolean(new URL(value).hostname); }
catch { return false; }
}

export function guardCorrection(record, current) {
if (!record.slug || !record.fields || Object.keys(record.fields).length === 0) throw new Error("A slug and reviewed fields are required");
if (!record || typeof record.slug !== "string" || !record.slug.trim() || !record.fields || Array.isArray(record.fields) || typeof record.fields !== "object" || Object.keys(record.fields).length === 0) throw new Error("A slug and reviewed fields are required");
const reviewTimestamp = Date.parse(record.reviewed_at);
if (!/^\d{4}-\d{2}-\d{2}$/.test(record.reviewed_at ?? "") || !Number.isFinite(reviewTimestamp) || new Date(reviewTimestamp).toISOString().slice(0, 10) !== record.reviewed_at) throw new Error("A valid review date is required");
for (const [field, value] of Object.entries(record.fields)) {
if (!correctionFields.has(field)) throw new Error(`${field}: partial correction not allowed`);
if (field === "install_methods") {
if (!Array.isArray(value) || !Array.isArray(current.install_methods) || new Set(value).size !== value.length || value.some(method => typeof method !== "string" || !current.install_methods.includes(method))) {
throw new Error("install_methods: this partial workflow only removes unsupported methods from the existing list");
}
} else if (value !== null && typeof value !== "string") throw new Error(`${field}: expected text or null`);
if (Object.hasOwn(arrayValues, field)) {
if (!Array.isArray(value) || new Set(value).size !== value.length || value.some(item => typeof item !== "string" || !item.trim() || (arrayValues[field] && !arrayValues[field].has(item)))) throw new Error(`${field}: expected unique allowed text values`);
if (field === "categories" && (value.length < 1 || value.length > 3)) throw new Error("categories: expected 1 to 3 reviewed values");
} else if (value === null ? !nullableFields.has(field) : typeof value !== "string" || !value.trim()) throw new Error(`${field}: expected nonempty text${nullableFields.has(field) ? " or null" : ""}`);
if (Object.hasOwn(scalarValues, field) && !scalarValues[field].has(value)) throw new Error(`${field}: invalid value`);
if (!Object.hasOwn(record.expected ?? {}, field)) throw new Error(`${field}: expected value required`);
if (!isDeepStrictEqual(current[field], record.expected[field])) throw new Error(`${record.slug}: ${field} changed since review`);
if (!/^https:\/\//.test(record.sources?.[field] ?? "")) throw new Error(`${field}: reviewed source URL required`);
if (field.endsWith("_url") && value !== null && !/^https:\/\//.test(value)) throw new Error(`${field}: HTTPS URL required`);
if (["summary", "name"].includes(field) && !value?.trim()) throw new Error(`${field}: cannot be empty`);
// A committed chunk can be replayed, but a third value still blocks the chunk.
if (!isDeepStrictEqual(current[field], record.expected[field]) && !isDeepStrictEqual(current[field], value)) throw new Error(`${record.slug}: ${field} changed since review`);
if (!isHttps(record.sources?.[field])) throw new Error(`${field}: reviewed source URL required`);
if (field.endsWith("_url") && value !== null && !isHttps(value)) throw new Error(`${field}: HTTPS URL required`);
if (field === "repository_url") githubRepository(value);
if (field === "summary") {
const { errors } = validateSummary(value);
if (errors.length) throw new Error(`${record.slug}: ${errors.join("; ")}`);
}
}
}

export function validateBatch(batch, chunkSize) {
if (!Number.isSafeInteger(chunkSize) || chunkSize < 1 || chunkSize > 25) throw new Error("chunk-size must be an integer from 1 to 25");
if (!Array.isArray(batch?.apps) || batch.apps.length === 0) throw new Error("Batch must contain reviewed apps");
if (new Set(batch.apps.map(app => app?.slug)).size !== batch.apps.length) throw new Error("Duplicate slug in correction batch");
// Validate all input before committing even the first chunk. DB comparisons happen under row locks.
for (const record of batch.apps) guardCorrection(record, record?.expected ?? {});
}

function githubRepository(url) {
if (new URL(url).hostname.toLowerCase() !== "github.com") return null;
const parsed = parseRepo(url);
const owner = parsed.owner.trim().toLowerCase();
const repo = parsed.repo.trim().toLowerCase();
if (!owner || !repo) throw new Error("GitHub owner and repository are required");
return { owner, repo };
}

async function repositoryProjectId(client, url) {
const repository = githubRepository(url);
if (!repository) return null;
const { owner, repo } = repository;
// Older imports preserve casing, and some repos already have multiple case variants.
// Prefer the most recently checked row, with a stable ID tie-breaker.
const find = () => client.query("select id from public.github_projects where lower(owner) = $1 and lower(repo) = $2 order by checked_at desc nulls last, id limit 1", [owner, repo]);
const existing = (await find()).rows[0];
if (existing) return existing.id;
// There is only a case-sensitive unique index. Serialize missing-project inserts
// with every writer (including importers that do not use advisory locks), then
// recheck after the lock has waited for any in-flight inserts to commit.
await client.query("lock table public.github_projects in share row exclusive mode");
const afterLock = (await find()).rows[0];
if (afterLock) return afterLock.id;
const inserted = await client.query(`insert into public.github_projects (owner, repo, full_name, repository_url, checked_at)
values ($1, $2, $3, $4, null) on conflict (owner, repo) do nothing returning id`,
[owner, repo, `${owner}/${repo}`, `https://github.com/${owner}/${repo}`]);
// A concurrent importer may have created the project after the first lookup.
return inserted.rows[0]?.id ?? (await find()).rows[0].id;
}

export async function applyCorrections(client, batch, { apply = false, chunkSize = 25, report = console.log } = {}) {
validateBatch(batch, chunkSize);
const totals = { apps: 0, changedApps: 0, changedFields: 0, skippedApps: 0 };
const chunkCount = Math.ceil(batch.apps.length / chunkSize);
for (let offset = 0; offset < batch.apps.length; offset += chunkSize) {
const records = batch.apps.slice(offset, offset + chunkSize);
const label = `Chunk ${offset / chunkSize + 1}/${chunkCount}`;
const counts = { apps: records.length, changedApps: 0, changedFields: 0, skippedApps: 0 };
let finishing = false;
try {
await client.query("begin");
// No trigger suppression: public dates and cache jobs commit/roll back with the app changes.
for (const record of records) {
const current = (await client.query("select * from public.terminal_apps where slug = $1 for update", [record.slug])).rows[0];
if (!current) throw new Error(`Missing CLI app: ${record.slug}`);
guardCorrection(record, current);
const fields = Object.keys(record.fields).filter(field => !isDeepStrictEqual(current[field], record.fields[field]));
if (!fields.length) { counts.skippedApps++; continue; }
// Identifiers come exclusively from the guard's fixed allowlist.
const updateFields = [...fields];
const updateValues = fields.map(field => record.fields[field]);
if (fields.includes("repository_url")) {
updateFields.push("github_project_id");
updateValues.push(await repositoryProjectId(client, record.fields.repository_url));
}
await client.query(`update public.terminal_apps set ${updateFields.map((field, index) => `${field} = $${index + 2}`).join(", ")}, updated_at = now() where id = $1`, [current.id, ...updateValues]);
// One small provenance query per app rather than one round trip per field.
await client.query(`insert into public.terminal_app_field_sources (app_id, field_name, source, source_url, confidence, checked_at, note)
select $1, data.field_name, 'stack127_primary_source_review', data.source_url, 'high', $2::timestamptz, data.note
from unnest($3::text[], $4::text[], $5::text[]) as data(field_name, source_url, note)
on conflict (app_id, field_name) do update set source = excluded.source, source_url = excluded.source_url,
confidence = excluded.confidence, checked_at = excluded.checked_at, note = excluded.note`,
[current.id, record.reviewed_at, fields, fields.map(field => record.sources[field]), fields.map(field => record.notes?.[field] ?? "Partial field review; other app facts remain unverified by this batch.")]);
counts.changedApps++;
counts.changedFields += fields.length;
}
finishing = true;
await client.query(apply ? "commit" : "rollback");
} catch (error) {
let rolledBack = false;
try { await client.query("rollback"); rolledBack = true; } catch { /* Preserve the original error if the connection is lost. */ }
const outcome = finishing || !rolledBack ? "failed; commit outcome may be uncertain" : "rolled back";
report(`${label}: ${outcome}; stopped. Earlier committed chunks remain applied. Re-run the same file to resume (identical fields skip).`);
throw new Error(`${label}: ${error.message}`, { cause: error });
}
for (const key of Object.keys(totals)) totals[key] += counts[key];
report(`${label}: ${apply ? "committed" : "dry-run rolled back"}; ${counts.apps} apps, ${counts.changedApps} changed, ${counts.changedFields} fields, ${counts.skippedApps} identical apps skipped.`);
}
return totals;
}
Loading
Loading