From a1563786f2466bf5c80dd882e76202da08c947a7 Mon Sep 17 00:00:00 2001 From: test Date: Thu, 13 Aug 2026 23:54:50 +0000 Subject: [PATCH 1/3] Escape captured control sequences where a query result is rendered for a person (#752) Co-Authored-By: Claude --- ...ed-text-is-escaped-for-display.decision.md | 170 ++++++++++++++++++ src/core/query/format.js | 73 +++++++- src/core/query/overview.js | 15 +- src/core/util/index.js | 1 + src/core/util/json_util.js | 112 ++++++++++-- test/core/query-format-escaping.test.js | 115 ++++++++++++ test/core/query-overview.test.js | 26 +++ test/core/util-json-util.test.js | 52 ++++++ 8 files changed, 533 insertions(+), 31 deletions(-) create mode 100644 llp/0224-captured-text-is-escaped-for-display.decision.md create mode 100644 test/core/query-format-escaping.test.js diff --git a/llp/0224-captured-text-is-escaped-for-display.decision.md b/llp/0224-captured-text-is-escaped-for-display.decision.md new file mode 100644 index 00000000..5c87bfa3 --- /dev/null +++ b/llp/0224-captured-text-is-escaped-for-display.decision.md @@ -0,0 +1,170 @@ +# LLP 0224: captured text is escaped where it is rendered for a person, never where it is rendered for a program + +**Type:** Decision +**Status:** Accepted +**Systems:** CLI, Query, Observability +**Author:** Claude +**Date:** 2026-08-13 +**Related:** LLP 0189 (#choke-point: severity colour is applied where stderr is bound, the same one-place-per-stream shape), LLP 0164 (why `sanitizeLabel` strips, and why the zero-width group is in its class), LLP 0054 (the context controls that already clip every cell before it is rendered) + +> `hyp query sql` wrote captured bytes straight to `process.stdout`. Every +> string column of every dataset is verbatim captured text, so an `ESC` in a +> prompt, a log body, an HTTP header value or a filename reached the operator's +> terminal and was interpreted: a row could hide itself, overwrite the row above +> it, or reorder what was read. This settles that the *output format* decides +> whether captured text is escaped, that the escape is visible rather than a +> silent strip, and that one character vocabulary serves both this and the +> existing label-stripping policy. + +## Context {#context} + +`formatCell` returned a string value unchanged, and the `query sql` verb wrote +the rendered table to `process.stdout`, which is not wrapped the way `dispatch` +wraps stderr ([LLP 0189](./0189-cli-severity-colour.decision.md) #choke-point). +`--format markdown` escaped only `|` and newline. `json` and `jsonl`, and the +MCP `query_sql` path, were safe only incidentally, through `JSON.stringify` +([#752](https://github.com/hyparam/hypaware/issues/752)). + +Three facts make this worth a decision rather than a patch. + +**The data is captured, so there is no upstream to fix.** `content_text` is +prose a model or a user wrote. A `logs` body is whatever the observed process +emitted. `traces` carries HTTP header values and filenames. None of it passes +through a regex or a parser that could have bounded it, and none of it is ours. +The base64 strip marker in the gateway's message projector is how this surfaced +([#748](https://github.com/hyparam/hypaware/pull/748) round 1), and fixing that +regex would have fixed nothing: an `ESC` arrives in a cell with no regex +involved at all. + +**The repo already treats this threat as real, one plane over.** `sanitizeLabel` +strips control, bidi and zero-width characters out of status-file labels, and +its doc comment names terminal repainting as the reason. The query plane is +where far more attacker-influenced text reaches a terminal, and it was the plane +with no policy at all. + +**The operator reading a doctored table is the person triaging hostile +traffic.** That is exactly who must not be shown a row that lies about itself. + +## Decision {#decision} + +**A render escapes captured text if and only if a person reads it.** + +- `table` and `markdown` go through `formatCell`, which escapes. +- `json` and `jsonl` stay byte-exact, so the data stays extractable and a + pipeline still receives what was captured. +- The `--output` spill receipt escapes its preview even though the file it + describes may be `jsonl`: the receipt is a human render in its own right, and + the file is the machine copy. + +**The switch is the format, never `process.stdout.isTTY`.** {#format-not-tty} + +A TTY gate was the obvious alternative and is worse. It makes one command print +different bytes depending on whether it is piped, so what an operator saw and +what they saved to a file disagree, and a bug reproduces differently under +redirection. That is a second, invisible mode in a command whose whole job is +reporting what was captured. A format flag is already in the operator's hand, +already documented, and already the thing that decides every other rendering +question; the pipeline that wants exact bytes asks for `--format jsonl` and gets +them on a TTY too. + +Doing nothing was the third option (declare `hyp query sql` output untrusted, +like `cat` on a binary file). Rejected: `cat` has no idea what its bytes are, +while this renderer knows every cell it is laying out and has already clipped +each one for context budget. A renderer that structures output into aligned +columns has already promised the columns mean something. + +**Escape, do not strip.** {#escape-not-strip} + +`sanitizeLabel` drops the bytes because a label *names* a surface and a stripped +name is still a usable name. A rendered cell *is* the payload the operator asked +to see, so dropping bytes silently turns a query into a lie about what was +captured. The row stays honest: an `ESC` that was in the data renders as +`\u001b`, visibly and greppably. + +Consequences accepted with this: + +- The escape is the familiar JavaScript spelling (`\n`, `\r`, `\t`, `\uXXXX`), + which is pure ASCII and one column per character, so column widths measured on + escaped text are still display widths. +- A backslash already present in the value is **not** doubled. Captured data is + full of Windows paths, regexes and JSON blobs; mangling all of them to + disambiguate a literal `\n` from an escaped newline would cost far more + legibility than the ambiguity does. The ambiguity is cosmetic, and neither + spelling can move a cursor. +- A newline inside a `table` cell now prints as `\n` instead of breaking the + row. That is a visible change to multi-line prose, and it is the point: a + newline in a cell is how a captured value forges a row. + +**One vocabulary, two policies.** {#one-vocabulary} + +The unsafe-character class is *not* duplicated. It is decomposed into three +named groups (terminal control, bidi formatting, invisible formatting) in +`src/core/util/json_util.js`, and both policies are built from those groups in +that one file: + +- `sanitizeLabel` strips all three, exactly as before. The recomposed class is + the same set of code points as the literal it replaces, and a test asserts + that over the whole BMP so the refactor cannot have quietly widened or + narrowed it. +- `escapeForDisplay` escapes the first two only. + +A second hand-written class somewhere else would be a second chance for the two +to drift apart about what "unsafe" means, which is the failure mode this shape +exists to prevent. + +**The display plane escapes control and bidi, not zero-width.** {#escape-class} + +| group | in a label | in a cell | why | +| --- | --- | --- | --- | +| C0, DEL, C1, U+2028/2029 | stripped | escaped | moves the cursor, erases lines, opens escape sequences, forges rows | +| bidi marks, embeddings, overrides, isolates | stripped | escaped | reorders text that follows, past the end of the value | +| zero-width and default-ignorable (ZWSP, ZWNJ, ZWJ, word joiner, BOM, soft hyphen, variation selectors) | stripped | **left alone** | see below | + +The zero-width group is stripped from a label because a label is a **map key**: +two labels that render identically but compare unequal dilute the entrypoint +tracker's eviction cap. A query cell is not a key, so nothing downstream is +diluted. Against that, ZWJ and the variation selectors are load-bearing inside +ordinary emoji (a family emoji *is* a ZWJ sequence, and U+FE0F is what makes a +heart red), so escaping them would visibly corrupt legitimate captured prose on +a large fraction of real rows, in exchange for defending against a character +that cannot repaint anything. + +Confusables remain out of scope on both planes, for the reason already recorded +against the label class: this bounds what a value *does*, not what it looks +like. + +## Scope {#scope} + +The rule stated above is general. What is *implemented* with this document is +the query plane, which is the whole of `src/core/query/`: + +- `format.js`, the renderer behind `hyp query sql`, the `--output` receipt, and + the `vector-search` plugin's result output. +- `overview.js`, the block behind `hyp query overview` and the wizard's closing + first look, whose `provider`, `model`, `date`, `tool_name` and `repo_root` + columns are the same captured strings. Here the escape sits on each captured + value rather than on the assembled row, because this block paints its own + bars and headings: a sweep over the finished table would strip the colour + along with the attack. + +Every other CLI surface that prints a captured value is a separate change and +is deliberately not swept in. `hyp graph neighbors` renders node labels lifted +straight off `ai_gateway_messages` (`client_name`, `model`, `tool_name`, a +`file_path` basename out of a `tool_use` block) with a 48-character clamp and no +stripping. `hyp status`, `hyp daemon status`, `hyp purge`, `hyp policy show`, +`hyp session`, `hyp backfill plan` and the client attach adapters each echo +values read back out of files this product does not own. Those are label-plane +surfaces where `sanitizeLabel` is already the established answer, so each needs +its own argument about strip-versus-escape rather than an automatic import of +this one. + +## Verification {#verification} + +Unit tests, each shown failing against the pre-fix source before being kept: +`ESC` in a `table` cell, `ESC` and bidi in a `markdown` cell, `json`/`jsonl` +byte-exactness, non-ASCII and emoji survival, and column alignment with an +escaped cell. + +**Not verified:** no test here renders into a real terminal emulator. The claim +that `\u001b` is inert, and that a raw `ESC` sequence is not, rests on the +escape output being pure printable ASCII rather than on an observed terminal. diff --git a/src/core/query/format.js b/src/core/query/format.js index 1b520026..616ce8de 100644 --- a/src/core/query/format.js +++ b/src/core/query/format.js @@ -1,5 +1,7 @@ // @ts-check +import { escapeForDisplay } from '../util/json_util.js' + /** * @import { ContextControls, ContextControlsResult, QueryFormat, QueryResultSet } from '../../../src/core/query/types.js' */ @@ -139,6 +141,14 @@ function rowBytes(row) { * - `markdown` GitHub-flavoured table * - `table` fixed-width text table (default) * + * The format, and only the format, decides whether cell values are escaped + * for a terminal: `table` and `markdown` are read by a person and go through + * `formatCell`, which escapes; `json` and `jsonl` are read by a program and + * stay byte-exact. Deliberately not a TTY test, so a query redirected to a + * file holds exactly the bytes its terminal showed. + * + * @ref LLP 0224#format-not-tty [implements]: escaping is chosen by format, never by isTTY + * * @param {QueryResultSet} result * @param {QueryFormat} format * @returns {string} @@ -182,7 +192,11 @@ export function jsonReplacer(_key, value) { function renderTable(result) { const columns = result.columns.length > 0 ? result.columns : inferColumns(result.rows) if (columns.length === 0) return '(no rows)\n' - if (result.rows.length === 0) return columns.join(' ') + '\n(no rows)\n' + // Headers are schema names rather than captured data, so escaping them is + // defence in depth, not the fix; it costs one map and means no part of the + // rendered table is exempt. + const headers = columns.map(escapeForDisplay) + if (result.rows.length === 0) return headers.join(' ') + '\n(no rows)\n' const cells = result.rows.map((row) => { /** @type {Record} */ @@ -190,14 +204,17 @@ function renderTable(result) { for (const column of columns) out[column] = formatCell(row[column]) return out }) - const widths = columns.map((column) => { - let width = column.length + // Widths are measured on the escaped text, which is what actually gets + // padded and printed. Every escape this file emits is ASCII, one column + // per character, so a measured length is still a display width. + const widths = columns.map((column, i) => { + let width = headers[i].length for (const row of cells) width = Math.max(width, row[column]?.length ?? 0) return Math.min(width, 80) }) const lines = [ - columns.map((column, i) => column.padEnd(widths[i])).join(' ').trimEnd(), + headers.map((header, i) => header.padEnd(widths[i])).join(' ').trimEnd(), columns.map((_, i) => '-'.repeat(widths[i])).join(' ').trimEnd(), ] for (const row of cells) { @@ -213,7 +230,7 @@ function renderTable(result) { function renderMarkdown(result) { const columns = result.columns.length > 0 ? result.columns : inferColumns(result.rows) if (columns.length === 0) return '_(no rows)_\n' - const header = '| ' + columns.join(' | ') + ' |' + const header = '| ' + columns.map((c) => mdEscape(escapeForDisplay(c))).join(' | ') + ' |' const divider = '| ' + columns.map(() => '---').join(' | ') + ' |' if (result.rows.length === 0) return [header, divider, '_(no rows)_'].join('\n') + '\n' const body = result.rows.map((row) => { @@ -232,10 +249,37 @@ function inferColumns(rows) { } /** + * One cell of a human-facing render, escaped so it cannot drive the + * terminal it is about to be written to. + * + * Every string column of every dataset is captured verbatim: `content_text` + * is prose a model or a user wrote, and a log body, an HTTP header value or + * a filename is whatever the observed process emitted. So an `ESC` reaches + * a cell with no parser or regex involved, and `hyp query sql` writes the + * rendered table straight to `process.stdout`, which (unlike stderr at + * `dispatch`) wraps nothing. A row could otherwise hide itself, overwrite + * the row above it, or reorder what the operator reads. + * + * The escape is applied to the finished text rather than only to the string + * branch, because the object branch is not safe either: `JSON.stringify` + * escapes C0 but passes C1 and every bidi override through untouched. + * Running it over already-stringified JSON is a no-op for the parts + * `JSON.stringify` already handled. + * + * @ref LLP 0224#decision [implements]: human-facing formats escape, machine formats do not + * * @param {unknown} value * @returns {string} */ function formatCell(value) { + return escapeForDisplay(rawCell(value)) +} + +/** + * @param {unknown} value + * @returns {string} + */ +function rawCell(value) { if (value === null || value === undefined) return '' if (typeof value === 'string') return value if (typeof value === 'bigint') return value.toString() @@ -245,12 +289,14 @@ function formatCell(value) { } /** - * Pipe characters break Markdown tables; backslash-escape them. + * Pipe characters break Markdown tables; backslash-escape them. Newlines + * cannot reach here any more: every caller passes text that has already + * been through `escapeForDisplay`, which spells a newline `\n`. * * @param {string} value */ function mdEscape(value) { - return value.replace(/\|/g, '\\|').replace(/\n/g, ' ') + return value.replace(/\|/g, '\\|') } /** @@ -297,6 +343,13 @@ export function buildQuerySqlOutput(full, opts) { * * @param {string} outputPath * @param {{ columns: string[], rows: Record[] }} full + * The receipt is a human-facing render in its own right, whatever + * `--format` the file got, so its preview is escaped too. It is escaped one + * rendered line at a time: the newlines between preview rows are structure + * this function produced, not captured bytes, and must survive. + * + * @ref LLP 0224#decision [constrained-by]: the receipt is a human render, so it escapes + * * @param {string} content the already-rendered file content (sized for the receipt) * @returns {string} */ @@ -306,7 +359,7 @@ function renderSpillReceipt(outputPath, full, content) { const lines = [ `wrote ${full.rows.length} rows · ${cols.length} cols · ${bytes}B → ${outputPath}`, ] - if (cols.length > 0) lines.push(`schema: ${cols.join(', ')}`) + if (cols.length > 0) lines.push(`schema: ${cols.map(escapeForDisplay).join(', ')}`) const previewRows = full.rows.slice(0, 3) if (previewRows.length > 0) { const { result: preview } = applyContextControls( @@ -314,7 +367,9 @@ function renderSpillReceipt(outputPath, full, content) { { maxCell: 80, maxBytes: 0 } ) lines.push(`preview (first ${previewRows.length}, cells clipped):`) - lines.push(renderResult(preview, 'jsonl').trimEnd()) + for (const line of renderResult(preview, 'jsonl').trimEnd().split('\n')) { + lines.push(escapeForDisplay(line)) + } } return lines.join('\n') + '\n' } diff --git a/src/core/query/overview.js b/src/core/query/overview.js index 331260ab..a9e4b91e 100644 --- a/src/core/query/overview.js +++ b/src/core/query/overview.js @@ -15,6 +15,7 @@ * @import { OverviewNotice, OverviewRows, OverviewQueryRunner, OverviewWindow } from '../../../src/core/query/types.js' */ +import { escapeForDisplay } from '../util/json_util.js' import { executeQuerySql } from './sql.js' import { renderLocalOnlyNotice } from './verb.js' // @ref LLP 0189#palette [implements]: one ANSI table for the whole CLI @@ -702,7 +703,7 @@ export function renderProviderMix(rows, color, showSql = false, sql = '') { // An unlabelled row only reaches the table if it carried tokens, which // today it never does. If that changes, name it for what it is rather // than dropping measured tokens on the floor. - truncate(hasModelLabel(r) ? String(r.model).trim() : '(model not recorded)', MAX_MODEL_WIDTH), + truncate(escapeForDisplay(hasModelLabel(r) ? String(r.model).trim() : '(model not recorded)'), MAX_MODEL_WIDTH), formatCount(r.input_tokens), formatCount(r.cached_tokens), formatCount(r.output_tokens), @@ -806,7 +807,7 @@ export function renderRepoMix(rows, color, showSql = false, sql = '') { // same thing here as in the models and daily tables. const max = Math.max(...shown.map((r) => toNumber(r.input_tokens) + toNumber(r.output_tokens))) const body = shown.map((r) => [ - shortRepo(String(r.repo_root)), + escapeForDisplay(shortRepo(String(r.repo_root))), formatCount(r.sessions), formatCount(r.input_tokens), formatCount(r.cached_tokens), @@ -1015,12 +1016,20 @@ function bar(value, max) { * (an unnamed model, an undated row), so render the absence explicitly * rather than emitting a blank column the reader has to interpret. * + * It is also where a captured group key (`provider`, `date`, `tool_name`) + * is escaped for the terminal. The escape belongs on the captured value and + * not on the assembled row, because this block emits `ESC` of its own: the + * bars and headings are painted, so a sweep over the finished table would + * erase the colour along with the attack. + * + * @ref LLP 0224#decision [implements]: the same rule on `hyp query overview`, per captured cell + * * @param {unknown} value * @returns {string} */ function cell(value) { if (value === null || value === undefined) return '(none)' - const text = String(value).trim() + const text = escapeForDisplay(String(value).trim()) return text.length === 0 ? '(none)' : text } diff --git a/src/core/util/index.js b/src/core/util/index.js index f2e24c6b..75da4c0c 100644 --- a/src/core/util/index.js +++ b/src/core/util/index.js @@ -20,6 +20,7 @@ export { VOLATILE_BLOCK_FIELDS, canonicalJson, errCode, + escapeForDisplay, isPlainObject, parseMaybeJson, sanitizeLabel, diff --git a/src/core/util/json_util.js b/src/core/util/json_util.js index e0eb2603..de608b0a 100644 --- a/src/core/util/json_util.js +++ b/src/core/util/json_util.js @@ -34,28 +34,62 @@ export function stringValue(value) { */ export const MAX_LABEL_CHARS = 120 -// Everything a label may contain that either drives the terminal or -// occupies no width on it. Three groups, one class: +// Everything a captured value may contain that either drives the terminal +// or occupies no width on it. Three groups, named separately because two +// different policies are built out of them (strip, in `sanitizeLabel`; +// escape, in `escapeForDisplay`), and a second hand-written character class +// would be a second chance for the two to disagree about what "unsafe" +// means. // -// - C0/DEL/C1 and the Unicode line/paragraph separators, so a label can -// never move the cursor, erase a line, open an escape sequence, or -// split into a second line. -// - Bidirectional formatting (embeddings, overrides, isolates, marks). -// These print nothing but reorder what follows, and an unterminated -// one keeps reordering past the end of the label into the rest of the -// status line. A label that renders as a different string than the one -// it stores defeats the point of naming a surface at all. -// - Zero-width and default-ignorable formatting (ZWSP/ZWNJ/ZWJ, word -// joiner, BOM, soft hyphen, variation selectors). These render as -// nothing, so keeping them lets two labels be distinct map keys while -// being indistinguishable on screen, which is what would otherwise -// dilute the tracker's eviction cap. -// -// Confusables are deliberately out of scope: this bounds what a label +// @ref LLP 0224#one-vocabulary: one class, two policies over its named groups + +// C0/DEL/C1 and the Unicode line/paragraph separators, so a value can +// never move the cursor, erase a line, open an escape sequence, or split +// into a second line. +const TERMINAL_CONTROL_CHARS = '\\u0000-\\u001F\\u007F-\\u009F\\u2028-\\u2029' + +// Bidirectional formatting (embeddings, overrides, isolates, marks). +// These print nothing but reorder what follows, and an unterminated one +// keeps reordering past the end of the value into the rest of the line. A +// value that renders as a different string than the one it stores defeats +// the point of showing it at all. +const BIDI_FORMATTING_CHARS = '\\u061C\\u200E-\\u200F\\u202A-\\u202E\\u2066-\\u2069' + +// Zero-width and default-ignorable formatting (ZWSP/ZWNJ/ZWJ, word +// joiner, BOM, soft hyphen, variation selectors). These render as +// nothing, so keeping them lets two labels be distinct map keys while +// being indistinguishable on screen, which is what would otherwise dilute +// the tracker's eviction cap. +const INVISIBLE_FORMATTING_CHARS = '\\u00AD\\u180E\\u200B-\\u200D\\u2060-\\u2064\\uFE00-\\uFE0F\\uFEFF' + +// Confusables are deliberately out of scope: this bounds what a value // *does*, not what it looks like. Two labels built from different but // similar-looking real letters stay distinct, as they must. -const UNSAFE_LABEL_CHARS = - /[\u0000-\u001F\u007F-\u009F\u00AD\u061C\u180E\u200B-\u200F\u2028-\u2029\u202A-\u202E\u2060-\u2064\u2066-\u2069\uFE00-\uFE0F\uFEFF]/g +const UNSAFE_LABEL_CHARS = new RegExp( + `[${TERMINAL_CONTROL_CHARS}${BIDI_FORMATTING_CHARS}${INVISIBLE_FORMATTING_CHARS}]`, + 'g' +) + +// What a rendered cell may not contain: a strict subset of +// `UNSAFE_LABEL_CHARS` holding only the characters that drive or reorder a +// terminal. The zero-width group is deliberately left alone here, unlike in +// a label, for two reasons. A query cell is prose, not a map key, so nothing +// downstream is diluted by two cells looking alike. And ZWJ and the +// variation selectors are load-bearing inside ordinary emoji (a family emoji +// is a ZWJ sequence), so escaping them would visibly corrupt legitimate +// captured text on a large fraction of real rows, to defend against a +// character that cannot repaint anything. +// +// @ref LLP 0224#escape-class [implements]: display escapes control and bidi, not zero-width +const DISPLAY_UNSAFE_CHARS = new RegExp(`[${TERMINAL_CONTROL_CHARS}${BIDI_FORMATTING_CHARS}]`, 'g') + +// The three C0 characters an operator reads more easily by name than by code +// point. Everything else in the class falls through to a `\uXXXX` escape. +const DISPLAY_NAMED_ESCAPES = new Map([ + ['\n', '\\n'], + ['\r', '\\r'], + ['\t', '\\t'], +]) // A high surrogate left stranded by the clamp below. Slicing counts UTF-16 // code units, so a cut can land between the halves of an astral character @@ -85,6 +119,10 @@ const TRUNCATION_MARKER = '...' * bytes: a label of astral characters is still roughly 4x `max` bytes * once encoded, which is why callers cap the count of labels too. * + * Sibling policy: `escapeForDisplay`, for the places where the value *is* + * the payload rather than a name for one, and losing bytes would be worse + * than showing them. + * * @param {unknown} value * @param {number} [max] * @returns {string | undefined} Cleaned non-empty string, else `undefined`. @@ -100,6 +138,42 @@ export function sanitizeLabel(value, max = MAX_LABEL_CHARS) { return head.length === 0 ? undefined : `${head}${TRUNCATION_MARKER}` } +/** + * Make a captured string safe to *print* without losing any of it: replace + * every character that drives or reorders a terminal with a visible escape, + * and change nothing else. + * + * This is the display-plane sibling of `sanitizeLabel`, over the same + * character vocabulary but under a different policy, because the two have + * different jobs. A label *names* a surface, so a stripped name is still a + * usable name and the shortest safe answer is to drop the bytes. A rendered + * cell *is* the captured payload the operator asked to see, so silently + * dropping bytes turns a query into a lie about what was captured: the row + * must stay honest about the ESC that was there. Hence escape, not strip, + * and hence no truncation and no `undefined` for empty (a cell that held an + * empty string is a cell, and the caller has already applied its own + * `--max-cell` clip). + * + * Escapes are the familiar JavaScript spellings: `\n`, `\r`, `\t`, and + * `\uXXXX` for everything else. The output is pure ASCII, one column per + * character, so a caller that pads to a computed column width stays aligned. + * A backslash already in the value is deliberately *not* doubled: captured + * data is full of Windows paths, regexes and JSON blobs, and mangling every + * one of them to disambiguate a literal two-character `\n` from an escaped + * newline would cost far more legibility than the ambiguity does. The + * ambiguity is cosmetic; neither spelling can move a cursor. + * + * @ref LLP 0224#escape-not-strip [implements]: the display plane escapes where the label plane strips + * + * @param {string} value + * @returns {string} + */ +export function escapeForDisplay(value) { + return value.replace(DISPLAY_UNSAFE_CHARS, (ch) => { + return DISPLAY_NAMED_ESCAPES.get(ch) ?? `\\u${ch.charCodeAt(0).toString(16).padStart(4, '0')}` + }) +} + /** * Parse `value` as JSON when it is a string, falling back to the * original value when it is not a string or does not parse. Projectors diff --git a/test/core/query-format-escaping.test.js b/test/core/query-format-escaping.test.js new file mode 100644 index 00000000..d96ae898 --- /dev/null +++ b/test/core/query-format-escaping.test.js @@ -0,0 +1,115 @@ +// @ts-check + +// The human-facing query formats must not hand a captured control sequence to +// the terminal, and the machine-facing ones must not alter a single byte. Both +// halves are asserted here, because the whole decision is that the *format*, +// and nothing else, picks between them. +// +// @ref LLP 0224#decision [tests]: table/markdown escape, json/jsonl stay byte-exact +// @ref LLP 0224#format-not-tty [tests]: no assertion here touches isTTY, and none needs to + +import test from 'node:test' +import assert from 'node:assert/strict' + +import { buildQuerySqlOutput, renderResult } from '../../src/core/query/format.js' + +const ESC = '\u001b' +const RLO = '\u202e' +const CSI = '\u009b' // C1 control introducer: JSON.stringify leaves it raw + +/** @param {Record[]} rows */ +const set = (rows) => ({ columns: rows[0] ? Object.keys(rows[0]) : [], rows }) + +// A hidden row, an overwritten row, and a reordered row: the three effects the +// issue names, in one cell each. +const attacks = set([ + { id: 1, content_text: `hide${ESC}[8m me${ESC}[0m` }, + { id: 2, content_text: `overwrite${ESC}[1A` }, + { id: 3, content_text: `reorder ${RLO}txetdesrever` }, +]) + +test('table format escapes every control and bidi character in a cell', () => { + const out = renderResult(attacks, 'table') + assert.equal(out.includes(ESC), false, 'no raw ESC reaches the rendered table') + assert.equal(out.includes(RLO), false, 'no raw bidi override reaches the rendered table') + assert.match(out, /hide\\u001b\[8m me\\u001b\[0m/) + assert.match(out, /overwrite\\u001b\[1A/) + assert.match(out, /reorder \\u202etxetdesrever/) +}) + +test('markdown format escapes control and bidi, and still escapes pipes', () => { + const out = renderResult(set([{ c: `a${ESC}[31mb|c${RLO}d` }]), 'markdown') + assert.equal(out.includes(ESC), false) + assert.equal(out.includes(RLO), false) + assert.match(out, /\| a\\u001b\[31mb\\\|c\\u202ed \|/) +}) + +test('a newline in a table cell cannot forge a row', () => { + const out = renderResult(set([{ id: 1, c: 'real\nnot-a-row' }]), 'table') + // Three lines only: header, divider, one row. + assert.equal(out.trimEnd().split('\n').length, 3) + assert.match(out, /real\\nnot-a-row/) +}) + +test('tab and carriage return get their familiar spellings', () => { + const out = renderResult(set([{ c: 'a\tb\rc' }]), 'table') + assert.match(out, /a\\tb\\rc/) +}) + +test('json and jsonl stay byte-exact, control characters included', () => { + const rows = [{ c: `x${ESC}[8m${RLO}${CSI}y` }] + const jsonl = renderResult(set(rows), 'jsonl') + assert.deepEqual(JSON.parse(jsonl.trimEnd()), rows[0]) + const json = renderResult(set(rows), 'json') + assert.deepEqual(JSON.parse(json), rows) + // Byte-for-byte: exactly what JSON.stringify produces, with nothing added. + assert.equal(jsonl, JSON.stringify(rows[0]) + '\n') + assert.equal(json, JSON.stringify(rows, null, 2) + '\n') + // And the bidi override really is still in there, unescaped by JSON. + assert.equal(jsonl.includes(RLO), true) +}) + +test('ordinary non-ASCII text is not touched by either human format', () => { + // Accents, CJK, RTL letters (not overrides), a ZWJ family emoji, and an + // emoji whose colour depends on a variation selector. All of this is + // legitimate captured prose and must survive verbatim. + const legit = 'caf\u00e9 \u65e5\u672c\u8a9e \u05e9\u05dc\u05d5\u05dd \ud83d\udc68\u200d\ud83d\udc69\u200d\ud83d\udc67 \u2764\ufe0f' + for (const format of /** @type {const} */ (['table', 'markdown', 'json', 'jsonl'])) { + assert.equal(renderResult(set([{ c: legit }]), format).includes(legit), true, format) + } +}) + +test('column widths and alignment survive an escaped cell', () => { + const out = renderResult( + set([ + { flag: `${ESC}[8m`, tail: 'A' }, + { flag: 'short', tail: 'B' }, + ]), + 'table' + ) + const [header, divider, first, second] = out.trimEnd().split('\n') + // The escaped cell is the widest, so it sets the column width. + const width = '\\u001b[8m'.length + assert.equal(divider.split(' ')[0].length, width) + assert.equal(header.indexOf('tail'), width + 2) + assert.equal(first.indexOf('A'), width + 2) + assert.equal(second.indexOf('B'), width + 2) +}) + +test('the spill receipt escapes its preview but the file it wrote does not', () => { + const rows = [{ c: `a${RLO}b` }, { c: `d${ESC}e` }] + const out = buildQuerySqlOutput(set(rows), { + format: 'jsonl', + output: '/tmp/spill-752.jsonl', + maxCell: 200, + maxBytes: 0, + }) + assert.equal(out.stdout.includes(RLO), false) + assert.equal(out.stdout.includes(ESC), false) + assert.match(out.stdout, /a\\u202eb/) + assert.match(out.stdout, /d\\u001be/) + // One preview line per row: the structural newlines are not escaped away. + assert.equal(out.stdout.trimEnd().split('\n').length, 5) + // The file is the machine copy and keeps every byte. + assert.equal(out.file?.content.includes(RLO), true) +}) diff --git a/test/core/query-overview.test.js b/test/core/query-overview.test.js index a138badf..4f693a6f 100644 --- a/test/core/query-overview.test.js +++ b/test/core/query-overview.test.js @@ -1016,3 +1016,29 @@ test('hyp query overview: the usage line lists every flag the codec accepts', as assert.match(stdout.text(), new RegExp(flag.replace(/-/g, '\\-')), flag) } }) + +// @ref LLP 0224#decision [tests]: the overview's captured columns are escaped too +test('renderOverview escapes captured columns and keeps its own colour', () => { + const ESC = '\u001b' + const RLO = '\u202e' + const usage = { input_tokens: 10, cached_tokens: 0, output_tokens: 5 } + const out = renderOverview({ + providerRows: [{ provider: `anth${ESC}[8mropic`, model: `claude${RLO}-opus`, ...usage }], + dailyRows: [{ date: `2026-08-1${ESC}[1A`, sessions: 1, ...usage }], + repoRows: [{ repo_root: `/a/b${ESC}[2K`, sessions: 1, ...usage }], + toolRows: [{ tool_name: `Ba${ESC}[8msh`, calls: 3, sessions: 1 }], + // Colour on, so the painted bars keep their own ESC. That is what proves + // the escape is applied per captured cell rather than swept over the + // finished block, which would take the colour with it. + color: true, + }) + assert.match(out, /anth\\u001b\[8mropic/) + assert.match(out, /claude\\u202e-opus/) + assert.match(out, /2026-08-1\\u001b\[1A/) + assert.match(out, /b\\u001b\[2K/) + assert.match(out, /Ba\\u001b\[8msh/) + // Every raw ESC that is left is one of our own colour codes. + for (const found of out.match(/\u001b.{0,4}/g) ?? []) { + assert.match(found, /^\u001b\[[0-9;]*m/, JSON.stringify(found)) + } +}) diff --git a/test/core/util-json-util.test.js b/test/core/util-json-util.test.js index 4392232f..202e5a84 100644 --- a/test/core/util-json-util.test.js +++ b/test/core/util-json-util.test.js @@ -6,8 +6,10 @@ import assert from 'node:assert/strict' import { canonicalJson, errCode, + escapeForDisplay, isPlainObject, parseMaybeJson, + sanitizeLabel, sha256Hex, sortKeys, stringValue, @@ -64,3 +66,53 @@ test('errCode extracts string codes and nothing else', () => { assert.equal(errCode(null), undefined) assert.equal(errCode('ENOENT'), undefined) }) + +// The character class `sanitizeLabel` strips is now composed from three named +// groups so that `escapeForDisplay` can reuse two of them instead of a second +// hand-written literal. A refactor of a security class has to prove it moved no +// code point, so this is the literal it replaced, held here as the oracle. +// +// @ref LLP 0224#one-vocabulary [tests]: the recomposed class is the same set of code points +const UNSAFE_LABEL_CHARS_BEFORE_LLP_0224 = + /[\u0000-\u001F\u007F-\u009F\u00AD\u061C\u180E\u200B-\u200F\u2028-\u2029\u202A-\u202E\u2060-\u2064\u2066-\u2069\uFE00-\uFE0F\uFEFF]/ + +test('sanitizeLabel strips exactly the code points it stripped before LLP 0224', () => { + /** @type {number[]} */ + const disagreements = [] + for (let code = 0; code <= 0xffff; code++) { + const ch = String.fromCharCode(code) + const stripped = sanitizeLabel(`a${ch}b`) === 'ab' + if (stripped !== UNSAFE_LABEL_CHARS_BEFORE_LLP_0224.test(ch)) disagreements.push(code) + } + assert.deepEqual(disagreements, []) +}) + +// @ref LLP 0224#escape-not-strip [tests]: nothing is dropped, and the output is printable ASCII +test('escapeForDisplay replaces control characters with visible escapes', () => { + assert.equal(escapeForDisplay('a\u001b[31mb'), 'a\\u001b[31mb') + assert.equal(escapeForDisplay('a\nb\rc\td'), 'a\\nb\\rc\\td') + assert.equal(escapeForDisplay('a\u0000b\u007fc\u009bd'), 'a\\u0000b\\u007fc\\u009bd') + assert.equal(escapeForDisplay('a\u2028b\u2029c'), 'a\\u2028b\\u2029c') +}) + +// @ref LLP 0224#escape-class [tests]: bidi is escaped, zero-width formatting is not +test('escapeForDisplay escapes bidi formatting and leaves zero-width formatting alone', () => { + assert.equal(escapeForDisplay('a\u202eb'), 'a\\u202eb') + assert.equal(escapeForDisplay('a\u2066b\u2069c'), 'a\\u2066b\\u2069c') + assert.equal(escapeForDisplay('a\u200eb\u200fc'), 'a\\u200eb\\u200fc') + assert.equal(escapeForDisplay('a\u061cb'), 'a\\u061cb') + // Left alone: a family emoji is a ZWJ sequence and U+FE0F colours a heart. + const emoji = '\ud83d\udc68\u200d\ud83d\udc69 \u2764\ufe0f' + assert.equal(escapeForDisplay(emoji), emoji) + assert.equal(escapeForDisplay('a\u200bb\ufeffc\u00add'), 'a\u200bb\ufeffc\u00add') +}) + +test('escapeForDisplay never truncates, never drops, and passes clean text through', () => { + const clean = 'caf\u00e9 \u65e5\u672c\u8a9e plain/path\\with\\backslashes' + assert.equal(escapeForDisplay(clean), clean) + assert.equal(escapeForDisplay(''), '') + // Unlike sanitizeLabel, no clamp: a long value comes back whole. + const long = 'z'.repeat(500) + assert.equal(escapeForDisplay(long), long) + assert.equal(sanitizeLabel(long)?.length, 120) +}) From 588bf91a526616ad3346d5970eec5b0c0e6814c4 Mon Sep 17 00:00:00 2001 From: test Date: Fri, 14 Aug 2026 00:07:40 +0000 Subject: [PATCH 2/3] Renumber LLP 0224 to 0225: 0224 is claimed by PR #759 (#752) --- ...captured-text-is-escaped-for-display.decision.md} | 2 +- src/core/query/format.js | 6 +++--- src/core/query/overview.js | 2 +- src/core/util/json_util.js | 6 +++--- test/core/query-format-escaping.test.js | 4 ++-- test/core/query-overview.test.js | 2 +- test/core/util-json-util.test.js | 12 ++++++------ 7 files changed, 17 insertions(+), 17 deletions(-) rename llp/{0224-captured-text-is-escaped-for-display.decision.md => 0225-captured-text-is-escaped-for-display.decision.md} (99%) diff --git a/llp/0224-captured-text-is-escaped-for-display.decision.md b/llp/0225-captured-text-is-escaped-for-display.decision.md similarity index 99% rename from llp/0224-captured-text-is-escaped-for-display.decision.md rename to llp/0225-captured-text-is-escaped-for-display.decision.md index 5c87bfa3..9b3336b7 100644 --- a/llp/0224-captured-text-is-escaped-for-display.decision.md +++ b/llp/0225-captured-text-is-escaped-for-display.decision.md @@ -1,4 +1,4 @@ -# LLP 0224: captured text is escaped where it is rendered for a person, never where it is rendered for a program +# LLP 0225: captured text is escaped where it is rendered for a person, never where it is rendered for a program **Type:** Decision **Status:** Accepted diff --git a/src/core/query/format.js b/src/core/query/format.js index 616ce8de..b6cf814e 100644 --- a/src/core/query/format.js +++ b/src/core/query/format.js @@ -147,7 +147,7 @@ function rowBytes(row) { * stay byte-exact. Deliberately not a TTY test, so a query redirected to a * file holds exactly the bytes its terminal showed. * - * @ref LLP 0224#format-not-tty [implements]: escaping is chosen by format, never by isTTY + * @ref LLP 0225#format-not-tty [implements]: escaping is chosen by format, never by isTTY * * @param {QueryResultSet} result * @param {QueryFormat} format @@ -266,7 +266,7 @@ function inferColumns(rows) { * Running it over already-stringified JSON is a no-op for the parts * `JSON.stringify` already handled. * - * @ref LLP 0224#decision [implements]: human-facing formats escape, machine formats do not + * @ref LLP 0225#decision [implements]: human-facing formats escape, machine formats do not * * @param {unknown} value * @returns {string} @@ -348,7 +348,7 @@ export function buildQuerySqlOutput(full, opts) { * rendered line at a time: the newlines between preview rows are structure * this function produced, not captured bytes, and must survive. * - * @ref LLP 0224#decision [constrained-by]: the receipt is a human render, so it escapes + * @ref LLP 0225#decision [constrained-by]: the receipt is a human render, so it escapes * * @param {string} content the already-rendered file content (sized for the receipt) * @returns {string} diff --git a/src/core/query/overview.js b/src/core/query/overview.js index a9e4b91e..88b85787 100644 --- a/src/core/query/overview.js +++ b/src/core/query/overview.js @@ -1022,7 +1022,7 @@ function bar(value, max) { * bars and headings are painted, so a sweep over the finished table would * erase the colour along with the attack. * - * @ref LLP 0224#decision [implements]: the same rule on `hyp query overview`, per captured cell + * @ref LLP 0225#decision [implements]: the same rule on `hyp query overview`, per captured cell * * @param {unknown} value * @returns {string} diff --git a/src/core/util/json_util.js b/src/core/util/json_util.js index de608b0a..017bc66f 100644 --- a/src/core/util/json_util.js +++ b/src/core/util/json_util.js @@ -41,7 +41,7 @@ export const MAX_LABEL_CHARS = 120 // would be a second chance for the two to disagree about what "unsafe" // means. // -// @ref LLP 0224#one-vocabulary: one class, two policies over its named groups +// @ref LLP 0225#one-vocabulary: one class, two policies over its named groups // C0/DEL/C1 and the Unicode line/paragraph separators, so a value can // never move the cursor, erase a line, open an escape sequence, or split @@ -80,7 +80,7 @@ const UNSAFE_LABEL_CHARS = new RegExp( // captured text on a large fraction of real rows, to defend against a // character that cannot repaint anything. // -// @ref LLP 0224#escape-class [implements]: display escapes control and bidi, not zero-width +// @ref LLP 0225#escape-class [implements]: display escapes control and bidi, not zero-width const DISPLAY_UNSAFE_CHARS = new RegExp(`[${TERMINAL_CONTROL_CHARS}${BIDI_FORMATTING_CHARS}]`, 'g') // The three C0 characters an operator reads more easily by name than by code @@ -163,7 +163,7 @@ export function sanitizeLabel(value, max = MAX_LABEL_CHARS) { * newline would cost far more legibility than the ambiguity does. The * ambiguity is cosmetic; neither spelling can move a cursor. * - * @ref LLP 0224#escape-not-strip [implements]: the display plane escapes where the label plane strips + * @ref LLP 0225#escape-not-strip [implements]: the display plane escapes where the label plane strips * * @param {string} value * @returns {string} diff --git a/test/core/query-format-escaping.test.js b/test/core/query-format-escaping.test.js index d96ae898..4bc12dd1 100644 --- a/test/core/query-format-escaping.test.js +++ b/test/core/query-format-escaping.test.js @@ -5,8 +5,8 @@ // halves are asserted here, because the whole decision is that the *format*, // and nothing else, picks between them. // -// @ref LLP 0224#decision [tests]: table/markdown escape, json/jsonl stay byte-exact -// @ref LLP 0224#format-not-tty [tests]: no assertion here touches isTTY, and none needs to +// @ref LLP 0225#decision [tests]: table/markdown escape, json/jsonl stay byte-exact +// @ref LLP 0225#format-not-tty [tests]: no assertion here touches isTTY, and none needs to import test from 'node:test' import assert from 'node:assert/strict' diff --git a/test/core/query-overview.test.js b/test/core/query-overview.test.js index 4f693a6f..696c8a08 100644 --- a/test/core/query-overview.test.js +++ b/test/core/query-overview.test.js @@ -1017,7 +1017,7 @@ test('hyp query overview: the usage line lists every flag the codec accepts', as } }) -// @ref LLP 0224#decision [tests]: the overview's captured columns are escaped too +// @ref LLP 0225#decision [tests]: the overview's captured columns are escaped too test('renderOverview escapes captured columns and keeps its own colour', () => { const ESC = '\u001b' const RLO = '\u202e' diff --git a/test/core/util-json-util.test.js b/test/core/util-json-util.test.js index 202e5a84..47497f29 100644 --- a/test/core/util-json-util.test.js +++ b/test/core/util-json-util.test.js @@ -72,22 +72,22 @@ test('errCode extracts string codes and nothing else', () => { // hand-written literal. A refactor of a security class has to prove it moved no // code point, so this is the literal it replaced, held here as the oracle. // -// @ref LLP 0224#one-vocabulary [tests]: the recomposed class is the same set of code points -const UNSAFE_LABEL_CHARS_BEFORE_LLP_0224 = +// @ref LLP 0225#one-vocabulary [tests]: the recomposed class is the same set of code points +const UNSAFE_LABEL_CHARS_BEFORE_LLP_0225 = /[\u0000-\u001F\u007F-\u009F\u00AD\u061C\u180E\u200B-\u200F\u2028-\u2029\u202A-\u202E\u2060-\u2064\u2066-\u2069\uFE00-\uFE0F\uFEFF]/ -test('sanitizeLabel strips exactly the code points it stripped before LLP 0224', () => { +test('sanitizeLabel strips exactly the code points it stripped before LLP 0225', () => { /** @type {number[]} */ const disagreements = [] for (let code = 0; code <= 0xffff; code++) { const ch = String.fromCharCode(code) const stripped = sanitizeLabel(`a${ch}b`) === 'ab' - if (stripped !== UNSAFE_LABEL_CHARS_BEFORE_LLP_0224.test(ch)) disagreements.push(code) + if (stripped !== UNSAFE_LABEL_CHARS_BEFORE_LLP_0225.test(ch)) disagreements.push(code) } assert.deepEqual(disagreements, []) }) -// @ref LLP 0224#escape-not-strip [tests]: nothing is dropped, and the output is printable ASCII +// @ref LLP 0225#escape-not-strip [tests]: nothing is dropped, and the output is printable ASCII test('escapeForDisplay replaces control characters with visible escapes', () => { assert.equal(escapeForDisplay('a\u001b[31mb'), 'a\\u001b[31mb') assert.equal(escapeForDisplay('a\nb\rc\td'), 'a\\nb\\rc\\td') @@ -95,7 +95,7 @@ test('escapeForDisplay replaces control characters with visible escapes', () => assert.equal(escapeForDisplay('a\u2028b\u2029c'), 'a\\u2028b\\u2029c') }) -// @ref LLP 0224#escape-class [tests]: bidi is escaped, zero-width formatting is not +// @ref LLP 0225#escape-class [tests]: bidi is escaped, zero-width formatting is not test('escapeForDisplay escapes bidi formatting and leaves zero-width formatting alone', () => { assert.equal(escapeForDisplay('a\u202eb'), 'a\\u202eb') assert.equal(escapeForDisplay('a\u2066b\u2069c'), 'a\\u2066b\\u2069c') From 5bcd50ebd017efbd547b708504cd6572b098a781 Mon Sep 17 00:00:00 2001 From: test Date: Fri, 14 Aug 2026 00:43:47 +0000 Subject: [PATCH 3/3] Fix round-1 review findings on escape-for-display (#752) - LLP 0225 Verification: separate the six tests that discriminate the fix from the two that pass pre-fix by construction and guard preserved behaviour, confirmed by re-running the three new test files against the pre-fix sources and by mutation. - format.js: move the receipt rationale and its @ref out of the `full` param's description and into the function description, so the @param list is contiguous again. - overview.js: clip before escaping for the model and tool columns, matching format.js's order, so a `\uXXXX` escape can no longer be cut in half at MAX_MODEL_WIDTH. cell() gains an optional width argument that truncates the raw value before escapeForDisplay. - LLP 0225 Consequences: record the markdown newline behaviour change alongside the table one. - Add a test pinning table column width to the escaped header rather than the raw column name (the one surviving mutant from review). --- ...ured-text-is-escaped-for-display.decision.md | 17 +++++++++++------ src/core/query/format.js | 4 ++-- src/core/query/overview.js | 15 +++++++++++---- test/core/query-format-escaping.test.js | 11 +++++++++++ 4 files changed, 35 insertions(+), 12 deletions(-) diff --git a/llp/0225-captured-text-is-escaped-for-display.decision.md b/llp/0225-captured-text-is-escaped-for-display.decision.md index 9b3336b7..6f73fbcd 100644 --- a/llp/0225-captured-text-is-escaped-for-display.decision.md +++ b/llp/0225-captured-text-is-escaped-for-display.decision.md @@ -92,8 +92,9 @@ Consequences accepted with this: legibility than the ambiguity does. The ambiguity is cosmetic, and neither spelling can move a cursor. - A newline inside a `table` cell now prints as `\n` instead of breaking the - row. That is a visible change to multi-line prose, and it is the point: a - newline in a cell is how a captured value forges a row. + row, and a newline in a `markdown` cell prints as `\n` rather than being + flattened to a space. That is a visible change to multi-line prose, and it + is the point: a newline in a cell is how a captured value forges a row. **One vocabulary, two policies.** {#one-vocabulary} @@ -160,10 +161,14 @@ this one. ## Verification {#verification} -Unit tests, each shown failing against the pre-fix source before being kept: -`ESC` in a `table` cell, `ESC` and bidi in a `markdown` cell, `json`/`jsonl` -byte-exactness, non-ASCII and emoji survival, and column alignment with an -escaped cell. +Unit tests. Six were shown failing against the pre-fix source before being +kept: `ESC` in a `table` cell, `ESC` and bidi in a `markdown` cell, a newline +that cannot forge a row, tab/CR spellings, column alignment with an escaped +cell, and the `--output` receipt. Two more pass pre-fix by construction and +guard preserved behaviour rather than the fix: `json`/`jsonl` byte-exactness +(fails if the escape is ever applied to a machine format) and non-ASCII and +emoji survival (fails if the display class is widened to the zero-width +group). Both were confirmed by mutating the code they guard. **Not verified:** no test here renders into a real terminal emulator. The claim that `\u001b` is inert, and that a raw `ESC` sequence is not, rests on the diff --git a/src/core/query/format.js b/src/core/query/format.js index b6cf814e..494e1cce 100644 --- a/src/core/query/format.js +++ b/src/core/query/format.js @@ -341,8 +341,6 @@ export function buildQuerySqlOutput(full, opts) { * result went, its shape, and a small truncated preview so the caller can * sanity-check without ingesting the file. * - * @param {string} outputPath - * @param {{ columns: string[], rows: Record[] }} full * The receipt is a human-facing render in its own right, whatever * `--format` the file got, so its preview is escaped too. It is escaped one * rendered line at a time: the newlines between preview rows are structure @@ -350,6 +348,8 @@ export function buildQuerySqlOutput(full, opts) { * * @ref LLP 0225#decision [constrained-by]: the receipt is a human render, so it escapes * + * @param {string} outputPath + * @param {{ columns: string[], rows: Record[] }} full * @param {string} content the already-rendered file content (sized for the receipt) * @returns {string} */ diff --git a/src/core/query/overview.js b/src/core/query/overview.js index 88b85787..43db9f43 100644 --- a/src/core/query/overview.js +++ b/src/core/query/overview.js @@ -703,7 +703,7 @@ export function renderProviderMix(rows, color, showSql = false, sql = '') { // An unlabelled row only reaches the table if it carried tokens, which // today it never does. If that changes, name it for what it is rather // than dropping measured tokens on the floor. - truncate(escapeForDisplay(hasModelLabel(r) ? String(r.model).trim() : '(model not recorded)'), MAX_MODEL_WIDTH), + escapeForDisplay(truncate(hasModelLabel(r) ? String(r.model).trim() : '(model not recorded)', MAX_MODEL_WIDTH)), formatCount(r.input_tokens), formatCount(r.cached_tokens), formatCount(r.output_tokens), @@ -841,7 +841,7 @@ export function renderRepoMix(rows, color, showSql = false, sql = '') { export function renderToolMix(rows, color, showSql = false, sql = '') { const max = Math.max(...rows.map((r) => toNumber(r.calls))) const body = rows.map((r) => [ - truncate(cell(r.tool_name), MAX_MODEL_WIDTH), + cell(r.tool_name, MAX_MODEL_WIDTH), formatCount(r.calls), formatCount(r.sessions), paint(bar(toNumber(r.calls), max), ANSI.cyan, color), @@ -1022,14 +1022,21 @@ function bar(value, max) { * bars and headings are painted, so a sweep over the finished table would * erase the colour along with the attack. * + * `width`, when given, clips the raw trimmed value before escaping rather + * than after: escaping first can cut a `\uXXXX` escape in half at the + * boundary, so the clip always runs on the raw text. + * * @ref LLP 0225#decision [implements]: the same rule on `hyp query overview`, per captured cell * * @param {unknown} value + * @param {number} [width] * @returns {string} */ -function cell(value) { +function cell(value, width) { if (value === null || value === undefined) return '(none)' - const text = escapeForDisplay(String(value).trim()) + const trimmed = String(value).trim() + const clipped = width === undefined ? trimmed : truncate(trimmed, width) + const text = escapeForDisplay(clipped) return text.length === 0 ? '(none)' : text } diff --git a/test/core/query-format-escaping.test.js b/test/core/query-format-escaping.test.js index 4bc12dd1..53900ec6 100644 --- a/test/core/query-format-escaping.test.js +++ b/test/core/query-format-escaping.test.js @@ -96,6 +96,17 @@ test('column widths and alignment survive an escaped cell', () => { assert.equal(second.indexOf('B'), width + 2) }) +test('table column width is measured on the escaped header, not the raw column name', () => { + // The column name itself carries the control character here, not a cell + // value: widths must come from `headers[i]`, the already-escaped text, + // or the header and its column drift out of alignment. + const out = renderResult(set([{ [`h${ESC}`]: 'x', tail: 'A' }]), 'table') + const [, divider, row] = out.trimEnd().split('\n') + const width = `h${ESC}`.replace(ESC, '\\u001b').length + assert.equal(divider.split(' ')[0].length, width) + assert.equal(row.indexOf('A'), width + 2) +}) + test('the spill receipt escapes its preview but the file it wrote does not', () => { const rows = [{ c: `a${RLO}b` }, { c: `d${ESC}e` }] const out = buildQuerySqlOutput(set(rows), {