[Bug] LanceDbIndex interpolates paths into LanceDB SQL predicates — apostrophes in file/workspace paths break index deletes (stale chunks stay retrievable) and retrieval

[bossjoker1]

Description

LanceDbIndex builds its LanceDB SQL predicates by string interpolation of on-disk paths and workspace directories, and a single quote is a legal character in both:

• delete path (core/indexing/LanceDbIndex.ts, update()):

  await lanceTable.delete(
    `cachekey = '${cacheKey}' AND path = '${path}'`,
  );

• retrieval path (_retrieveForTag):

  query = query.where(`path LIKE '${directory}%'`).limit(300);

A path or workspace directory containing an apostrophe (don't.py, it's a test/…) produces an unparseable predicate and LanceDB throws Unterminated string literal. Consequences:

1. Deleted/renamed files stay retrievable. compute/insert paths use parameterized sqlite and Arrow row adds (no interpolation), so a file with a quote in its name indexes fine. When the file is later deleted or renamed, update() reaches the toDel loop (removeTag + del) and the interpolated delete throws. CodebaseIndexer catches per batch, logs a warning, and moves on — markComplete never runs for that batch, so the tag-catalog rows survive and every subsequent sync retries the same throw. The stale chunks remain in that tag's LanceDB table and continue to surface in @Codebase retrieval even though the source file is gone.
2. Batch aborts. Every other file sharing the failing file's filesPerBatch batch also never completes and is re-processed on every sync.
3. Workspace directories with an apostrophe break retrieval. _retrieveForTag interpolates the directory into path LIKE '…%', so retrieval throws for the whole workspace.

This was found in a correctness study of derived-representation lifecycle handling (what happens to index entries when their source is revoked).

Environment

• continue pinned revision 5522c6f44ca0ac3528b37244818fbfa39b5af470 (2026-09), core/
• vectordb@0.4.20 (the version pinned in core/package.json), Node 24, Linux x64

Steps to Reproduce

Standalone script against the pinned vectordb version (same predicate strings as the source):

const lancedb = require("vectordb");
const fs = require("fs");
const DIR = fs.mkdtempSync("/tmp/lance-quote-");

async function main() {
  const db = await lancedb.connect(DIR);
  const makeArrowTable = lancedb.makeArrowTable;
  const rows = [
    { uuid: "u-1", cachekey: "ck-1", path: "src/normal.ts", vector: [1, 0], startLine: 1, endLine: 2, contents: "a" },
    { uuid: "u-2", cachekey: "ck-2", path: "src/don't.ts", vector: [0, 1], startLine: 1, endLine: 2, contents: "b" },
  ];
  const table = await db.createTable("chunks", await makeArrowTable(rows));

  // The exact predicate construction from LanceDbIndex.ts
  await table.delete(`cachekey = 'ck-2' AND path = 'src/normal.ts'`); // ok
  await table.delete(`cachekey = 'ck-2' AND path = 'src/don't.ts'`);  // throws
}
main().catch((e) => { console.error(String(e.message).split("\n")[0]); process.exit(1); });

Observed output:

apostrophe-path delete:      FAILED -> lance error: LanceError(IO): Unterminated string literal at Line: 1, Column 65 …
apostrophe-directory (LIKE): FAILED -> lance error: LanceError(IO): Unterminated string literal at Line: 1, Column 58 …
escaped variant:             ok

normal-path delete and the ''-escaped delete both succeed, isolating the interpolation as the cause.

Expected Behavior

Filesystem paths are arbitrary byte strings (modulo / and NUL); predicates built from them must be escaped (e.g. double the single quotes: 'don''t.ts' — verified to work against vectordb@0.4.20), or the filters should use a parameterized API. All interpolated sites in LanceDbIndex.ts (update() delete loop, _retrieveForTag path LIKE) need it; other artifacts (FullTextSearchCodebaseIndex, sqlite) already use bound parameters and are unaffected.

Happy to provide the full standalone script (including the retrieval-side case) or a patch.