[Ruby184]

A site can now keep what it asked a source for between runs, so a nightly refetch doesn't ask for every programme's details again. homepi asked for this: three of its sites carry the same ~40 lines to do it by hand, and with them webtv.sk went from 9,643 requests a cold day to 1,543. One commit, a feature.

ctx.remember(key, { until, miss? }, task)

Every context a site is handed has it: requests, parseDay, and a stream's pass. It answers with what an earlier run kept under key, or runs task and keeps its answer until until.

const details = await remember(
  `detail:${item.id}|${item.start}`,
  {
    until: Date.parse(item.end) + 86_400_000,
    miss: (error) => error instanceof HTTPError && error.response.status === 404,
  },
  async () => trim(await paced(({ signal }) => http.get(`detail/${item.id}`, { signal }).json())),
);

• until is required, a Date or epoch milliseconds, since how long something is worth keeping is the site's to say. One already past keeps the value for this run only, with one warning per run; it usually means seconds where milliseconds were meant.
• miss turns a failure into a kept null, so "no details" is asked once. Any other failure is thrown and not kept, so the next run asks again.
• The return type has null in it only when the ask has a miss. A "none" kept for an ask with a miss is never handed to one without; that ask runs its own task.
• A value must be plain JSON, as a channel's data must. A Date or a Map is refused with the site, the key and where in the value. Each caller gets its own copy.

Two asks at once

Two asks of one key while the first is still running share one task, and each reads the outcome with its own miss. That covers a broadcast past midnight, which appears in two days' listings. A parse waiting on another parse's ask gives up its localWork slot meanwhile, as it does for paced, so localConcurrency: 1 can't deadlock. A request can't give up its site-queue slot yet, so it asks for itself instead of waiting.

Storage

• Values live in a state group of their own, remembered, beside the site's state, so a token isn't rewritten for the details' sake.
• Expired values are dropped at the end of every site run, whether or not anything asked. That's one extra state read per site per run; for a site that never remembers anything, it finds nothing and writes nothing.
• --refresh reads nothing an earlier run kept, and still keeps new answers, as it does for cached listings.
• A failed read of a site's state is no longer kept as the group, so the next ask tries again rather than the run failing when it saves.

Smaller changes

• The plain-JSON check moved from the channel module to src/core/plain.ts, since remember uses it too.
• StateKey.REMEMBERED names the new group, and the API docs now list all four state groups.

Testing

• 17 new tests, covering:
  • keeping and reading across runs;
  • miss, other failures and each caller's own miss;
  • expiry, and pruning when nothing asks;
  • a shared ask, including the slot-lending case, which hung before the fix;
  • copies through a store that really serializes, --refresh, a past until, option validation, a failed read, and the return types.
• Each behaviour was broken in turn to confirm a test fails.
• 1,788 tests, passing on Node 22.18, 24.15 and 26.12.
• The docs example is in test/readme-examples.ts and typechecks.