Repository navigation
feat!: --explain reads back what a run said - #35
Merged
Merged
Conversation
Four things a run said only as counts are now said in full, and read:
`--explain unmatched` on `epg merge` and `epg build`, and `--explain empty`,
`requests` and `held` on `epg grab` and `epg build`. Repeatable, and written on
stderr once the run is done. Each lists what the run said and what it most
likely means, said as likely, never as found: it has the events and the
config, not the source. Refused with `--dry-run` and `--format json`; a
cancelled run explains nothing.
- unmatched: every programme the merge nearly joined to another source's and
did not, grouped by channel, source pair and what kept them apart, with the
`startToleranceMs`, `windowMs` or `normalizeTitle` that would have joined
them. Each is a new `merge:unmatched` event at `debug`, with a signed
`shiftMs` and a `why`: `start`, `title`, `both`, `runs`, or `matcher` for a
config's own `match`.
- empty: the channel-days a source answered with nothing, read against the
rest of the run: a parse that no longer reads its source, a source that has
not published that far, a channel another source has, one every source
asked is empty on, a gap. A day served from the cache, unchanged or failed
is never taken as evidence.
- requests: every site's requests by label, with how long they were out, how
long they sat in the site's queue (`RequestCounts.waitedMs`) and what failed
them; then where each site's time went, what would change it, and what its
failures mean, as its `backoff` has them.
- held: each site the source told to slow down, with what, how each hold
ended, and the pace that earned it beside its `rateLimit`.
`site:done.msBy` says where a site's wall clock went: `source`, `held`,
`rateLimit`, `working`, `queued` and `other`, which add up to `ms`. A clock
measures it as the site runs, reading the time only when what holds changes.
A site in a thread writes it to the memory it shares with the run, and a cache
call's wait for a slot on the run's thread is told back to it. A streaming
site's pass reading its document is `source`. `pacing:held` says the
`retryAfterMs` the source asked for, and `lengthened` where it made a running
hold longer.
A reporter may say which kinds it wants (`wants`). A run asks once, the first
time it needs to, and makes its frequent events (`entry:*`,
`request:started`/`done`, `site:note`, `pacing:rateLimit`, `merge:unmatched`)
only where one is wanted; a site in a thread neither makes nor sends them. At
`info` that is 7% of a grab's time, and 4% in a thread. The shipped reporters
want what they show or count, `emitterReporter` what its emitter has
listeners for or what `{ wants }` says, and `combineReporters` what any of
them does. A config's `events` may now be a reporter, for an emitter that
hears more than its listeners do.
Also: merge:summary's `combined` no longer counts a source joining its own
programme held over from the day before, and `nearMiss` counts only another
source's left unjoined; a reporter given an event of a kind this version does
not know filters it by its own level.
BREAKING CHANGE: `ResolvedMatch.matches(a, b)` is replaced by `apart(a, b)`,
which answers why two programmes are not one broadcast, or `undefined` where
they are. `site:done.msBy` and `RequestCounts.waitedMs` are required. `Emit`
carries a required `hears` set: make one with `emitter({ reporter })`, as
`sitePacing` now insists. A run no longer makes its frequent events for kinds
an `emitterReporter`'s emitter has no listener for, so an emitter whose own
`emit` passes every event on needs `{ wants: () => true }`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four things a run said only as counts are now listed in full and read back:
--explain unmatchedepg merge,epg buildstartToleranceMs,windowMsornormalizeTitlethat would have joined each group--explain emptyepg grab,epg build--explain requestsepg grab,epg build--explain heldepg grab,epg buildrateLimit--explaincan be repeated and writes to stderr once the run is done. It's refused with--dry-runand--format json, and a cancelled run explains nothing.New on events:
merge:unmatched(debug), with a signedshiftMsand awhy.site:done.msBy: the site's wall clock split intosource,held,rateLimit,working,queuedandother. It's measured as the site runs, including in threads through shared memory.RequestCounts.waitedMs.pacing:held.retryAfterMs, andlengthenedon a hold that was extended.A reporter may now say which event kinds it
wants. The run asks once, the first time it needs to. It then makes its frequent events only where one is wanted:entry:*,request:started/done,site:note,pacing:rateLimitandmerge:unmatched. A site in a thread neither makes nor sends them.emitterReporterwants what its emitter has listeners for, or what{ wants }says.combineReporterswants what any of its reporters wants.eventsmay now be a reporter, for an emitter that hears more than its listeners do.Also:
merge:summary.combinedno longer counts a source joining its own programme held over from the day before.nearMisscounts only another source's programmes left unjoined.Breaking changes
ResolvedMatch.matches(a, b)is replaced byapart(a, b), which returns why two programmes are not one broadcast, orundefinedwhen they are.site:done.msByandRequestCounts.waitedMsare required.Emitcarries a requiredhearsset. Make one withemitter({ reporter });sitePacingthrows at the call on a plain function.emitterReporter's emitter has no listener for. An emitter whose ownemitpasses every event on needs{ wants: () => true }.Measured
Plain Node on the built
dist, 1000 channels × 7 days, one request per channel-day, text reporter atinfo:Checks
Format, lint, typecheck and build are clean. Tests: 2329 passed, 2 skipped, on Node 22.18, 24.15 and 26.12.