Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/constructed-stream-independent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@tanstack/oracles': minor
---

A campaign's generated histories no longer depend on its constructed histories, and each constructed history runs several bases from a stream of its own. The `generated` unit draws only from the property's arbitrary. Constructed history k builds from each of the first `bases` draws of the arbitrary's sample at the campaign's seed plus k, and every base must enter the states it names. The new `histories.bases` option sets that number. It must be a whole number of at least 1 (default 5), or the property fails to load. So appending a constructed history keeps every fixed seed's generated histories and every other constructed history's bases. Before, half of the generated unit's runs were constructed histories, weighted by how many there were, so appending one moved every run's draw, and each constructed unit ran one history.

This shifts each fixed seed once, for properties that have at least one constructed history. Their `generated` unit now runs other histories, and a seed-and-path replay line printed before this change for that unit no longer replays the same history. A constructed unit keeps its first base, and its first replay path index still selects it. Replay lines for properties without constructed histories stay valid: their `generated` unit already drew from the arbitrary alone. Each constructed unit now runs `bases` histories, so a campaign takes longer; set `bases` lower for a property whose constructed histories are slow.
12 changes: 8 additions & 4 deletions packages/oracles/docs/oracle-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,10 +275,14 @@ No requirement here can be met by declaring something out of reach.
on, and a history MUST be constructed to enter each one.
- **Framework:** Enforced. Every state in `states` must be named by a
constructed history. Every campaign runs each constructed history, and one
that does not enter its named state fails. Half of the generated histories
are also constructed, so the histories after each state are explored. A
state that random histories reach only by chance goes in `measured`, which
is reported but not required.
that does not enter its named state, on any of its bases, fails. Each
constructed history runs `bases` bases (5 by default) from a stream of its
own, and the generated histories are drawn apart from the constructed ones.
So at a fixed seed, appending a constructed history keeps the generated
histories and every other constructed history's bases, and the histories
after each required state are explored, not visited once. A state that random histories
reach only by chance goes in `measured`, which is reported but not
required.
- **Author:** Reach is a predicate over the plan or trace, or a production
probe. A literal example is not reach. A `fixed` history is a regression,
not reach. A law that needs a state entered at least N times in each
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3230,6 +3230,7 @@ const required = [
`a survivor whose work fell`,
`extra-work named past an unregistered rise`,
`a rise exactly at the declared bound`,
`authored agreement-only-kill`,
]

// One constructed history per outcome class: one mutant judged by one
Expand Down Expand Up @@ -4000,6 +4001,18 @@ const constructed = [
baseline: `load`,
},
),
// An authored mutant that only the reach agreement rejects: a G11
// failure-gap class, so a constructed history enters it (review of #122;
// the fixed seed's generated histories stopped reaching it once the
// constructed histories left the generated unit).
single(
`an authored mutant killed by reach agreement alone`,
[`authored agreement-only-kill`],
{
effect: { kind: `verdict`, law: `reach agreement` },
authored: [{}],
},
),
]

oracleProperty({
Expand All @@ -4016,7 +4029,6 @@ oracleProperty({
measured: [
...[
`timeout`,
`agreement-only-kill`,
`unregistered-kill`,
`crashed`,
`framework-error`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4790,6 +4790,11 @@ oracleProperty({
`a check that ran more than one batch`,
],
runs: 2,
// One base per constructed history: each runs the CLI end to end, often
// for minutes, and five bases took a fixed-seed shard to 18 to 30
// minutes on an Arclet, past the 10-minute CI limit (#122). Random seeds
// vary each one's base from run to run.
bases: 1,
fixedSeeds: [20261105],
constructed: withDeclaration([
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1453,6 +1453,10 @@ oracleProperty({
arbitrary: argumentHistories,
states: required,
runs: 40,
// Three bases per constructed history: each runs the CLI on a fixture
// package, and five bases took the larger fixed-seed shard from 116 s to
// 530 s on an Arclet, close to the 10-minute CI limit (#122).
bases: 3,
fixedSeeds: [20261105],
constructed: withDeclaration([
{
Expand Down
94 changes: 64 additions & 30 deletions packages/oracles/src/oracle-property.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,22 +165,32 @@
* driver must await the work it starts.
* - A **constructed history** is built from a generated history so that it
* reaches states that the laws need, and it names those declared states.
* Every campaign runs every constructed history once, in its own unit, and a
* constructed history that does not enter a named state fails. Only
* Every campaign runs every constructed history in its own unit, over a
* number of bases (`bases`, 5 by default), and a constructed history that
* does not enter a named state, on any base, fails. Only
* constructed histories make reach required: a state that random histories
* reach is measured, never gated. So every state in `states` must be named
* by a constructed history, and a state the property only measures is
* declared in `measured` instead (ORC-015). Half of a campaign's generated
* histories are also constructed, so the histories after each required
* state are explored, not visited once.
* declared in `measured` instead (ORC-015). The constructed histories and
* the generated ones are independent streams: the `generated` unit draws
* only from the arbitrary, and constructed history k builds from each of
* the first `bases` draws of the arbitrary's sample at the campaign's seed
* plus k. So at a fixed seed the generated histories do not depend on the
* constructed list, its length or its contents, appending a constructed
* history changes no other constructed history's bases, and each
* constructed history is explored over `bases` histories, not visited once
* (register M, 2026-11-16, "N bases per constructed history"). A history
* inserted mid-list moves the bases of the ones after it, so new
* constructed histories go at the end of the list.
* - A **fixed history** is a named history, such as a fixed regression. It may
* name the states it `enters`, which its run must reach. Fixed and
* constructed histories must be histories the arbitrary can take back
* (`canShrinkWithoutContext`), so they shrink like generated ones and cannot
* cover a dimension that the grammar lacks; such a dimension belongs in the
* grammar, or in the property's stated limits. The property checks this when
* it registers, for every fixed history and for the constructed histories
* built from the first fixed seed (or seed 0 when there is none), so a
* it registers, for every fixed history and for the first base of each
* constructed history at the first fixed seed (or seed 0 when there is
* none); a campaign checks every base it builds. So a
* constructed history's `build` runs when the test file loads. It fails to
* load naming the history, and a `build` that throws there fails to load
* naming its constructed history
Expand Down Expand Up @@ -598,6 +608,13 @@ export interface GeneratedHistories<THistory> {
/** One fixed-seed campaign per seed, beside the random campaign. */
fixedSeeds: ReadonlyArray<number>
constructed?: ReadonlyArray<ConstructedHistory<THistory>>
/**
* How many bases each constructed history runs in each campaign: the first
* `bases` draws of the arbitrary's sample at the campaign's seed plus the
* history's index, each built into one history. A whole number of at least
* 1; 5 when absent.
*/
bases?: number
fixed?: ReadonlyArray<FixedHistory<THistory>>
}

Expand Down Expand Up @@ -1771,20 +1788,33 @@ function buildConstruction<THistory>(
seed: number,
registering = false,
): THistory {
let history: THistory
return buildBases(id, arbitrary, construction, seed, 1, registering)[0]!
}

/** A construction built from each of the first `count` draws at `seed`. */
function buildBases<THistory>(
id: string,
arbitrary: fc.Arbitrary<THistory>,
construction: ConstructedHistory<THistory>,
seed: number,
count: number,
registering = false,
): Array<THistory> {
let histories: Array<THistory>
try {
const [generated] = fc.sample(arbitrary, { seed, numRuns: 1 })
history = construction.build(generated!)
histories = fc
.sample(arbitrary, { seed, numRuns: count })
.map((generated) => construction.build(generated))
} catch (error) {
if (!registering) throw error
throw new Error(
`oracleProperty ${id}: constructed history "${construction.name}" threw while it was built for registration: ${error instanceof Error ? error.message : reportText(error)}`,
{ cause: error },
)
}
if (!arbitrary.canShrinkWithoutContext(history))
if (histories.some((history) => !arbitrary.canShrinkWithoutContext(history)))
throw untakeable(id, `constructed history "${construction.name}" built`)
return history
return histories
}

/** Stable across list insertion and reordering; no process-random replay route. */
Expand Down Expand Up @@ -2380,6 +2410,13 @@ function generatedTests<THistory, TExpected, TPlan>(
)
const options: BoundOptions<THistory, TExpected, TPlan> =
probes.size > 0 ? { ...unbound, probes } : unbound
// Each constructed history runs this many bases in every campaign (register
// M, 2026-11-16, "N bases per constructed history").
const bases = source.bases ?? 5
if (!(Number.isInteger(bases) && bases >= 1))
throw new Error(
`${id}: \`bases\` must be a whole number of at least 1, the number of bases each constructed history runs in a campaign; it is ${reportText(bases)}.`,
)
const required = (source.states ?? []).map(stateName)
const constructions = source.constructed ?? []
const declared = declaredOf(source)
Expand All @@ -2391,18 +2428,6 @@ function generatedTests<THistory, TExpected, TPlan>(
throw new Error(
`${id}: no constructed history enters ${unconstructed.join(`, `)}. A required state needs a constructed history; a state that is only counted belongs in \`measured\`.`,
)
// Generated histories, half of them constructed. A constructed history
// shrinks through its generated input, as a mapped arbitrary does.
const arbitrary =
constructions.length === 0
? source.arbitrary
: fc.oneof(
{ arbitrary: source.arbitrary, weight: constructions.length },
...constructions.map((construction) => ({
arbitrary: source.arbitrary.map(construction.build),
weight: 1,
})),
)
for (const construction of constructions) {
const undeclared = construction.enters.filter(
(state) => declared !== undefined && !required.includes(state),
Expand Down Expand Up @@ -2500,6 +2525,11 @@ function generatedTests<THistory, TExpected, TPlan>(
registering,
)
}
// A constructed unit's histories: its construction over the first `bases`
// draws of its own stream, the sample at seed + index, which neither the
// generated draws nor the other constructed histories touch.
const constructedBases = (seed: number, index: number) =>
buildBases(id, source.arbitrary, constructions[index]!, seed + index, bases)
// Registration checks every fixed history, then the constructed histories
// built from the first fixed seed (or seed 0 when there is none), so a
// history that cannot shrink fails the load rather than a full run.
Expand All @@ -2514,8 +2544,8 @@ function generatedTests<THistory, TExpected, TPlan>(
const unitNames = [...constructedNames, `generated`]
/**
* Runs one unit of a campaign whose parameters are `base`. A constructed
* unit runs its one history, built from the campaign's seed, as an
* example with no generated runs; the `generated` unit runs the
* unit runs its bases, built from the campaign's seed, as examples with
* no generated runs; the `generated` unit runs the
* campaign's generated runs, with no examples, so fast-check draws the
* same histories that the whole campaign drew. A unit reports its reach
* counts after it passes, and its rerun and replay commands when it fails.
Expand All @@ -2540,12 +2570,13 @@ function generatedTests<THistory, TExpected, TPlan>(
const { reporter: _reporter, asyncReporter: _asyncReporter, ...rest } = base
let run: typeof rest = rest
if (index >= 0) {
const history = constructedAt(seed, index)
const histories = constructedBases(seed, index)
// fast-check passes example inputs by reference, which identifies them.
constructionOf.set(history, constructions[index]!)
for (const history of histories)
constructionOf.set(history, constructions[index]!)
run = withOracleFixedHistories(
run.path === undefined ? { ...run, numRuns: 0 } : run,
[[history]],
histories.map((history) => [history]),
)
}
oracleRoute(oracleRouteOf(run))
Expand All @@ -2570,9 +2601,12 @@ function generatedTests<THistory, TExpected, TPlan>(
: timeout > 0
? started + timeout / 2
: Infinity
// Every unit runs the property's own arbitrary: the `generated` unit
// draws from it alone, so the constructed list moves none of its
// histories, and a constructed unit's one history shrinks through it.
const details = await fc.check(
fc.asyncProperty(
arbitrary,
source.arbitrary,
predicateFor(constructionOf, counts, shrinkUntil, rerun, violations),
),
run,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@
* task through `scheduleFrom`, waits until idle, and records the release
* order and each task's outcome, or that `scheduleFrom` threw. The check
* compares them. The declared state `a choice that names no queue` is entered
* by a constructed history that puts one first, so half of each campaign
* reaches it; the generated choices are mostly valid, so the release order is
* checked in most of the rest.
* by a constructed history that puts one first, which each campaign runs
* over its bases (5 by default); the generated choices are mostly valid, so
* they reach it only by chance, and the release order is checked in most of
* them.
*
* Limits: tasks are scheduled up front; tasks scheduled by released tasks'
* continuations are not generated.
Expand Down
7 changes: 7 additions & 0 deletions packages/oracles/tests/oracle-property-self-oracle-fixture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,12 @@ export type SyntheticSpec = {
| { kind: `enumerated`; histories: ReadonlyArray<SyntheticHistory> }
/** A second fixed seed, ten times the first, whose text the first's extends. */
secondSeed?: boolean
/**
* The bases each constructed history runs (the property's `bases`); absent
* leaves the framework's default. 0 and 1.5 are not whole numbers of at
* least 1, so they fail the load.
*/
bases?: number
/** Required states; each must be entered by a constructed history. */
states: ReadonlyArray<string>
constructed: ReadonlyArray<{
Expand Down Expand Up @@ -1481,6 +1487,7 @@ export function syntheticProperty(spec: SyntheticSpec): void {
.filter((state) => !spec.states.includes(state))
.map(declare),
runs: spec.source.runs,
...(spec.bases === undefined ? {} : { bases: spec.bases }),
fixedSeeds: [
spec.source.seed,
...(spec.secondSeed ? [spec.source.seed * 10] : []),
Expand Down
Loading
Loading