LibRed: a fully managed, cross-platform Jet/ACE engine and EF Core provider - #293
Merged
Conversation
ExpressionEvaluator.Arithmetic had no DateTime branch, so DateSerial(...) + TimeSerial(...) fell through to Convert.ToInt32(DateTime) and threw InvalidCastException. Date/time arithmetic now operates on the OLE Automation serial (days since 1899-12-30, fractional part = time), verified vs ACE: date+time and date±N days yield a DateTime, date−date yields a day count (Double); the result is rounded to a whole second (Jet has no sub-second) to shed the serial round-trip's float drift. Unblocks the three DateOnly.ToDateTime translations (all threw the same cast); their AssertSql baselines were still SQL Server and never exercised — updated to LibRed's actual SQL. New DateArithmeticTests locks in the ACE-verified behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Normalize DateTimeOffset values to UTC in test fixtures. Update SQL translation tests to use Jet/ACE-compliant syntax (backticks, DATEPART abbreviations, TIMEVALUE) instead of SQL Server style, ensuring dialect compliance.
…folder identity/default Mirror the EFCore.Jet functional-test changes onto the LibRed suite: - delete the tests removed on the Jet side (ComplexJson/OwnedJson associations, primitive-collection variants, Updates TPC/TPT/StoredProcedure/ComputedColumn), - port the structural test changes (F1 List<byte> rowversion + OptimisticConcurrency, ComplexTypesTracking disables, projection/added-method merges, JetEndToEnd / MismatchedKeyTypes / MiscellaneousTranslations, removed Skip_reflexive_foreign_key), - add CompiledModelLibRedTest and the Scaffolding\Baselines csproj rules (baselines themselves not copied — regenerated on run), - extend the compliance test's ignored-test-base list to match. Scaffolder (LibRedDatabaseModelFactory.GetColumns): - add the identity seed/increment annotations (JetAnnotationNames) as EFCore.Jet does; LibRed AutoNumber columns are always (1, 1), - report the column default as DefaultValue (a parsed CLR literal) instead of DefaultValueSql — Jet/ACE stores a literal default value, not a SQL expression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… increment)) Access stores an AutoNumber column's config in the TDEF header: 0x18 = increment (default 1 — earlier mislabelled a constant), 0x14 = last-assigned value initialized to Seed-Increment so the first insert yields Seed. Probed vs ACE (COUNTER(1000, 7) → 0x18=7, 0x14=993). - Parse COUNTER(seed, increment) / AUTOINCREMENT(...) / INTEGER IDENTITY(...) — the (size, scale) pair carries seed/increment (AccessTypeMapper); ColumnSpec gains Seed/Increment. - TdefBuilder writes 0x18 = increment and 0x14 = Seed-Increment from the counter column. - TableDefinitionPage reads them back into ColumnDef (Seed = 0x14 + increment). - RowInserter advances the AutoNumber by the increment, not by 1. - Scaffolder reports the real seed/increment in the identity annotations (was 1/1). Verified: LibRed generates the seeded sequence (5,8,11 / 1000,1007,1014 / …) and ACE opens a LibRed-written custom-counter file and continues it. Engine suite 276/276. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egative Follow-up to the COUNTER seed/increment work. Verified vs ACE that TDEF 0x18 is a full signed 32-bit int, not the 1-byte "autonumber enable" flag mdbtools/Jackcess describe: COUNTER(1,300)→0x18=2C 01 00 00, COUNTER(5,100000)→3 bytes, and decisively COUNTER(100,-5)→FB FF FF FF (-5) with a descending 100,95,90 sequence. - Grammar: allow a signed size/scale (new signedInteger rule) so COUNTER(seed, -inc) parses; regenerate ANTLR. - RowInserter: advance the last-value (0x14) in the increment's direction (max for +, min for -) so a descending counter doesn't reissue the previous id. - Spec §TDEF-header: document 0x18 as signed int32 with the multi-byte/negative evidence. Verified: LibRed generates ascending and descending sequences, and ACE opens a LibRed-written negative-increment counter and continues it. Engine suite 277/277. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Those bytes are the high three bytes of the AutoNumber increment int32 at 0x18, not a separate unknown field. Header now reads 0x14 (last value) → 0x18 (increment) → 0x1C (complex-type autonumber), all int32. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Random AutoNumber (New Values=Random) is an ordinary AutoNumber column plus a DefaultValue extended-property holding GenUniqueID() — no special flag, and the TDEF 0x14/0x18 counter fields stay at defaults and are ignored. Byte-identical descriptor to an increment counter; distinction is entirely the default expression. Decoded from a modern Office-365-authored file; LibRed already surfaces it via the normal DefaultValue read path. Documented in spec § LvProp/AutoNumber. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…a marker Probed ACE: SELECT GenUniqueID() errors (undefined function), but an unquoted `long DEFAULT GenUniqueID()` is valid SQL (numeric only; text rejected) and generates a random signed Long per row (incl. negative). Quoting it makes it a literal string. So a Random AutoNumber carries the unquoted GenUniqueID() default; LibRed reads the text but doesn't evaluate the generator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…GenUniqueID() Probed ACE: CREATE TABLE T (Id COUNTER DEFAULT GenUniqueID(), ...) (also AUTOINCREMENT / COUNTER PRIMARY KEY forms) is accepted and produces genuinely random signed-Long AutoNumber IDs, reading back byte-identical to a UI-authored Random AutoNumber. Corrects the earlier "UI/DAO-only" claim in the format spec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ID() LibRed can now create and insert into a "Random" AutoNumber — an AutoNumber column carrying DefaultValue=GenUniqueID() (Access's "New Values = Random"). - ColumnDef.IsRandomAutoNumber gates the behaviour off the persisted default. - ExecuteInsert excludes AutoNumber columns from default-expression evaluation (GenUniqueID() is not a callable expression). - RowInserter assigns a random non-zero Int32 per row instead of the sequential seed/increment counter, and leaves the TDEF high-water (0x14) unadvanced — as ACE does for a Random AutoNumber. The persisted descriptor is byte-identical to a UI/ACE-authored Random AutoNumber: verified ACE opens a LibRed-written one without repair and issues its own random ids into it (RandomAutoNumberAccessTests), plus engine-level round-trip/insert tests (RandomAutoNumberTests). Spec + format notes updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probed ACE across BYTE/SHORT/SINGLE/DOUBLE/CURRENCY/DECIMAL/GUID/DATETIME/BIT/
TEXT: all reject DEFAULT GenUniqueID() ("Cannot place this validation expression
on this field"). Only a LONG (Int32) column — the COUNTER width — accepts it and
yields a random signed Long per row. Corrects the earlier "numeric columns" note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends Random-AutoNumber support to a plain (non-AutoNumber) LONG column with DEFAULT GenUniqueID() — the other form ACE accepts (LONG only). GENUNIQUEID is now a real function in ExpressionEvaluator returning a random non-zero Int32, so the default-expression path handles it: an omitted value defaults to a random Long, a supplied value is kept (user-writable, unlike an AutoNumber). Verified ACE reads and applies a LibRed-written plain-long default (RandomAutoNumberAccessTests) plus engine round-trip (RandomAutoNumberTests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CREATE TABLE / ALTER TABLE ADD COLUMN now reject DEFAULT GenUniqueID() on any non-LONG (non-Int32) column with ACE's own message, "Cannot place this validation expression on this field" (StatementExecutor.ValidateColumnDefault). Verified rejected across BYTE/SHORT/DOUBLE/CURRENCY/GUID/DATETIME/BIT/TEXT. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Access date/time function defaults on a DATETIME column now work in LibRed:
NOW()/bare Now → current timestamp, Date() → today midnight, Time() → current
time on the Jet epoch. bare Now is recognised as a niladic function
(ExpressionEvaluator.TryNiladicFunction, tried only after column resolution so a
real column named Now still shadows it).
Verified vs ACE that ONLY bare Now is niladic — bare Date/Time are rejected
("Type mismatch" as a default, parameter in a SELECT) because DATE/TIME are
reserved type keywords, so they require parentheses. ACE reads and applies a
LibRed-written NOW() default (DateTimeDefaultAccessTests); engine semantics +
column-shadows-function covered by DateTimeDefaultTests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nested)
A Jet default is a per-row expression, not a static value. LibRed's SQL front-end
and evaluator handle compound defaults — "INV-" & Year(Now()) -> INV-2026,
Now()+30, 1+2, UCase('hi') — which ACE's OLE DB DDL parser rejects ("Syntax error
in CREATE TABLE") even though ACE's read-time expression service evaluates them
fine. Verified ACE reads and applies a LibRed-written compound default, so
LibRed's SQL surface is a superset of ACE's DDL here.
No engine change needed (already supported); adds AceCompoundDefaultTests and
DateTimeDefaultTests coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…refs
Probed two ACE questions:
- Parentheses do NOT unlock compound defaults in ACE's OLE DB DDL the way SQL
Server's DEFAULT (expr) does — (1+2), (Year(Now())), ("INV-" & Year(Now())) all
still error. LibRed accepts parenthesised defaults uniformly (superset grammar).
- A default cannot reference another column (same table or other): ACE errors
"The database engine does not recognize ... the field 'A' ... or the default
value in the table 'T'" — a fundamental Jet limit. LibRed matches (defaults
evaluate against an empty scope; a column ref throws "Column 'A' was not found").
Adds coverage to DateTimeDefaultTests (parenthesised defaults + column-ref
rejection). Findings recorded in the libred-datetime-defaults memory.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LibRed can write a column-reference default ([A] + 2) straight to LvProp — one
ACE's own DDL parser refuses to create. This does NOT corrupt the file: ACE opens
it cleanly and reads the stored default, but its expression service enforces the
"no field reference in a default" rule at INSERT (evaluation) time — the insert is
rejected ("does not recognize ... the field 'A' ... or the default value" /
"Type mismatch") and no row is written. So a column-ref default can't be smuggled
past the engine regardless of how it got into the catalog.
Adds AceSmuggledColRefDefaultTests; finding recorded in memory.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probed the two remaining edges the DAO Field2.DefaultValue doc names:
- Aggregate functions in a default are rejected, and the restriction is broader
than the doc's "SQL aggregate functions": SQL Sum/Count AND the domain aggregate
DCount all fail as "Unknown function ... in validation expression or default
value" — defaults share a restricted function whitelist with field validation
rules. LibRed rejects them too.
- The 255-char DefaultValue limit is a DAO-API cap, not an engine/file-format one:
ACE accepts and applies a 300-char string-literal default; LibRed round-trips
300+ char defaults through LvProp. ("Expression too complex" is a separate
operator-count limit.)
The DAO doc otherwise corroborates the whole session's default findings (defaults
are expressions, no column/query refs, GenUniqueID() = random Long, Random
AutoNumber is a Long field not a DAO AutoNumber). Recorded in memory. Adds
AceDefaultExpressionLimitsTests + DateTimeDefaultTests coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Confirms LibRed rejects a subquery default (the default-expression sub-parser doesn't accept a full SELECT), matching ACE's "no references to queries" rule. Completes the forbidden-category parity set: columns, tables/queries, aggregates and unknown/user-defined functions are all rejected by both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Translating a SQL Server table (CHOOSE / CONVERT([bit],...)) to Jet/ACE-native
(Choose / CBool) surfaced that ACE has the VBA Choose(index, choice-1, ...)
function but LibRed did not ("Function Choose is not supported") — a
harmless-direction parity gap (LibRed too restrictive).
Implemented in ExpressionEvaluator.Choose, verified vs ACE: 1-based selection,
out-of-range index -> NULL, NULL index -> error, any value type. Covered by
ChooseFunctionTests (incl. the translated YESNO DEFAULT Choose(...)/CBool(Choose)
table) and AceChooseDefaultTests (ACE reads/applies a LibRed-written Choose
default, incl. the nested CBool(Choose(...)) form).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Confirms IIF(cond, t, f) is evaluated in a DEFAULT — both branches, an environment-function condition, and string results — matching ACE (verified via DDL and smuggle+insert). No engine change (IIF was already supported); adds IifDefaultTests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch(cond-1, value-1, cond-2, value-2, ...) was missing (another harmless-direction gap like Choose). Implemented in ExpressionEvaluator.Switch, verified vs ACE: returns the first true condition's value, NULL when none match, and requires an even argument count (odd -> "Wrong number of arguments"). Completes the IIF/Choose/Switch conditional trio for defaults. Covered by SwitchFunctionTests (engine) and AceSwitchDefaultTests (ACE round-trip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The conditionals are ordinary scalar functions and work anywhere an expression is allowed. Unlike in a DEFAULT (row-blind), a query evaluates against a row scope so they can read the row's own columns. Adds ConditionalsInSelectTests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds src/LibRed/docs/default-values.md — the authoritative reference for Jet/ACE column DEFAULT behaviour: the "more than a constant, less than a computed value" model, the allowed/forbidden matrix, the two-parser (DDL parser vs expression service) split, GenUniqueID/Now specifics, and the full function whitelist. Full LibRed-vs-ACE scalar-function cross-check (~80 functions swept through both). Added the missing straightforward built-ins to ExpressionEvaluator: Chr, Space, String, StrReverse, StrComp, Str, Val, Hex, Oct, InStrRev, Rnd, Timer, MonthName, IsNull, IsNumeric, IsError, TypeName, VarType — each value verified against ACE (AddedFunctionsTests). Deferred: Format, Partition, StrConv, WeekdayName, and Asc (grammar-blocked). Noted divergences (GenUniqueID/CDec evaluate in a LibRed SELECT but ACE rejects them there) and that Nz is correctly absent from both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Access exposes VBA variant spellings of string functions; probed which ACE's
expression service actually supports and matched them in LibRed:
- $ (string-returning): Left$, UCase$, Chr$, Space$, String$, Str$, Hex$, Oct$,
... — same value as the base function. Grammar now allows a trailing '$' on an
identifier (longest-match makes "Left$" an identifier, not the LEFT keyword) and
the evaluator strips it before dispatch, so every base function gains its $ form.
- B (byte, UTF-16 2 bytes/char): AscB LenB LeftB RightB MidB InStrB implemented
with byte<->char mapping (LenB('abc')=6, InStrB(1,'abc','b')=3). ChrB omitted to
match ACE (which has no ChrB).
- W (wide/Unicode): AscW, ChrW (ChrW(233)='é').
Also unblocks base Asc(): functionName now accepts the ASC keyword (unambiguous
with ORDER BY ... ASC since a call is always followed by '('). Grammar regenerated
(AccessSql.g4 -> Generated). Covered by FunctionVariantTests; full engine suite
(349) still green. Doc + memory updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probed Format/Partition/StrConv/WeekdayName against ACE to characterise them before implementing: - Partition: "lower:upper" right-justified to width=max(len(start-1),len(stop+1)); below/above ranges blank one side. Deterministic. - StrConv: modes 1/2/3 (Upper/Lower/ProperCase); >=4 -> "Invalid procedure call". - WeekdayName: deterministic with explicit firstdayofweek, but the OMITTED default follows the OS regional first-day (locale-dependent) — the deferral reason. - Format: custom numeric formats map ~directly to .NET; custom date formats need VBA->.NET token translation (mm/nn/hh, q); named formats are OS-locale-sensitive (Currency/Short Date/Long Date). A real sub-project. Doc (default-values.md) + memory updated with the pinned semantics; no code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The three bounded members of the deferred-four, implemented in ExpressionEvaluator against their probed ACE semantics: - Partition(n, start, stop, interval): "lower:upper" range label, right-justified to a fixed width, blank side below/above range. Deterministic, matches ACE. - StrConv(s, mode): 1=Upper, 2=Lower, 3=ProperCase; modes >=4 -> "Invalid procedure call". Matches ACE. - WeekdayName(wd, [abbrev], [firstDayOfWeek]): day name counting from firstDay; matches ACE with an explicit first day. Omitted first day is fixed to vbSunday for determinism (ACE follows the OS regional first day) - the one documented divergence. Covered by DeferredFunctionsTests (20 cases); full engine suite green at 387. Only Format now remains deferred. Doc + memory updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Format(value, format) driven off CurrentCulture (as ACE drives it off the OS regional settings), so it reproduces ACE's output on a matching-locale host: - Named formats: General Number/Currency/Fixed/Standard/Percent/Scientific/ Yes-No/True-False/On-Off, and General/Long/Medium/Short Date + Long/Medium/Short Time. - Custom numeric format strings map to .NET directly (0.00, #,##0.00, 0%, \#0). - Custom date format strings via a VBA->.NET token translator (m=month, n=minute, h=24h unless AM/PM present, q=quarter emitted as a literal). - String formats '>' (upper) / '<' (lower). Verified against the ACE probe values. FormatFunctionTests (26 cases) pins CurrentCulture to en-US so the locale-dependent named date/currency formats are deterministic on any host. Full engine suite green at 413. Doc + memory updated; nothing remains deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compared LibRed's InStr to ACE across the documented edge cases: case-insensitive default, not-found -> 0, empty string1 -> 0, empty string2 -> start, null args -> null, start > length -> 0, and binary vs textual compare modes. All 12 match ACE exactly. Locks them in as InstrEdgeCasesTests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous InStrRev handled only the 2-arg form and ignored start/compare.
Rewritten to the full InStrRev(string1, string2, [start=-1], [compare]) contract,
verified byte-identical to ACE:
- start bounds the search to Left(string1, start) (a match must end at/before
start): InStrRev('abcabc','bc',5) = 2, not 5.
- empty needle -> the effective start position (InStrRev('abc','') = 3).
- start = 0 (or < -1) -> "Invalid procedure call".
- a NULL argument -> "Data type mismatch" error (unlike InStr, which propagates
NULL - an ACE asymmetry).
- compare: 0 = binary/case-sensitive, else case-insensitive (default).
Covered by InstrRevEdgeCasesTests (16 cases); engine suite green at 441.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A correlated subquery is evaluated once per outer row, so an EXISTS whose body is a join re-runs that join for every candidate row. Measured on Northwind with the shape EF's ExecuteDelete generates for a predicate over a navigation -- a three-table join inside EXISTS -- deleting 164 of 2155 rows took 92,394 ms. Running that join ONCE costs 101 ms. Correlating barely narrowed the work (61.5 ms per iteration against 101 ms for the whole unfiltered join), because the correlation filters after the joins have happened. Decorrelate instead: strip the `inner = outer` conjuncts, run the body once projecting the inner sides, hash them, then test each outer row against that set. Same statement now takes 436 ms -- ~212x -- deleting the identical 164 rows. This is the standard transform (SQL Server's Left Semi Join, PostgreSQL's JOIN_SEMI, Oracle's subquery unnesting). Reuses the existing hash-join machinery rather than inventing a second notion of key equality: HashKeyComparer, and the same rules that a null key can never match and that a hash only agrees with the evaluator's `=` within one type kind. Declines -- falling back to per-row evaluation -- on TOP/GROUP BY/HAVING (whose result depends on which rows the correlation admitted), a cross-kind correlation, or any outer reference outside the key equalities, including inside a nested subquery. Also fixes a real gap this exposed in QueryPlanner.SubtreeAliases: it had no case for ProjectNode, and every planned SELECT has a projection at its root, so it reported NO aliases for any subquery -- which made the first version of this optimisation decline everything while looking correct, because falling back gives the same answers. Sort/Limit/ Distinct/DistinctRow/IndexScan were missing too. A DerivedTableNode deliberately stays non-pass-through: above it only its own alias is visible. Tests pin the semantics -- nulls on both sides, residual predicates, duplicate inner matches, NOT EXISTS, multi-column keys, an inner LEFT JOIN, and the two fallback paths -- plus a performance guard on the real shape, since every semantics test passes whether or not the rewrite engages and that is exactly how the no-op version hid. Engine suite 776/776 (SubtreeAliases is shared with index selection). BulkUpdates: same 20 failures by name as before, class time 2m48s -> 1m19s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e per row
A subquery that doesn't reference the outer row has the same result every time, yet was
re-evaluated for every candidate row. EF emits exactly that for a GROUP BY predicate:
DELETE FROM `Order Details` AS `o`
WHERE `o`.`OrderID` < (SELECT TOP 1 (...) FROM `Orders` AS `o0`
GROUP BY `o0`.`CustomerID` HAVING COUNT(*) > 11)
That subquery never mentions `o`, but ran 2155 times. The GROUP BY only made each repeat
expensive; repeating it at all was the defect. Cache the result per statement, keyed by AST
node, for all three entry points (scalar, IN, EXISTS). Measured on the four affected
Northwind tests: 42.8s, 11.9s, 5.5s and 1.6s all become ~1s, taking the BulkUpdates class
from 1m19s to 19s (6.7 min before the EXISTS decorrelation).
Deciding "does this depend on the outer row" takes two checks, because each covers what the
other cannot:
* A static walk rejects any QUALIFIED reference to an outer alias, at any depth. Needed
because a conditional can hide one from evaluation -- IIF(x, o.Col, 1) may never touch
o.Col on the row a trial run happens to see. Unrecognised AST shapes count as
correlated, so a missed case costs speed, never correctness.
* A trial run with NO OUTER SCOPE settles UNQUALIFIED references by deferring to
EvalScope.TryResolve: a bare name binding to an inner table succeeds, one that would
bind outward walks out, finds nothing and throws. Reusing the evaluator's own resolver
rather than re-implementing name resolution is the point -- the optimiser and the
evaluator then cannot disagree about what a bare column name means.
Catching broadly is safe in the harmless direction: a body throwing for an unrelated reason
is recorded as correlated and re-run per row, raising the same error the caller would have
seen. A subquery is a SELECT, so the abandoned attempt has no side effects. The negative
verdict is cached too, so a correlated subquery pays one failed attempt, not one per row.
Tests cover both directions of the bare-name case -- textually identical queries where the
column exists on the inner table (hoisted) versus only on the outer one (not hoisted) --
plus the conditional-hidden correlation the static check exists for. Engine suite 776/776;
BulkUpdates same 20 failures by name as the benchmark.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removed [ConditionalTheory(Skip = "LibRed fails")] and related overrides to re-enable several tests in NorthwindBulkUpdatesLibRedTest.cs. Updated the expected SQL for Update_Where_Join_set_property_from_joined_table to use an INNER JOIN with a subquery for setting the City property.
…dependent subquery The residual test asked "does this reference ONLY inner aliases?" via IndexSelection.ReferencesOnly, which reports false for any expression containing a subquery -- subqueries are opaque to it, deliberately, because one may be correlated. So an EXISTS was declined whenever its residual merely mentioned a subquery, even an entirely outer-independent one. Soundness actually needs the narrower question: does the residual reference any OUTER alias? SubqueryHoisting's walk already answers that properly by descending INTO nested subqueries, so use it. Delete_Where_predicate_with_GroupBy_aggregate_2 is exactly this shape -- its residual is an IN over a GROUP BY that never mentions the outer row -- and goes from 90,728 ms to 401 ms, same 109 rows. That query had never been observed to finish before today; hoisting uncorrelated subqueries brought it down to 90s, and this brings it to 0.4s. Unqualified columns in the residual are refused outright. A bare name may bind outward and only the evaluator's resolver can say; the hoisting path settles that with a trial run, but this rewrite commits before the body is ever executed (the key set is built lazily on first probe), so guessing wrong would surface as a query error rather than a fallback. EF always qualifies, so nothing real is lost. Tests pin all three outcomes: a residual whose subquery is outer-independent decorrelates, one whose subquery references the outer row falls back (and yields a different, correct answer, so the two are genuinely distinguished), and an unqualified column falls back. Engine suite 788/788. BulkUpdates: same 20 failures by name as the benchmark. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TOP n for n >= 1 cannot change whether ANY row exists: if the body matches at all it still returns a row, and no limit conjures one where nothing matched. EF emits EXISTS (SELECT TOP 1 ...) for Any(), so declining on any TOP at all left that very common shape evaluating per outer row. Drop the TOP and decorrelate; the key query must not carry it either, since it needs every matching key rather than the first n. TOP 0 genuinely does change existence -- the body returns nothing regardless of the correlation -- so it still declines, as does a non-literal TOP that cannot be judged here. TOP n PERCENT is refused outright rather than reasoned about: for n > 0 it rounds up to at least one row and would be safe, but that is a rule worth verifying against ACE first. Tests cover all three: TOP 1 gives the same rows as no TOP, TOP 0 admits nothing (which discriminates -- wrongly dropping it would admit three rows), and TOP PERCENT falls back. Since the TOP-1 result is identical whether the TOP is dropped or the whole rewrite is declined, only timing separates those, so the performance guard is duplicated for the TOP form: the pair of guards now runs 2155-row deletes twice and the class completes in ~1s against ~92s per-row. Engine suite 791/791. BulkUpdates: same 20 failures by name as the benchmark. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The PERCENT branch has to materialise its whole input before it can compute the take, so `TOP 0 PERCENT` buffered every row only to discard all of them — and did so once per outer row when it sat inside a correlated subquery, which is exactly the shape the EXISTS decorrelation declines. Exit before touching the input when the limit is zero. Plain `TOP 0` was already cheap, since Take(0) returns empty without pulling from the source; the early exit also makes a negative limit well defined rather than relying on Take's tolerance. Tests assert both forms return no rows. Engine suite 793/793. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A GROUP BY body was declined outright, on the grounds that its result depends on which rows the correlation admitted. It does — but grouping on the correlation columns as well reproduces exactly that dependency: each group is split by key, giving the same partition the correlated body saw one key at a time, so HAVING decides each group identically. The keys have to be grouping columns to be projectable anyway. What genuinely cannot be decorrelated is a body that yields a row over an EMPTY input, because then "the key is absent" no longer means "EXISTS is false". That is an aggregate with no GROUP BY (one group even over no rows, so `EXISTS (SELECT COUNT(*) …)` is true for every outer row) or a HAVING with no GROUP BY, which filters that same lone group and can admit it (`HAVING COUNT(*) = 0`). Both now decline explicitly, via EmptyInputMeansNoRows. They were already declining, but only by accident: SubtreeAliases had no AggregateNode case, so no inner alias was found and every conjunct fell to the residual, which then held an outer reference. Supporting grouped bodies means adding that case, which removes the accident — hence the explicit test. GROUP BY keys and HAVING now also face the residual's outer-reference/unqualified-column test, since they stay in the key query. Tests: a theory asserting which shapes the analysis accepts, queried directly rather than inferred from a runtime — the semantics tests all pass either way, which is how an earlier version of this optimisation looked correct while never firing. Engine suite 811/811; NorthwindBulkUpdates unchanged at 14 failures; GroupBy/Where/AggregateOperators 1289/1289. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A correlated IN is a semi-join too — membership is just one more equality, between the subquery's
output column and the value being tested — so the value joins the correlation columns as the hash
key's last element and the probe supplies it alongside them. The body runs once per statement
instead of once per outer row: measured on the Order-Details-by-customer shape, ~26 s to ~0.2 s.
IN is harder than EXISTS because it is three-valued, and the rewrite has to reproduce that exactly
rather than approximate it:
- No match, but the column yielded a NULL, is UNKNOWN and not FALSE — which matters because NOT
IN is then also not satisfied. So rows whose output column is null aren't dropped from the build
phase (as an equi-key null would be); their correlation prefix goes to a second set, and a failed
probe consults it. A null in the CORRELATION prefix still drops the row, since no outer value can
equal it.
- An empty body is FALSE, not UNKNOWN, so NOT IN admits the row. A null correlation value produces
exactly that empty body, and reports (false, false).
- A null left side is UNKNOWN whatever the set holds; the caller already returns early for it.
Stricter than the EXISTS form in two ways, both because IN asks for the body's VALUES and not merely
for a row: any TOP declines (TOP n >= 1 can't change whether a row exists, but it certainly changes
which values are in the set), and GROUP BY/HAVING decline rather than have the output column added to
the grouping on a guess about what a non-grouping column projects out of a group.
Verified against the per-row loop, not just against my own expectations: with the rewrite forced off,
all 8 semantics tests give the same answers — including the pair that pins row 3 as absent from both
the IN and the NOT IN result.
Engine suite 828/828. NorthwindBulkUpdates unchanged at 14 failures;
Where/Navigations/GearsOfWar/Select 4108 passed with the 2 pre-existing
Correlated_collection_after_distinct failures (confirmed failing without this change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`(SELECT COUNT(*) FROM I WHERE I.K = o.K)` was one aggregate per outer row. Grouping the body BY the correlation columns computes every outer row's answer in a single pass: the correlated form aggregates over the rows with K = this row's key, and grouping by K aggregates over precisely those rows, one key at a time. Measured on Northwind (a correlated COUNT over a two-table join, 2155 outer rows): 21,280 ms to 224 ms. The trap is the missing key. Absence from the map is NOT null — a bare aggregate over zero rows still returns a row, so an outer row with no partner gets COUNT(*) = 0, while SUM/MIN/MAX do give null there. Rather than encode a table of per-aggregate empty values, which could drift from what the evaluator actually computes, the miss value comes from asking the executor to compute this very aggregate call over an empty group — the same ComputeAggregate the per-row form would have reached, so the two cannot disagree. (Every branch of it is already empty-safe: COUNT returns 0, FIRST/LAST guard on the count, the rest fall through to "of nothing is NULL".) Only a lone aggregate call is accepted. A non-aggregate body has first-row semantics, whose answer depends on an ordering this rewrite discards; an expression around an aggregate would need its own empty-input evaluation; a GROUP BY body yields several rows for the outer row to take the first of. All decline. The correlation analysis — the WHERE split, the outer-reference and unqualified-column tests, the type-kind check — is now shared, extracted to SubqueryCorrelation with three callers (EXISTS, IN, this) instead of being copied. The extraction was verified behaviour-neutral on its own before the new rewrite went in. Verified against the per-row path with the rewrite forced off: all 8 semantics tests give the same answers, including the pair that distinguishes COUNT's 0 from SUM's null on the same absent key. Engine suite 846/846. Regression set (BulkUpdates, AggregateOperators, GroupBy, Where, Select, GearsOfWar) 5008 passed / 16 failed — the same 14 + 2 known before this change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All three decorrelation rewrites fired on the first probe, which is sound but not always cheaper. The rewrite runs the body once WITHOUT the correlation, so a body that per-row would have been a single index seek pays a full pass instead. Measured on a 3-row outer against a 20,000-row indexed inner: EXISTS 0.2 ms per-row -> 32.8 ms decorrelated IN 0.1 ms -> 34.5 ms COUNT 0.2 ms -> 117.7 ms 160x to 590x the wrong way, in the same engine where decorrelating turns 92 s into 0.4 s the other way. I introduced that regression with the rewrites and hadn't measured this direction. Choosing up front would need the outer row count, and nobody has it at the first probe: rows stream, and the subquery is asked for its answer long before the outer scan finishes. So don't choose — measure. Evaluate per row, accumulate the time actually spent, and switch once it reaches a budget (25 ms). The loss is bounded whichever way the query leans: a per-row-friendly query never reaches the budget and wastes nothing, and a decorrelation-friendly one wastes at most the budget before switching. The budget only has to separate a handful of index seeks from a query that is actually slow, so its value is not delicate — three orders of magnitude sit between them. Switching mid-scan is sound because both forms answer identically, and because DELETE and UPDATE both materialise their whole row set (JoinRows returns a List) before mutating anything, so no probe and no build ever observes a partly-modified table. Checked, not assumed. Known limit, recorded rather than guessed at: the case in between — a medium outer with cheap seeks over a very large inner, where per-row would have finished just above the budget. Bounding that needs the inner's cardinality, and the TDEF does carry a row count, which is where a refinement would start. Tests pin both directions: the tiny-outer shape stays per-row for all three rewrites, and the same body with outer and inner swapped still switches (so the cost check can't degenerate into disabling decorrelation). Engine suite 850/850, with the three existing engagement guards still passing. NorthwindBulkUpdates unchanged at 14 failures and 9 s; wider regression set unchanged at 16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`SELECT TOP 1 c.… FROM Customers c, Orders o, Employees e ORDER BY c.CustomerID` took 31.2 s. The identical query without the ORDER BY takes 6 ms, so the 679,770-row cross join was never the problem — it streams, and TOP 1 short-circuits it. Two defects in the sort, both fixed here: 31.2 s -> 1.4 s. 1. SortNode evaluated the ORDER BY key expression INSIDE the comparer, allocating an EvalScope and an ExpressionEvaluator for both operands of every comparison. A sort of n rows therefore paid O(n log n) evaluations instead of n — here ~13.4M comparisons, so ~27M evaluations of `c.CustomerID`. Keys are now evaluated once per row and compared as values (31.2 s -> 3.4 s). ExecuteAggregate already precomputed its per-group sort keys this way, so this also settles a disagreement between our own two sort paths; they now share CompareEvaluatedKeys. 2. A TOP above a sort still ordered the whole input. SortNode takes an optional row bound, which the planner attaches from an enclosing TOP, and keeps only the smallest n as it goes — trimming back to n whenever the buffer reaches 2n, so each trim is amortised over the n rows it discards and the input is never fully ordered (3.4 s -> 1.4 s). This is the shape behind every `FirstOrDefault` with an OrderBy, so it matters well beyond this query. The bound is only pushed down when nothing between the sort and the limit changes the row count: a projection is 1:1, but DISTINCT/DISTINCTROW collapse rows, so the n rows reaching the limit would not be the n the sort kept. PERCENT is excluded too — it needs the full input count to work out its take. Stability: ties must keep input order (EF's reference and SQL Server both preserve it), which used to come from Enumerable.OrderBy being a documented stable sort. Each row now carries its input position and compares it when the keys tie, making the ordering total — so the unstable sort gives the stable answer, and rows can be discarded early without "first among equals" drifting. Tests: ties come out in input order; a bounded sort agrees exactly with sorting everything and taking n, across bounds of 1/2/7/150/499/500/550 (spanning several trim cycles, exactly the row count, and more than exists) and with DESC + multiple keys; and a TOP over a DISTINCT is left unbounded. Engine suite 861/861. Six functional query classes (Misc, Select, AggregateOperators, GroupBy, Join, Include) verified against a stashed baseline: the same 10 failures before and after, by name — and the run went from 2 m 20 s to 1 m 3 s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EF emits `a = b OR (a IS NULL AND b IS NULL)` wherever a correlation involves a nullable column, so this is the common form, not an edge case. The conjunct walk only recognised a plain `=`, so the whole disjunction fell to the residual, the residual mentioned the outer row, and the rewrite declined — leaving the body to run per outer row. Measured on Northwind, the Late_subquery_pushdown shape: 9,632 ms written this way against 55 ms with a plain `=`, the SQL otherwise identical. Now 202 ms. It is exactly null-safe equality: `a = b` is UNKNOWN if either side is null, so the disjunction is true precisely when both sides are non-null and equal, or both are null. That one difference has to be carried, because a plain `=` key drops null-bearing rows from the hash and fails a null probe outright, while a null-safe key must hash the null and match it. CorrelationSplit therefore records null-safety PER KEY, and the two builders and both probes consult it — so a correlation mixing the two forms treats each key by its own rule. HashKeyComparer becomes null-total (a null element equals only a null element) rather than widening KeyEqual/KeyHash, which are documented for non-null keys. Nothing else is affected: for a plain `=` no null ever reaches the comparer. Neither connective's operand order is depended on, and the near-miss shapes are refused: testing only ONE side's null is not null-safe equality (`a = b OR a IS NULL` is true whenever a is null, whatever b is), nor is a null test on a different column, so both decline and evaluate per row. Tests: NULL matches NULL where plain `=` matches nothing; the identical predicate in a shape that declines reaches the same answer via the per-row path; all four operand arrangements; the two near misses; one plain and one null-safe key in the same correlation; and the IN and scalar-aggregate forms. Engine suite 876/876. Eight functional classes incl. NullSemanticsQueryLibRedTest — which adds zero failures — are unchanged at the 24 known ones. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`SELECT TOP 1 c.… FROM Customers c, Orders o, Employees e ORDER BY c.CustomerID` still built all 679,770
rows of the cross product (91 x 830 x 9) to order them and return one. The previous commit stopped it
SORTING them all; it did not stop it enumerating them, because the key was evaluated on joined rows even
though it only depends on `c`.
When every ORDER BY key comes from one side of a join, sorting that side and letting the join stream
gives the same order as sorting the join's output. Sorting the 91 customers lets the TOP stop the join
after its first row: 1,363 ms -> 172 ms. It pays off without a TOP too — EF's Include emits
`LEFT JOIN … ORDER BY {principal key}`, which now sorts the principal table instead of the whole joined
result, and the join streams rather than being materialised.
Why the tie order is unchanged, which is the crux: the product is enumerated left-major, so tied left
rows already appear in left order within it — the order a stable sort of the product preserves, and the
order sorting the left side alone produces. Verified on a fixture whose sort key ties in pairs and is
anti-correlated with the primary key, so any reordering would show.
Only the LEFT input drives output order — the nested loop iterates it in the outer loop, and the hash
join probes with it (building the right) for INNER/LEFT. A RIGHT join probes with the right, so its
output follows the right side and a left-side sort would be lost: excluded by the kind check. Checked
that IndexSelection never swaps a join's Left/Right, so a pushed sort cannot end up on a build side.
Qualifiers() already returns null for anything it can't pin to a table — an unqualified column, a
subquery, a qualified star — and those decline.
A pushed sort deliberately gets no row bound: BoundSort only walks row-preserving nodes and stops at the
join, and bounding it would be wrong in general since an INNER join can drop left rows, so the first n
sorted rows need not yield n joined rows. (Sound for LEFT/CROSS; noted, not done.)
Tests: cross, inner, left and three-way joins each match the order sorting the product would give, with
DESC and with a TOP; and the four shapes that must not push (keys spanning both sides, right-side-only
keys, an unqualified key) still order correctly.
Engine suite 886/886. 4,244 functional tests across the Include, split-Include, collection-navigation,
Northwind query and BulkUpdates classes, diffed by name against a stashed baseline: 30 failures before,
30 after, empty diff both directions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The test in NorthwindBulkUpdatesLibRedTest.cs is now skipped using [ConditionalTheory(Skip = "LibRed fails")], documenting that it fails with LibRed.
…rom Defender Both changes target the Windows leg of LibRedFunctional, which ran 14m01 against ubuntu's 9m30 and macOS's 10m22 — and that is xunit's own Duration line, so it is test execution, not restore/build. ACE is no longer installed. It was there as a control, on the reasoning that a test failing only off Windows would be a real cross-platform gap. All three legs now agree exactly — 35,578 passed, 179 failed, 2,016 skipped — so nothing in this suite needs an Access engine and the control has served its purpose. LibRedAccess still installs ACE, which is where cross-checks against the real engine belong. Defender is excluded from the workspace, TEMP and RUNNER_TEMP, and for .accdb/.mdb. This leg spends its time writing database files, and real-time scanning of every write is the remaining candidate for the gap: the work is identical (no platform-conditional code anywhere in src/LibRed, and Linux passing identically rules out a Windows-only COM path), macOS sits within 9% of ubuntu so it isn't a kernel-family effect, and durable flushing was measured locally as not the cause (11 s vs 10 s on a write-heavy class with flushToDisk off). Tolerant of failure — a policy-managed Defender just logs and continues rather than failing the job. If the gap survives this, the exclusion can come out; the hypothesis will have been wrong. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Windows went 14m01 -> 8m18 on the run that added it, a 41% drop on the only leg that got the change, with results byte-identical across all three (179 failed / 35,578 passed / 2,016 skipped). Real-time scanning of every .accdb write really was the cost, so the comment no longer invites removing the exclusion as a failed experiment. Not claimed: a split between this and the three engine commits in the same push. Ubuntu improved 18% (9m30 -> 7m47) but macOS did not move at all (10m22 -> 10m20) on the same platform-neutral managed code, so one run cannot separate engine work from runner variance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends the exclusion from the LibRed functional leg to all four jobs that can run on Windows — BuildAndTest (the EFCore.Jet ACE matrix, 8 combinations), LibRed, LibRedAccess and LibRedFunctional — in both the push and pull_request workflows. That is 8 call sites, so the step moved into .github/actions/exclude-defender rather than being copied with its rationale eight times. The OS guard lives inside the action, so it is a no-op off Windows and a cross-platform matrix job can call it unconditionally without a call site being able to forget the condition. It is now called immediately after checkout rather than before the test step, so restore, build and the ACE install are covered too — all of which write a great many files — not just the test run. Measured basis, from the LibRed functional leg: 14m01 -> 8m18, a 41% drop on the only leg that changed, results byte-identical across all three OSes. The same file-write pattern is what the Jet suites do, so BuildAndTest is the one most likely to benefit next; it has ACE installed and runs the full Jet functional suite eight times over. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BuildAndTest moved a couple of minutes out of ~30 (2010/x86/ODBC 29m16, OLE DB 33m32) against LibRed's 41%. Recorded so the uneven benefit doesn't later read as the exclusion having stopped working. Likely cause (user's, and a hypothesis rather than a measurement, but it fits both numbers): the Jet suites run sequentially — MaxParallelThreads = 1, because the ACE provider crashes concurrently — so they spend much of their time blocked on ACE/COM round trips, and Defender scans during time the test was going to wait anyway. LibRed's suite is unlimited-parallel, so every core is busy and scanning contends directly. Kept on those jobs regardless: a couple of minutes across eight matrix combinations is ~15 minutes of runner time a push, for a step that costs nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All five Jet test invocations passed it; no LibRed one did. That single flag is why the Jet logs are enormous while LibRed's are not, despite identical fixtures and identical logging categories: it makes the console logger emit every test's captured output, and EF logs the compiled shaper expression tree per query under the Query category (which BuiltInDataTypes*Test opts into via ShouldLogCategory). One shard reached ~62,000 lines in 43 seconds and GitHub truncated the step. Nothing consumed it. The green-tests extraction parses the trx files, and crash detection looks for Sequence_* blame dumps in the results directory — neither reads console output, and green tests only need the test name and pass/fail. So it bought console I/O for a log too large to read in the UI. `--logger trx` is untouched on all five, so artifacts, green-test extraction and crash detection are unaffected. Put the flag back temporarily if a specific run ever needs the detail. Plausibly worth more here than on LibRed: console writes are synchronous, so on suites pinned to MaxParallelThreads = 1 they sit directly on the critical path rather than overlapping other threads' work — the same asymmetry that made the Defender exclusion matter less on these legs, cutting the other way. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Windows LibRed functional leg ran 14m01 before the exclusion and 8m18 immediately after, which I recorded as confirmation. A later run of the same code came back at 13m, so that leg ranges roughly 8-14 minutes on identical work and the pair proved nothing: GitHub runners are VMs on shared hardware, and a single before/after comparison cannot separate the change from whoever else was on the host. Both paragraphs rewritten to say what is actually known - the exclusion plausibly helps and demonstrably costs nothing, and establishing it would need several runs each way, compared as the windows:ubuntu ratio rather than one absolute number. The same caveat now applies to the couple of minutes seen on the Jet legs, which is equally inside the noise band; the sequential-execution reasoning built on top of that difference is dropped, since there may be no difference to explain. The exclusion itself stays. It is free, and nothing here argues against it - only against the certainty I attached to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EF's Contains-over-a-GroupBy shape puts the correlation in HAVING rather than WHERE:
WHERE EXISTS (SELECT 1 FROM Orders o0 GROUP BY o0.CustomerID
HAVING COUNT(*) > 30 AND (o0.CustomerID = o.CustomerID OR (both IS NULL)))
TrySplit only looked at WHERE — and this body has none — so it declined and re-grouped all 830 orders once
per outer row. On Northwind: 3,030 ms, now 219 ms for the same 31 rows.
Sound only when the correlation's subquery side is a GROUPING KEY, which is the condition enforced. Such a
predicate is constant within a group, so it selects whole groups rather than filtering rows inside them —
which is what makes lifting it equivalent to a WHERE correlation, and what leaves COUNT(*) computed over
the same rows as before. A correlation against an aggregate, or against a non-grouping column (which reads
an arbitrary row of the group), would change what the aggregate sees, so both stay in HAVING and the
rewrite declines. With no GROUP BY at all nothing can be lifted, and that falls out of the same test.
The key query now takes the RESIDUAL having, not the original — leaving the original behind would keep the
body referencing the outer row. Conjuncts() accepts null so an absent WHERE or HAVING simply contributes
none.
Tests: the grouped-key form decorrelates, including in the null-safe spelling (both of today's extensions
at once); an aggregate condition alongside it still applies over the full group; and the three shapes that
must decline — non-grouping column, aggregate correlation, no GROUP BY — fall back and are pinned by the
analysis theory.
Engine suite 895/895. Eight functional classes name-diffed against a stashed baseline: 18 failures before,
18 after, empty diff both directions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The file was last modified 2020-05-01, and before that 2017 by the original maintainers — six commits in total. Nothing in the codebase referenced it; the only two files that did were CLAUDE.md and AGENTS.md, which described it as "the catalog of known unsupported patterns" and skips as `[Fact(Skip = "Unsupported by JET: ...")]`, a string that appears zero times. It survived purely because those two files pointed at it. Their line now just says unsupported tests are skipped with a reason, without describing a convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sql_variant is a SQL Server type with no Access equivalent, so the tests built on it could never pass here. Removed from Jet and LibRed together so the two don't diverge: - BuiltInDataTypes*: two blocks that were already commented out — dead code, no behaviour change. - AdHocMiscellaneous*: Batch_insert_with_sqlvariant_different_types and its Context12482, which insert a double, an int, a string and a DateTime into one `object` column. Nothing in Access stores heterogeneous types in a column. - ModelBuilding*: Can_avoid_attributes_when_discovering_properties and SqlVariantEntity. This one looked incidental — it tests PropertyDiscoveryConvention and touches no database — but both branches throw the same CoreStrings.PropertyNotAdded from ModelValidator: with attributes off the test expects it and passed, with attributes on it expects `[Column(TypeName = "sql_variant")]` to make an `object` property mappable, which only happens on a provider that knows the type. It was SQL Server-specific end to end. - Scaffolding*: the sql_variantColumn line and its assertion, from a table of SQL Server types. Both green-test lists lose the two `Can_avoid_attributes_when_discovering_properties(useAttributes: False)` entries, which were the only removed tests they contained — the others were failing, so were never listed. Verified on LibRed against a baseline worktree at HEAD, same filter both sides: 22 failures before, 19 after, no new ones. The three that went are the two useAttributes:True cases and the batch insert. Test count 1851 -> 1846: three of the five removed were red, two green (the useAttributes:False branch, whose coverage of "an unmappable property is rejected" is the only thing genuinely lost — the same test works with any store type the provider can map, if it's ever wanted back). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
The ReportTestResults workflow failed on the first fork PR: actions/checkout refuses to place a fork's code in a workflow_run context, which runs with the base repository's GITHUB_TOKEN, secrets and runner access. That refusal is correct — it is the "pwn request" pattern — and the documented opt-in (allow-unsafe-pr-checkout) would have executed a fork's code in that trusted context to obtain files this workflow never reads. Nothing depends on the checkout. Every path the job touches — test-results.zip, ./test-results, test-results/**/*.trx — is created at runtime in github.workspace, which exists with or without a checkout: the artifact is fetched through the API by github-script, expanded in place, and handed to dorny/test-reporter as local files. Its provenance agrees. It was added in c04284a (one of four commits in two days titled "Fix test related CI workflows") in the same diff as two console.log debug lines and a PAT, while the cross-repo artifact download was being got working. The token line has since been dropped; this removes the rest. Replaced with a comment recording why there is no checkout, since it was cargo-culted in once already. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
|
Wow!!! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
LibRedis a from-scratch reimplementation of the Microsoft Jet/ACE database engine in fullymanaged C#. It reads and writes
.mdb/.accdbfiles directly — no ODBC, no OLE DB, no DAO,no ADOX — so it lifts the three constraints the existing provider lives with:
The existing
EntityFrameworkCore.Jetprovider is unchanged and still the supported path.LibRed is additive:
LibRed.AdoreferencesEFCore.Jet.DataandLibRed.EFCorereferencesEFCore.Jet, reusing their provider plumbing rather than duplicating it.Architecture
Five projects, one-way dependencies (
EFCore → Ado → Engine → Sql, andEngine → Core):LibRed.CoreMSysObjects→ table/column/index defs), storage, cryptoLibRed.EngineLibRed.EFCoreAddEntityFrameworkLibRed()/UseLibRed(), connection, database creator, scaffoldingLibRed.SqlISchemaProviderLibRed.AdoEvery query runs the full pipeline, even trivial ones, so new features add plan nodes rather
than special cases:
text → parser → AST → binder → bound statement → planner → plan → executor.src/LibRed/docs/format/holds 13 documents specifying the on-disk Jet 4 / ACE format, one perpage type plus cross-cutting topics. It records only facts verified against real files or against
Access itself.
Status
The functional suite is EF Core's own specification tests, adapted:
Identical on Windows, Linux and macOS, test-for-test — and two of those three have no Access
engine present, which is the substantive proof that the engine is genuinely self-contained.
Also implemented: transactions with a deferred-write overlay, index access paths (equality and
range), hash joins, subquery decorrelation, native database creation, and decryption of legacy
Jet RC4, ACE Agile and Office Standard encryption.
The
LibRedFunctionalCI job is markedcontinue-on-errorwhile those 179 remain, so it reportswithout blocking. The other LibRed jobs gate normally.
Known gaps
Access is not implemented.
OFFSET/LIMITis emulated via the existing double-TOPrewrite, which over-returns whenskip + takeexceeds the row count.Side benefit
Running EF Core's specification suite through a strict, from-scratch engine surfaced real defects
in
EntityFrameworkCore.Jetthat OLE DB had been quietly tolerating — query translation, updategeneration and type mappings. Those fixes are in this branch and benefit the ACE path too.
Trying it