Repository navigation
Draft
Conversation
…11/K12 excuses + find a signed-zero divergence Second-pass audit of ufunc out=/where= coverage, then closed the gaps found. NumPy 2.4.2 was the ground truth throughout (probed live). == API exposure (the engine was already out=/where=-ready; only the np.* overloads weren't) == - np.maximum / np.minimum / np.fmax / np.fmin: merged to the house NumPy-shaped overload f(x1, x2, NDArray out = null, NDArray where = null, DType dtype = null). Their engine methods (DefaultEngine.Maximum/Minimum/FMax/FMin) already routed out=/where= through ExecuteBinaryOp -> ExecuteBinaryUfuncInto; the public overloads exposed only `dtype`, so out=/where= were unreachable and had ZERO coverage. BREAKING: the 3rd positional is now `out` (NumPy's order), not dtype — but every in-repo caller uses a positional NDArray out or the `dtype:` keyword, so nothing breaks (verified: 0 positional-dtype call sites in src/ or test/). == Oracle out_where tier extended (29 -> 46 ufuncs; 4737 -> 6243 cases) == gen_oracle.py OUT_BINARY_UFUNCS += maximum/minimum/fmax/fmin/gcd/lcm/fmod/copysign/nextafter/ heaviside; OUT_UNARY_UFUNCS += signbit/isinf/isfinite/fabs/positive/trunc/conjugate. All are pure compare/select/bit-op/integer ops (no libm) -> bit-exact AND portable, safe for the strict all-platform RunCorpus tier. OpRegistry.Kinds.ApplyBinaryOut/ApplyUnaryOut wired for each. == Platform-fragility fix (latent Linux/macOS-only red) == out_where is RunCorpus (strict, runs on all 3 CI OSes) yet OUT_UNARY carried float64 exp/log/sin = Math.Exp/Log/Sin = the host CRT libm, bit-exact vs NumPy ONLY on win-amd64 (ucrtbase). Dropped float64 from exp/log/sin (now float32-only: NumSharp's float32 exp/log/sin are BIT-EXACT ports of NumPy's own SIMD kernels — pure managed, identical bits on every platform). Their float64 plain path stays gated by the host-pinned unary.jsonl. This makes the tier legitimately all-platform. == Excuse corrections (MisalignedRegistry) == - K11 REMOVED: the "out=/where= float32 transcendental within 2 ULP [open question]" branch fired 0x — probed exp/sin(1.0f) now agree bit-for-bit between the plain and out= paths at every corpus shape incl. (4,5) (its own example), so it was a stale excuse that would silently swallow a future regression. Dropping float64 exp/log/sin (above) leaves only bit-exact float32, so no transcendental excuse is needed at all. - K12 REFRAMED (was mislabeled a NumSharp bug) + generalized to isnan/isinf/isfinite/signbit: audit proved NumSharp is CORRECT and NumPy 2.4.2 has the bug. np.isnan(x, out=view) into a rank>=2 [..., ::2]-strided bool `out` with a non-zero prior LEAKS the out's prior contents (its buffered predicate loop fails to cover the gap-strided slots on write-back) — probed exhaustively: confined to those 4 unary float predicates, rank>=2, the inner skip-stride view only (comparisons/ logical/arithmetic, 1-D, f/negstride/offset/transposed, and a zeros prior are all correct on both sides). NumSharp overwrites correctly and MUST NOT reproduce the bug, so those cells are an excused NumSharp-is-more-correct divergence. == New divergence FOUND by the new coverage (min/max signed-zero) == - K13 (new): the SIGN of a +-0 result from maximum/minimum/fmax/fmin is NON-CONTRACTUAL. Since +0.0 == -0.0, IEEE maxNum/minNum may return either, and NumPy picks the surviving zero's sign by SIMD LANE — probed: fmax(+0,-0) returns +0 inside a 20-element array yet -0 inside a 4-element one, so it is not bit-reproducible across array contexts. NumSharp's per-element out=/where= scalar path picks a consistent sign that can differ from whichever NumPy vectorized. Excused ONLY when EVERY diff is a pure +-0 sign flip (both values exactly zero) via the new BitDiff.IsSignedZeroFlip helper (a wrong non-zero result or a NaN-bit flip still fails). Same class as the documented float32-sum NaN-bit order-dependence excuse. == Tests == - test/NumSharp.Tests/Math/UfuncMinMaxSignbitOutWhereTests.cs (new, 12): maximum/minimum/fmax/fmin out=/where=/cast/NaN-propagate-vs-ignore/out-cast-error-names-ufunc, signbit out=(bool+numeric)/ int-input/where=/strided-out/dtype-validate-only, and divmod out=(q,r)/where= (both previously UNTESTED though implemented — verified correct). - Gate: OutWhere green (excused 983x K10 broadcast-out cosmetic + 34x K13 signed-zero + 20x K12 numpy-bug). Full FuzzMatrix: only the pre-existing, unrelated leak-gate red remains (piecewise/logspace/geomspace, 38 families / 54 cases — not touched here). 117 out/where + 51 min/max/signbit/divmod unit tests pass. == Deliberately deferred (separate feature work, documented) == out=/where= for left_shift/right_shift (dedicated ExecuteShift kernel path, needs shift-count edge probing), logical_and/or/xor/not (compose over bitwise but with bool-only dtype semantics), and modf out=(out1,out2)/where= (the one true API gap: multi-output kernel with special inf/nan handling that does not compose from trunc/subtract). Each needs its own NumPy-parity probing + kernel work, out of scope for this coverage-hardening pass.
…ent to full rows The out=/where= family "smoke" tests pinned only the LAST element (o[3]) per ufunc — a whole row of a wrong kernel could pass. Widened them to the full contract, values re-probed against NumPy 2.4.2: - UfuncOutWhereTests.Ufunc_Family_OutSmoke (16 ufuncs): each now asserts out= REFERENCE IDENTITY (the provided instance is returned), EVERY element of the result, and the float64 out dtype — via a new AssertRow(o, params double[]) helper (size + dtype + per-element within 1e-12). - UfuncUnaryBatchOutWhereTests.UnaryBatch_Family_OutSmoke (18 ufuncs incl. the aliases + the int-loop bitwise_not): same treatment — reference identity + every element + dtype, with the pi-based interior values pinned (arcsin/arccos/arctan/deg2rad/rad2deg full rows), plus an AssertRowInt helper for the int32 bitwise_not row. - UfuncComparisonOutWhereTests.Family_OutSmoke_AllSixOps: added out= reference identity per op (all six already checked every element); CollectionAssertInts now also asserts size + int32 dtype (all its callers use int32(4) outs, verified). Pure test hardening — no product change. The remaining single-element assertions elsewhere are intentional (masked-off slots are uninitialized garbage per NumPy, or dtype-selection pins). 144 out/where unit tests green (net8.0).
…a=True), matching NumPy + pin the intended ownership divergences
Second-pass differential flags sweep (numpy 2.4.2 vs NumSharp, 100+ flag-affecting ops)
found one genuine OWNDATA bug amid a family of intended NumSharp-owns-vs-NumPy-view
representation choices, and closes the test-coverage gap on both.
THE BUG (fixed) — Default.BooleanMask.cs
arr[boolmask] where mask.ndim < arr.ndim (a row / prefix mask) gathers the selected
trailing sub-tensors into a FRESH, independent buffer, then returned result.reshape(...)
— a VIEW of that internal buffer — so .flags.owndata read False on what is a genuine
copy (num=1281 instead of 1285). NumPy's boolean selection is an owned copy (probed:
np.arange(12).reshape(3,4)[[True,False,True]].flags.owndata is True, num=1285). The
full-element-mask branch (mask.ndim==arr.ndim) already returned the owned `result`
directly; the partial branch now reshapes that owned buffer IN PLACE (a pure relabel of
the C-contiguous gather buffer, which preserves OWNDATA — see
OwnDataFlagTests.InPlaceMutations) and returns it, consistent with the full-mask branch
and the leading-integer fancy path. Values, shape, writeability and independence-from-
source are unchanged; only the mislabeled owndata bit is corrected. Verified: 2-D row
mask, 3-D row mask, ndim-2 partial mask all now own; full-element mask unchanged; the
result does not alias the source.
COVERAGE — OwnDataFlagTests.cs
These op-level ownership divergences were listed in the class-doc as "KNOWN open, do not
assert" and so were SILENTLY untested. Now:
* BooleanPartialMask_NowOwns_LikeNumpy — green gate for the fix.
* OwnDataDivergences_NumpyInternalRepresentations_Documented [Misaligned] — pins the 12
remaining divergences (current NumSharp num, with NumPy's value documented), so the
surface is regression-gated: flatnonzero / argwhere / np.resize(function) / squeeze
no-op-on-owner (2-D and 1-D) / np.cov (inherits its terminal np.squeeze) / the
advanced-index "transpose-back" family — fancy X[:,[i,j]], single-column X[:,[i]],
column boolean mask X[:,mask], 3-D mid-axis A[:,[i,j],:], ellipsis A[...,[i,j]],
slice+fancy A[0:2,[i,j]], newaxis+fancy M[na,[i,j]]. In every case NumSharp returns a
clean OWNED copy where NumPy returns a non-owning view of a composition temp (or self
for a no-op squeeze); VALUES are identical, only the ownership/layout flag differs.
They are kept OUT of the bit-exact flags_oracle corpus deliberately: matching NumPy
would pessimize correct owned results into views of internal temps (flatnonzero's owned
index buffer, which NumSharp computes directly), or return `self` from a no-op squeeze
(changing ARC disposal semantics NumSharp keeps distinct). A future op fix that changes
any of these records turns the [Misaligned] gate red for re-evaluation.
* class-doc updated: the former "5 KNOWN open divergences, do not assert" note replaced
by the fixed-bug note + the pointer to the documented gate.
No regression: the fix touches owndata only; index values/shapes are unchanged (advanced-
indexing oracle + metamorphic invariants green), and the buffer-pool leak balance is
unchanged (the owned result replaces the view-of-result at the same scope exit).
… every read-only write — resolves oracle K10
NumSharp's central write guard, NumSharpException.ThrowReadOnly, raised a
`NumSharpException` for a read-only destination while NumPy 2.4.2 raises
`ValueError` — with the *identical* message text on both sides. The only
divergence was the exception TYPE. This was the last read-only hold-out on
NumSharpException: np.choose, np.nditer, AxisPartition, byteswap and getfield
already raised NumSharp's `ValueError` for the same condition, so the codebase
was internally inconsistent. ThrowReadOnly now throws `ValueError`, unifying
the whole in-place-write surface on NumPy's exact type AND text.
Probed against NumPy 2.4.2: every read-only write raises ValueError —
assignment destination / output array / put:/putmask: output array /
WRITEBACKIFCOPY base / array — all "... is read-only". NumSharp already
emitted those exact strings (via the `name` argument to ThrowIfNotWriteable);
only the type is now corrected.
Why this is safe and correct:
- ValueError : ArgumentException, INumSharpException. Handlers catching
INumSharpException or ArgumentException (or Exception) still catch it;
only a handler catching NumSharpException *specifically* for a read-only
write needed to move to ValueError.
- No write-through regression: ThrowIfNotWriteable already refuses the write
before any store; a broadcast view (stride-0) reports IsWriteable == false.
Oracle (K10) resolved, excuse deleted not narrowed:
- The out_where tier's 983 `out_broadcast` cases record NumPy's
ValueError("output array is read-only"). NumSharp now raises exactly that,
so CheckError's type+text match passes on its own. The MisalignedRegistry
K10 branch (which excused the type mismatch across ErrorText/Value/Arity/
Shape) is removed; a future write-through, wrong type, or wrong wording now
turns the OutWhere tier red instead of being silently absorbed.
- Fuzz/README.md K10 row marked FIXED.
Verified:
- Direct runtime probe (broadcast out= on np.add/np.sqrt, +where, indexer set,
fill, copyto, put): all raise ValueError with NumPy's exact per-op text;
ValueError still catchable as INumSharpException and ArgumentException.
- Unit gates green (net8.0, --no-build): ReadonlyWriteGuardTests +
FlagsOracleTests write-guard matrix (exact typeof(ValueError) for indexer/
fill/copyto/ufunc-out/put/place/shuffle/flatiter) + WriteabilityMatrixTests
(57) and the remaining read-only classes (153) = 210 read-only tests pass.
The 4 AuditV2 finfo/iinfo failures in the tree are pre-existing and unrelated
(no code path touches ThrowReadOnly).
Test updates (~50 assertions across 20 files): read-only assertions moved
Throw<NumSharpException>/ThrowsException<NumSharpException>/typeof(NumSharpException)/
catch(NumSharpException) -> ValueError; stale "house convention" comments and
src <exception cref> doc tags (NDIter, put_along_axis, fill, copyto, indexer
setter) corrected to ValueError. The NumSharpException.ThrowIfNotWriteable
*method* stays on the NumSharpException class (only what it throws changed).
Parallel-session note: FlagsOracleTests.cs is co-edited by a concurrent
session's in-flight "wave 4" recipe-catalog work; only my read-only-type hunks
were staged (index-only landing via git apply --cached from the HEAD base), so
that session's uncommitted changes to the same file are preserved untouched in
the working tree. The Oracle project's untracked FuzzCorpusTests.Ma.cs (that
session's masked-array work) currently blocks a full Oracle build, so the
OutWhere tier was verified by construction + the direct runtime probe rather
than by running the tier here.
…ex-addressable map, value stored once Adds a fourth point in the ordered-concurrent-map design space beside the two shipped siblings (ConcurrentOrderedDict / ConcurrentOrderedCompactDict) and the experimental ConcurrentPointingDict: a thread-safe, insertion-ordered, index-addressable dictionary that stores each value EXACTLY ONCE. It is the LEAN counterpart of ConcurrentOrderedDict (COD) — COD holds the value twice (inline in the hash node AND in the contiguous array) to buy a one-load key hit; OrderedDict holds it a single time in one dense array, so it uses ~1.5x less memory per entry and builds markedly faster, at the cost of a key lookup that touches three arrays instead of one inline node. LAYOUT (src/NumSharp.Core/Collections/Concurrent/OrderedDict.cs): - A purpose-built open-addressed index (int[] of `slot+1` words, Fibonacci-hashed home + linear probe, 0 = empty) maps a key to a dense insertion-order slot; the slot indexes two contiguous arrays, keys[] and values[]. A key lookup is: acquire-read the index word -> confirm keys[slot] -> return values[slot]. Enumeration / ToArray / AsValuesSpan scan the dense value array directly, so they read like a List<TValue>. Because the value lives only in values[] (never inlined beside the key), a value replace is a single store — no second copy to keep in sync. - All shape lives in one immutable-in-shape `Tables` generation published through a single volatile field; a reader captures a consistent snapshot in one acquiring read. CONCURRENCY (lock-free reads, one write lock, lock-free atomic replace): - Reads need no count-gate: an in-place append writes keys[n]/values[n] BEFORE the index word and publishes the word with a release store, so any word a reader acquire-reads already exposes its slot's key and value, and the keys[slot] equality confirm proves identity. - An atomically-writable value replace is lock-free and rigorous via a `_resizing` flag held for the ENTIRE duration of any generation-replacing resize (set before its array copy, cleared after its publish): a lock-free store commits only if, after the store, `!_resizing && ReferenceEquals(_t, t)` — the flag catches a resize whose copy is mid-flight (which might miss the store), the generation check catches a resize that already published (leaving the store on an abandoned generation); together they cover the whole resize lifecycle so no update is lost. Either check failing falls back to the lock. A non-atomically-writable value type always takes the lock (no torn value ever reaches a reader). - Structural mutation is serialized by one lock (a global insertion order cannot be maintained per-stripe — the deliberate cost of ordering). Reentrant writes from user code the collection runs under the lock (an AddRange source enumerator, a comparer) are detected via the owner-thread id and refused with LockRecursionException rather than silently interleaving two half-applied mutations (Monitor is reentrant). SURFACE (deliberately leaner than the shipped siblings — the point is minimal overhead): TryGetValue, GetByKey, ContainsKey, IndexOf, TryAdd, SetByKey, GetOrAdd, TryRemove, Clear, AddRange, AsValuesSpan, ToArray, KeysToArray, CopyTo, Count, Comparer, and a struct Enumerator over the dense value array. It does NOT carry the order-breaking swap-back, RemoveWhere, or positional removal of the full siblings. STATUS: compiles clean in NumSharp.Core (Release, 0 errors); fully XML-documented (every public/internal member + the concurrency rationale). It is a design-space variant — NO dedicated test suite or benchmark yet, and nothing in-tree references it, so it ships inert (no behavior change to any existing type). The measured comparison of the shipped siblings and the compact design lives in ConcurrentOrderedDict.COMPACT.md; this lean variant is a candidate for the same per-member treatment once gated.
…ariant (hash -> int32 index -> value list), value stored once on the proven CD
Adds the third design point the discovery set out to compare: an EXPERIMENTAL,
thread-safe, insertion-ordered, index-addressable map whose key path is the
vendored ConcurrentDictionary clone used as ConcurrentDictionary<TKey,int> — the
hash node stores only an int32 INDEX that "points" into a contiguous value list,
so each value is stored EXACTLY ONCE (in the list) rather than denormalized into
the node. This is the "pointing dictionary" lead: hash -> int32 index ->
List<TValue>. Built to be MEASURED, not yet shipped.
WHERE IT SITS BETWEEN THE TWO SHIPPED SIBLINGS:
- ConcurrentOrderedDict (COD) stores the value TWICE — inline in the hash node
(ValueIndex{value,index}) AND in the contiguous array — buying a one-load key
hit at the cost of the denormalized copy (~57 B/entry for <int,int>, growing
with value width).
- ConcurrentOrderedCompactDict (COCD) folds the whole index into a bespoke
open-addressed table (no per-entry node) — value stored once, ~21-25 B/entry,
key hit two loads.
- ConcurrentPointingDict (CPD, this type) keeps the PROVEN vendored CD for the
hash part but moves the value out of the node into the list — value stored once,
but the ~40 B node stays. Its niche is WIDE-VALUE memory (no inline value copy)
reached with the LOWEST novel-concurrency risk (it reuses CD's battle-tested
lock-free machinery rather than a hand-rolled table), at the cost of a key hit
that is inherently the most load-heavy of the three.
THE VALIDATED READ (why the key hit costs three dependent loads):
COD reads the value straight out of the node (one atomic load, always consistent).
CPD's key->value path spans TWO independently-published objects — the node's int
index and the Store's value array. A concurrent order-preserving interior removal
rewrites a moved key's node index and THEN publishes a fresh Store; in the window
between them a lock-free reader can pair a stale index with a fresh store (or vice
versa) and read a NEIGHBOUR's value. The fix is a validated read that, after
loading the value, verifies the slot actually holds the requested key
(comparer.Equals(store._keys[idx], key)) and retries otherwise. It is correct with
NO writer-side changes because every published Store is immutable in its live
slots (COD's rule 2), so a consistent (store, idx) pair holding the key is stable
for the whole read and an inconsistent pair is always caught. A hit therefore
reads the node index, then keys[idx] (validate), then values[idx] — three
dependent loads plus one comparer call, versus COD's one and COCD's two.
DESIGN (src/NumSharp.Core/Collections/Concurrent/ConcurrentPointingDict.cs):
- Key path: ConcurrentDictionary<TKey,int> (the vendored clone, concurrencyLevel:1
since all map writes serialize under the one write lock), read through the
clone's GetValueRefOrNullRef ref seam so a lookup is allocation-free and the
in-place int-index rewrite on a removal shift needs no node reallocation.
- List path: a Store{ TKey[] keys, TValue[] values, int count } published through
one volatile field, following COD's memory-model rules (count monotonic per
instance; live slots never structurally rewritten; shrinking transitions publish
a fresh Store). Full sibling surface — GetByKey/SetByKey/ContainsKey/
TryGetValue/TryAdd/Add/TryRemove/Remove/TryRemoveSwapBack/RemoveWhere/SetAt/
TryGetAt/GetKeyAt/IndexOf/TryGetIndex/RemoveAt/Clear/ToArray/Snapshot+ValuesView/
Enumerator/Count/IsEmpty/Comparer.
- Reentrancy guard (Monitor.IsEntered -> LockRecursionException) on every mutator,
matching the shipped siblings' enforced callback contract.
STATUS: EXPERIMENTAL (its own class doc says "built to be measured, not yet
shipped"); compiles clean in NumSharp.Core (Release, 0 errors); fully
XML-documented incl. the validated-read correctness argument. NO dedicated test
suite or benchmark yet, and nothing in-tree references it, so it ships inert (no
behavior change to any existing type). Its purpose is to complete the three-way
memory/latency comparison against COD and COCD documented in
ConcurrentOrderedDict.COMPACT.md; it is a candidate for the shipped siblings'
mirrored gates + the per-member measurement once promoted past the experimental
stage.
…se the coverage gap (fix a real strided-mask bug it found)
WHAT & WHY
==========
np.ma (~205 module functions + the MaskedArray instance surface) had ZERO presence in
the committed NumPy 2.4.2 oracle corpus: MaskedArray is a separate TYPE, absent from
gen_oracle.py/OpRegistry, and the np.ma facade does not trip OracleSurfaceCoverageTests.
It was byte-validated only by throwaway dotnet-run differentials (the pass-13 6055-case
run) + 90 MaskedArrayTests — nothing in CI replayed NumPy bytes for it. This closes that
gap with a new committed corpus family, and the bigger, layout-direct coverage revealed
that np.ma was NOT actually byte-identical on strided/non-contiguous views (one real bug,
now fixed) and a handful of documented combos.
Full design + the divergence ledger: docs/MA_ORACLE_DESIGN.md.
THE NEW CORPUS (26,586 cases, 9 tiers, GREEN net8.0 + net10.0)
=============================================================
A MaskedArray is data + an optional bool mask, which the single-buffer NDArray corpus
cannot carry, so the schema is extended minimally and additively:
* operand += optional `mask` hex (C-contiguous bool) -> nomask when absent
* expected = kind:"masked" with buffer=filled(0) + mask=getmaskarray
The comparator gates filled(0) bytes (BitDiff, NaN-tokenized) + getmaskarray bytes
(bit-exact) — the pass-13 "M-token + hex" scheme, now committed and CI-gated. Op keys are
prefixed "ma." so they never collide with the np.* key of the same name; value divergences
reuse the SHARED MisalignedRegistry with the prefix stripped (np.ma delegates to the same
np.* op on the data). Tiers: ma_{unary,binary,reduce,scan,manip,construct,select,
sortsetops,extras}.
Generator (standalone, like gen_nan_oracle.py): test/oracle/gen_ma_oracle.py
Registry: OpRegistry.Ma.cs (ApplyMasked / ApplyMaskedTuple)
Harness: FuzzCorpusTests.Ma.cs (RunMaCorpus + CompareMasked + tight, labeled excuses)
Schema: FuzzCorpus.{Operand.Mask, Expected.Mask, ReconstructMask}
REAL BUG FOUND + FIXED (src/NumSharp.Core/Ma/MaskedArray.cs)
===========================================================
Mask/scratch allocation from a NON-CONTIGUOUS Shape produced GARBAGE. np.zeros(a.Shape)
for a strided/offset/transposed `a` builds a STRIDED zeros over the base buffer and
materializes the uninitialized tail as non-zero bytes — so getmaskarray of a nomask [::2]
view returned 01-bytes, and every manip/constructor op (transpose/squeeze/reshape/
diagonal/masked_invalid/masked_array/masked_all_like/…) plus `count` inherited a corrupt
mask/count on non-contiguous inputs. Root-caused with a NATIVE (non-reconstructed) strided
view, so it is a genuine library bug, not a harness artifact. Fixed by sizing every mask/
scratch allocation by the DIMENSIONS (new `DimsOf(a)` helper / new Shape(a.shape) /
CountUnmasked(…, DimsOf(…))), matching NumPy's make_mask_none(a.shape). The whole strided
manip/construct class went green; the 90 MaskedArrayTests still pass.
DOCUMENTED KNOWN GAPS (excused with [known gap]/[documented] labels; ledger in the design doc)
=============================================================================================
np.ma is NOT byte-identical for: argsort ndim>=2 (NumSharp uses -1, numpy's documented
FUTURE default; current axis=None is deprecated) and masked/negstride-float tie ordering;
masked unique (numpy treats each masked element as distinct) and complex-NaN unique; flat
keepdims on all/any/median (0-d vs (1,)); var/std/mean/average/anom masking of non-finite
reduction results; average/anom/median narrow-float->float64 widening; anom/median value on
non-contiguous 1-D; sort NaN-in-unmasked ordering; isin/in1d mask propagation; complex
arctanh/power finiteness edges; domained divide/mod signed-zero. Each is scoped tightly so
a regression elsewhere still fails; the byte-exact surface (the common case) is gated.
WIRING
======
The corpus-replay leak/coverage gates (UndisposedIntermediateTests,
OracleCoverageStrengthTests) skip ma_*.jsonl — those drive OpRegistry.Apply over the
ordinary-op model, which ma bypasses via ApplyMasked + RunMaCorpus. (The pre-existing
piecewise/logspace/geomspace leak reds in UndisposedIntermediateTests are unrelated —
a separate session's uncommitted work.)
… cbrt/floor/ceil/trunc/deg2rad/rad2deg/floor_divide/mod — resolves oracle K4/K5
The scan for gaps like K10 (right refusal, wrong exception shape) surfaced K4/K5:
the eight ufuncs NumPy has no complex loop for refused a complex operand with a
kernel `NotSupportedException("... not supported for Complex")` in NumSharp's own
wording, where NumPy raises `TypeError("ufunc '<name>' not supported for the
input types, and the inputs could not be safely coerced to any supported types
according to the casting rule ''safe''")`. Two divergences, one root cause:
- K4 (wording/type): NotSupportedException + NumSharp text vs NumPy's TypeError +
exact text. NotSupportedException already type-mapped to TypeError in the oracle,
so this was a text-only excuse over ~230 corpus cases (errors_full).
- K5 (zero-size skip): on a zero-size complex operand the kernel never ran, so
NumSharp returned an empty array where NumPy still raises — NumPy validates the
LOOP (is this dtype handled at all?), not the data.
Fix (one guard per op, the np.fabs/np.signbit/np.arctan2 template — "the machinery
exists, these kernels simply did not use it"): each `Default.<Op>` method now guards
`typeCode is null && <operand is complex>` and throws NumSharp's `TypeError` with
NumPy's verbatim message BEFORE reaching the kernel. mod's ufunc name is 'remainder'
(np.mod IS np.remainder). Because the guard keys off the INPUT DTYPE rather than the
data, it fires for a zero-size complex operand too — fixing K5 in the same line.
`TypeError : NumSharpException` and is what the oracle already accepts for NumPy's
TypeError, so `except TypeError` (ported Python) now catches these.
Verified against NumPy 2.4.2:
- Direct runtime probe — all 8 ops × {non-empty, zero-size} complex raise TypeError
with NumPy's exact text (16/16); non-complex paths intact (floor(f64), mod(i32));
catchable as TypeError.
- Unit gates green (net8.0, --no-build): B11_Complex_Cbrt_NotSupported (now asserts
TypeError + message, was NotSupportedException), NDEvaluateParityTests (floor/mod
complex still raise), the invert/positive/sign complex-reject tests, and a 622-test
sweep of the eight ops' classes — 0 regressions.
Oracle: K4 and K5 excuses DELETED from MisalignedRegistry (not narrowed), along with
the now-unused ComplexRejectOps set; a regression (wrong type, wrong wording, or a
zero-size skip) now turns the errors_full tier red. Fuzz/README.md rows marked FIXED.
The ~230 errors_full cases for these ops pass on their own (NumSharp emits exactly
NumPy's recorded type+text).
Test update: NewDtypesBattletestRound6Tests.B11 moved Throw<NotSupportedException>()
-> Throw<TypeError>().WithMessage("ufunc 'cbrt' not supported for the input types*")
(its own comment already noted NumPy raises TypeError).
Not fixed here (separate, latent — reported): the dtype=complex "No loop matching the
specified signature" path still throws IncorrectTypeException (unmapped to TypeError),
but it is not exercised by the committed corpus.
Parallel-session note: MisalignedRegistry.cs is co-edited by a concurrent session's
in-flight excuse work; only my K4/K5/ComplexRejectOps-removal hunks were staged
(index-only landing via git apply --cached from the HEAD base), leaving that session's
uncommitted edits to the same file untouched in the working tree.
…nc type errors are catchable as NumPy's TypeError The K10-style scan turned up one more wrong-exception-TYPE gap: the ufunc "No loop matching the specified signature and casting was found for ufunc X" error — plus "not supported for the input types" via dtype=, np.i0's "i0 not supported for complex values", and np.digitize's "x may not be complex" — was raised as IncorrectTypeException, which derived from the generic NumSharpException. NumPy raises TypeError for every one of these, so Python code ported as `catch (TypeError)` (mirroring `except TypeError`) silently missed them. All 25 IncorrectTypeException throw sites (19 files) use one of those three TypeError-class messages — verified — so re-parenting is semantically correct everywhere it is thrown. Change: `IncorrectTypeException : TypeError` (was : NumSharpException). Strict improvement, no breakage: - still a NumSharpException transitively (TypeError : NumSharpException), so `catch (NumSharpException)` and every `Throw<NumSharpException>` assertion keep working; - the concrete type is unchanged, so `ThrowsExactly<IncorrectTypeException>` / `ThrowsException<IncorrectTypeException>` / `Throw<IncorrectTypeException>` (exp/log/signbit/heaviside/hypot/arctan2/…) keep working; - NEW: `catch (TypeError)` now catches them, matching NumPy. Latent, not corpus-active: the committed corpus does not currently exercise the dtype= "No loop matching" cells (0 cases), so this was a runtime-parity gap rather than a red gate. To keep the oracle honest if such a cell is ever added, "IncorrectTypeException" is added to the errors-tier ErrorTypeMap["TypeError"] allow-list (its .GetType().Name stays "IncorrectTypeException" even though it is now a TypeError). The DivMod/comparison/bitwise-for-complex NotSupportedException sites are a SEPARATE class (kernel NotSupportedException, already type-mapped to TypeError) and are left as-is. Verified: runtime probe — np.exp(x, dtype=Byte) (no-loop) and np.digitize(complex) now catch as TypeError, IncorrectTypeException, and NumSharpException alike; unit gates green (net8.0, --no-build) across all 14 IncorrectTypeException-asserting classes (440 tests, 0 regressions).
…logical_and/or/xor/not, modf — gain out=/where=/dtype=, plus modf per-width promotion + a byte-exact float16 kernel + an out-of-place refactor
Closes the deferred feature work from the out=/where= coverage audit (the three ufuncs that
lacked the NumPy-shaped `f(x1[,x2], out=None, *, where=True, dtype=None)` surface entirely).
Every behaviour probed against NumPy 2.4.2.
SHIFTS (left_shift / right_shift)
- np.{left,right}_shift and TensorEngine.{LeftShift,RightShift} gain (dtype, @out, where). The engine
ALREADY emitted the shift kernel through the ufunc out= path — EmitScalarOperation handles
LeftShift/RightShift (→ EmitShiftFromStack) and EmitMixedScalarBody inherits it — so ExecuteShift
routes out=/where=/dtype= straight through ExecuteBinaryOp's out/where machinery (ExecuteBinaryUfuncInto).
- Loop-selection contract enforced up front on every path: a float/complex/decimal dtype= is the verbatim
"No loop matching the specified signature and casting was found for ufunc {left,right}_shift"; a
uint64×signed pair promotes to float64 (NEP50, no loop) and raises the "not supported for the input
types" TypeError. UfuncName(BinaryOp) gained left_shift/right_shift (was falling to "leftshift").
- Perf (NPY/NS, out=): 2.45× @100k; @10m 1.18× (memory-bandwidth wall). The shift SIMD variable-shift
vector body (EmitShiftVectorBody) is now supplied in ExecuteBinaryUfuncInto too, so the hot out=/where=
path vectorizes instead of falling to the scalar body.
LOGICAL (logical_and / or / xor / not)
- Gain (out, where, dtype); return type is now NDArray (was NDArray<bool>), matching the comparison-op
shape — a plain call still yields a bool array, but out=<numeric> returns that numeric out (True→1).
No caller relied on the NDArray<bool> return (verified); NDArray.operator! re-views via MakeGeneric<bool>.
- Composition, NOT a fused kernel: EmitConvertTo(float→Boolean) mixes an int-0 compare against a float on
the IL stack (a landmine), so the truth reduction stays the proven `x != 0` (NaN is truthy) and the
bitwise engine's out=/where= does the combine + masked write. This is fully NumPy-faithful because
bool→X out cast NEVER fails (bool is safe-castable to every dtype → the out-cast error can't fire, so
the composed-over bitwise ufunc name is invisible). dtype= validated bool-only at the np layer naming
the LOGICAL ufunc ("No loop matching…ufunc logical_and"). logical_not: bool→Invert, else→Equal(x,0).
MODF (two-output ufunc) — three things
1. PROMOTION FIX (correctness): modf REJECTED int/float16/bool inputs before ("modf only supports
floating-point types"). Now promotes per NumPy's NEP50 narrowest-float tier — bool/int8/uint8→float16,
int16/uint16/char→float32, int32+→float64 (NOT a blanket float64); float16/32/64 preserved; Complex
refused ("not supported for the input types"). float16 rides a byte-exact port of NumPy's HALF_modf
(widen→modff→narrow both); verified bit-exact over all 65,536 float16 values and NumPy's 10 reference
values (quiet-NaN sign preserved; only a signalling-NaN integral part differs — NumPy keeps raw sNaN,
NumSharp quiets — an exotic edge avoided in coverage).
2. out=(frac,integral) / where= / dtype=: signature modf(x, outFrac=null, outIntegral=null, where=null,
dtype=null) — the positional form matches NumPy's modf(x, out1). Either output may be null (auto-alloc);
the input broadcasts UP to a bigger out; the out-cast error numbers the OPERAND ("Cannot cast ufunc
'modf' output 1 …" for frac, "output 2 …" for integral); read-only out → "output array is read-only";
a non-bool where → the safe-rule cast error; dtype must be a float loop.
3. OUT-OF-PLACE kernel refactor (perf): ModfHelper now reads the input ONCE and writes frac+integral to
their own buffers (NumPy's 3-touch pattern), replacing the old in-place kernel that copied the input
TWICE (fracTemp + intTemp both = input copy). The source is a C-contiguous logical-order buffer
(ascontiguousarray never returns a non-owning view; a 0-d input stays 0-d via the Cast branch since
ascontiguousarray would promote it to 1-D); a provided out that is C-contiguous, the loop dtype, the
iteration shape and unmasked is written DIRECTLY (no temp, no copy-back); a fresh source copy is reused
as the frac buffer in-place to keep the promotion/F/strided paths at two allocations.
Perf (NPY/NS): modf out= 0.40×→1.82× @10m; plain 1.06×→1.26–4×; promote 0.63×→1.38–3.9×.
TYPO (audit finding T1.32): the modf tuple element "Intergral" → "Integral" across np.modf /
TensorEngine.ModF / Default.ModF (no by-name caller). The two AuditV2 T1_32 tests are un-[OpenBugs]'d and
their stale (NDArray, NPTypeCode?) signature lookup updated to the new shape.
Tests: test/NumSharp.Tests/Math/UfuncShiftLogicalModfOutWhereTests.cs (34 — out=/where=/dtype=, cast + error
taxonomy, promotion tier, float16 bit-exactness, special values, all pinned to NumPy 2.4.2). Full main suite
16194/0 green; the modf/binary/logic/multioutput fuzz tiers green (the modf tier's existing float16/int32
cases now bit-compare instead of being W8-A-excused, since modf no longer throws).
Follow-up (deferred, blocked/entangled with concurrent oracle work): add left_shift/right_shift +
logical_and/or/xor/not to the out_where fuzz tier (verified 0 non-broadcast divergences; the tier is
currently blocked on the shared broadcast-out read-only reconstruction, K10), expand the modf tier to the
full int/bool width family, and drop the now-dead W8-A excuse.
… params), find + fix a real median-empty heap-corruption crash Broadens the committed np.ma differential corpus from 26,586 → 68,860 cases to exercise the paths pass 1 did not, and triages every new discrepancy. Design + full ledger: docs/MA_ORACLE_DESIGN.md "Pass 2". COVERAGE ADDED ============== - EDGE LAYOUTS across unary/reduce/scan/manip/construct/sortsetops: c_contiguous_3d, transposed_3d, f_contiguous_3d, broadcast_1d_to_2d, scalar_0d, one_element_1d, empty_2d, negstride_2d_offset, strided_2d_cols, reshape_view_2d (rank>2, broadcast/read-only, degenerate, 2-D non-contiguous — the strided/rank/empty mask paths). - BINARY: scalar + broadcast pair layouts (pp_broadcast_col, pp_scalar_right/left) and 6 more dtype pairs (narrow-int width mixing, signed/unsigned crossing, int/complex + float widths). - SELECT: the previously-skipped `choose`, and `take` mode=wrap/clip (out-of-range indices). - PARAMS: diagonal offset, diag k, moveaxis, diff n/axis, cumsum axis, trailing expand_dims. REAL BUG FOUND + FIXED (src/NumSharp.Core/Ma/MaskedArray.cs) =========================================================== ma.median on an EMPTY array → intermittent HEAP-CORRUPTION CRASH (fatal, uncatchable AccessViolation; same class as the pass-13 arctan2(complex) OOB). MedianAxisMasked guarded only the *reduced* axis being 0-length, so median(axis=1) of a (0,3) array (0 rows, reduced axis length 3) slipped past and ran sort / take_along_axis / middle-averaging on 0-row data, writing out of bounds — cumulative corruption that faulted a later allocation (non-deterministic; needed op diversity + ~2000 complex128 reduces to surface). Localized by op-exclusion bisection to `median`, then to the empty layout. Fixed with a top guard: ANY empty dimension short-circuits to mean (NumPy's _median empty-slice value) BEFORE the OOB-prone path. 90 MaskedArrayTests still green; MaReduce now runs to completion reliably (was aborting every run). HARNESS FIX =========== ma.around dispatch: NumPy's ma.around IS the round ufunc (collapses a 0-d all-masked input to the float64 `masked` singleton); the registry called the two-arg around(a, decimals)=round path (which preserves the input dtype). Routed decimals=0 to the one-arg ufunc. (The two-arg ma.round not collapsing a 0-d all-masked is a latent library inconsistency, noted in the doc.) DOCUMENTED KNOWN GAPS ADDED (excused, tight, printed at run time) ================================================================ - anom/median VALUE on 2-D non-contiguous (strided_2d_cols/negstride_2d_offset): internal mean/sort over a strided view reads the wrong stride (anom yields -inf) — extends the pass-1 anom/median non-contiguous excuse. - complex128 cumprod ~ULP: host-FMA-contracted npy_cmul vs .NET Complex operator* (bounded ≤256 ULP/component) — the nanscan complex-cumprod carve-out class. - NOTED (code-read, not yet plumbed): minimum/maximum/default_fill_value widen a FLOAT fill to double where NumPy keeps the input width. CLEAN (broadened with NO new divergence): manip/construct/select/sortsetops/binary stayed byte-exact across the edge layouts, scalar/broadcast operands and new params — the pass-12 strong-scalar-widening fix holds across the added pairs, and complex128 sort/unique on empty/ edge shapes did NOT crash (only median did). All 9 ma tiers green net8.0 + net10.0; the leak gate reports no ma.* leaks.
… tiers, applicability-matrix gate, all-8-facade surface gates, writeable-flag schema + 6 real Core bugs and 3 leaks the new cells found Executes docs/plans/oracle-coverage-expansion.md (authored earlier today) end to end: every phase lands except the cells that require NEW NumSharp feature work (reduction-family out=, sum-family tuple-axis), which are now WARN cells in the new applicability gate rather than silent absences. Full ledger: test/NumSharp.Tests.Oracle/Fuzz/README.md → "The 2026-09-18 coverage expansion". Plan checkboxes updated in place with a status header. NEW TIERS ========= * instance.jsonl (5,414; gen_oracle.py `instance` mode; OpRegistry.Instance.cs) — the whole `ndarray.*` INSTANCE surface (plan row G0). Op keys carry the "ndarray." prefix (the ma. convention); MisalignedRegistry strips it at Classify entry so the shared excuse branches (float var/std order, complex envelopes, repeat-offset known bug…) cover the instance spellings. Covers: the dual-form methods through their INSTANCE defaults (a.max(axis), a.reshape(-1), a.round(n), a.astype/view/byteswap/getfield, argsort/argmax, cumsum/cumprod, clip/take/compress/searchsorted/choose/dot, flatten orders, transpose perms, diagonal/trace); instance-only members (item→scalar kind with NumPy's Python-scalar dtype mapping, tobytes→uint8 vector for C AND F order, __len__, view(dtype) same-size reinterprets, getfield windows); the §D4 property reads (T/mT/real/imag/flat/nbytes/itemsize/ndim/size/strides); nonzero's tuple (arity == ndim asserted); and the IN-PLACE mutators sort/fill/put/resize — §D3's operand-after comparator expressed with the EXISTING tuple machinery as [post-call view, post-call WHOLE base buffer], so a mutator that writes outside a strided view's window is caught (resize is array-kind: after resize the old base no longer exists; the registry resizes an OWNING copy because reconstructed operands don't own their buffer — the ownership/refcheck contract stays unit-suite territory). partition/argpartition are deliberately ABSENT: whole-output bytes between kth anchors are introselect-implementation- specific on BOTH sides (the sort tier pins DERIVED kth-values; the instance spelling delegates to the same kernel) — first generation proved it with 40+ spurious reds. * emath.jsonl (329; `emath` mode) — np.emath promoted into the differential corpus (§A2/E5): the real→complex promotion DECISION (any(x<0), |x|>1) + promoted values over the complex128/float64 lanes; int8/16/uint16/float32/float16 promote to NumPy complex64 (numpy _scimath _tocomplex) which NumSharp cannot represent (#569) — those lanes stay sibling-owned. Host-pinned (RunHostLibmCorpus). logn/power ride pair operands. 329/329 bit-exact first run. WIDENED TIERS ============= * out_where.jsonl 6,243→6,463: §B2 — out= beyond the elementwise ufuncs, for the surface NumSharp actually implements: out_scan (cumsum/cumprod; complex128 cumPROD carved exactly as nanscan carves it — host-FMA product chain + NumPy copies element 0 where a 1*z seed poisons both lanes through NaN), out_round (round_ decimals×out), out_clip (scalar 0-D bounds as operands; the read-only broadcast-out refusal included — np.clip refuses first, verbatim), out_nanarg (nanargmax/nanargmin axis×out). Same two-slot contract (returned view + ENTIRE out base buffer). The sum/prod/min/max family has NO out= overload yet — a tracked FEATURE gap the applicability matrix warns on, deliberately not generated. * params.jsonl 1,208→1,496: §C1 — multi-axis (tuple-axis) reduction cells, "axes" int[] param binding the int[] overloads, for exactly the reductions that HAVE them (median/average/ nanmedian × {[0,1],[-2,-1],[0,2],[0,1,2]} × keepdims × 3 layouts × 4 dtypes). Bit-exact. OpRegistry median/average/nanmedian gained the axes branch. * errors_full.jsonl 714→801 over 22→50 distinct NumPy messages: §B1 curated recipes over ops with verbatim-parity claims — reshape rejection family (wrong size, multi-unknown), expand_dims OOB both signs, flip axis-OOB + repeated-axis, matrix_transpose ndim, take/put OOB + float-index dtype rejection (same_kind vs safe rule split), partition kth OOB, percentile/quantile q-range, linalg validation (1-D / non-square / float16-unsupported — backend-free cells), fft n-guard. The regeneration ALSO picked up 29 fmod×complex cells the tier had never generated (fmod postdates the last regen) — exposing the missing guard fixed below. * nan.jsonl 120→176: §B3 — the binary special-pair CROSS grid (FLOAT_VALS × FLOAT_VALS, BOTH NaN signs), closing the audit's finding that the pair layouts align A[i] with B[i] so cross-operand NaN interactions (nan+finite, ±inf vs both NaN signs, mod-by-0, copysign-from-NaN, fmax NaN priority) never occurred: 19 ops at f64/f32, the 13-op widen-compute-narrow/bit-level subset at f16 (power/arctan2/hypot/logaddexp* are f32/f64-only — their f16 loops bridge differently), plus 5 complex128 binaries against a rolled grid (divide/multiply byte-exact ports; power under the registry's complex-binary excuse; complex-binary NaN sign stays tokenized — only the unary NDComplexMath family carries the sign pin). SCHEMA — operand.writeable (closed a SILENTLY-RED gate hole) =========================================================== Since 60024b4 deleted the K10 excuse, the OutWhere tier had been red on 293 cells and nobody had run it whole: a SAME-SHAPE np.broadcast_to keeps its source's ordinary strides, so the (dtype,shape,strides,offset,bytes) serialization reconstructed the read-only out as WRITEABLE and NumSharp computed a result where NumPy raises ValueError("output array is read-only"). Bisected via a worktree walk (green at d41eb4a, red from 60024b4 on; NOT caused by either in-flight session). layout_catalog.describe() now emits "writeable": false ONLY when the view is read-only beyond what its strides convey, and FuzzCorpus.Reconstruct honors it through the public setflags(write:false) route — emitted-only-when-False keeps every pre-flag corpus row byte-identical. This also UNBLOCKS the shift/logical out_where oracle wiring 59f9932 deferred on exactly this reconstruction gap. REAL BUGS THE NEW CELLS FOUND (all fixed, cells retained as regression proof) ============================================================================ * np.trace / ndarray.trace promoted the UNSIGNED narrow lanes (uint8/uint16/Char) to int64 where NumPy's add.reduce default-integer rule gives uint64 — wrong result dtype AND wrong wrap point. Invisible until now because the pre-existing trace corpus covered only int16/int32/int64/floats/ complex. Both maps fixed (DirectILKernelGenerator.Trace: TraceTypeInfo + GetTraceAccumTypeCode; Char→UInt64 per the uint16-proxy rule); np.trace docs corrected. * ndarray.flat over a 0-d array returned 0-d; NumPy yields shape (1,) (== np.ravel(0-d), probed 2.4.2). The IsScalar branch now reshapes to (1,); the two stale test pins in NDArray.flat.Test.cs updated with the parity note. * np.fmod with a complex operand leaked the kernel NotSupportedException — fmod landed AFTER the K4/K5 sweep and missed the verbatim no-loop TypeError guard every sibling has. Default.Fmod now raises NumPy's exact "ufunc 'fmod' not supported for the input types…" (dtype-keyed, so zero-size complex is rejected too). * np.take mode='raise' OOB text matched NEITHER NumPy spelling: an axis take says "index N is out of bounds for axis {k} with size M", the flat form drops the axis clause ("for size M"); NumSharp said "for axis with size". axis threaded into ExecuteTakeKernel (full docs added); both spellings now exact. * ndarray.item() on size>1 and __len__() on 0-d now raise NumPy's verbatim texts ("can only convert an array of size 1 to a Python scalar" / "len() of unsized object"); IncorrectSizeException added to the ValueError type map. * THREE pooled-buffer leaks (UndisposedIntermediateTests' zero-leak rule — fixed, never allowlisted; all PRE-EXISTED, the instance tier was simply the first corpus to replay those exact op×layout families): tobytes' materialized reorder copy on every non-directable layout; the f16 SUM's ascontiguousarray temp (flat strided/negstride) and float32 wide/halfResult intermediates (axis path); the f16 MEAN axis path's double intermediate before the narrowing Cast. The leak gate is back to exactly its pre-existing baseline (logspace/geomspace/piecewise/ loadtxt/fromfile/frombuffer — another session's claimed work per b14ff3a's commit message). * net8-ONLY: SIMD hypot returned |other|·√2 for hypot(NaN, finite). .NET 8's Vector256.Max/Min lower to the raw x86 vmaxpd/vminpd, which return the SECOND operand on an unordered compare — so a NaN lane produced a FINITE hi inside the safe band and skipped the scalar completion (.NET 9+ Max propagates NaN and hid it; found by the new binary NaN grid, which pairs NaN with finite for the first time). All three safe-gates in NDHypotMath.Simd.cs now AND in Vector256.Equals(v,v) ordered-operand guards — false only for NaN on every TFM. NEW GATES (plan §A / §M — the durability levers) ================================================ * OracleSurfaceCoverageTests now reflects ALL 8 [ModuleName] facades: `ndarray` (lowercase = NumPy-named, must be ndarray.*-corpus-covered or classified; PascalCase = C# infra by convention; the 25-property row included, T exempted from the uppercase rule), `np.emath` (100% corpus-covered — no classifications permitted), `np.ma` (ma.* keys + 13 aliases + the sibling ledger), `np.dtypes` (the 29 class properties, DTypes-suite-owned) — each with the stale-classification self-retirement rule (§A3), so a renamed member or newly-covered op fails instead of rotting. * OracleApplicabilityTests — the §M1-M3 mechanization: a 22-row declarative manifest of (taxonomy-group × assertion-kind) cells. Enforced cells assert op-count floors (MinCases-style ratchets — e.g. G1×outwhere ≥10 ops, G2×nan ≥15, G0×inplace ≥4); Warn cells PRINT their missing ops (the driven checklist for the next session: reduction out=, nan-reduce errors, G3/G9/G13/ma error recipes, precision widening); NotApplicable REQUIRES a reason and fails if coverage appears anyway; kinds a row omits entirely are covered by its one-line InapplicableNote. Applied kinds are computed from the corpus itself (out_* vehicle keys credit the ufunc named in their params; masked/masked_tuple normalize to array/tuple; zerod from 0-d operands/layouts; multiaxis from the axes param; inplace from the mutator set). EXCUSE BRANCHES (MisalignedRegistry) ==================================== * ndarray./emath. prefixes stripped at Classify entry (shallow-copy recurse — the ma. pattern). * argsort tie order: NumPy's default introsort is UNSTABLE, so ±0.0/duplicate ties resolve arbitrarily while NumSharp's radix argsort is stable. Excused ONLY when ArgsortPermutationsEquivalent re-derives from the operand bytes that both index vectors are in-range permutations selecting pairwise IEEE-equal values (self-limiting; the main sort tier uses distinct values and never reaches it). Trap encoded: identical indices must short-circuit BEFORE the IEEE compare, or an agreed NaN position vetoes the excuse (NaN != NaN). * K13 signed-zero extended to out_clip: clip IS maximum(minimum(x,hi),lo), so a ±0 result's sign inherits the same non-contractual SIMD-lane dependence; still guarded by every-diff-is-a-pure- ±0-flip. VERIFICATION ============ FuzzMatrix net10.0: FuzzCorpusTests 88/88 green (all tiers incl. the 6 touched + ma), gate classes green (surface ×5, applicability, strength, index, metamorphic, harness-self, random- isolation, journey3, BLAS delta); leak gate at its pre-existing baseline with ZERO new families. net8.0: 90/90 (the corpus + gate classes; the binary NaN grid found and now pins the net8 hypot fix). Main suite (CI filter, net10.0): 16,193 passed / 11 skipped / 1 failed — the one failure was the Indexing.KernelRoute pin of the OLD flat-take wording, updated to NumPy's verbatim flat spelling (with the two flat-0-d pins in NDArray.flat.Test.cs) — 193/193 green on the affected classes after. MinCases floors added/bumped (instance 4400, emath 260, out_where 6200, params 1190, errors_full 720, nan 140). The known teardown-crash flake (AccessViolation after all-passed) reproduced twice on full-category runs; per-class reruns are clean, per the documented protocol.
…harp's 15 dtypes, widen the narrow axes, and gate the spread (>=4 dtypes/op + per-dtype floors + a 61-entry reviewed ledger) Answers "do all oracles cover all dtypes NumSharp supports?" by measurement, closes the gaps that were generator artifacts, and turns the answer into a ratcheted gate so it stays true. Model: the NumPy corpus expresses 13 dtypes directly, Char rides the uint16 proxy weave, Decimal rides the independent C# oracle (decimal_*.jsonl); a dtype is COVERED for an op when it appears as an operand of a value OR error cell (a gated rejection IS coverage), with the params/expected dtype as the fallback for zero-operand creation ops. Ledger section: test/NumSharp.Tests.Oracle/Fuzz/README.md -> "The dtype-spread validation". THE SCAN ======== 612 op keys x 15 dtypes across every committed tier (+ the sibling oracles). Pre-pass spread: char 173 ops, decimal 124, uint64 217, uint32 245, int16 252 ... float64 579. The thin tail (< 4 dtypes: 75 ops) split cleanly into generator artifacts (fixable) and by-design narrowness (reviewable) — the two are now mechanically separated by the new gate. AXES WIDENED (all bit-exact on regeneration — ZERO new NumSharp divergences, itself a result: the char instance weave re-proves Char == uint16 across ~50 instance methods incl. the in-place mutators; the modf integer lanes gate 59f9932's per-width promotion differentially for the first time) ============================================================================================ * instance.jsonl 5,414 -> 9,463: INSTANCE_DTYPES 8 -> the 13 NumPy-expressible dtypes, plus char_tier("instance") — gen_instance is now parameterized (dtypes=None) so the whole tier re-runs on the uint16 proxy, dedicated dot/searchsorted/choose and resize jobs included; ndarray.view gains the int16/uint16/uint32/uint64 same-size reinterpret pairs. FIX in _relabel_dtype: it never recursed into kind=tuple SLOTS, so every char in-place mutator (sort/fill/put — slots [post-call view, base buffer], both proxy-dtyped) kept "uint16" slot dtypes after the relabel and failed on a dtype mismatch. Slots now relabel. * modf.jsonl 128 -> 208: MODF_DTYPES was the stale pre-promotion list (f16/f32/f64/int32); 59f9932's per-width promotion (bool/i8/u8->f16, i16/u16->f32, i32+->f64) makes every non-complex lane computable — all 12 generated + char via the new char_tier("modf") branch. * nanreduce.jsonl 10,266 -> 10,416: nanpercentile/nanquantile gain the integer/bool lanes (legal NumPy — no NaN can occur, so it degenerates to percentile; the degenerate path is exactly what was ungated). Generator trap fixed: the NaN-laced float pool RAISES at array construction for unsigned/bool (NumPy 2.x bounds-checks python ints) and a float->uint astype is C-UNDEFINED — integer lanes build an int64 pool and modular-astype it instead. * emath.jsonl 329 -> 482: the unsigned lanes (uint16/uint32/uint64) — safe by construction: no negatives means the any(x<0) promotion never fires, and |x|>1 cells that would land complex64 are auto-skipped by the existing result-dtype guard. * out_where.jsonl 6,463 -> 6,631: float16 + uint8 lanes through out_scan/out_round/out_clip/ out_nanarg (clip keeps its complex skip, cumprod its complex carve). THE GATE — OracleCoverageStrengthTests.EveryOrdinaryOp_MeetsItsDtypeSpreadFloor ============================================================================== Same file scope as the case-count gate (host pins, index_, ma_ excluded — ma has its own coverage model). Two rules, both ratchets: 1. every op reaches >= 4 distinct dtypes OR carries a one-line reviewed entry in FixedDtypeOps (61 entries in five classes: fixed-output generators — windows/fftfreq/ index-coordinate builders; dtype-axis-in-PARAMS — can_cast/promote_types/min_scalar_type sweep dtype PAIRS the operand metric cannot see; PRNG protocol; the host-pinned LAPACK/CBLAS f32/f64/c128 family; variant keys whose primary op carries the axis, e.g. modf-tuple vs modf_frac/modf_int's 13). Self-retiring BOTH ways: an entry whose op vanished, or whose spread grew past the floor, fails — the ledger tracks reality. 2. fifteen per-dtype GLOBAL op-count floors (~95% of the post-widening spread: char 216, decimal 117, uint64 261 ... float64 417), so a regeneration that silently drops a dtype axis turns the gate red instead of shrinking quietly. SIBLING ORACLES (validated as-is, posture recorded in the README) ================================================================= .npy format oracle: full dtype-map coverage incl. <U1/Char, big-endian variants and the <c8 widening; Decimal's rejection is itself a gated case. Advanced-indexing oracle: a dedicated 13-dtype tier (index_dtype.jsonl) + setter-dtype tier. Flags + layout-parity oracles: 13-dtype spot sweeps around an int64/f64 bulk — correct for their contract (flags/view semantics depend on itemsize, not lane type; the flags oracle is under active parallel work and untouched here). Decimal stays gated exclusively by decimal_*.jsonl (124 op keys) by design. VERIFICATION ============ net10.0: FuzzCorpusTests 88/88 (all widened tiers green), the full gate family green (surface x5, applicability, strength incl. the new dtype gate, case-count, index floors), leak gate exactly at its pre-existing baseline (logspace/geomspace/piecewise/loadtxt/fromfile/frombuffer — another session's claimed work) with ZERO new families from the ~5,700 new rows. net8.0: corpus + gates green (dtype gate + Instance/Modf/NanReduce re-run explicitly after the known teardown-abort flake truncated the batch run). MinCases floors bumped: instance 7500, emath 385, modf 165, out_where 6300, nanreduce 8300.
…lections.Concurrent An API-proposal + design doc pitching OrderedDict as a new BCL collection: the concurrent, insertion-ordered, index-addressable dictionary the framework currently lacks. .NET 9's OrderedDictionary<TKey,TValue> is ordered+indexed but NOT thread-safe; ConcurrentDictionary is thread-safe but unordered — no member is all three at once. This is the reference implementation's case for filling that cell. docs/proposals/ConcurrentOrderedDictionary.md: - Gap-analysis table (Dictionary / OrderedDictionary<,> / ConcurrentDictionary vs proposed) + the "manually locking an OrderedDictionary serializes reads too" motivation. - Goals/non-goals — deliberately scopes out lock-free writers, arbitrary positional insert, and O(1) interior remove, so reviewers are not surprised. - Full ref-assembly API surface proposed as System.Collections.Concurrent.ConcurrentOrderedDictionary<TKey,TValue>: the ConcurrentDictionary atomic combinators (TryAdd / TryUpdate / TryRemove / GetOrAdd / AddOrUpdate incl. the TArg overloads) fused with the OrderedDictionary positional read surface (this[int], IndexOf, GetKeyAt/GetValueAt), the right interface set, and the int-key indexer footgun noted. - Concurrency specification (the load-bearing section): lock-free reads, lock-free atomic value replace, serialized structural writes, Volatile/release-acquire (Arm64-safe), snapshot enumeration, a per-op complexity table including the honest O(n) order-preserving remove, and a "what is NOT a fault" list (last-writer-wins, add-not-yet-visible, indexer-races-remove). - Measured evidence: vs OrderedDictionary<,> reads ~parity + enum 1.53x (build/replace the modest thread-safety tax); vs ConcurrentDictionary 2.3-41x single-thread + 2.8x concurrent-read throughput (~914M vs ~325M lookups/sec); concurrency gun ~280M ops with 0 faults + a focused concurrent-get-plus-write test (195.7M reads + 6.0M writes on the SAME keys under resize churn, 0 faulty reads, 0 lost writes). - Alternative designs, risks, open questions (naming, positional mutation, swap-back removal). Reference implementation already on master: src/NumSharp.Core/Collections/Concurrent/OrderedDict.cs (6cf21af).
Upgrades the coverage pipeline to schema v5 by adding `objectTypes` reflection data (including inherited member ownership), object-surface cataloging, and stricter scope projection for dashboard/API counts. It also introduces an offline NumPy docs-link snapshot with anchor-aware auditing, new CI contract tests (Python + Node), and refreshed coverage docs/dashboard behavior. Additionally, the flags oracle was extended with a new recipe wave (87 -> 100 recipes) and regenerated corpus/test expectations to cover previously missing view/copy and boolean-mask flag scenarios.
…harp2 Reference-only clone (pinned at 7334cc3b, PR #218) matching the refs/ convention — every reference tree here is a committed submodule (numpy, pocketfft, OpenBLAS, onnxruntime, parquet-dotnet, data). Added via `git submodule add https://github.com/SciSharp/TensorSharp2.git refs/TensorSharp2`. WHAT IT IS (for future reference): Zhongkai Fu's native .NET LLM inference engine for GGUF models — the Cceusor -> Seq2SeqSharp -> TensorSharp lineage grown into a full local-inference + agent stack (~519K LOC C#, 43 projects). Ships a CLI, browser chat UI, and Ollama/OpenAI-compatible HTTP APIs; runs a dozen autoregressive families (DeepSeek V4/V4.1, GLM 5.x, Qwen 3.5/3.6/3.8, Gemma 4, GPT-OSS, Nemotron-H, Mistral 3, Muse-Glimmer) plus text-diffusion, image edit, and video+audio generation. Benchmarked head-to-head against llama.cpp on identical GGUFs. WHY IT IS A USEFUL REFERENCE FOR NUMSHARP: it is the OTHER SciSharp tensor library, and its core (TensorSharp.Core, only ~11K LOC) is the opposite design point to NumSharp's NumPy port — a Torch/TH-style tensor worth studying by contrast: - Tensor = mutable `class : IDisposable` (sizes[] + strides[] + refcounted Storage + storageOffset), NOT a value-ish NumPy NDArray; `RefCounted` base is CAS+SpinWait AddRef/Release with a finalizer safety net (cf. NumSharp ARC). - DType is a 6-entry enum — Float32/Float16/Float64/Int32/UInt8 + GGML block quants Q8_0/Q4_0 — i.e. exactly what GGUF inference needs, vs NumSharp's 15 dtypes + full NEP50 promotion. No Complex/Decimal/Char/bool, no promotion table. - Ops are result-tensor-IN (`Ops.Add(result, lhs, rhs)`, Torch convention) routed through a reflection-based OpRegistry: backends tag classes `[OpsClass]` and methods `[RegisterOp*]`/`[ArgConstraint]`; dispatch matches op-name + storage type. Device = IAllocator (BlasEnum DotNet/MKL/CUDA/GGML_METAL/GGML_CPU/MLX), so one Tensor type transparently lives on CPU/CUDA/Metal. Contrast NumSharp's TensorEngine + IL-emitted SIMD kernels + bit-exact-NumPy-parity goal. - Backends: managed CPU (from-scratch SGEMM under Cpu/LinearAlgebra, LSAME/XERBLA BLAS-reference lineage), direct CUDA (Driver API + cuBLAS + PTX), Apple MLX, and a native C++ ggml bridge (TensorSharp.GGML.Native). AGENTS.md: keep upstream ggml unmodified; TensorSharp-owned code implements anything ggml lacks. Layering above Core: Runtime (GGUF/tokenizers/sampling, paged KV cache, continuous-batching scheduler, speculative decoding) -> Models (13 architectures + multimodal) -> AgentHost (skills + sandboxed code exec) -> Chat (host-neutral pipeline) -> Server/Cli/TensorAgent(iOS). Not built or referenced by NumSharp; it sits in refs/ purely for study, like the other reference trees.
…ng[]/Shape overloads)
np.indices and np.indices_sparse accepted ONLY int[] for the grid dimensions,
which is at odds with the whole house shape vocabulary — NDArray.shape is long[],
Shape.Dimensions is long[], and np.zeros/np.ones/np.empty already offer the full
int[]/long[]/Shape overload set. A caller holding their own array's shape had to
manually narrow it to 32 bits to reach np.indices, risking silent truncation of a
dimension that only fits in 64 bits.
This mirrors the ndindex/zeros overload pattern: one long-native core with thin
spellings on top.
## API surface (both dense np.indices and sparse np.indices_sparse)
- indices(int[] dimensions, DType dtype = null) — KEPT for back-compat + the
oracle harness; widens to the long core (int[] does not convert to long[] by
array covariance, so it needs its own entry).
- indices(long[] dimensions, DType dtype = null) — NEW, THE house shape type.
`np.indices(arr.shape)` now binds here with no down-cast — the friction this
fixes. Carries `[NDScoped]` (the astype temp is born here).
- indices(Shape dimensions, DType dtype = null) — NEW. Accepts an NDArray.Shape
directly, and lets a C# value tuple read like Python: `np.indices((2, 3))`.
Overload resolution is unambiguous: an exact-type parameter (int[]->int[],
long[]->long[]) always wins over Shape's implicit int[]/long[] conversions, so
every existing `np.indices(intArray)` call still binds the int[] overload (verified:
OpRegistry, benchmark, MaskedArray all recompile clean).
## Core made long-native
BuildDenseIndices / BuildSparseIndicesAsArray now take long[]. The IL fill kernel
(DirectILKernelGenerator.Indices) was ALREADY fully long-typed (dimStrides/dims/
prod/result are long), so nothing changed there — only the public param and the
`int dim` reads were the 32-bit chokepoint. The dense builder now pins the long[]
dims array directly for the kernel (dropping the per-call int->long scratch copy),
and the sparse builder uses arange(0L, dim). Overflow detection on the running
element-count product is retained (long-native).
Behaviour parity: values, dtype, empty-dims (-> (0,)), and negative-dim rejection
all match the former int[] path bit-for-bit (differential-fuzz creation tier green).
A dimension > int.MaxValue paired with a 0 now yields shape (2, 2147483648, 0),
size 0, allocation-free — the old API could not even express it, and a silent
(int) narrowing would have flipped it negative and thrown.
Deliberate, documented convenience divergence (Shape overload only): because a
scalar converts implicitly to a 1-D Shape, `np.indices(5)` is accepted as the grid
(1, 5); NumPy's `np.indices(5)` raises (an int is not a sequence). Spell a 1-D grid
as `np.indices((5,))` to stay verbatim. Same class as zeros' scalar convenience.
## Internal consumers cleaned up (they existed BECAUSE of the int[]-only limit)
- np.mgrid.Build: was `int[] sizes = checked((int)MeshAxisSize(...))` then re-widened
to `long[] full` right after — a pure long->int->long round-trip forced by the
int[]-only indices, plus an artificial 2^31 grid ceiling. Now `long[] sizes`
straight through; `fullShape = new Shape(sizes)`. Negative-size and zero-step
behaviour unchanged (indices still rejects negatives; MeshAxisSize still throws
on a zero real step).
- MaskedArray.notmasked_edges: `np.indices(d.shape.Select(x => (int)x).ToArray())`
-> `np.indices(d.shape)` (d.shape is long[]).
- MaskedArray.indices wrapper: added long[] and Shape overloads beside the int[] one.
## Tests
- test/NumSharp.Tests/Indexing/np.indices.Test.cs (NEW, 14 tests): spelling parity
(int[]==long[]==tuple==arr.shape==arr.Shape), known 2x3 values, dtype, empty,
negative-throws, the >int.MaxValue non-truncation case, sparse parity + shapes,
the mgrid regression, and the ma.indices long/Shape overloads.
- Green: np_indices (14), mgrid+ogrid+MaskedArray (123, incl. notmasked_edges with
axis), creation FuzzMatrix tier (indices/indices_sparse differential), and
OracleSurfaceCoverageTests (5). Core/test/Oracle build 0W/0E.
…ples, dtype-safe
Convert the DocFX documentation examples from `new[]{...}` / `new int[,]{...}`
array-creation syntax to C# 12 collection expressions `[...]` (and `()` tuples
where a shape/tuple overload exists) wherever it is BEHAVIORALLY IDENTICAL —
so the examples read like NumPy's own `np.array([...])` — while carefully
preserving every case where `[]` would change semantics or dtype.
Every replacement class was verified empirically against NumSharp.Core
(compile + run), not assumed:
np.array(new[]{1,2,3}) -> np.array([1,2,3]) int32, identical
np.array(new int[,]{{..}}) -> np.array([[..],[..]]) 2-D, identical
np.array(new long[]{1,2,3}) -> np.array([1L,2L,3L]) int64 PRESERVED (L suffix)
np.array(new long[,]{..}) -> np.array([[10L],[20L]]) int64 PRESERVED
np.array(new float[,]{..}) -> np.array([[5.1f,..],..]) float32 PRESERVED (f suffix)
np.transpose(x, new[]{..}) -> np.transpose(x, [..]) int[] arg (no long[] twin)
np.hsplit(x, new[]{3,4}) -> np.hsplit(x, [3,4])
np.indices(new[]{3,3}) -> np.indices([3,3])
np.loadtxt(usecols:new[]{..})-> usecols:[..]
nd.AsDataView(new[]{"a",..}) -> nd.AsDataView(["a",..]) string[] overload
np.concatenate(new[]{a,b}) -> np.concatenate([a,b]) still an array (prose stays valid)
DELIBERATELY KEPT AS `new[]` — these are NOT convertible (verified):
- Fancy indexing x[new[]{0,2}] stays: a collection expr `[0,2]` binds to the
COORDINATE indexer (x[0,2]), not the fancy int[] one — it returns a scalar or
throws "too many indices", NOT the row-select. Only the np.array(...) DATA
wrapped inside such index expressions was converted; the raw index was not.
- Narrow integers new sbyte[]/byte[]/short[] stay: no C# literal suffix, so
`[-1,0,1]` infers int (int32), not int8. Added a note in dtypes.md and
array-creation.md documenting the suffix rule (L/u/f/m widen inline; the
narrow types need `new T[]` or a per-element cast).
- var names = new[]{...} stays: `var x = [...]` has no target type (won't compile).
- np.zeros(new[]{3,4}) in NDArray.md stays: it teaches int[]->Shape, and
`np.zeros([3,4])` is C#12-ambiguous between the int[]/long[] overloads.
- Empty typed (new double[0]); NDIter / nested_iters operand/dtype/flag arrays;
reflection Type[]; np.block object[][]; ORT session.Run; System.Numerics.Tensors;
and byte[] buffers — all left as-is.
Method: 148 bulk 1-D conversions applied by a tightly-scoped, dry-run-reviewed
regex anchored to `np.array(` (so it can never reach a fancy index, np.concatenate,
or an NDIter operand array), plus ~16 hand-edited special cases (2-D literals,
long/float suffixes, int[]-only args, concatenate, the dtype-table comments).
Comments that named the now-hidden array type were reworded to be literal-accurate
(e.g. "from int[]" -> "int literals").
Pre-existing uncommitted changes left untouched (not part of this work):
docs/website-src/docs/toc.yml, templates/numsharp/public/main.js,
docs/extensions/bitmap.md, docs/advanced/index.md.
…invalid (5,) in doc
Follow-up to the long-friendly np.indices overloads: prove the API is idiomatic from
C#/.NET, and bring the one DocFX example that uses it in line with the new surface.
## .NET-friendliness tests (np.indices.Test.cs, +7)
Exercise the API through real C# idioms, all verified compiling AND agreeing:
- Collection expressions: np.indices([2, 3]) binds int[] (unambiguous — an identity
element conversion beats int->long, so adding the long[] overload did NOT make the
documented [3, 3] spelling ambiguous), np.indices([2L, 3L]) binds long[].
- Value tuple: np.indices((2, 3)) reads like NumPy.
- The `row, col = np.indices(...)` unpack idiom via the leading-axis indexer.
- Result is a normal NDArray: fluent chaining (…[1].reshape(6L)).
- Sparse result is a plain NDArray[]: foreach + LINQ (Select over shapes, sum of sizes).
- dtype via every .NET spelling: typeof(int), NPTypeCode.Int32, "i4", np.int32.
- The documented scalar convenience: np.indices(5) -> (1, 5) grid (NumPy raises), with
the valid C# 1-D spellings np.indices([5]) / np.indices(new long[]{5}).
Green on BOTH net8.0 and net10.0 (collection expressions / tuples compile on both).
## DocFX example coverage + accuracy
docs/website-src/docs/fundamentals/array-creation.md was the ONLY page with an np.indices
code example, and its comment ("takes an int[] of dimensions (not a tuple)") became false
with the new overloads. Updated it to show every .NET shape spelling (collection expr,
tuple, long[], an array's own .shape) plus the sparse form.
FundamentalsArrayCreationDocTests: the mirror test Intrinsic_Indices_TakesIntArray ->
Intrinsic_Indices_AcceptsAllShapeSpellings, asserting all four spellings agree on the
(2,3,3) grid, the documented grid[0]/grid[1] values, and the sparse (3,1)/(1,3) shapes.
## Doc fix
The np.indices(Shape) XML <remarks> suggested spelling a 1-D grid as np.indices((5,)) —
INVALID C# (no 1-element tuple). Corrected to np.indices([5]) / np.indices(new long[]{5}),
which is what the new scalar-convenience test pins.
Green: np_indices (20) + FundamentalsArrayCreationDocTests (15) on net8.0 AND net10.0;
whole Documentation namespace (192). Core/test build 0W/0E.
Convert every em-dash (U+2014) to an ASCII hyphen-minus (-) in the hand-authored docfx source under docs/website-src/. 1574 occurrences across 42 tracked markdown/YAML files. Scope: only git-tracked *.md / *.yml content in docs/website-src/. The generated api/*.yml (gitignored) and the refs/data report content pulled in at build time are untouched. Method: byte-level replacement of the UTF-8 encoding E2 80 94 -> 2D, so line endings (CRLF/LF), any BOM, and every other byte are preserved exactly. The diff is a balanced 1438/1438 (each touched line is one delete + one add), so there is no whitespace or EOL churn. UTF-8 is self-synchronizing, so the 3-byte sequence cannot collide with part of another character. Safety checks made before converting: - No em-dash starts any line (after leading whitespace), so none could turn into a stray "- " markdown list item. - The only YAML occurrence was mid-value (docs/toc.yml, "Under the hood - internals"); the list-item dash sits at line start and is unaffected. - Em-dashes inside fenced code blocks were all prose comments (// ... - ...), never executable syntax. En-dashes (U+2013, 27 occurrences) are intentionally left as-is; only em-dashes were requested.
Nucs
changed the title
…p.ma.masked stays as NumPy alias
NumSharp's port of numpy.ma.MaskedArray now follows the ND* type convention of NDArray.
Renames (symbols, done with the IDE rename)
- class MaskedArray -> NDMaskedArray (file Ma/MaskedArray.cs -> Ma/NDMaskedArray.cs)
- class MaskedConstant -> NDMaskedConstant
- tests MaskedArrayTests -> NDMaskedArrayTests (file Ma/MaskedArrayTests.cs -> Ma/NDMaskedArrayTests.cs)
- prop np.ma.masked -> np.ma.NDMasked
- prop np.ma.masked_singleton -> np.ma.NDMaskedSingleton
- the ma oracle harness (FuzzCorpusTests.Ma / OpRegistry.Ma) and three gate comments follow the type rename.
MaskedArrayModule (the np.ma facade) and MaskedPrintOption keep their names.
NumPy API parity kept: np.ma.masked / np.ma.masked_singleton
- NumPy 2.4.2 (probed): np.ma.masked is the ONE numpy.ma.core.MaskedConstant instance (a MaskedArray
subclass), a 0-D float64 0.0 under a True mask, `masked is masked_singleton` is True, repr 'masked',
and a fully-masked reduction returns it (`ma.array([1.,2.], mask=[1,1]).sum() is masked`).
- Both NumPy names are re-added on MaskedArrayModule as EXPRESSION-BODIED aliases of NDMasked, so all four
spellings (NDMasked, NDMaskedSingleton, masked, masked_singleton) are the same object. A second stored
`new NDMaskedConstant()` would have broken the reference-identity contract: the ufuncs/reductions return
NDMasked, and `ReferenceEquals(r, np.ma.masked)` (NumPy's `r is np.ma.masked`) must hold. Ported NumPy
code (`x[i] = np.ma.masked`) compiles unchanged; NumSharp code can use the ND* name.
- New test NDMaskedArrayTests.MaskedConstant_NumPyNames_AliasTheNDMaskedSingleton pins identity across all
four names, the NDMaskedConstant type, the repr token, the 0-D float64 0.0 / True-mask payload, the
fully-masked sum returning np.ma.masked, and `y[1] = np.ma.masked` masking without touching data.
Repaired: the IDE's "text occurrences" pass damaged prose, NumPy-verbatim strings, and the oracle gate
The rename ran with text-occurrence replacement, which rewrote EVERY whole word `masked` -> `NDMasked`
(1,104 comment/string hits) and `MaskedArray` -> `NDMaskedArray` even where the text names NumPy's own
class. Left as-is it would have shipped:
1. A red oracle gate. FuzzCorpusTests.Ma's `case "masked":` became `case "NDMasked":`, but the committed
ma_* corpus spells the kind "masked" (~62K cases), so every masked case fell to `default` ->
"unknown ma expected.kind 'masked'". Verified: MaReduce / MaScan / MaExtras all failed that way.
2. Broken NumPy-verbatim messages: "A masked array does not own its data and therefore cannot be
resized.", "Cannot process masked data." (x2), "Unavailable for masked array.", NumPy's own
NotImplementedError "MaskedArray.tofile() not implemented yet.", the view() itemsize message, and the
NDMaskedConstant.ToString() token "masked" (NumPy's repr(np.ma.masked)). Their tests had been rewritten
in lockstep, so the unit suite stayed green while parity broke.
3. English turned into nonsense, including in files the rename never needed to touch: "masked-off slots
keep prior contents" -> "NDMasked-off slots" in Default.Fabs/Frexp/Ldexp/Positive/Reciprocal and
DefaultEngine.CompareOp, "NumPy's MaskedArray._insert_masked_print" -> "NDMaskedArray._insert_…" in
ArrayFormatter, the masked-corpus schema docs in FuzzCorpus.cs, "masked positives" in
UndisposedIntermediateTests, and the pythonnet decode tests that describe the PYTHON-side numpy
MaskedArray. Those 9 files outside the ma rename are byte-identical to HEAD again (not in this commit);
UndisposedIntermediateTests keeps only its correct NDMaskedArray type mention.
How the repair was done (mechanical, then reviewed)
- HEAD and the renamed tree were tokenized (words + single non-word chars). The rename was strictly 1:1
word-for-word, so both token streams have equal length and zip into exactly 1,889 substitutions.
- Each substitution was classified with Roslyn at its position:
code identifier / doc-comment cref ....... 701 kept (real references to the renamed symbols)
string / interpolated-string literal ....... 19 reverted (NumPy-verbatim text, corpus kind, repr)
comment / XML doc text: `masked` ......... 1,087 reverted (the English word / NumPy's constant)
comment / XML doc text: masked_singleton ..... 3 reverted (NumPy's name)
comment / XML doc text: MaskedArray ......... 79 split by context: 29 name NumSharp's own class
(kept as NDMaskedArray / NDMaskedArrayTests);
50 name NumPy's class (reverted: numpy.ma.MaskedArray,
"NumPy's <c>MaskedArray.x</c>", MaskedArray._mask, …).
10 of those 50 were caught in manual review (the "NumPy's" sat on the previous doc line, which the
context heuristic did not see).
- The +2-char rename had pushed aligned trailing `//` comments right; the 7 affected lines are put back
on HEAD's comment column. Two articles fixed to house style ("an NDMaskedArray"; the codebase writes
"an NDArray" 196x vs "a NDArray" 2x).
Docs
- docs/MA_ORACLE_DESIGN.md, docs/MA_MODULE_AUDIT.md: current-state pointers now name
Ma/NDMaskedArray.cs / NDMaskedArrayTests, plus a dated rename note; the historical pass logs keep the
names that were in use at the time.
Verification (isolated detached worktree at b779d95 + exactly this changeset; the shared tree carries
an unrelated in-flight collections rename that is NOT part of this commit)
- Build: NumSharp.Tests + NumSharp.Tests.Oracle (Core, OpenBLAS interop) - 0 errors.
- NumSharp.Tests `NumSharp.Tests.Ma.*`: 91/91 on net10.0 and net8.0 (90 existing + 1 new).
- Oracle (net10.0), each in its own test host: MaUnary, MaBinary, MaReduce, MaScan, MaManip,
MaConstruct, MaSelect, MaSortSetops, MaExtras, MaSurface_IsCoveredOrExplicitlyClassified,
OracleCoverageStrengthTests (3), OracleApplicabilityTests (1) - all pass.
- No remaining code references to the old identifiers in any tracked .cs; no API snapshot, coverage
mapping (coverage/object_surfaces.py maps ma.MaskedArray with no CLR owner) or website page names them.
Pre-existing, NOT introduced here (reproduced 3/3 on a clean b779d95 worktree)
- Running several ma oracle tiers in ONE test host intermittently kills the host: it vanishes mid-test
(the crash point varies from after MaReduce to during OracleApplicabilityTests), with no managed
exception, no stderr, and no Application event-log entry. It is timing-sensitive: a --diag run of the
9 tiers completed with exit code 0. Every test passes in its own host.
Not included (other work present in the tree): the staged OrderedDict/ConcurrentOrderedDict ->
*Dictionary collections rename and ConcurrentPointingDict.cs deletion, benchmark BenchmarkBase.cs /
Program.cs / ConcurrentOrderedDictBenchmarks.cs edits, and the dirty refs/data submodule.
…entPointingDict removed, docs and crefs fixed
Lands the rename of NumSharp's ordered concurrent maps (an IDE rename, finished here) and deletes the experimental
ConcurrentPointingDict, with every doc, cref, test, benchmark and probe brought in line.
TYPE / FILE RENAMES (namespace NumSharp.Collections unless noted)
- ConcurrentOrderedDict<TKey,TValue> -> ConcurrentOrderedDictionary<TKey,TValue> (COD)
- ConcurrentOrderedCompactDict<TKey,TValue> -> ConcurrentOrderedCompactDictionary<TKey,TValue> (COCD)
- OrderedDict<TKey,TValue> -> OrderedDictionary<TKey,TValue> (lock-free value-once variant; the
proposal's reference implementation)
- ConcurrentOrderedDict.{TODO,COMPACT}.md -> ConcurrentOrderedDictionary.{TODO,COMPACT}.md (design ledgers)
- ConcurrentDictionary.NumSharp.cs -> ConcurrentDictionary.RefAccessors.cs (the vendored clone's one
NumSharp-only member). GetValueRefOrNullRef is now PUBLIC (was internal) - deliberate part of the rename; its
remarks now say the three caller obligations (lifetime inside one serialized section, single-field <=word writes
only, caller-side write serialization) bind EVERY caller, and that it bypasses the tear-free discipline the REST of
the public surface keeps.
- All 11 test classes/files now read ...Dictionary... (the IDE rename had covered only the two primary suites;
the Advanced/Concurrency/Memory/Randomized mirrors of both types and COCD's Specific suite still said "Dict").
- ConcurrentOrderedDictBenchmarks -> ConcurrentOrderedDictionaryBenchmarks. LOAD-BEARING, not cosmetic: the rename
changed menu option 15 to `--filter "*ConcurrentOrderedDictionary*"`, and BenchmarkDotNet's --filter globs
`Namespace.Class.Method` (categories are not matched), so with the class still named ...DictBenchmarks the menu
entry matched ZERO benchmarks. Verified: `--list flat` now lists all 43.
REMOVED
- ConcurrentPointingDict<TKey,TValue> (src/NumSharp.Core/Collections/Concurrent/ConcurrentPointingDict.cs,
1,030 lines). Experimental "hash -> int32 index -> value list" variant on the vendored ConcurrentDictionary; it had
no tests, no benchmark rows and no in-tree consumer, and its own /optimize run already recommended not shipping it.
It also carried two concurrency defects reproduced before removal (isolated worktree, x64): its validated read
checks keys[idx] BEFORE loading values[idx], so an in-place TryRemoveSwapBack let a reader of the removed key
return the moved entry's value (31,268 wrong-value hits / 6 s with the default comparer; COD/COCD 0) - the exact
chained-layout tear ConcurrentOrderedDictionary.COMPACT.md section 7 warns about - and its "lock-free" read spun
until an interior removal finished re-indexing (4.1-4.5 ms GC-free stall at N=1M vs ~0.35 ms for COD/COCD).
Nothing else referenced it; docs/MA_MODULE_AUDIT.md keeps its dated pass-11 mention (a historical log).
DOC / CREF FIXES
- The IDE rename double-applied inside crefs: 22 `<see cref="ConcurrentOrderedDictionaryionary{TKey,TValue}"/>` /
`ConcurrentOrderedCompactDictionaryionary` (src 4, benchmark 1, tests 17). All collapsed to the real names. The
csproj NoWarn suppresses the whole CS1574 family, so the compiler never flags a broken cref - docfx is the check.
- Crefs docfx cannot bind on net10.0, where the BCL signatures are `ref readonly`:
* `<see cref="Volatile.Read(ref int)"/>` x3 (OrderedDictionary class doc - rendered as unlinked plain text on the
public API page - plus COD/COCD private count docs) -> `<c>Volatile.Read</c>` (the paragraphs already link the
Volatile class; a `ref readonly` cref would break the net8.0 doc build instead).
* `<see cref="Unsafe.IsNullRef{T}(ref T)"/>` -> `(ref readonly T)` on GetValueRefOrNullRef - newly visible because
the method is now public, so docfx renders it.
- Ledgers (ConcurrentOrderedDictionary.TODO.md / .COMPACT.md): type, file, test-suite, benchmark-class and
menu-filter names; the seam's new file name and public signature.
- docs/proposals/ConcurrentOrderedDictionary.md: reference implementation is now
`NumSharp.Collections.OrderedDictionary<TKey, TValue>`; added a naming note because the rename makes three names
collide in one document - the reference impl shares its simple name with the BCL
`System.Collections.Generic.OrderedDictionary<TKey, TValue>` the proposal compares against, and NumSharp's own
node-based `NumSharp.Collections.ConcurrentOrderedDictionary<TKey, TValue>` shares the PROPOSED type's name while
NOT being the reference implementation. Section 8's comparator is now fully qualified.
- benchmark/collections/probes/*.cs (4 `dotnet run` reproduction scripts the ledgers cite; outside every build, so
nothing else would have caught them): type names updated - they no longer compiled against the renamed types.
NAMING CAVEAT (not changed, flagged): NumSharp.Collections.OrderedDictionary<TKey,TValue> now shares its simple name
with System.Collections.Generic.OrderedDictionary<TKey,TValue> (.NET 9+). A net10.0 consumer importing both
namespaces gets CS0104 and must alias/qualify (net8.0 has no generic BCL OrderedDictionary). Same situation as the
pre-existing public NumSharp.Collections.Concurrent.ConcurrentDictionary. Nothing in-tree is affected.
VERIFICATION (isolated `git worktree` = HEAD + exactly the paths in this commit; the shared tree holds a parallel
session's unrelated pending edits)
- Build: NumSharp.Core + NumSharp.Tests (net8.0 + net10.0) + NumSharp.Benchmark.CSharp - 0 errors. The benchmark
project needed that parallel session's still-uncommitted BenchmarkBase.cs fix to build at all: at HEAD it has
CS0160 (`catch (IncorrectTypeException)` after `catch (TypeError)`, and IncorrectTypeException : TypeError). That
fix is NOT part of this commit.
- Tests: `FullyQualifiedName~ConcurrentOrdered` -> 167/167 passed on net10.0 AND net8.0 (Release).
- BenchmarkDotNet: `--filter "*ConcurrentOrderedDictionary*" --list flat` -> 43 benchmarks (= every method).
- docfx metadata: 0 warnings for any collection file (repo total 423, all pre-existing elsewhere); the API
reference regenerates 11 collection pages under the new names.
- Probes: `dotnet build -c Release <probe>.cs` -> all 4 compile.
- Leftover sweep: no ConcurrentOrderedDict / ConcurrentOrderedCompactDict / OrderedDict / ConcurrentPointingDict /
ConcurrentDictionary.NumSharp / "Dictionaryionary" in tracked files outside the dated MA audit log and the
gitignored optimize-runs/.
NOT IN THIS COMMIT: the parallel session's pending work in the shared tree (Ma MaskedArray -> NDMaskedArray rename,
Default.* math / ArrayFormatter / Oracle fuzz / interop test edits, BenchmarkBase.cs, refs/data).
…n list, probe self-name, blame-ignore the rename A second, independent verification pass over 5635182 (the *Dict -> *Dictionary rename), running checks the first pass did not, plus the three small fixes it surfaced. VERIFIED, NO CHANGE NEEDED - Mirror-suite integrity. For each of the 5 COD/COCD test-suite pairs, substituting ConcurrentOrderedDictionary -> ConcurrentOrderedCompactDictionary in the COD file and diffing against its COCD mirror leaves exactly the documented residual deltas (the MIRROR header, cod-/cocd- thread names, the memory-doc pointer TODO -> COMPACT, the `== 0` vs `<= 72` presized-append bound, the AddRange comment variant) - and those residuals are IDENTICAL (name-normalized) before and after the rename, so the IDE rename kept both suites in lockstep. - Compiler-level cref validation, stronger than docfx (every member incl. private, BOTH TFMs): rebuilt NumSharp.Tests (pulls Core) in a clean HEAD worktree with `-p:NoWarn=NONE -p:GenerateDocumentationFile=true`. The override surfaces 782 CS1574 repo-wide, and ZERO XML-doc diagnostics (CS1574/1580/1581/1584/0419/1570-1573/1711/ 1712/1734) in any src or test collection file - their only warnings are 668 CS1591 (undocumented test members). - Cross-reference integrity: every file path / file name cited by the 26 files of 5635182 resolves to a tracked file (the only non-resolving tokens are upstream dotnet/runtime paths) - except one pre-existing stale self-reference, fixed below. - Broader leftover sweep (`Dict(?!ionary)` in scope; case-insensitive `ordered[ _-]?dict` repo-wide): only BCL Dictionary shorthand ("COD vs Dict", the DictC adapter), Python-dict semantics (AddRange_UpsertsWithPythonDictSemantics) and generic "ordered dict" prose remain - none name these types. - Rename-collateral audit (the Rider text-occurrence hazard): every changed string literal is the type naming itself (reentrancy messages, nameof, BDN category, menu text/filter); the rename touched nothing outside the collection scope (no Python OrderedDict exists in any tracked .py). FIXED - ConcurrentDictionary.cs header: its "deviations from the upstream source" list said the public key-API is otherwise unchanged, but the clone now carries a PUBLIC NumSharp-only member (GetValueRefOrNullRef in ConcurrentDictionary.RefAccessors.cs, made public in 5635182). Added the `partial` + public-accessor bullet and scoped the closing "unchanged" line to the upstream members. - benchmark/collections/probes/compact_ordered_dict_complexity.cs: its title comment named the file "complexity_probe.cs" (stale since it was committed under its real name). - .git-blame-ignore-revs: records 5635182, per the file's convention for mechanical renames (test-project rename, Npy* -> ND*, src -> refs relocation). Verified: a renamed test line (ConcurrentOrderedDictionaryConcurrencyTests.cs:110) blames to 5635182 without the file and back to its origin 1fcaea0 with it. The entry states the caveat measured while verifying: git also re-attributes the commit's few genuinely NEW doc lines to neighbouring older lines (the proposal's naming note blames to e3d5b77 with the file on); `git blame --no-ignore-revs-file` shows their true origin. Every earlier entry's SHA is an ancestor of origin/master (branches merge, not squash), and an unknown SHA in the file is harmless (probed: blame still exits 0). BUILD: NumSharp.Core (net8.0 + net10.0) and the edited probe (`dotnet build -c Release <probe>.cs`) compile in a clean HEAD worktree. LEFT AS IS, DELIBERATELY: ledger line widths (already 19/43 lines over 120 chars - tables and repo paths; the rename added one each), generic "ordered dict" prose, and the dated history entries in docs/MA_MODULE_AUDIT.md.
… bool-mask crash, leaks, host-pinned tests PR #631 (journey4) had never run on Linux/macOS CI. Run 35818966511 failed in five jobs, and CI stops at the first red step, so later steps hid more. Every job was replicated locally to find all of it: Windows directly, Linux in WSL Ubuntu 24.04 (every step of the `test` job plus the docs job's benchmark smoke), and ARM64 via DOTNET_EnableSSE41=0 (Vector128 stays accelerated, Sse41.IsSupported turns false, the same shape the kernel emitter sees on Apple silicon). Six root causes, two of them real Core bugs. 1. CORE BUG: np.argsort wrote past a zero-byte buffer (native heap corruption) AxisSort.ArgSortInto guarded only an empty AXIS (N == 0), not an empty ARRAY. For (0,3) along the last axis (or (2,0,3)/-1) there are zero lines, but the all-but-axis NDIter still makes one kernel call, and the radix line kernels ignore the per-call count: each call writes a full line of N int64 indices. Those 24 bytes landed in dst's ZERO-byte buffer. glibc never noticed. Windows' heap manager killed the test host with 0xC0000374 (STATUS_HEAP_CORRUPTION) in whatever test ran next, which aborted the whole Windows Oracle run inside IndexOracleTests.Index_Curated/Index_Dtype. Bisected 88 FuzzCorpusTests tiers -> MaSortSetops -> the single case ma.sort/empty_2d/bool.some/748 -> np.argsort (np.ma.sort = argsort of the filled keys + take_along_axis), then isolated in a standalone repro: every dtype, not just bool, at any zero-lines shape with a length >= 2 sort axis. This is the "OPEN pre-existing Ma-tier testhost crash" noted with 2ea7ba6. SortInPlace and AxisPartition (partition/argpartition) already carried the size guard; argsort was the one sibling without it. Fix: `if (N == 0 || src.size == 0) return;` in ArgSortInto, plus a defensive `ops[0].size == 0` early-out in DriveAllButAxis so a future caller cannot repeat it. Gate: Sorting/ArgsortZeroLinesTests (all 15 dtypes, covering every argsort kernel family, x 6 empty shapes/axes, axis=None, the ma.sort corpus case) with native-heap churn after each call. Verified to have teeth: the unfixed build crashes the test host; the fixed build passes 3/3 in 0.6 s. 2. CORE BUG: np.evaluate threw PlatformNotSupportedException on ARM64 (macOS) The fused NDExpr shell (vector v2) widens a STREAMED bool operand into a Vector<W> lane mask with DirectILKernelGenerator.EmitInlineMaskCreation, whose 2/4/8-byte path calls the x86-only Sse41/Avx2 ConvertToVector*Int*. The plan/shell gated bool operands on vector WIDTH only (FusedBoolLanesAvailable: 128 or 256 bits), and ARM64 has 128-bit AdvSimd Vector128 with every x86 ISA unsupported. The DynamicMethod compiles fine and throws only when it runs, so every tree mixing a bool array with a 2/4/8-byte lane (where(m1,a,b), a*m1, a+m1, where(m1&(a>b),a,-b)) threw on the macos-latest runner (30 cases across 7 NDEvaluateVectorTests). np.where / np.select / np.piecewise gate the same helper with their own needsX86 checks; only the NDExpr shell lacked one. Fix: new DirectILKernelGenerator.InlineMaskCreationSupported(simdBits, elementSize) (the helper's real ISA contract) + FusedBoolInputMasksAvailable, applied in both NDExprVectorPlan.TryPlan and FusedSimdViable. Such trees take the scalar shell on ARM64, which is the byte-for-byte oracle the vector path must reproduce, so results are unchanged. The gate is precise: a bool OUTPUT is packed portably and stays vectorized, and so does a bool 0-d PARAMETER (a constant Zero/AllBitsSet mask, no x86). TryPlan now receives isParam so it can tell the two apart. x86 behaviour is unchanged (SSE4.1/AVX2 present). Tests: VectorPlan_DecidesLaneAndScalarFallback now states the plan against the host capability, plus param/output cases and an InlineMaskCreationSupported truth table. Verified: with DOTNET_EnableSSE41=0 the 30 failures reproduce before the fix and all NDEvaluate tests pass after it. 3. Undisposed intermediates (Oracle Corpus_AllOps_LeaveNoUndisposedIntermediates; 38 families / 54 cases, all on this branch) - np.logspace / np.geomspace: CastSpacingResult dropped the float64 compute array after astype -> exactly 1 escaped buffer per non-float64 call. It now disposes the replaced array (ReferenceEquals-guarded). Ownership transfer is documented on the parameter. - np.piecewise (1-3 escaped per call): the bool-condition aliases, ToBoolCondition's astype temp, ComputeElseCondition's partial ORs and its logical_not result, the coerced 0-d scalar constants (composition path), and the fused path's scalar + astype pair were all dropped live. PiecewiseCore now owns and releases them in a finally, and also releases the half-written result on a throw. The bare-array and scalar-bool overloads release the condition arrays they build. Deliberately NOT [NDScoped]: an ambient scope would reclaim arrays a user callback allocates or keeps (the reason apply_along_axis stays unscoped too), so disposal is explicit and never touches caller-owned or user-callback arrays. asanyarray can hand back an EXISTING array for NDArray input, so the scalar-constant dispose is safe only because that case returns earlier. That is documented at the site. 4. Stale tests / benchmark gates behind the earlier failures - benchmark CS0160 (Deploy Docs build): IncorrectTypeException now derives from TypeError (3ad4360), so `catch (IncorrectTypeException)` after `catch (TypeError)` was unreachable. Reordered derived-before-base and kept it explicit so the benchmark survives a future re-parenting. - ModfBenchmarks (next failure in the same job once it compiled; benchmark smoke: "expected to be rejected, but the operation succeeded"): np.modf accepts every REAL dtype since 59f9932 (bool/int -> narrowest float per width, float16 kernel); measured 14/15 accept, only Complex is rejected, the same as NumPy 2.4.2 (per-dtype probe on both sides, result dtypes identical). The C# gate now rejects only Complex, and the NumPy twin (numpy_benchmark.py) times every non-complex dtype so the new C# rows have peers. The old comment explaining the float32/float64-only restriction was describing a C# rejection that no longer exists. - MisalignedRegistryTightnessTests B2/B3 asserted that two excuses STILL existed, but 074d4d0 (complex multiply now bit-exact) and 7fd70a3 (cumprod size<=1 widening) removed them on purpose. Inverted to *_NoLongerExcused so they pin the tight contract. 5. Tests that need an optional dependency CI does not provide - 25 Examples tests (Gist/Karpathy demos + polyfit companions) called OpenBlasEngine.Enable unconditionally, and CI's test job stages no native binary, so they threw DllNotFoundException on all 3 OSes. New Examples/ExampleBlasBackend follows the house LapackEigTests.RequireLapack pattern (Inconclusive where no CBLAS/LAPACK loads). KarpathyLstmTests binds in [TestInitialize] without failing: its 4 semantic/tolerance tests now run everywhere (verified passing with no backend), and only the 2 exact-value tests require the backend. With the binary staged, all 104 example tests still pass (0 skipped). - 9 interop tests import SciPy, which the interop job does not install ("No module named 'scipy'"). They now call the existing SkipUnless("scipy") as their FIRST statement: their mid-test failures had leaked export pins, which cascaded into NumpyNet_BootsOnTheSharedEngine / Lifetime_TheirWrapperIsTheOnlyHolder (LiveExports 9 != 0). 6. Host-dependent exact values (the tests, not the product) - np.emath arctanh(0.5) (glibc) and arccos(0.5)/arccos(0.3) (Apple libm), plus np.sinc(0.25) (both), differ from the win-amd64 ucrtbase bits by 1 ULP. The matching corpus tiers are already host-pinned (RunHostLibmCorpus: Inconclusive off-Windows). New Utilities/HostLibm gives unit tests the same pin but keeps checking: bit-exact on Windows, <= 4 ULP elsewhere, with NaN-ness, infinities and the sign bit (incl. signed zero) always strict. - 4 interop live-parity tests (NES final weights/trace, forecast mpe, ranking dcg) measured 1 ULP off live NumPy on macos-latest arm64 only. Their authors deliberately refuse to widen per-host budgets ("Other hosts deliberately expose a disagreement instead of silently widening the gate"), so they use the existing SkipByteExactOnArm64 x64 pin (the NEON 128-bit reduction-width class) with the measured drift in their docs. Verification (all local; macOS arm64 itself is only reachable through CI). Other sessions landed 2ea7ba6, 5635182 and 4499d3e while this was in progress (none touch these files), so each figure names the base it ran on: - Windows (CI filter): NumSharp.Tests net8.0 16022/0 (b779d95) and net10.0 16024/0 (2ea7ba6); Oracle net8.0/net10.0 182/0 without OpenBLAS (2ea7ba6; was: test-host crash) and 206/206 with it staged (4499d3e); Interop net8.0 733/733 with SciPy and 724 passed / 9 skipped / 0 failed with SciPy hidden, the CI path (2ea7ba6). At 4499d3e all four affected projects build and the 220 targeted tests pass. - Linux (WSL Ubuntu 24.04, CI test-job replica, every step run even after a failure, b779d95): NumSharp.Tests net8.0 15958/0, net10.0 15959/0; Oracle net8/net10 167/0; Analyzer net8/net10 321/0 (baseline before the fix: 28 + 28 + 3 + 3 failures). The docs job's benchmark body smoke runs all 499 bodies OK on net8.0, and the unary twin join check is OK. - ARM64 simulation (DOTNET_EnableSSE41=0): NDEvaluate 30 failures -> 0. Not changed here (noted for follow-up): benchmark/scenarios.html is a generated artifact that already disagrees with its own (non-CI) test_scenarios_audit.py totals (python 1296 vs 1699); regenerate it with benchmark/scripts/generate_scenarios_html.py. NumpyNet lifetime tests assert ABSOLUTE LiveExports counts, so any interop test that leaks on failure cascades into them; comparing deltas would localise such failures.
Nucs
added a commit
that referenced
this pull request
Sep 23, 2026
… bool-mask crash, leaks, host-pinned tests PR #631 (journey4) had never run on Linux/macOS CI. Run 35818966511 failed in five jobs, and CI stops at the first red step, so later steps hid more. Every job was replicated locally to find all of it: Windows directly, Linux in WSL Ubuntu 24.04 (every step of the `test` job plus the docs job's benchmark smoke), and ARM64 via DOTNET_EnableSSE41=0 (Vector128 stays accelerated, Sse41.IsSupported turns false, the same shape the kernel emitter sees on Apple silicon). Six root causes, two of them real Core bugs. 1. CORE BUG: np.argsort wrote past a zero-byte buffer (native heap corruption) AxisSort.ArgSortInto guarded only an empty AXIS (N == 0), not an empty ARRAY. For (0,3) along the last axis (or (2,0,3)/-1) there are zero lines, but the all-but-axis NDIter still makes one kernel call, and the radix line kernels ignore the per-call count: each call writes a full line of N int64 indices. Those 24 bytes landed in dst's ZERO-byte buffer. glibc never noticed. Windows' heap manager killed the test host with 0xC0000374 (STATUS_HEAP_CORRUPTION) in whatever test ran next, which aborted the whole Windows Oracle run inside IndexOracleTests.Index_Curated/Index_Dtype. Bisected 88 FuzzCorpusTests tiers -> MaSortSetops -> the single case ma.sort/empty_2d/bool.some/748 -> np.argsort (np.ma.sort = argsort of the filled keys + take_along_axis), then isolated in a standalone repro: every dtype, not just bool, at any zero-lines shape with a length >= 2 sort axis. This is the "OPEN pre-existing Ma-tier testhost crash" noted with 2ea7ba6. SortInPlace and AxisPartition (partition/argpartition) already carried the size guard; argsort was the one sibling without it. Fix: `if (N == 0 || src.size == 0) return;` in ArgSortInto, plus a defensive `ops[0].size == 0` early-out in DriveAllButAxis so a future caller cannot repeat it. Gate: Sorting/ArgsortZeroLinesTests (all 15 dtypes, covering every argsort kernel family, x 6 empty shapes/axes, axis=None, the ma.sort corpus case) with native-heap churn after each call. Verified to have teeth: the unfixed build crashes the test host; the fixed build passes 3/3 in 0.6 s. 2. CORE BUG: np.evaluate threw PlatformNotSupportedException on ARM64 (macOS) The fused NDExpr shell (vector v2) widens a STREAMED bool operand into a Vector<W> lane mask with DirectILKernelGenerator.EmitInlineMaskCreation, whose 2/4/8-byte path calls the x86-only Sse41/Avx2 ConvertToVector*Int*. The plan/shell gated bool operands on vector WIDTH only (FusedBoolLanesAvailable: 128 or 256 bits), and ARM64 has 128-bit AdvSimd Vector128 with every x86 ISA unsupported. The DynamicMethod compiles fine and throws only when it runs, so every tree mixing a bool array with a 2/4/8-byte lane (where(m1,a,b), a*m1, a+m1, where(m1&(a>b),a,-b)) threw on the macos-latest runner (30 cases across 7 NDEvaluateVectorTests). np.where / np.select / np.piecewise gate the same helper with their own needsX86 checks; only the NDExpr shell lacked one. Fix: new DirectILKernelGenerator.InlineMaskCreationSupported(simdBits, elementSize) (the helper's real ISA contract) + FusedBoolInputMasksAvailable, applied in both NDExprVectorPlan.TryPlan and FusedSimdViable. Such trees take the scalar shell on ARM64, which is the byte-for-byte oracle the vector path must reproduce, so results are unchanged. The gate is precise: a bool OUTPUT is packed portably and stays vectorized, and so does a bool 0-d PARAMETER (a constant Zero/AllBitsSet mask, no x86). TryPlan now receives isParam so it can tell the two apart. x86 behaviour is unchanged (SSE4.1/AVX2 present). Tests: VectorPlan_DecidesLaneAndScalarFallback now states the plan against the host capability, plus param/output cases and an InlineMaskCreationSupported truth table. Verified: with DOTNET_EnableSSE41=0 the 30 failures reproduce before the fix and all NDEvaluate tests pass after it. 3. Undisposed intermediates (Oracle Corpus_AllOps_LeaveNoUndisposedIntermediates; 38 families / 54 cases, all on this branch) - np.logspace / np.geomspace: CastSpacingResult dropped the float64 compute array after astype -> exactly 1 escaped buffer per non-float64 call. It now disposes the replaced array (ReferenceEquals-guarded). Ownership transfer is documented on the parameter. - np.piecewise (1-3 escaped per call): the bool-condition aliases, ToBoolCondition's astype temp, ComputeElseCondition's partial ORs and its logical_not result, the coerced 0-d scalar constants (composition path), and the fused path's scalar + astype pair were all dropped live. PiecewiseCore now owns and releases them in a finally, and also releases the half-written result on a throw. The bare-array and scalar-bool overloads release the condition arrays they build. Deliberately NOT [NDScoped]: an ambient scope would reclaim arrays a user callback allocates or keeps (the reason apply_along_axis stays unscoped too), so disposal is explicit and never touches caller-owned or user-callback arrays. asanyarray can hand back an EXISTING array for NDArray input, so the scalar-constant dispose is safe only because that case returns earlier. That is documented at the site. 4. Stale tests / benchmark gates behind the earlier failures - benchmark CS0160 (Deploy Docs build): IncorrectTypeException now derives from TypeError (3ad4360), so `catch (IncorrectTypeException)` after `catch (TypeError)` was unreachable. Reordered derived-before-base and kept it explicit so the benchmark survives a future re-parenting. - ModfBenchmarks (next failure in the same job once it compiled; benchmark smoke: "expected to be rejected, but the operation succeeded"): np.modf accepts every REAL dtype since 59f9932 (bool/int -> narrowest float per width, float16 kernel); measured 14/15 accept, only Complex is rejected, the same as NumPy 2.4.2 (per-dtype probe on both sides, result dtypes identical). The C# gate now rejects only Complex, and the NumPy twin (numpy_benchmark.py) times every non-complex dtype so the new C# rows have peers. The old comment explaining the float32/float64-only restriction was describing a C# rejection that no longer exists. - MisalignedRegistryTightnessTests B2/B3 asserted that two excuses STILL existed, but 074d4d0 (complex multiply now bit-exact) and 7fd70a3 (cumprod size<=1 widening) removed them on purpose. Inverted to *_NoLongerExcused so they pin the tight contract. 5. Tests that need an optional dependency CI does not provide - 25 Examples tests (Gist/Karpathy demos + polyfit companions) called OpenBlasEngine.Enable unconditionally, and CI's test job stages no native binary, so they threw DllNotFoundException on all 3 OSes. New Examples/ExampleBlasBackend follows the house LapackEigTests.RequireLapack pattern (Inconclusive where no CBLAS/LAPACK loads). KarpathyLstmTests binds in [TestInitialize] without failing: its 4 semantic/tolerance tests now run everywhere (verified passing with no backend), and only the 2 exact-value tests require the backend. With the binary staged, all 104 example tests still pass (0 skipped). - 9 interop tests import SciPy, which the interop job does not install ("No module named 'scipy'"). They now call the existing SkipUnless("scipy") as their FIRST statement: their mid-test failures had leaked export pins, which cascaded into NumpyNet_BootsOnTheSharedEngine / Lifetime_TheirWrapperIsTheOnlyHolder (LiveExports 9 != 0). 6. Host-dependent exact values (the tests, not the product) - np.emath arctanh(0.5) (glibc) and arccos(0.5)/arccos(0.3) (Apple libm), plus np.sinc(0.25) (both), differ from the win-amd64 ucrtbase bits by 1 ULP. The matching corpus tiers are already host-pinned (RunHostLibmCorpus: Inconclusive off-Windows). New Utilities/HostLibm gives unit tests the same pin but keeps checking: bit-exact on Windows, <= 4 ULP elsewhere, with NaN-ness, infinities and the sign bit (incl. signed zero) always strict. - 4 interop live-parity tests (NES final weights/trace, forecast mpe, ranking dcg) measured 1 ULP off live NumPy on macos-latest arm64 only. Their authors deliberately refuse to widen per-host budgets ("Other hosts deliberately expose a disagreement instead of silently widening the gate"), so they use the existing SkipByteExactOnArm64 x64 pin (the NEON 128-bit reduction-width class) with the measured drift in their docs. Verification (all local; macOS arm64 itself is only reachable through CI). Other sessions landed 2ea7ba6, 5635182 and 4499d3e while this was in progress (none touch these files), so each figure names the base it ran on: - Windows (CI filter): NumSharp.Tests net8.0 16022/0 (b779d95) and net10.0 16024/0 (2ea7ba6); Oracle net8.0/net10.0 182/0 without OpenBLAS (2ea7ba6; was: test-host crash) and 206/206 with it staged (4499d3e); Interop net8.0 733/733 with SciPy and 724 passed / 9 skipped / 0 failed with SciPy hidden, the CI path (2ea7ba6). At 4499d3e all four affected projects build and the 220 targeted tests pass. - Linux (WSL Ubuntu 24.04, CI test-job replica, every step run even after a failure, b779d95): NumSharp.Tests net8.0 15958/0, net10.0 15959/0; Oracle net8/net10 167/0; Analyzer net8/net10 321/0 (baseline before the fix: 28 + 28 + 3 + 3 failures). The docs job's benchmark body smoke runs all 499 bodies OK on net8.0, and the unary twin join check is OK. - ARM64 simulation (DOTNET_EnableSSE41=0): NDEvaluate 30 failures -> 0. Not changed here (noted for follow-up): benchmark/scenarios.html is a generated artifact that already disagrees with its own (non-CI) test_scenarios_audit.py totals (python 1296 vs 1699); regenerate it with benchmark/scripts/generate_scenarios_html.py. NumpyNet lifetime tests assert ABSOLUTE LiveExports counts, so any interop test that leaks on failure cascades into them; comparing deltas would localise such failures.
…ntOrderedDictionary ([OpenBugs]), pin the compact sibling's comparer exception safety Two new test classes. Every [OpenBugs] test asserts the CORRECT outcome promised by the type's own docs and by docs/proposals/ConcurrentOrderedDictionary.md, and fails today (CI excludes the category until each fix lands; remove the attribute together with the fix). Five passing controls pin the behaviour the compact sibling and OrderedDictionary already get right. Tests only - no product code changed. == test/NumSharp.Tests/Collections/OrderedDictionaryOpenBugsTests.cs (6 tests, all [OpenBugs]) == 1. Enumerate_WhileAWriterRegrowsTheTable_NeverThrowsAndNeverYieldsAnUnwrittenSlot GetEnumerator() = new Enumerator(_t._values, Volatile.Read(ref _t._count)) reads the volatile generation field TWICE: a growth published between the reads pairs the old (shorter) values array with the new count -> IndexOutOfRangeException or phantom default values. Violates "snapshot enumeration never throws / consistent prefix" (proposal 7.1). Fix: read `_t` once into a local. Storm (writer Clear+refill 64 keys, readers foreach); fails in ~3 ms; ~160K IOOR/s measured on the shipped code. 2. WideValueReplaceThenAppend_ReaderHoldingTheOlderGeneration_NeverReadsAnUnwrittenValue (DETERMINISTIC) A replace of a non-atomically-writable TValue (decimal/Guid/wide struct/long on 32-bit) clones ONLY the values array; the new generation SHARES _index/_keys with the old one, so the next in-place append writes a word + key into those shared arrays and a reader still holding the old generation resolves the new key and returns its own never-written values[slot] = default for a present key. Exactly the aliasing rule ConcurrentOrderedDictionary.COMPACT.md 4.7 forbids (the compact sibling copies the whole generation). Fix: full generation copy-on-write. Made deterministic with a comparer hook: TryGetValue captures its generation and only THEN calls GetHashCode, so the hook runs the writer (wide replace + append, on another thread) inside that window. Result today: found=True, value=0 for a key whose value is 150.25 (~47K/s with a trivial comparer). 3. LockFreeReplace_RacingCopyingResizes_NeverLosesACompletedWrite The lock-free SetByKey does a plain value store then volatile-reads _resizing/_t; a resize volatile-writes _resizing = true then copies the values array. That is the store-buffering (Dekker) litmus: release/acquire cannot forbid it on x86-TSO or ARM64, so the resize can copy the OLD value while the replacer sees the flag clear -> the completed write is lost when the resize publishes. Violates "no update is ever lost" (proposal 7.1). Fix: Interlocked.MemoryBarrier() after the store AND after every `_resizing = true` - verified in a scratch A/B: 3.26% lost -> 0 (0.84% -> 0 with one slot per cache line), +3.3 ns per replace (4.9 -> 8.2 ns), still ~5x a locked replace. The test uses one WRITER PER KEY with read-your-own-write re-reads: a storm with several writers per key (the original ~280M-op gun) cannot see this - last-writer-wins hides the lost intermediate write. 4. NullKey_IsRejectedOnEveryKeyEntryPoint_LikeTheSiblingsAndTheBcl No null check anywhere; EqualityComparer<string>.Default.GetHashCode(null) is 0, so a null key is stored, found and enumerated. All 11 key entry points accept it (COD/COCD/Dictionary/ConcurrentDictionary throw ArgumentNullException). Fix: the siblings' NullCheck (with the typeof(TKey).IsValueType Debug-boxing guard). 5. HugeCapacity_IsRefusedWithArgumentOutOfRange_InsteadOfSpinningForever `while (len < cap * 100 / 70 + 1) len <<= 1;` on an int: from capacity ~752M the shift wraps to int.MinValue then 0 and `0 << 1 == 0` spins forever (AppendGrow's `newLen <<= 1` has the same overflow). The compact sibling throws ArgumentOutOfRangeException above ~715M. Runs the ctor on a lowest-priority background thread with a 10 s join (a spinning managed thread cannot be stopped; no memory is allocated before the spin). 6. ComparerThrowDuringAResize_DoesNotLeaveTheReplacePermanentlyLocked Every generation-replacing resize sets _resizing = true, then allocates and rebuilds the index through the comparer, then clears the flag - with no try/finally. A comparer/OOM throw in that window leaves the collection consistent (nothing published) but _resizing stuck true forever, so every later "lock-free" replace fails its re-check and silently takes the write lock. Observable: SetByKey on an existing key no longer completes while another thread holds the lock (an AddRange whose source blocks inside the locked foreach). A healthy-instance baseline check in the same test proves the harness. Fix: clear the flag in a finally. == test/NumSharp.Tests/Collections/OrderedCollectionsComparerExceptionSafetyTests.cs == ConcurrentOrderedDictionary mutates its key map FIRST and then re-indexes the shifted/moved keys THROUGH the map (one comparer call per key) BEFORE publishing the new list-path store, in: the order-preserving interior removal (RemoveCoreUnderLock), both swap-back branches (in place for word-sized types, copying for wide ones) and RemoveWhere (whose finally re-runs the throwing re-index, so its "publish the consistent prefix" never happens). A comparer that throws there - including the collection's OWN LockRecursionException raised by the documented "evil comparer write-back" defence, which the docs say is "refused ... instead of being allowed to corrupt" - aborts the operation half-applied. 4 x [OpenBugs], deterministic, single-threaded: - interior TryRemove(3), poison key 7: key 3 still enumerates but ContainsKey(3) is false; IndexOf(4..6) wrong - in-place TryRemoveSwapBack(3), poison 9 (the moved key): key 9 enumerates TWICE - copying TryRemoveSwapBack(3) on <int,decimal>: key 3 enumerates but no longer resolves - RemoveWhere(even), poison 7: keys 0/2/4/6 enumerate but no longer resolve (15 disagreements) Fix: resolve every node ref to be re-indexed BEFORE the first map mutation, then mutate throw-free (the same "allocate before touching the map" discipline the type already applies to allocations). The existing Reentrancy_EvilComparerWritingDuringALockedLookup test only covers the refusal on the FIRST lookup (before any mutation), which is why this was not caught. 5 passing controls (run in CI, regression guards): ConcurrentOrderedCompactDictionary never consults the comparer after its probe (tags travel with the index words) and probes BOTH swap-back keys before mutating (in-place and copying branches), so the same attacks either never fire or change nothing; OrderedDictionary rebuilds a private generation and publishes last, so its published state survives the throw (its stuck-flag defect is test 6 above). The audit is generic over the value type: count/keys/values agree, no key enumerates twice, IndexOf/TryGetValue/ ContainsKey agree with every enumerated position, the value law holds, and no key resolves without enumerating. == Notes == - Both files put `using NumSharp.Collections;` INSIDE the namespace block: on net10.0 a file-level using next to `using System.Collections.Generic;` makes `OrderedDictionary<,>` ambiguous with the BCL's .NET 9 generic OrderedDictionary (CS0104 - verified with a compile check). The using attached to the namespace declaration is consulted first. Any net9+ consumer with ImplicitUsings that imports NumSharp.Collections and names OrderedDictionary<,> hits the same error. - Storm tests stop at the first anomaly (a broken build fails in milliseconds; a fixed one runs its 2-3 s budget). They are statistical by nature: on a 1-2 core box the races may not reproduce within the budget. Verification (isolated worktree, Release): - HEAD b7d9093 + these two files: net10.0 -> 10 failed / 5 passed (exactly the [OpenBugs] set fails). - At 5635182 + these files: net10.0 and net8.0 each 10 failed / 5 passed, every failure message checked for the right reason; CI filter (FullyQualifiedName~NumSharp.Tests.Collections & TestCategory!=OpenBugs) -> 172/172 on net10.0 and net8.0 (167 existing + 5 new controls). - A single-threaded differential fuzz of all three types against a List-of-pairs oracle (765K ops over int/string keys, constant-hash / mod-8 / case-insensitive comparers, int/long/decimal/Guid/string values, growth, tail pops, interior removals, AddRange, Clear, every read surface) found NO functional divergence: the defects are concurrency, exception-safety and edge-contract ones. Related findings NOT covered by these tests (for reference): - ConcurrentOrderedDictionary.ReplaceExistingUnderLock, non-atomic TValue: the key-map node is swapped BEFORE the values-array Clone() allocation, so an OutOfMemoryException there leaves the key path holding a value the list path never publishes - contrary to the type's own strand-proofing rule (AddRange's wide-value branch does it in the right order). Inspection only (no deterministic in-process OOM). - docs/proposals/ConcurrentOrderedDictionary.md still says the reference implementation is "concurrency-verified" and "complete" and that 7.1 holds (no lost updates / no garbage reads / never-throwing enumeration); tests 1-3 contradict it, and the proposed surface (KeyValuePair indexer, IDictionary family, TryUpdate/AddOrUpdate) is a superset of what OrderedDictionary implements. - AddRange semantics differ inside the family: OrderedDictionary = add-if-absent (first value wins), ConcurrentOrderedDictionary / ConcurrentOrderedCompactDictionary = upsert (last value wins). Both documented. - OrderedDictionary accepts a negative capacity (documented: <= 0 -> default); both siblings and the BCL throw ArgumentOutOfRangeException. - OrderedDictionary.TryRemove is O(n) even for the LAST entry (no O(1) tail path, so a pop-back drain is quadratic) and both TryRemove and growth re-hash every key through the comparer (no stored hashes/tags) - the compact sibling pops the tail in O(1) and rebuilds from stored tags. - GetValueRefOrNullRef on the public vendored ConcurrentDictionary is now public (documented as intentional in 4499d3e); it bypasses the class's "all public members are thread-safe" guarantee by design.
…ed copy read values it had already written (251 of 1,500 overlap probes gave silent wrong values), slice assignment copied an overlapping source first where NumPy walks its raw order, masks and sources that do not broadcast were accepted, weak C# scalars wrapped (np.copyto(int8, 1000) gave -24), and the checks ran in another order with other texts; now bit-exact over a new 13,462-case copyto_full oracle tier with zero excuses, and the masked copy is a bit-copy kernel per element width (165 masked cells: geomean 12.4x NumPy, minimum 1.51x; numpy.ma's masked reductions 2.83x -> 4.60x) (plan api-parameter-parity X16 + X11) Plan docs/plans/api-parameter-parity.md: X16 (new, found by X11's probe) and X11 (the masked copy's speed), both done here; X17 and findings 21-24 recorded; A6 re-measured. What was wrong (NumPy 2.4.2 vs NumSharp at 04e0e70) ---------------------------------------------------- A probe of 86 np.copyto calls: 54 differed (the house exception types mapped to NumPy's first). - Silent wrong values. The masked path never checked overlap: np.copyto(a[1:], a[:-1], where=m) gave [0, 0, 2, 2, 2, 5, 5, 5, 8, 8], NumPy [0, 0, 2, 2, 3, 5, 5, 6, 8, 8]. An overlap probe (1,500 random views of one buffer, 1-D and 2-D, stepped / reversed / transposed, masked or not) differed in 251 cases: 112 masked 1-D, 135 masked 2-D, 4 unmasked 1-D. The unmasked ones are NDIter.Copy's overlap rule, which copied the source first for every non-unit-stride 1-D overlap where NumPy walks its raw order and reads its own writes when the strides differ: a[3:19:2] = a[6:14] gives NumPy [.., 12, 10, 14, 12, 16, 10, ..], NumSharp gave [.., 12, 11, 14, 12, 16, 13, ..]. Slice assignment runs the same path, so a[6:14] = a[3:19:2] and a[0:15:3] = a[0:5] were wrong too. - Accepted where NumPy refuses: a mask that does not broadcast ((1,1) into (3,): NumPy never strips a mask's axes; a (1,) mask into a 0-d destination), a source that does not broadcast under a false, 0-d false or (1,) false mask (NumPy checks the source first). - Refused where NumPy accepts: a source with leading unit axes beyond the destination's rank ((1,3) and (1,1,3) into (3,), (1,) and (1,1) into ()), a redundant self-copy of a read-only array (np.copyto(b, b)), a C# float into float32 under 'safe' and 'no', a C# int into int64 under 'equiv' and into float32 under 'safe' (all legal for NumPy's weak Python scalars). - Wrapped where NumPy raises: np.copyto(int8, 1000) gave -24, masked or not (NumPy: "Python integer 1000 out of bounds for int8"); 'equiv' accepted an int into int32 (NumPy: "cannot cast Python int to int32 under the casting rule 'equiv'"); -1 into uint8 was a cast error (NumPy: OverflowError). - Other errors and orders: "where must be a boolean array, got dtype('int64')" instead of NumPy's cast text; the iterator's "operands could not be broadcast together with remapped shapes ..." instead of "could not broadcast input array / where mask from shape (...) into shape (...)"; the casting-name errors as ArgumentException / ArgumentNullException; "Cannot cast array data" where NumPy says "Cannot cast scalar"; a read-only destination and a bad cast reported before a bad mask; a bad source shape reported as a cast error. - Speed (X11): the masked copy ran a branch per element (2.5-6 ns per element under a random mask) and a generic per-element ConvertValue loop for float16 / char / bool / complex / decimal; the per-pair masked cast kernel's SIMD blend for same-dtype rows was dead code. The port (src/NumSharp.Core/Manipulation/np.copyto.cs) ------------------------------------------------------ array_copyto -> PyArray_AssignArray / PyArray_AssignRawScalar, statement for statement: - Arguments: PyArray_CastingConverter first (ValueError "casting must be one of 'no', 'equiv', 'safe', 'same_kind', 'unsafe' (got '...')", TypeError "casting must be str, not NoneType"), then the mask's bool check (NumPy's "Cannot cast array data|scalar from dtype(...) to dtype('bool') according to the rule 'safe'", the house ArgumentException). - A C# integer / float / double / Complex binds the new copyto<T> and is NumPy's WEAK Python scalar: npy_mark_tmp_array_if_pyscalar, npy_find_descr_for_scalar (the common DType of _PyLongDType / _PyFloatDType / _PyComplexDType and the destination's class: the destination's dtype for a kind it holds, int64 for an int into bool, float64 for a float into an integer), npy_update_operand_for_scalar's 'equiv' TypeError, then PyArray_Pack's range checks (ReductionInitial.TryPack: never wrapped). A new BigInteger overload is a Python int of any size. bool / Half / char / decimal and every NDArray (0-d too) are STRONG: a 0-d int64 into int8 wraps, as NumPy's np.array(1000) does. - PyArray_AssignArray's order: a 0-d source -> AssignRawScalar; a redundant self-copy (same data, dtype, shape, strides) returns before the writeable check; "assignment destination is read-only"; the cast rule (InvalidCastException, NumPy's text); an overlapping source copied first unless the destination is 1-D with both runs in the same direction (NumPy's arrays_overlap bounds test); the source broadcast with its leading extent-1 axes beyond the destination's rank stripped; the 0-d mask (false copies nothing); the mask broadcast, never stripped. PyArray_AssignRawScalar: writeable, the cast ("Cannot cast scalar from ..."), the mask broadcast, the value cast ONCE. - The masked copy: RawArrayIter.PrepareThree (new, Backends/Iterators/RawArrayIter.cs: a port of PyArray_PrepareThreeRawArrayIter - axes sorted by the destination's |stride|, negative destination strides flipped, coalescing) and raw_array_wheremasked_assign_array's 1-D backward rule, then the new masked-assign kernel. - A source of another dtype, or an overlapping source NumPy copies first, is staged in the destination's memory order: one element per distinct source element (an axis the source does not move along keeps one), on the stack up to 16 KB ([SkipLocalsInit]), else a SizeBucketedBufferPool buffer, through the cast kernels NDIter.Copy uses (contiguous cast kernel, strided cast kernel, the per-run inner cast); a same-dtype temporary is a raw bit copy (a bool byte other than 0/1 stays what it is, as in NumPy). - With more than one prepared axis the operands cannot overlap (an overlapping multi-axis source was copied first), so the walk puts innermost the axis along which most operands are contiguous (NumPy's order wins ties; the bits are the same either way). np.CopytoNumPyOrderOnly is the A/B hook. The masked-assign kernel (src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.MaskedAssign.cs, new) --------------------------------------------------------------------------------------------------------------- dst[i] = src[i] wherever mask[i] != 0 over prepared operands: a BIT copy of the destination's element width, so one kernel per width (1 / 2 / 4 / 8 / 16 bytes) serves all 15 dtypes - no per-dtype code. - An IL DynamicMethod per width walks the outer axes (an odometer of incremental byte offsets) and calls a C# run stage per inner run (the house "IL kernel calls a tidy C# helper" pattern). - MaskedAssignRun<T> / MaskedAssignRun16: a mask stride of 0 decides the whole run (nothing, memmove, a Span fill for a broadcast source, or an element loop); a contiguous destination + mask runs a SIMD blend (ConditionalSelect between the source vector and the destination's own, the mask widened with AVX2 vpmovzx, two vectors per pass; a broadcast source hoisted into one vector; 16-byte elements two per vector with the mask byte duplicated onto both lanes); everything else and every tail is a branch-free select under an all-ones lane mask ((ulong)-(long)((b + 0xFF) >> 8): a random mask mispredicts a branch on a third of the elements), four elements per pass off running pointers. - Order: each element's store precedes the next element's loads, and the vector paths run only forward with the source at or above the destination, so an overlapping 1-D run is walked exactly in NumPy's order. - MaskedAssignFallback (the same walk in C#) for hosts without dynamic code; AssignRaw1D (NumPy's unmasked 1-D raw order: the flip, then backward when the source starts below the destination inside it) for NDIter.Copy. NDIter.Copy (src/NumSharp.Core/Backends/Iterators/NDIter.cs) ------------------------------------------------------------- The overlap rule is NumPy's for every assignment through it (np.copyto's unmasked copy, slice assignment): NeedsAssignmentTemp copies first only for a multi-axis destination or an opposite-direction 1-D one (dst.strides[0] * src.strides[-1] < 0); RawAssignOrderMatters + AssignRaw1DLikeNumPy walk every other overlapping 1-D same-dtype copy in NumPy's raw order (a unit-stride forward run stays a memmove; a cross-dtype overlap - two reinterpreting views - is still copied first, NumPy's cast loop order being undefined). Also ---- - nanmean / nanvar's eager zero fill (ZeroWhereMask, A3 / A7) is NumPy's own np.copyto(arr, 0, where=mask, casting='unsafe') again: 3.7 us over a float16 (250, 400) with every 7th element masked, where the np.putmask route A7 took while the masked copy cost 253 us takes 22.7 us (NumPy's copyto 124 us). - AuditV2 T1_18 now expects NumPy's mask text. Oracle tier copyto_full (test/oracle/gen_oracle.py copyto_full -> copyto_full.jsonl, 13,462 cases, floor 12,000) ------------------------------------------------------------------------------------------------------------------ Each case records the destination's WHOLE base buffer after the call (operand 0 carries its prior contents), so a write outside the view, into a masked-off slot or through an overlapping source is caught; NumPy's errors are cases (2,662, verbatim texts). Sections: M1 (13 dtypes x 1-D/2-D/3-D x destination C / F / strided / reversed / offset / transposed x 14 mask forms incl. the C# literals true / false, 2,626), M2 (source layouts and ranks incl. an extra leading unit axis and a read-only broadcast_to view, 1,729), M3 (131 and 4x37 elements: the SIMD pairs, single vectors and tails at every width, 819), X (16 cast pairs x layouts x masks x same_kind / unsafe, 2,688), O (1,100 overlapping views, masked and unmasked), S (C# literal sources as NumPy's Python scalars under every casting rule, 4,266), Z (0-d and empty destinations, strong 0-d sources, 184), E (43 error jobs: every check against every later one, so the ORDER is gated), R (redundant self-copies, 7). MisalignedRegistry returns null for the op (zero excuses). The generator drops a Python int at or past 2**31 that NumPy rejects with its platform C long text (NumSharp pins LP64, as the other Python-int tiers do) and complex64 texts (one complex width). The C# harness (OpRegistry.Copyto.cs) binds a scalar source to copyto<T> / the BigInteger overload by its spec kind, so the weak / strong split is the binding a port gets. coverage/oracle_map.json maps the new op key to numpy.copyto. Proof ----- - copyto_full 13,462 / 13,462; the overlap probe 1,500 / 1,500 (251 before); the call probe 82 / 86 (the four left: NumPy's where=1 scalar mask - finding 2, a 0-d non-bool mask always raises here; a Python list of ints with no C# spelling; the LP64 text for a Python int past int64; the probe's own complex print format). - Unit tests Manipulation/np.copyto.NumPyPort.Test.cs (21, NumPy-probed literals): 12 of the first 18 fail on 04e0e70; the last three were added for mutants only the tier caught. - Planted bugs (a detached worktree, one exact edit each, scratchpad mutants_x16.py): 21 - the redundant self-copy, the 2-D overlap temporary, the source's axis stripping, the 0-d mask, the lane mask, the 1-D backward rule, the staging's coalescing and stride rewrite, the 'equiv' check, the weak scalar's dtype, NDIter's temporary rule and raw order, the 1-D flip, the SIMD blend, the 16-byte pair mask, the unrolled select, a scalar source's mask, the overlap staging, the 0-d source route, the strided staging copy, a broadcast run's memmove. The tier kills 20, the unit tests all 21 (the 21st needs a source view whose axes overlap, np.ndarray with strides (8, 8), which the tier cannot build). - Full suites: NumSharp.Tests 18,138 (net10.0) / 18,137 (net8.0) passed, oracle 239 / 239 on both, no warning in the touched code at WarningLevel 4; the [OpenBugs] pass set is unchanged (24 / 105, the same names as 04e0e70). Speed (NPY/NS, two pinned P-cores, best of rounds) --------------------------------------------------- 192 cells: bool / int8 / uint16 / float16 / int32 / float32 / float64 / int64 / complex128 x 1K / 100K / 1M x a 0-d source, an array source, a row source, a row mask, a strided destination, an F destination, a float32 -> float64 cast (each under a 35 % random mask), and no mask. Geomean 9.46x. - The 165 masked cells: geomean 12.4x, minimum 1.51x (row-mask complex128 1M streams 32 MB, DRAM / L3-bound on both sides, 1.21-1.51x across runs). By form: 0-d source 38x, array 23x, row source 23x, row mask 10.6x, strided destination 4.0x, F destination 4.6x, cast 5.5x. - The unmasked copy is memcpy on both sides (np.copyto equals a bare Buffer.MemoryCopy of the same bytes): 1K 4.3-7x, 100K / 1M 0.73-1.18x run to run (finding 23). - Levers measured on the way: staging the cast source on the stack instead of an NDArray took a 1K masked float32 -> float64 copy from 0.78 to 0.32 us (NumPy 1.15 us; finding 22); running pointers in the strided select loop took a 1K strided-destination copy from 0.77 to 0.48 us (NumPy 1.15 us; finding 24). - X11's original cell, np.copyto(float16 (250, 400), 0, where=every 7th): 253 us -> 3.7 us (NumPy 124 us). - numpy.ma (A6's 310 cells, whose filled is this copy): geomean 2.83x -> 4.60x, random-mask cells 1.89x -> 5.40x; six cells stay under 1.5x, none of them the copy (X15's float16 anom, the memory-bound 1M nomask means). Left open --------- X17: the old per-pair masked cast kernels (DirectILKernelGenerator.Cast.Masked.cs) have no caller left. The exception types stay the house's (TypeError -> ArgumentException / InvalidCastException, OverflowError -> OverflowException).
…(np.copyto's masked copy, a bit-copy kernel per element width) and X16 (np.copyto is NumPy's array_copyto) and their state-log entry cite 2dffb8d in place of "this commit"
…and strided layout run the SIMD widen-compute-narrow kernels - a row, column, outer-product, strided or reversed operand cost ~10 ns an element through a per-element coordinate decomposition (0.30-0.50x NumPy at 100K), a broadcast F set 0.22x, and an out= call went to the iterator's scalar float16 body; now 1.55-7.4x NumPy over 39 cells, numpy.ma's masked reductions 4.60x -> 4.80x (plan api-parameter-parity X15 + X4) Plan docs/plans/api-parameter-parity.md: X15 and X4 done; X18 and finding 25 recorded. What was slow (NumPy 2.4.2 vs NumSharp at 72696ec, two pinned P-cores) ------------------------------------------------------------------------ 100K float16 elements: np.subtract / add / multiply / divide with a column (250,1), a row (1,400), a strided or a reversed operand, or an outer-product pair took ~1,000 us (~10 ns an element) where NumPy takes 300-480 us (0.30-0.50x); same-shape operands took 73 us (4.1x) and a 0-d one 57 us (5.6x). X4's broadcast F set (30,40,20) F * (30,1,20) F took 328 us (NumPy 77 us, 0.23x), (100,100,100) F * (100,1,100) F 13.8 ms (NumPy 3.3 ms). numpy.ma's float16 anom along axis 1 sat at 0.86x. The values were NumPy's everywhere: speed only. Cause ----- (Half, Half) -> Half arithmetic declines the iterator (its scalar body bridges through double and double-rounds), so the direct route serves it, and its SimdChunk / General paths ran HalfArithStrided: for every output element a div and a mod per axis to recover the coordinates, then the scalar widen-compute-narrow. A broadcast all-F set was also walked in C order (1,200-element strides along the inner axis) and the C result copied to F. With out=, the trivial route declined a broadcast or strided input, so the call ran the iterator, whose float16 body is scalar (~5 ns an element) - numpy.ma's anom subtracts a column mean into an out= buffer, so fixing the plain call alone left it at 0.86x. The fix ------- - DirectILKernelGenerator.Binary.Arith.Half.cs: HalfArithStrided walks the outer axes with an incremental odometer (a carry rewinds an axis by stride * extent) and hands each inner run to HalfArithRun, which picks the SIMD kernel for the run's strides: both contiguous -> HalfArithFull; one broadcast -> HalfArithScalarRight / HalfArithScalarLeft with the operand ROLES kept (NumPy's NaN priority is in2-first for add / multiply, in1-first for subtract / divide); both broadcast -> one value filled; a strided or negative stride -> the operand gathered into a 256-element stack block ([SkipLocalsInit]) and the same kernels. Bit-identical: every lane runs NumPy's HALF loop (float32 compute, RTNE narrow, the explicit NaN-priority blend) and float16 has no call-schedule NaN rule. - DefaultEngine.BinaryOp.cs, ExecuteBinaryOp: a broadcast set of strictly F operands is walked with its axes REVERSED into an F-allocated result (the kernel's C-coordinate order over the reversed dims is the F memory order), so no C result is copied to F and every operand is read along its contiguous axis - NumPy's K-order walk for that set. - DefaultEngine.BinaryOp.cs, TryTrivialBinaryOpInto: float16 add / subtract / multiply / divide with broadcast or strided inputs run the same kernel straight into a C- or F-contiguous out (TryHalfArithBroadcastInto: right-aligned broadcast strides, the out's memory order, coalescing, the MixedType kernel); an input that IS the out (same first element, dims and strides) runs in place, any other overlap (a column view of the out, its transpose) goes to the iterator's copy (NumPy's COPY_IF_OVERLAP). Tests ----- - Math/HalfArithKernelTests.cs (+3): every layout (column / row on both sides, a 1-D row, strided, reversed, strided against reversed, the outer product, a broadcast view against another column) through the four ops, bit for bit against the contiguous kernel over the materialized operands, with random bit patterns (NaN payloads of both signs, sNaN, inf, subnormals, +-0) and runs of 300 that cross the gather block; a broadcast F set is F and exact; out= into C and F outs against the iterator route, in place, a column view of the out and the transpose of a square out. - Backends/Kernels/TrivialBinaryOutTests.cs: the trivial route now serves float16 add / subtract / multiply with F inputs into a C out (it still declines for every other dtype and op). - Planted bugs (detached worktree, scratchpad mutants_x15.py): 13 - the broadcast-right run with swapped roles, the gather stride, the odometer rewind, the reversed F walk, the gather block length, the double-broadcast operand order, the strided-left / broadcast-right pair, the right operand's offset, the out= broadcast alignment, the out= alias rule, the out= overlap check, the F out's walk order, the F result copied again. The unit tests kill 12, the oracle's binary / NaN-order / specials tiers 7; the last is equivalent (one extra copy). - Full suites: NumSharp.Tests 18,141 (net10.0) / 18,140 (net8.0) passed; oracle 239 / 239 on both. Speed (NPY/NS, two pinned P-cores, best of rounds) --------------------------------------------------- - 24 plain cells (column, row, left column, divide by a row, strided, reversed, the F set, the outer product x 1K / 100K / 1M): geomean 3.91x, minimum 1.55x (the first 1K cells of a fresh process; 3.3x after a warm pass); 100K 3.7-7.2x, 1M 3.4-7.9x. - 15 out= cells (column, row, strided, an F input into an F out, in place x 1K / 100K / 1M): 3.58-7.41x. - X4: (30,40,20) F * (30,1,20) F 328 -> 38.6 us (1.99x), (100,100,100) 13.8 -> 0.91 ms (3.62x). - numpy.ma (A6's 310 cells): geomean 4.60x -> 4.80x, minimum 0.86x -> 1.07x; float16 anom along axis 1 at 100K 579 -> 83 us (5.97x); the five cells under 1.5x are the memory-bound 1M nomask means. Found and recorded ------------------ X18: a fresh elementwise result is laid out C (F only for an all-F set) and walked in C order where NumPy uses NpyIter's K order - np.add(x.transpose(1,0,2), 1.0) has strides (60, 20, 4) here, (20, 80, 4) in NumPy, and np.subtract(float32 (400,250) F, (1,250)) reads strided and copies to F (66 us, NumPy 14 us). Finding 25: measure the call form the consumer uses (out=, where=, in place), not only the plain call.
…(float16 arithmetic on every broadcast and strided layout, out= included) and X4 (a broadcast F set walked in F order) and their state-log entry cite 1185f39 in place of "this commit"
… path exactly where NumPy's calls see a zero exponent step - the out= / where= routes called pow for a 0-d exponent, (1,) ** (1,) took the fast path where NumPy calls pow, column exponents never took it, float16 power took the fast path or a double bridge (7,161 of 1,114,112 values off), float_power took np.power's fast path, and a one-element operand with more dimensions lost its leading axes in every binary ufunc; now bit-exact over a new 7,902-case power_fast oracle tier with zero excuses, fast-path cells geomean ~6x NumPy, float16 power 1.52-1.56x, wide varying columns 1.7-4.4x (plan api-parameter-parity X13 + X19 + X20) Plan docs/plans/api-parameter-parity.md: X13, X19 and X20 done; X21, X22, X23 and findings 26-32 recorded; X23 is next (np.kaiser differs from NumPy by 1-2 ULP through np.evaluate's Power node). BREAKING (NumPy-aligned): np.float_power(x, 2.0) is pow, no longer x*x; float16 np.power is powf (x ** 0.5 of -0.0 is +0.0 and of -inf is +inf, as in NumPy); a one-element operand with more dimensions than the other operand now gives the broadcast shape ((7,) + (1,1) is (1,7)). What was wrong (NumPy 2.4.2 vs NumSharp at 6f00ae1) ---------------------------------------------------- NumPy's DOUBLE_power / FLOAT_power (loops_umath_fp.dispatch.c.src) test the exponent's STEP in each inner-loop call: with steps[1] == 0 they read the exponent once and write 1.0/x, 1.0, sqrt(x), x (raw bits) or x*x for -1, 0, 0.5, 1, 2; any other exponent, or a nonzero step, calls the CRT pow. The answers differ: 757.7617475757002 ** 2 is x*x 0x412185f5bb7003c1 but pow ...03c2; -0.0 ** 0.5 is sqrt -0.0 but pow +0.0; -inf ** 0.5 is sqrt NaN but pow +inf; sNaN ** 1 keeps the NaN signalling, pow quiets it. - A 34-call twin probe: the out= and where= routes called pow for a 0-d exponent; (1,) ** (1,) and 0-d ** (1,) took the fast path where NumPy's trivial loop calls pow, and 0-d ** (1,) returned a 0-d result (X19); float16 took the fast path or the double bridge (X20). - A special-value probe (31 values incl. +-0, +-inf, subnormals, finfo.max/tiny and six NaN patterns x 31 exponents x float16 / float32 / float64, through full-shape and 0-d exponents): 44 cells differed. - float16 (X20): NumPy's HALF_power is PyUFunc_ee_e_As_ff_f with powf and has NO fast path. Over every float16 base x 17 exponents, 7,161 of 1,114,112 results differed: the fast path (sqrt(-0.0) = -0.0, sqrt(-inf) = NaN, x ** 1 kept a sNaN), sNaN ** 0 gave 1 (the BCL widen quiets the NaN before pow), integer powers of a negative NaN lost its sign, and one result rounded once from double instead of from powf's float. - float_power (X20): NumPy's loop is PyUFunc_dd_d with pow, no fast path; NumSharp delegated to np.power and took it (float_power(757.7617475757002, 2.0) gave ...03c1, NumPy ...03c2). - Shapes (X19): a 165-call probe (24 binary ufuncs, comparisons, logical ops, shifts and C# operators x 5 shape pairs with a one-element operand of higher rank): 100 wrong shapes, values right. The scalar-broadcast shortcuts of the binary, comparison and shift routes gave the result the array operand's shape. - The new power_fast tier: 1,785 of its first 7,773 cases differ on the parent. Cause ----- Which calls NumPy makes decides the bits, and three stages decide the calls: - check_for_trivial_loop: a cast or unaligned 0-d input, or a 1-D input of at most bufsize elements, is copied into a contiguous loop-dtype array; any other such input, and an out of another dtype or unaligned, refuses the trivial loop. - try_trivial_single_output_loop: a 0-d input has step 0, a 1-D operand its own stride, an N-D operand the item size (contiguous in one shared order). - NpyIter: fill_axisdata, the axis sort (C wins ties), negative-stride flips, COPY_IF_OVERLAP, coalescing and npyiter_find_buffering_setup's cost model (a read-only 0-d cast input is copied eagerly; a buffered single-stride broadcast keeps step 0). A column exponent (R,1) reaches the loop unbuffered - step 0 - only for rows longer than bufsize / 2 at the default; with every input cast, rows of exactly bufsize stop the core and stay at step 0. NumSharp's plain call took the fast path for every one-element exponent, its ufunc out= / where= routes never did, and float16 and float_power were in it. The fix ------- - DefaultEngine.UfuncInnerSteps.cs (new): the step decision ported once - NumPyUfuncInputSteps (check_for_trivial_loop's copies and refusals, TryTrivialLoopSteps = try_trivial_single_output_loop with PyArray_EQUIVALENTLY_ITERABLE_OVERLAP_OK, NpyIterInputSteps = NpyIter's axis data, sort, flips, overlap copy, coalescing and buffering cost model) over UfuncLoopOperand (dims, byte strides, dtype, item size, contiguity, alignment from memory, data start). Generalized from np.clip's const-loop decision: Default.ClipNDArray.ConstLoop.cs now builds its operands and calls it (the moved members were removed there; np.clip's tiers pass unchanged). - Default.Power.StrideZero.cs (new), Default.Power.cs: np.power's float32 / float64 loops (ResolvePowerLoop) go to PowerFloatLoop. A single-valued exponent (0-d, one element, or a stride-0 broadcast of one) is classified in the loop dtype (0.50000001 is 0.5 in a float32 loop); a fast value runs the SIMD unary kernel (square / sqrt / reciprocal / positive) or a fill with 1 into the result or out; where= computes, then copies with the mask (np.copyto: 2.5-9x faster than one kernel call per mask run, the same values). A 0-d plain call skips the shape prologue (it can neither add a dimension nor fail to broadcast) but keeps dtype='s input-cast checks. A varying exponent that NumPy's calls see with step 0 runs ONE NDIter pass (the ufunc out= route's iterator flags: COPY_IF_OVERLAP, a buffered cast out, the mask as ARRAYMASK) whose kernel is NumPy's per-call branch per element. Everything else is pow (ExecuteBinaryOp). Validation keeps the general route's order (dtype= casts, read-only out, mask dtype, out cast, shapes). The old fast path stays for the integer and decimal loops only. - DirectILKernelGenerator.Power.cs (new): PowerStrideZero<T> (the branch chain, generic over float / double; T.Pow / T.Sqrt are the CRT functions NumPy calls; the float32 reciprocal is a float division - C's 1.0 / in1 divides in double, and that double rounding is innocuous, verified over all 2^32 float32 patterns), the vector bodies PowerStrideZeroVector128 / 256 / 512 (one per width: a uniform fast exponent is one compare and one vector op, any other vector goes lane by lane), EmitPowerStrideZeroBody / EmitPowerStrideZeroVectorBody, and PowerHalfNumPy. InnerLoopKernelKey gains the PowerStrideZero family ("npy_powz_...") so the kernel never shares the plain power kernel's cache slot. - float16 (X20): EmitHalfOperation routes Power to PowerHalfNumPy (exact widen, MathF.Pow = the CRT powf, NumPy's npy_float_to_half narrow). Whole arrays run the widen-compute-narrow kernels (TryGetHalfArithKernel accepts Power; IsHalfArithOp and the iterator's (Half, Half) -> Half decline include it, so every X15 layout and out= route serves it) through the new HalfPower: a vector widen of each 256-element block, ONE MathF.Pow loop over the block, a vector narrow; no operand-order NaN rule (powf decides). A per-8-lane form (extract lanes, powf, rebuild) reached only 1.32-1.41x NumPy. - float_power (X20): Default.FloatPower ends in ExecuteBinaryOp(BinaryOp.Power, ..., loop): the power kernel, never the fast path. - Shapes (X19): TryTrivialContiguousBinaryOp, TryTrivialContiguousComparisonOp and TrySimdScalarShift decline when the one-element operand has more dimensions than the other; the broadcasting routes compute NumPy's shape. Tests ----- - Oracle tier power_fast.jsonl (gen_oracle.py power_fast, 7,902 cases, floor 7,500, zero excuses - MisalignedRegistry returns null for the op): sections A plain calls (3,640), B integer / bool / complex bases (1,542), C trivial-loop copies and iterator buffers at np.setbufsize(256) and the default (244), D varying exponents across the buffering threshold with out= / where= (388), E out= / where= layouts with single-valued exponents (1,536), F dtype= (180), G float_power (216), I two decision boundaries a mutation pass found (129), H empties and errors (27); 219 cases are NumPy errors. OpRegistry.PowerFast.cs replays them (np.setbufsize per case, out views of the base buffer, the out's whole buffer compared). Generator drops, each another item: a 0-d float32 / float16 base with a Python-scalar exponent (the NEP 50 0-d divergence), complex bases with non-integer exponents (F5), unsigned bases with -1 (X21), an out= that does not broadcast (X22). coverage/oracle_map.json maps the op (coverage tests 15 / 15). - Math/np.power.StrideZero.Test.cs (11 tests, NumPy-probed raw bits): the plain / out= / where= routes, one-element exponents through the trivial loop and NpyIter, columns on both sides of the buffering threshold and the trivial loop's cast copy, dtype=, float16, float_power, the vector bodies against the scalar body on every lane (uniform and mixed exponents, every width), and the two mutation-found boundaries. The old OpenBugs.UfuncOut.cs pin moved here (file deleted). - Backends/Kernels/OneElementOperandRankTests.cs (2 tests: 165 shapes + values; 100 shapes fail before the fix). - InnerLoopKernelKeyTests: the new key's legacy string and distinctness; Production_Route_Registers_Kernel_Under_Both_Keys now uses a strided out - since 2026-10-04 a contiguous out takes the trivial out= loop and never compiles the kernel, so the test passed only after another test had compiled it (finding 31). - Probes: every float16 base x 17 exponents through five routes (full, 0-d exponent, reversed base, odd slice, 0-d base) 0 differences; the special-value probe 0 of 44; the shape probe 0 of 100; the twin probe 0 differences. - Planted bugs (scratchpad x13_mutants.py, one exact edit each): 13 - the trivial-loop copy bound, the cast-out refusal, the buffering stop, the cost comparison, the buffered broadcast's step 0, the trivial 1-D step, the float32 exponent comparison, the 0-d shortcut, where='s mask, the vector square, the scalar reciprocal, float16's tail, float16's broadcast exponent. The tier and the unit tests kill all 13. Two survived the first pass (no case had one-element operands with a cast out, none had rows of exactly bufsize with every input cast); NumPy's answer flips at both, so they became section I and a unit test (finding 27). - Full suites: NumSharp.Tests 18,154 (net10.0) / 18,153 (net8.0) passed, 0 failed; oracle 240 / 240 on both. Speed (NPY/NS, two pinned P-cores, best of rounds) --------------------------------------------------- - 90 fast-path cells (float64 / float32 x 1K / 100K / 1M x -1, 0, 0.5, 1, 2 x plain / out= / where=): geomean 6.04x and 6.21x over two runs; 100K and 1M 2.0-20x. - The 1K plain float64 cells sit at the NDArray allocation floor: 1.3-2.9x from run to run. np.empty(1000) alone costs 0.20 us disposed and 0.42 us left to the finalizer, against NumPy's 1.0-1.9 us per call; with disposed results x ** 0 is 3.0x, x ** -1 1.9x (division-bound on both sides), x ** 2 6.2x. Power's own cost over np.ones / np.reciprocal / np.square is 0.01-0.04 us. - The 24 pow cells (x ** 3, full-shape exponents): 0.92-1.15x - the CRT pow on both sides. - float16 power: 1.52-1.56x at 1K / 100K / 1M (the old fast path 2.6-5.8x for 2.0 and 0.5 but with other bits; the double bridge 0.54x for 3.0). powf is ~5 ns an element on both sides and NumPy's scalar half conversions ~3.5 ns more: the ceiling is ~1.7x. - float_power x ** 2.0: 0.96-1.06x, pow on both sides (the old 14-28x computed x*x, not NumPy's value). - Varying column exponents: (2, 5000) 1.67x, (16, 8192) 4.36x, (100, 10000) 3.52x - the old route called pow there (0.16-0.29x); the scalar stride-0 body alone gave 0.85-1.75x, the vector body takes the rest. Found and recorded ------------------ - X21: np.power(unsigned, -1) raises ArgumentException "Integers to negative integer powers are not allowed." where NumPy raises OverflowError "Python integer -1 out of bounds for uint8" (X2's NEP 50 packing class). - X22: the out= broadcast error omits the out's shape ("... shapes (3,) (4,) " vs NumPy's "(3,) (4,) (3,) "). - X23: np.evaluate's Power node computes pow for a constant exponent where NumPy's x ** 2.0 takes the fast path; np.kaiser evaluates ((n - alpha) / alpha) ** 2.0 through it, so 146 of 4,796 windows (M 2..1200 x 4 betas) differ by 1-2 ULP (np.kaiser(272, 5.0)[13]: NumPy 0x3fb79bb596a301c3, NumSharp ...c5). Next item. - Findings 26-32: a loop that reads its own steps makes the call schedule part of the answer; a mutation survivor is a missing case until proved equivalent; a restored source must look newer than the mutant's build (shutil.copy2 kept the old mtime and MSBuild kept the last mutant compiled in - the "real" tier failed 3 cases until a touch + rebuild); a float NaN constant is not a test input (RyuJIT folds float constants through double, so a constant sNaN can arrive quiet); C's float32 1.0 / x is an innocuous double rounding; a test can pass only because an earlier test compiled a kernel; numpy.ma's plain-call workaround for power can now be a plain out=.
…(np.power's float loops take NumPy's scalar-exponent fast path exactly where NumPy's calls see a zero exponent step), X19 (a one-element operand with more dimensions keeps its leading axes) and X20 (float16 power is HALF_power's powf, float_power never takes the fast path) and their state-log entry cite 4116b60 in place of "this commit"
…s families A 2026-10-06 error-parity sweep (every call probed against Linux NumPy 2.4.2, the oracle NumSharp's LP64 integer model matches) found four functions whose ERROR MESSAGE is not NumPy's, while sibling functions already produce the right text. NumSharp deliberately uses its own exception TYPES for NumPy's ValueError / AxisError (house convention), but the message TEXT is meant to be NumPy's verbatim - the library asserts verbatim texts across its surface and has an errors_full oracle tier for exactly this. These are the holes, each now an [OpenBugs] test that asserts NumPy's text and so FAILS today; being [OpenBugs] they are excluded from CI (TestCategory!=OpenBugs) and never turn the build red. A broad value sweep ran first (97 edge cases across diff / ediff1d / nan_to_num / gradient / percentile / quantile / interp / digitize / bincount / pad modes / tile / repeat / insert / delete / kron / vander / take / choose / searchsorted / partition / nanpercentile / median / convolve / corrcoef and more) and found ZERO value divergences - NumSharp's value/parameter coverage of these is complete - which is why this pass pins error-text gaps instead. Verified on a clean worktree at this HEAD (3bb5769): all three tests compile and FAIL with exactly the documented divergences; the TestCategory=OpenBugs filter runs them, TestCategory!=OpenBugs excludes them. Gaps (NumPy text -> what NumSharp produces): - Empty operand list to np.vstack / np.hstack / np.dstack / np.stack: NumPy ValueError "need at least one array to concatenate" (vstack/hstack/dstack) or "need at least one array to stack" (stack); NumSharp leaks .NET's generic guard "Value cannot be an empty collection. (Parameter '...')". np.concatenate([]) and np.column_stack() already carry NumPy's text, so this is an inconsistency, not an open question. - Out-of-range axis in the Transpose-delegating family (np.moveaxis destination & source, np.rollaxis, np.swapaxes) and np.squeeze: NumPy AxisError "axis 5 is out of bounds for array of dimension 2", prefixed with the operand role where it normalizes more than one axis ("destination:" / "source:" / "axis2:"). NumSharp throws a parenthesized "axis (5) is out of bounds for array of dimension (2)" (AxisOutOfRangeException), and np.squeeze leaks .NET's "Specified argument was out of the range of valid values.". np.sum / np.flip already give the house AxisError text with no parentheses. - np.moveaxis with mismatched source/destination lengths: NumPy ValueError "`source` and `destination` arguments must have the same number of elements"; NumSharp throws a bare Exception whose message ends in a stray backtick (the one defect in an otherwise-correct text), asserted by exact equality. The pins check the message by SUBSTRING on NumPy's core text (tolerating the house "(Parameter 'axis')" suffix a fixed AxisError would add), except the trailing-backtick case which is exact. Exception TYPES are not asserted - only the texts, which is the real parity target and the part the library already holds to NumPy verbatim elsewhere. Docs: .claude/CLAUDE.md lists OpenBugs.ErrorParity.cs in the OpenBugs-files enumeration.
…he object an API returns (view vs copy vs the operand itself, out= identity, C/F/OWNDATA/WRITEABLE/ALIGNED flags, byte strides, shape arguments in every C# form, secondary array arguments in non-C layouts); 9,226 gate cases at zero excuses plus 717 carved cases under 47 gap ids, and the gaps they found (silent wrong values from offset views, read-only copies, layouts, leaks) recorded, not fixed
WHY
Every earlier differential tier compares a result as (dtype, shape, C-order bytes). That contract cannot tell a
view from a copy, the operand itself from a new object, an F result from a C one, or a read-only array from a
writeable one, and the shape / secondary-array parameters were only ever passed in one C# form or C-contiguous.
This commit adds coverage for those three axes and identifies where NumSharp differs from NumPy. It changes no
NumSharp.Core code: the divergences are carved and pinned, not fixed.
GENERATOR (test/oracle/gen_viewflags_oracle.py, needs numpy==2.4.2; deterministic: a rerun is byte-identical)
- view_flags.jsonl (op "viewflags", params.fn = the API; 7,282 gate + 622 carved): 194 APIs over 21 source
layouts (C and F owners, transposed, permuted 3-D, stepped, reversed, a C-contiguous view at an offset, a
read-only owner and a view of it, a stride-0 broadcast, a same-shape broadcast, 0-d owner and 0-d offset view,
1x1, a unit axis, empty). Sections V1 reshape / transpose / flip / diagonal views, V2 complex parts, V3 the
conversions (asarray / array / copy / require / astype / view / getfield ..., every order and copy flag), V4 a
13-dtype sweep, V5 indexing (basic, integer, fancy, mask), V6 tuple results, V7 creators, V8 einsum's view path,
V9 the copy family, V10 tril / triu / diagflat, partitions, the nan-reductions, count_nonzero, the FFT family,
eye / identity / tri / mgrid / ogrid, the index-tuple generators, ndarray methods; K1-K3 the layout of ufunc
and reduction results.
- out_flags.jsonl (op "outflags"; 1,072 gate + 4 carved): f(..., out=o) for 149 out-capable APIs (every ufunc
NumSharp gives an out=, the reductions, scans, products, the FFT family, multi_dot, around / concat, the ndarray
methods) with o in six layouts (C and F owners, offset, stepped, transposed, reversed views). Slot 0 is the
returned object (its facet must say it IS o), slot 1 the whole base buffer behind o after the call (prefilled
with markers: a write outside the view or a missed slot shows). NumPy's refusals are verbatim error cases.
- shape_args.jsonl (op "shapearg"; 537 gate + 54 carved): zeros / ones / empty / full / reshape /
ndarray.reshape / broadcast_to / resize / tile / indices / indices_sparse / empty_like with the same dims as a
Shape, long[], int[], int and view:<layout> (a view's Shape, built by RandomApi/ViewSizeLayouts.cs).
- array_args.jsonl (EXISTING op keys, valueclass "arrayarg"; 335 gate + 37 carved): 55 secondary operand
positions (indices, condition, v, sorter, bins, weights, xp / fp, b, x2, values, a_max, x / y, choice lists,
ar2, cov's y, ...) in up to nine layouts (C, F, transposed, stepped, reversed, offset, read-only, broadcast, 0-d).
- Values are -1..3 in every dtype, so a value difference can only mean a wrong element was read. The offset view's
owner holds the sentinel 7 outside the window, so an order-free result (a sum, a unique set) shows a misread too
(with the periodic fill, np.unique of an offset view read the wrong window and still passed).
- Portability (the corpus holds Windows NumPy bits, CI runs three OSes): the out= rows of the libm-backed loops
take an all-zero input - on -1..3, win-amd64 NumPy's arccosh(2), expm1(1), cbrt(2), logaddexp(-1, 0) are not
correctly rounded and log2(3) / logaddexp2 lie 0.02 ULP from a midpoint (mpmath check). The view tier's exp /
power / arctan2 cells are correctly rounded and >= 0.17 / 0.17 / 0.06 ULP from a midpoint; logspace /
geomspace's libm calls >= 0.079. FFT lengths stay in {2, 3, 4, 6}, whose twiddles fft.jsonl proves on every host.
SCHEMA + HARNESS (test/NumSharp.Tests.Oracle/Fuzz)
- FuzzCorpus.cs: operand "own" (rebuild as an OWNER, OWNDATA set; without it every operand is a view and an
identity result reported OWNDATA false - a harness artifact), expected "mem" (MemFacet: flags, normalized byte
strides - extent <= 1 axes and empty results read 0 on both sides -, shares[i], same[i], scalar), tuple
"xshare" / "xsame", case "gap".
- FuzzCorpusTests.Kinds.cs: CompareMem / CompareCrossSlots (np.shares_memory, ReferenceEquals); the tuple path
compares every slot's facet. FuzzCorpusTests.cs: ReplayOne extracted from RunCorpus (shared with the gap
replay), NUMSHARP_ORACLE_FAILURE_DUMP writes every failure line of a tier (classification needs all of them),
MinCases floors for the four files.
- OpRegistry.ViewFlags.cs: ApplyViewFlags / ApplyViewFlagsTuple / ApplyOutFlags / ApplyShapeArg /
ApplyShapeArgTuple call each API in NumSharp's spelling and return its result OBJECT untouched (the ordinary
registry composes calls, and a composed result carries the harness's flags). OpRegistry.cs / .Kinds.cs route
the three vehicle keys. MisalignedRegistry: zero excuses for the vehicle keys and valueclass arrayarg.
- FuzzCorpusTests.ViewFlags.cs: the four FuzzMatrix gates + four [TestCategory("OpenBugs")] *_KnownGaps replays of
corpus/gaps/*_gaps.jsonl (per gap: how many carved cases still diverge, how many now match NumPy).
- RandomApi/ViewSizeLayouts.cs: Build disposes the SOURCE array of the view it derives a Shape from (it disposed
only the view, and its doc said the sources were disposed); the leak sweep counted that harness buffer against
every shapearg view:<layout> case.
- UndisposedIntermediateTests: families of the vehicle keys are labelled op:fn (FamilyOp), so a leak is pinned to
one API; KnownEscapes.Classify also accepts an "op [layout]" key (one family). OracleCoverageStrengthTests: a
reviewed FixedDtypeOps reason for shapearg (its axis is the shape form).
THE CARVE
_gap_of files every case NumSharp answers differently into corpus/gaps/<tier>_gaps.jsonl under a GAPS id; the
gates hold the rest with zero excuses. Drawn from a replay of the uncarved matrix (--uncarved DIR +
NUMSHARP_ORACLE_FAILURE_DUMP) and EXACT: every carved case diverges (the replay prints N/N per id, 0 matching), no
kept case does.
GAPS FOUND (47 ids, 717 cells; full text in GAPS; NumSharp.Tests/OpenBugs.ViewFlags.cs reproduces 18 through public
calls with NumPy 2.4.2 literals, all failing for the bug):
- SILENT WRONG VALUES: a C-contiguous view at a non-zero offset (np.split / np.unstack / np.array_split children)
is read from its parent's start by np.searchsorted (v, sorter, the method: VA01), np.where (condition / x / y:
VA03), np.ldexp (x2: VA05), np.nansum / np.nanprod with axis=None (VF30), np.repeat / ndarray.repeat (VF39),
np.unique and np.intersect1d / setdiff1d / setxor1d (VF40, np.unique(child) returned the parent's [9.]);
o[:, idx] on such a view throws IndexOutOfRangeException (VF22).
- READ-ONLY: copies of a read-only array stay read-only (VF14); b[1] of a read-only broadcast through the object[]
indexer is writeable (VF23); np.broadcast_arrays returns a read-only broadcast operand writeable (VF17);
np.median / np.nanmedian of a 0-d with keepdims is read-only (VF29).
- VIEW / COPY / IDENTITY: VF04 squeeze, VF13 array(copy=False), VF15 asmatrix, VF16 astype(copy=False, order),
VF18 argwhere / flatnonzero / unravel_index, VF26 NumPy's internal-temporary views (resize, tile, kron,
getfield, nanmedian), VF27 matvec / vecmat, VF28 argmax family on a length-1 axis, VF35 diag / tril / triu
_indices, VF37 nanmean keepdims, VA04 cov / corrcoef (squeeze); VF01 / VF02 / VF03 NumPy scalars as writeable
0-d arrays (no scalar type).
- LAYOUT (C where NumPy keeps K, or the reverse): VF05 ravel('K') of a permuted array returns C-order values,
VF06-VF12, VF19-VF21, VF31-VF34, VF36 the FFT family, VF38 tril / triu, VA02 take_along_axis, VS01
np.zeros(view.Shape) keeps the view's F layout.
- ERRORS: VF24 np.reshape(-1) of a broadcast view throws; VO01 np.linalg.multi_dot writes into an out= that
np.dot refuses; VF25 np.stack of 0-d arrays.
- LEAKS (the scope gate replays these tiers too; recorded in KnownEscapes with per-family ceilings, probed through
public calls): ndarray.astype with an order the input lacks (1), np.ascontiguousarray / np.asfortranarray with a
dtype change (1), np.eye(order='F') (1), np.nan_to_num(copy=False) of a non-C input (3), np.argpartition of a
non-C input (1), np.pad / np.roll of an empty array (1 / 2), and with the OpenBLAS backend np.inner /
ndarray.dot with an operand BLAS cannot read in place (1 per copy).
- API SURFACE (not expressible as cases): no NDArray-as-shape overloads; zeros_like / ones_like / full_like take
no shape=; np.reshape no order= / copy=; np.squeeze no tuple axis; np.zeros / np.empty take a dtype only beside
a Shape (np.ones beside a Shape or an int[]); np.sum / prod / mean / std / var / argmax / argmin / nansum take
no out=.
- REMAINING COVERAGE GAP (survey of the committed corpus): of 1,738 (op, operand position) pairs with a layout,
549 only ever see a C-contiguous operand at offset 0; outside the vehicle keys' deliberately-C inputs (196) that
is 353 pairs of the ordinary (153), masked ma.* (59), numpy.polynomial (~120) and other tiers.
COVERAGE JOIN (coverage/)
oracle_map.json: param-driven fuzz.keys entries viewflags / outflags / shapearg (params.fn -> numpy.{value}; getitem
-> numpy.ndarray, where1 / where3 -> numpy.where, the FFT names -> numpy.fft.*, multi_dot -> numpy.linalg,
indices_sparse -> numpy.indices). The tiers now gate 11 rows that only borrowed evidence before, so their reviewed
aliases are stale and removed (absolute, around, bitwise_not, concat, degrees, fix, ndarray.argpartition,
ndarray.conjugate, radians, remainder, true_divide). test_oracle_evidence.py follows (numpy.ndarray's keys, absolute
now direct, row_stack still borrows, the C#-alias mirror floor 30 -> 20). generate_coverage.py accepts the tiers:
603,410 contracts, headline oracle-verified 497/560.
VERIFIED
Oracle suite 272 passed / 1 skipped / 0 failed on net10.0 AND net8.0 (the four gates, the leak sweeps, surface /
strength / applicability / survey gates); NumSharp.Tests with the CI filter 18,401 passed on net10.0; the four
*_KnownGaps replays: every carved case still diverges, 0 match; coverage/test_oracle_evidence.py 15 passed; the
generator rerun is byte-identical.
Docs: Fuzz/README.md section "The array-object oracle" + a Table 2 row + the regeneration command;
.claude/CLAUDE.md fuzz-pipeline bullet.
…, dtype class) cell moved into the ndarray kinds it never saw (C / F / permuted / stepped views at an offset, reversed, stepped-and-reversed, broadcast at an offset, 0-d views, one-element 1-D, empty views inside a buffer), NumPy 2.4.2's answer recomputed: 16,692 variants plus a layout-invariance gate with no excuse table; 179 gap cells under 7 ids carved exactly (offset-view misreads in digitize / isin / nancumsum / nancumprod / ldexp / unique_values / the set routines / repeat; the comparison operators and logaddexp skip broadcasting for an empty operand), recorded, not fixed
WHY
---
The corpus pins each call on operands of one layout, chosen per tier. A survey of every operand descriptor of every
root corpus file (1,775 (op key, operand position) pairs) found most ops had seen only some of the ndarray kinds a
caller can pass: 42 % had ever seen a C-contiguous view at an offset (the np.split-child kind behind the array-object
oracle's silent wrong values), 0.1 % a permuted view at an offset, 4.9 % a stepped view at an offset, 2 % a
stepped-and-reversed view, 0 % a broadcast at an offset, 0.2 % an empty view inside a buffer, 27 % a read-only
operand. The task (/np-function "Validate and increase the coverage with edge cases and various kinds of ndarray,
extensively") is coverage and gap identification; no library code changes here.
THE GENERATOR (test/oracle/gen_relayout_oracle.py)
--------------------------------------------------
* kind_of: 19 kinds -- c, c_offset, f, f_offset, permuted, permuted_offset, reversed, strided, strided_offset,
strided_neg, bcast, bcast_offset, scalar_bcast, scalar_bcast_offset, 0d_owner, 0d_view, one_1d, empty, empty_view
-- plus the tags highrank / readonly. The *_offset kinds count only the EXTRA offset (a reversed axis needs one to
reach its first element). Cells are (op, position, kind, dtype class: bool / signed / unsigned / float16 /
float32-64 / complex).
* A source qualifies only when a plain NumPy call reproduces its recorded outcome byte for byte (the self-validating
dispatcher: resolve() maps the op key to np.* / np.linalg.* / np.fft.* / a module-qualified callable / an ndarray
method, every param must be one of its keywords, SPECIAL adapts the corpus conventions that are not a plain call;
the exact error type and text count for an error case). No variant guesses a call.
* Transforms (TRANSFORMS): value-preserving for every op (c_offset, f, f_offset, permuted, permuted_offset, reversed,
strided, strided_offset, strided_neg, readonly); derived-value only for the elementwise and reduction families
(gen_oracle's op tables): bcast, bcast_offset, scalar_bcast, scalar_bcast_offset, one_1d and 0d_owner (owning,
"own": true), empty and empty_view (axis 0 cut to length 0; the empty view built with np.ndarray(buffer=, offset=)
because NumPy's own base[3:3] does not advance its data pointer). An emptied operand fills TWO cells: a value one
and a "could not be broadcast" one (EMPTYING / BCAST_ERR), the edge where an early return for an empty operand
skips the shape check. View buffers hold the sentinel 7 wherever the view does not reach.
* Each variant records layout_sensitive when NumPy's own outcome changes once the moved operand is a fresh C-ordered
copy (60: reductions over F / permuted layouts, isfortran, read-only operands of in-place ops).
* _upstream_residual drops what another tier already documents: ufunc / np.where variants whose NumPy answer changes
when their 0-d operands become Python scalars (NEP 50's weak promotion -- the registry's NEP50 divergence), and the
flat np.mean of a broadcast float operand that is not order-free (mean_loops drops the same cells, finding 10).
The cell stays open for another source.
* Excluded: random streams, index_* / ma_* / decimal_* schemas, backend-pinned and array-object tiers, np.evaluate,
iterator traces, char / decimal operands, and np.frombuffer (its operand stands for Python bytes: the replay passes
NumSharp FuzzCorpus.ResultBytes, so no layout of it reaches NumSharp, while NumPy refuses a non-C-contiguous
ndarray buffer -- the first run's 18 frombuffer "failures" were this harness artifact).
* Deterministic: a rerun is byte-identical (sha256 checked). Runs in ~30 s. Run it after every other generator.
FILES
-----
corpus/relayout.jsonl 14,964 variants, 329 op keys from 47 portable tiers -> FuzzCorpusTests.Relayout
corpus/relayout_host.jsonl 1,728 variants of the host-libm tiers -> RelayoutHost (RunHostLibmCorpus)
corpus/gaps/relayout_gaps.jsonl 179 carved, 7 ids -> Relayout_KnownGaps [OpenBugs]
corpus/gaps/relayout_host_gaps.jsonl 0 (kept for the host carve rules to come) -> RelayoutHost_KnownGaps [OpenBugs]
Kinds: strided_offset 2,061, strided_neg 2,020, readonly 1,989, permuted_offset 1,474, c_offset 1,248, f_offset
1,235, empty_view 1,015, scalar_bcast_offset 739, bcast_offset 724, permuted 672, 0d_view 645, reversed 611, empty 603,
strided 497, scalar_bcast 407, f 317, bcast 312, one_1d 259, 0d_owner 43. 666 are NumPy error cases.
TWO VALIDATION HOLES CLOSED
---------------------------
1. Excuse inheritance. A variant keeps its source's valueclass and op, so the source tier's MisalignedRegistry rules
travel with it -- and an unbounded excuse swallows a layout misread as readily as the divergence it documents:
the "[known bug]" repeat branch excused five 0-d offset misreads. New gate FuzzCorpusTests.RelayoutLayoutInvariance
([FuzzMatrix]): for every variant NOT layout_sensitive, NumSharp's outcome on the variant must equal its own
outcome on a C-ordered copy of the moved operand -- same dtype, shape, bytes (NaN tokenized), or a throw on both
sides -- with NO registry in between. The copy is FuzzCorpus.ReconstructCanonical: the values gathered by index
arithmetic over the buffer, so a copy kernel sharing the bug cannot hide it. 16,632 compared, 60 skipped; it
flagged 70 layout-caused variants on the uncarved matrix (all carved now). Floor 14,000.
2. The NEP50 0-d blanket. MisalignedRegistry's two NEP50 branches excuse ANY dtype difference, and any value / error
difference, of a multi-operand case with a 0-d operand. They hid np.logaddexp returning its empty int32 operand
where NumPy returns float64. Relayout cases are now exempt (IsRelayoutCase); the generator drops instead the
variants that genuinely depend on weak 0-d promotion (point above), so whatever difference is left is real.
HARNESS CHANGES (test/NumSharp.Tests.Oracle/Fuzz/)
--------------------------------------------------
* FuzzCorpus.Reconstruct: an EMPTY operand descriptor that carries a buffer rebuilds as an empty VIEW over it (it was a
fresh empty array whatever the descriptor said). 176 existing empty operands with a buffer (manip ix_,
minmax_kwargs) became real views; they still pass.
* FuzzCorpus.ReconstructCanonical (above); Case.Layout_Sensitive; RunGapCorpus(file, extraVerdict): the relayout gap
replays add the invariance verdict, so a carved layout bug the registry would excuse still reads as open.
* MinCases floors: relayout.jsonl 14,000, relayout_host.jsonl 1,600.
* UndisposedIntermediateTests.KnownEscapes: the leaks the sweep found in the new tiers (below).
WHAT IT FOUND (carved by _gap_of, each rule emulating its bug so the carve is exact)
-------------------------------------------------------------------------------------
Offset-view misreads (the C-contiguous view at an offset is read from its OWNER'S START -- the array-object oracle's
VA01 / VA03 / VA05 / VF39 / VF40 class, reached through more ops, dtypes and positions). The rules require
_misread_observable: NumPy's answer for the window at offset 0 must differ, else the misread cannot be seen:
RL01 (15) np.searchsorted v / sorter, ndarray.searchsorted v, np.digitize x (searchsorted's v), np.isin of a
complex128 / float16 element already of the common dtype (its sort + searchsorted route).
RL02 (17) np.where condition (bool) / x / y already of the result dtype (a cast operand is copied first and read
right) and its all-0-d route; np.nancumsum / np.nancumprod of float16 / complex128 (they run
np.where(np.isnan(a), fill, a)), C-contiguous and 0-d views.
RL03 (3) np.ldexp mantissa already of the result dtype, or an int32 / int64 exponent (MaterializeContiguous hands
np.ascontiguousarray's unchanged view to a kernel that reads .Address without Shape.offset).
RL04 (27) np.unique, np.unique_values, and BOTH operands of np.intersect1d / setdiff1d / setxor1d.
RL05 (5) ndarray.repeat of a 0-d view at an offset -- seen only by the invariance gate.
Empty operands (carved by emulating the early return):
RL06 (72) the NDArray comparison operators (==, !=, <, >, <=, >=; the registry replays the six comparison keys
through them) return np.empty(lhs.Shape, bool) for an EMPTY left operand: no error where NumPy raises
"operands could not be broadcast together", the left operand's shape where NumPy broadcasts both, and
MakeGeneric<bool>() leaves the untyped wrapper to the finalizer (one pooled buffer per call). np.equal & co.
are right.
RL07 (40) np.logaddexp / np.logaddexp2 return a CLONE of an empty operand at either position: no broadcast error,
no float promotion (int32 / uint8 where NumPy gives float64 / float16), the empty operand's shape.
Leaks (the scope sweep replays the new tiers; recorded in KnownEscapes with ceiling 1):
* the comparison operators' empty-left early return -- 12 families "<op> [relayout:empty@0|empty_view@0]";
* with the OpenBLAS backend, np.inner / ndarray.dot with a stepped or reversed operand (the known backend-copy leak)
-- 16 families "inner|ndarray.dot [relayout:reversed|strided|strided_offset|strided_neg@0|1]".
Validated clean: F, permuted, stepped and broadcast views at an offset, one-element broadcasts at an offset and empty
views inside a buffer showed NO misread in any op -- every misread is the C-contiguous-offset fast path.
Public-call repros with NumPy 2.4.2 literals: test/NumSharp.Tests/OpenBugs.Relayout.cs (8 tests, [OpenBugs], each
fails today): digitize, complex isin, float16 nancumsum + nancumprod, complex nancumsum, ldexp mantissa,
unique_values + intersect1d first operand, the comparison operators, logaddexp / logaddexp2.
COVERAGE (share of the 1,775 (op, position) pairs that have seen each kind, before -> after)
------------------------------------------------------------------------------------------
c_offset 41.8 -> 55.3 %, f_offset 10.8 -> 26.0 %, permuted 27.6 -> 35.0 %, permuted_offset 0.1 -> 16.0 %, reversed
60.1 -> 66.4 %, strided_offset 4.9 -> 28.7 %, strided_neg 2.0 -> 25.2 %, bcast 29.1 -> 32.6 %, bcast_offset 0 -> 7.7 %,
scalar_bcast_offset 0 -> 7.9 %, 0d_view 9.7 -> 17.8 %, empty 31.8 -> 35.7 %, empty_view 0.2 -> 8.1 %, readonly 27.2 ->
48.5 %. The remaining pairs belong to tiers with their own schemas (polynomial, ma, index, vehicle keys), which no
plain call reproduces.
VERIFICATION
------------
* Carve exact: the uncarved matrix replayed with NUMSHARP_ORACLE_FAILURE_DUMP gave 174 ordinary failures (62 offset
misreads, 72 RL06, 40 RL07) and 67 invariance failures (the same 62 misreads + 5 RL05): a union of 179, the
carved set; carved, Relayout / RelayoutHost / RelayoutLayoutInvariance pass and Relayout_KnownGaps reports 179/179
still open, 0 "now match".
* Full oracle suite (TestCategory!=OpenBugs&TestCategory!=HighMemory): net10.0 and net8.0 275 passed, 1 skipped.
NumSharp.Tests (net10.0, CI filter): 18,401 passed. net8.0 builds.
* Linux x64 (WSL, CI's NoWarn and filter, this working tree on journey4): Relayout and RelayoutLayoutInvariance pass
on net10.0 and net8.0. Two EXISTING tiers fail there -- PowerFast (385 of 7,902) and Binary_DivModPower (2 of
2,371) -- and fail identically on the unchanged HEAD 44c1af4 (cloned and run alone): the power_fast tier pins
win-amd64 ucrtbase pow bits (e.g. 757.76...**2 is pow's ...03c2) that glibc's correctly rounded pow does not give.
Not touched here; the tier needs its libm-dependent pow cells host-pinned.
* Recomputing every portable variant with Linux NumPy 2.4.2: 82 differ (NaN tokenized), all transcendental / hypot /
pow cells (Linux NumPy's SVML paths and glibc round some of them otherwise, also for value-preserving layouts of the
same values, so Linux NumPy is itself layout-dependent there); NumSharp's Linux replay of the whole tier passes.
* Coverage join: coverage/test_oracle_evidence.py 15 passed; generate_coverage.py writes (strict join OK).
DOCS
----
Fuzz/README.md: section "The relayout tiers", the regeneration line, a Table 2 row. .claude/CLAUDE.md: a bullet in the
Differential-Fuzz Pipeline section.
…the coverage inventory listed as available but unproven (np.linalg's Array-API forms, the histogram family, the memory predicates, the text formatters, the ndarray members, the finfo / iinfo / dtype / flags / poly1d attributes, the np.broadcast / flatiter / np.nditer members, the print and buffer options, frompyfunc and the inventory's aliases): 15,141 gate cases at zero excuses plus 909 cells carved exactly under 40 gap ids, and the gaps they found (silent wrong values, crashes, refusals, error texts, two leaks) recorded, not fixed
Why
---
The coverage artifact (coverage/generate_coverage.py over NumSharp.Tools.ApiInventory) joins every committed NumPy
oracle contract onto its API row. Before this commit it listed 40 headline APIs, and about 200 object members, that
NumSharp exposes but no contract proves. Most were "sibling-owned" in OracleSurfaceCoverageTests because the operand
corpus had no form for their inputs: a function argument, two views of one buffer, a shape list, an option dict, an
object stepped k times. After this commit no in-scope available row lacks a contract (generate_coverage.py:
"0 available headline APIs lack contracts"; oracle-verified headline 537/560, expanded 1078/2519).
What
----
test/oracle/gen_surface_oracle.py (new) writes corpus/surface.jsonl and corpus/gaps/surface_gaps.jsonl. Inputs the
corpus could not express ride params, and the C# side maps each back to the same call:
* a FUNCTION argument is a name from a fixed menu (APPLY_FUNCS, OVER_FUNCS, FROMPY_FUNCS in the generator;
AlongAxisFunc, OverAxesFunc, FromPyFunc in the registry);
* two VIEWS of one buffer are step recipes ("s" slice string, "T", "r" reshape, "sw" swapaxes, "rev") applied to one
owner operand (shares_memory, may_share_memory, ndarray.base);
* an object's state is its construction params plus k steps (np.broadcast, flatiter, np.nditer);
* global options are set and restored inside the case (print options, buffer size).
Every array operand meets the relayout tier's ndarray kinds (C / F / permuted / stepped views at an offset, reversed,
stepped-and-reversed, read-only, broadcast, 0-d views at an offset, empty views inside a buffer), every API its
refusals with NumPy's exception type and text.
Sections (15,141 kept cases, 135 op keys; nditer.close is carved whole):
linalg np.linalg.cross / diagonal / outer / trace (the Array-API forms) over 13 dtypes and 13 kinds
histogram histogram, histogram_bin_edges, histogram2d, histogramdd: int / estimator / edge bins, range, density,
weights, degenerate samples, every refusal
memory array_equiv; shares_memory / may_share_memory (10 view pairs x 5 max_work x 5 dtypes); broadcast_shapes
apply apply_along_axis / apply_over_axes (13 dtypes x the menus), bmat (nested lists, strings + ldict / gdict)
text array2string (every option), format_float_positional / scientific (3 widths x 12 values x every option,
plus a rounding grid: carry / round-down / tie values x precision x trim), base_repr, binary_repr,
typename, datetime_data
ndarray tolist (Python repr), tofile (raw and text), setfield, base, dtype, shape, device, to_device, data
(the memoryview, 12 attributes)
attrs finfo / iinfo (names and spellings), dtype (21 attributes x 24 specs), ndarray.flags (14 x 6 dtypes x 21
kinds), poly1d (coeffs and aliases, order, deriv, integ, variable)
objects np.broadcast (index / iters / reset after k steps), flatiter (index / coords after k steps, copy, base),
np.nditer (23 members x 9 configs x 14 operand sets after k steps, refusals, closed and delayed
iterators), set_printoptions / printoptions / get_printoptions, setbufsize / getbufsize, dtype.type,
frompyfunc
aliases numpy.cumulative_sum / cumulative_prod (the inventory credits np.cumsum / np.cumprod), numpy.shape
Harness (test/NumSharp.Tests.Oracle/Fuzz):
* OpRegistry.Surface.cs + OpRegistry.Surface.Objects.cs (new): the adapters, reached from the default branches of
ApplyExtended / ApplyTuple / ApplyText / ApplyDtype and the ndarray.* instance dispatchers. A converter formats or
wraps NumSharp's answer (PyRepr: CPython's float / complex / list / tuple repr; a live iterator view is copied; a CLR
element type becomes NumPy's class repr) and never repairs it. apply_along_axis / apply_over_axes run in an
NDScope (their callbacks' results are the caller's), poly1d temporaries are disposed, view recipes dispose their
intermediates and results (an empty slice owns a buffer: the leak sweep caught the undisposed ones).
* FuzzCorpusTests.Surface.cs (new): Surface ([FuzzMatrix]) and Surface_KnownGaps ([OpenBugs], RunGapCorpus).
* MisalignedRegistry: valueclass "surface" returns null (zero excuses, error cases included).
* MinCases["surface.jsonl"] = 14400.
* OracleSurfaceCoverageTests: 24 np and 4 ndarray SiblingOwned entries and 5 property entries retired (each now a
direct op), with dated notes.
* OracleCoverageStrengthTests: 14 FixedDtypeOps reasons for the param-axis APIs (base_repr, binary_repr, typename,
datetime_data, broadcast_shapes, get/setbufsize, get_printoptions, the float formatters) and the finfo attributes
whose kept spread is narrow until SU11-SU14 are fixed (self-retiring).
* UndisposedIntermediateTests.KnownEscapes: two leaks found by the sweep (below).
coverage/:
* oracle_map.json: prefix rules linalg / finfo / iinfo / dtype / poly1d / broadcast / flatiter / nditer.
* overrides.json: numpy.cumulative_sum / cumulative_prod marked partial with an accurate note (NumSharp has no such
functions; np.cumsum / np.cumprod lack include_initial and flatten N-D input without axis), the stale ndarray.data
note replaced (it returns np.MemoryView now; the buffer protocol itself has no C# counterpart).
* test_oracle_evidence.py: the np.linalg Array-API rows are now credited by their own linalg.<name> keys only (the
bare np.* keys still never credit them).
test/oracle/gen_relayout_oracle.py: skips the surface tier (its vehicle params are no plain NumPy call); regenerating
the relayout tiers stays byte-identical.
What it found (40 ids, carved exactly: replayed uncarved, every carved case diverges and no kept case does, on
net8.0, net10.0 and Linux x64; 37 public-call repros in test/NumSharp.Tests/OpenBugs.Surface.cs, all failing):
Silent wrong values
SU38 np.cumsum / np.cumprod of an integer input with dtype=complex128 return the input's raw bytes as complex values
SU39 np.cumsum(x, dtype=bool) stores counts (2, 3, ...) in the bool result; NumPy's bool add is logical OR
SU12 np.finfo(float32 / float64 / complex128).minexp is one too high (-125 / -1021; NumPy -126 / -1022)
SU13 np.finfo(float16 / float32).resolution is the decimal 10**-p, not that value rounded to the dtype
SU14 np.finfo(float32).tiny / smallest_normal is 1.17549435e-38, not exactly 2**-126
SU20 np.broadcast.iters restart at element 0; NumPy's share the broadcast's cursor
SU25 np.nditer in K order over an F operand plus an operand broadcast along an axis walks C order (NumSharp sorts
axes by the smallest stride over all operands; NumPy lets the stride-0 operand abstain): other values,
chunks and itviews
SU31 np.nditer.remove_multi_index re-sorts the axes by stride and coalesces (order='F' over C becomes a C walk; a
broadcast row's stride-0 axis moves innermost)
SU6 the float formatters print .NET's Infinity. / NaN. / I.nfinitye+07 (or throw) for inf / nan
SU18 the float formatters keep the zeros a precision cutoff carries into (1.99999 at 2 digits is '2.00'; NumPy '2.')
SU19 format_float_scientific with trim='-' drops the point after zero-only fraction digits ('1e-01'; NumPy
'1.e-01')
SU9 format_float_positional ignores fractional=False
Crashes where NumPy computes
SU5 np.apply_along_axis over a one-element np.split child broadcast to 2-D (every stride 0 at Shape.offset k):
ArgumentOutOfRangeException (Parameter 'length')
SU30 np.nditer.remove_axis under order='F' or over an F operand: IndexOutOfRangeException
Refusals and error texts
SU4 histogram of complex with an estimator, SU8 negative formatter precision, SU10 an out-of-range Python int in
ndarray.setfield, SU15 iinfo(bool), SU28 np.nditer reads past the end, SU32 a delayed-buffer read before reset,
SU33 remove_axis without multi_index, SU34 enable_external_loop with a multi-index: NumPy refuses, NumSharp
computes. SU22 set_printoptions(precision=-1) over integers and SU35 the dtype name 'cdouble': NumSharp refuses,
NumPy computes. Texts differ for SU1 / SU2 linalg.cross, SU11 finfo, SU16 iinfo, SU29 the closed nditer, SU40
np.cumprod's axis.
Reporting
SU3 complex histogram edges are float64 (NumPy complex128), SU17 dtype.newbyteorder()'s repr, SU21
get_printoptions lacks formatter / override_repr, SU23 edgeitems=0 summarization picks the format from the last
element (NumPy's a[-0:] is the whole array), SU24 the datetime dtype.type is null, SU26 np.nditer.shape lists axes
outermost first (NpyIter_GetShape innermost first), SU27 c_index / f_index stop NumSharp's coalescing.
API shape (no repro: not a defect of a call)
SU7 the float formatters take only a double, so float16 / float32 values print with float64 digits; SU36 / SU37
numpy.cumulative_sum / cumulative_prod do not exist (np.cumsum / np.cumprod lack include_initial and flatten N-D
input without axis).
Leaks (KnownEscapes, 1 pooled buffer per call)
np.MemoryView.tobytes / hex of a non-contiguous array never dispose their C-order copy (every dtype);
np.array_equiv with an empty operand reaches the comparison operators' empty-left leak (relayout RL06).
Validated clean: tolist (incl. 0-d views at an offset), tofile, shares_memory / may_share_memory, broadcast_shapes,
bmat, array2string, base_repr, binary_repr, typename, datetime_data, the ndarray.flags attributes, poly1d,
np.linalg diagonal / outer / trace, histogram2d / histogramdd, frompyfunc, getbufsize / setbufsize.
How the carve is drawn: fixed predicates where the gap is a fixed set, and a model of the bug where it is not - the
float formatters on the widened double with NumSharp's Dragon4 carry and trim rules (_ns_format_model,
_rounding_carries), np.nditer's C walk for an F + broadcast set via NumPy with order='C', the edgeitems=0 text from the
last element (_edge0_model). Platform pins left out so the tier is portable: memoryview.format and dtype.char /
dtype.num of the C-long-sized integers, the C-long spellings and 'longdouble'.
Verification
* Windows: NumSharp.Tests.Oracle 276 passed / 1 skipped on net10.0 and net8.0 (OpenBugs / HighMemory excluded);
Surface_KnownGaps reports 909/909 open on both; NumSharp.Tests 18,401 passed (net10.0); the 37 new repros fail on
both frameworks; the leak sweep (UndisposedIntermediateTests + LeakSurfaceCoverageTests) passes.
* Linux x64 (WSL clone with this change overlaid): Surface passes and 909/909 carved cases still diverge. PowerFast
(385/7902) and Binary_DivModPower (2/2371) fail on Linux on the PARENT commit too (pristine clone of 4741a81):
pre-existing, not caused by this change.
* coverage: generate_coverage.py reports 0 available headline APIs without contracts; pytest coverage/ 32 passed.
* Generators are deterministic: gen_surface_oracle.py reruns byte-identical; gen_relayout_oracle.py regenerates the
committed relayout files byte-identical.
Regenerate: python test/oracle/gen_surface_oracle.py [--uncarved DIR] (needs numpy==2.4.2). Redraw recipe in
test/NumSharp.Tests.Oracle/Fuzz/README.md "The surface tier".
…he 51 numpy.ma functions the coverage inventory still listed as available but unproven (creation wrappers, joins, fill values, predicates, ids, hard masks, masked_object, putmask, convolve / correlate / cov / corrcoef, the edge finders, the compressors, ndenumerate, apply_*): 24,021 gate cases at zero excuses plus 10,123 cells carved exactly under 37 gap ids, and the gaps they found (unsafe casts, fill values never converted, dropped masks, a rowvar bug, crashes, refusals, error texts, the result form) recorded, not fixed Why --- After the surface tier (2de4328) the coverage artifact (coverage/generate_coverage.py) still listed 51 numpy.ma functions that NumSharp exposes (np.ma.<name>) without a NumPy contract. Seven of them were even registered in OpRegistry.Ma.cs (diagflat, dstack, column_stack, resize, getmask, putmask, hsplit) but no corpus case ever reached them: gen_ma_oracle's putmask block described two different arrays as (base, view), so layout_catalog.describe raised into the block's skip, and every catalog 2-D layout has an odd last axis, so the hsplit block never ran. After this commit the expanded oracle-verified count is 1128/2519 (was 1078); only numpy.ma.polyfit (lstsq needs a LAPACK backend), np.nditer close / debug_print, poly1d r / roots and the bit generators' lock remain unproven among the non-class rows. What ---- test/oracle/gen_ma_surface_oracle.py (new) writes corpus/ma_surface.jsonl and corpus/gaps/ma_surface_gaps.jsonl. Every masked operand is a DATA view in one of the relayout tier's kinds (C, F, permuted, stepped and C views at an offset, reversed, stepped-and-reversed, read-only, broadcast, one-element 1-D, 0-d, empty, an empty view inside a buffer) plus a C-contiguous mask in one of eight patterns (nomask, an explicit all-False mask, mixed, all, first, last, alternating, runs). NumPy computes each record on masked arrays rebuilt from the descriptors NumSharp reconstructs. Inputs with no operand form are params: * a FUNCTION argument is a name from a fixed menu (ALONG_FUNCS, OVER_FUNCS, FROMFUNCTION_FUNCS in the generator; AlongAxisMaFunc, OverAxesMaFunc, FromFunctionFunc in the registry); * a Python scalar is a JSON value typed as the C# literal it stands for (MaPyValue: int -> long, float -> double, bool, complex -> {"re", "im"}, None -> null, str); * an object's state is steps applied before the call (set a fill value, harden the mask, assign through the indexer). Sections (24,021 kept cases, 50 op keys): create arange (every overload, dtype=), zeros / ones / empty (7 shapes x 14 dtypes, order=), the *_like forms (13 dtypes x shapes x data kinds x mask patterns, dtype=), identity, indices (sparse=), frombuffer (13 dtypes x count / offset in and out of range), fromfunction (7 menu functions); empty / empty_like compare a "nodata" facet (the uninitialized data zeroed on both sides) join column_stack, dstack (8 shape pairs x 6 dtype pairs x mask pairs, three operands), hsplit (sections and index lists), diagflat (k), resize (8 shapes incl. () and a negative dimension) fill default_fill_value, minimum_fill_value / maximum_fill_value, set_fill_value (13 dtypes x 19 values), common_fill_value (8 dtype pairs x 9 fill-value pairs) query getmask, nonzero, ndim, shape, size (axis=), is_mask / isarray / isMA / isMaskedArray (8 argument forms), ids (the byte delta between a view and its array), harden_mask / soften_mask (returns self, the flag, five assignments under a hard and a re-softened mask), masked_object (fill_value facet, shrink=) putmask a Python int / float / bool / complex, a full array, a masked one, a broadcast row, a short array, another dtype, the masked singleton, soft and hard targets products convolve / correlate (9 dtypes x 4 length pairs x 7 mask pairs x mode / propagate_mask, the refusals), cov / corrcoef (rowvar, bias, ddof, allow_masked, y) on values whose every unmasked mean is exact (_exact_cov_values), so any summation order gives NumPy's bits edges clump_*, flatnotmasked_*, notmasked_* (axis incl. out of range), compress_nd (tuple axis) / compress_rowcols, ndenumerate (compressed=False), every mask pattern x 8 shapes incl. 0-d and empty apply apply_along_axis (8 menu functions x every axis incl. out of range), apply_over_axes (5 menu functions x int and list axes) Harness (test/NumSharp.Tests.Oracle/Fuzz): * FuzzCorpusTests.Ma.cs: RunMaCorpus's per-case body moved into ReplayMaOne (unchanged logic), shared with the new RunMaGapCorpus (one verdict per carved case, grouped by gap id); two result kinds, "tuple" (plain arrays: ma.nonzero) and "text" (a normalized Python repr: a NumPy scalar or 0-d array as its .item(), a None slice step as 1); an error case keeps its kind and CheckMaError dispatches by it; valueclass "masurface" is strict (none of the harness's own excuses run); RunMaCorpus honors NUMSHARP_ORACLE_FAILURE_DUMP; NUMSHARP_MA_SKIP_FORM=1 (diagnostic only) skips the form comparison so a replay shows the divergences a form carve could hide; MaMinCases["ma_surface.jsonl"] = 22800. * OpRegistry.MaSurface.cs (new): ApplyMaskedSurface / ApplyMaskedText / ApplyMaskedArrayTuple and their helpers. A NumPy call the C# signature cannot express throws NotSupportedException naming the missing piece (carved as an API-shape gap); the adapters dispose exactly what they build (no ambient scope that would hide a NumSharp leak from the sweep), except around the apply_* callbacks, whose results are the caller's by np.ma's contract. * OpRegistry.Ma.cs: the default branch routes to ApplyMaskedSurface; getmask takes "plain", putmask goes through MaPutmask (values from a param or the third operand, "hard" hardens first), isMaskedArray takes "arg", hsplit takes an index list. * FuzzCorpusTests.MaSurface.cs (new): MaSurface ([FuzzMatrix]) and MaSurface_KnownGaps ([OpenBugs]). * MisalignedRegistry: valueclass "masurface" returns null (the ma harness strips "ma." before classifying, so the plain ops' branches would otherwise meet convolve / cov / ... here). * OracleSurfaceCoverageTests.MaSiblingOwned: 50 entries retired (each now a direct op); polyfit, moveaxis and shrink_mask remain. * UndisposedIntermediateTests.Families.cs: the masked sweep dispatches by result kind (text, tuple, masked_tuple). * UndisposedIntermediateTests.KnownEscapes: ma.fromfunction = 1 (the gt2 menu callback over an empty grid reaches the comparison operators' empty-left leak, relayout RL06). test/oracle/gen_ma_oracle.py: a comment on the putmask block that has never emitted a case (its output is unchanged: ma_select.jsonl regenerates byte-identical). What it found (37 ids, carved exactly: replayed uncarved, every carved case diverges and no kept case does, on net10.0; the gap replay reports 10,123/10,123 open on net8.0, net10.0 and Linux x64; 30 public-call repros in test/NumSharp.Tests/OpenBugs.MaSurface.cs, all failing on both frameworks for the documented reason) Silent wrong values MS11 ma.putmask casts unsafely (2.5 into an int64 array writes 2); NumPy's np.copyto(where=mask) is same_kind and raises TypeError (float into int, complex into float, the masked singleton's 0.0 into int, int into bool) MS31 a hard-masked setitem of several elements casts unsafely; NumPy's hard path is the same same_kind copyto MS12 ma.set_fill_value stores the raw value (1.5 stays 1.5 on an int array, 7 stays 7 on a bool array, 'abc' is accepted); NumPy converts to the dtype or raises TypeError MS13 ma.common_fill_value compares raw C# values (the long 3 != the double 3.0) where NumPy compares converted ones MS7 ma.masked_object never sets fill_value=value (NumSharp keeps 1e20) MS3 zeros_like / ones_like / empty_like drop the prototype's mask; NumPy keeps a copy MS4 ma.resize to 0-d keeps the mask; NumPy sets it only when the result has dimensions MS37 ma.cov / ma.corrcoef with a 1-D x and y under rowvar=False decide rowvar after concatenating y: one variable of 2N observations (cov 0-d, corrcoef throws) where NumPy has two MS34 ma.corrcoef's masked slots hold +0.0; NumPy's in-place corr /= outer(std, std) keeps cov's -0.0 there MS36 ma.ids reports the buffer start for a strided or reversed view (and any view of a view at an offset) MS17 ma.is_mask(a masked bool array) is False (NumPy reads the dtype: True) MS14 minimum_fill_value / maximum_fill_value of a bool array are True / False; NumPy's are the ints 1 / 0 (a bool array plus the fill promotes to int64 in NumPy) Crashes and refusals where NumPy computes MS33 ma.corrcoef of one variable throws ArgumentException (diag of a 0-d); NumPy returns the masked singleton MS35 ma.convolve of a 0-d first operand throws; NumPy reads it as one element (correlate's 0-d and both 2-D texts differ too) MS22 ma.frombuffer rejects a buffer whose whole length is not a multiple of the item size before count / offset select a range (count=0 is an empty array in NumPy) MS26 notmasked_edges of a 0-d array with an axis raises AxisError; NumPy returns [(), ()] MS15 minimum_fill_value / maximum_fill_value of a dtype throw NotSupportedException NumPy's refusals and texts MS6 apply_along_axis over a narrow-int result (NumPy sets the default fill 999999, which does not fit, and raises), MS8 masked_object with a value that does not fit the dtype, MS23 nonzero of a 0-d array, MS24 clump_* of a 0-d or empty array with a mask, MS28 compress_nd / compress_rowcols with an out-of-range axis (NumPy validates even without a mask), MS30 hsplit into unequal parts (NumPy's _fromnxfunction_single wraps the parts in one masked array): NumPy raises, NumSharp computes. MS21 arange's zero step (ZeroDivisionError vs ArgumentException), MS27 notmasked_contiguous's out-of-range axis (IndexError vs AxisError), MS32 the setitem broadcast errors: types / texts differ. Reporting (the result form) MS1 convolve / correlate / cov / apply_along_axis drop an all-False result mask to nomask (NumPy's masked_array(data, mask=mask) keeps it); MS2 column_stack / dstack / hsplit / diagflat keep nomask where NumPy's _fromnxfunction builds a full mask; MS9 masked_object keeps an all-False mask where NumPy shrinks to nomask; MS5 apply_over_axes returns a masked array where NumPy returns a plain ndarray; MS25 flatnotmasked_edges / notmasked_edges of an empty array return None (NumPy: array([0, -1])). API shape (no repro) MS10 masked_object shrink=, MS16 default_fill_value takes a DType only, MS18 ma.size axis=, MS19 ma.ndenumerate compressed=, MS20 arange dtype= / a lone float stop / the creation wrappers' order= / indices' sparse=, MS29 compress_nd's tuple axis. Validated clean: flatnotmasked_contiguous, fromfunction, getmask, identity, isMA / isarray / isMaskedArray, ndim and shape on every case; the rest outside their carved cells. How the carve is drawn: fixed predicates over the case (op, params, operand kinds and masks, NumPy's recorded outcome) and two models where the gap is not a fixed set (NumSharp's raw stored fill values for set_fill_value / common_fill_value; which ids views re-seat the storage address). Hidden divergences checked: replayed with NUMSHARP_MA_SKIP_FORM=1, the form-carved cases (MS1-MS5, MS9) show only the mask divergences their own ids describe. Data trap fixed on the way: a bool cov / corrcoef input undoes the exact-mean property (True / False counts over an odd number of observations), so its sums depend on the GEMM order; the products section uses int16 instead. Verification * Windows: NumSharp.Tests.Oracle 277 passed / 1 skipped on net10.0 and net8.0 (OpenBugs / HighMemory excluded); MaSurface_KnownGaps 10,123/10,123 open on both; NumSharp.Tests 18,401 passed (net10.0); the 30 repros fail on both frameworks with the documented message; the leak sweep (UndisposedIntermediateTests) passes. * Linux x64 (WSL clone of journey4 with this change overlaid): MaSurface passes, the gap replay is 10,123/10,123 open; PowerFast and Binary_DivModPower fail there on the parent commits too (pre-existing, recorded in 2de4328). * coverage: generate_coverage.py reports 0 available headline APIs without contracts and the expanded oracle-verified count 1128/2519 (was 1078); pytest coverage/ 32 passed. * Generators are deterministic: gen_ma_surface_oracle.py reruns byte-identical; gen_ma_oracle.py select regenerates ma_select.jsonl byte-identical. Regenerate: python test/oracle/gen_ma_surface_oracle.py [--uncarved DIR] (needs numpy==2.4.2). Redraw recipe in test/NumSharp.Tests.Oracle/Fuzz/README.md "The numpy.ma surface tier".
…s, np.nditer's iteration protocol and close()'s write-back contract, driven by the coverage inventory's remaining rows; 15,482 gate cases at zero excuses plus 986 cells carved exactly under 42 gap ids, and the gaps it found (SU41 an updateifcopy operand refused without buffering, SU42 the iteration protocol one element ahead, which loses a buffered readwrite loop's last write per buffer, and NDArray.SetAtIndex<T>'s Release-mode memory corruption) recorded, not fixed Why --- The coverage inventory (coverage/generate_coverage.py -> coverage/generated/coverage.json, summary.md) still listed rows as available but unproven after the surface tier (2de4328) and the numpy.ma surface tier (b08d075). This pass adds contracts for the ones a NumPy-observable contract exists for, and records why the rest stay unproven. As before: coverage and gaps only; no library code changes. Inventory after this commit (python coverage/generate_coverage.py): headline 537 / 560 oracle-verified (every available headline row has a contract; the other 23 are not implemented), expanded 1131 / 2519 (44.9 %), 659,601 contracts; `pytest coverage/` 32 passed. Rows still available but unproven, by design: * numpy.ma.polyfit: computes through lstsq, which needs a LAPACK backend (the ma harness runs managed); * numpy.nditer.debug_print: a C-level dump to stdout, no return value to compare; * numpy.random.{BitGenerator,MT19937,PCG64,PCG64DXSM,Philox,SFC64}.lock: a threading.Lock; the random-API oracle exempts it ("a threading primitive; nothing NumPy-observable", gen_random_oracle.py fam_bitgen_base). What is new in the tier (test/oracle/gen_surface_oracle.py) ----------------------------------------------------------- * poly1d.r / poly1d.roots: np.roots(coeffs) over coefficient sets that keep it off the companion-matrix path (no LAPACK): a constant, a linear term with a zero root, a power of x (trailing zeros become zero roots), leading zeros stripped, negatives, the zero polynomial, all-zero coefficients, x**5 - crossed with float64 / int64 / complex128 / float32 / int8 and the c, c_offset, reversed and strided_neg kinds. * np.nditer's iteration protocol: iterindex / value (and multi_index under the multi_index flag) read after k Python next() calls (k = 1, 2, 3, 6) for a plain, a multi_index and a buffered float32 -> float64 iterator. C# drives the same protocol through the iterator's enumerator (MoveNext). NumPy's __next__ (npyiter_next in nditer_pywrap.c) advances ON ENTRY once started, so after one next() iterindex is 0 (probed against 2.4.2). * close()'s write-back contract (nditer.close with params.writeback): an iterator over the operand, x = x * 2 + 1 written through every element or chunk it yields, then the operand read before close() ("before") or after two close() calls ("after"). Configs: updateifcopy (unbuffered copy), buffered, buffered external loop (buffersize 4), plain float32 / float64 / int64 / int16 / uint8 operands, an int32 operand under casting unsafe and same_kind - crossed with the c, f, reversed, strided_offset, c_offset and readonly kinds. _nditer now passes buffersize. * Two gap ids, carved by rule in _gap_of: - SU41: np.nditer refuses a readwrite updateifcopy operand of another dtype without buffering ("Casting between different dtypes requires the BUFFERED flag"), where NumPy iterates a copy that close() writes back (and raises its own same_kind TypeError for an int32 operand) - 30 cells; - SU42: the iteration protocol publishes the element and THEN advances: after k next() calls iterindex / value / multi_index are one element ahead, and in a buffered readwrite loop the advance flushes the buffer before the loop body writes its last element, so that write never reaches the operand (a = [1, 3, 5, 7, 9, 5] where NumPy has [1, 3, 5, 7, 9, 11]) - 47 cells. The one protocol cell that agrees (value after the sixth next() of a buffered iterator) stays in the gate. - SU29 now applies only to the "after" member-read facet, not to the write-back facet. Result: 16,468 cases -> surface.jsonl 15,482 over 139 op keys (was 15,141 over 135) + gaps/surface_gaps.jsonl 986 under 42 ids (was 909 under 40). The carve is exact: replayed uncarved, carved 986 = failed 986, over 0, under 0. Harness (test/NumSharp.Tests.Oracle/Fuzz) ----------------------------------------- * OpRegistry.Surface.cs: poly1d.r / poly1d.roots (the polynomial is disposed; the roots array is the caller's). * OpRegistry.Surface.Objects.cs: SurfNditer passes buffersize; NditerMember consumes params.next through the enumerator and reads multi_index; NditerClosed routes params.writeback to NditerWriteback, which writes each element through SetElement - each dtype with its OWN type, because NDArray.SetAtIndex<T>(T, long) is unchecked in Release (below). * FuzzCorpusTests.cs: MinCases["surface.jsonl"] 14400 -> 14700. * README: the tier's rows, counts and findings (np.nditer's write-backs, the SetAtIndex<T> finding). What it found ------------- * SU41 and SU42 above. SU42 also corrects .claude/CLAUDE.md: its np.nditer paragraph said NumPy's __next__ "returns self.value and then calls iternext()"; nditer_pywrap.c and a 2.4.2 probe say NumPy advances on entry (iterindex 0 after one next()). The copy() observation it drew stays true (a copy taken there yields from the second element on both sides), but the claim that publishing first is "safe" is not: a buffered readwrite loop loses writes. * Outside NumPy's contract, found by the harness itself: NDArray.SetAtIndex<T>(T, long) checks T against the dtype only with a Debug.Assert. In Release, a.SetAtIndex(7.5, 1) on a float32 array writes the double's 8 bytes at byte offset 8 - over elements 2 and 3 - and leaves element 1 unchanged; for the last element the write runs past the buffer's end. The first write-back adapter did exactly that and corrupted the operands it was meant to observe. Repros (test/NumSharp.Tests/OpenBugs.Surface.cs, now 41 tests; all fail on net10.0 and net8.0 for the documented reason): * Surface_Nditer_UpdateIfCopy_WritesBackAtClose (SU41); * Surface_Nditer_BufferedReadWriteLoop_KeepsTheLastWrite (SU42); * Surface_Nditer_AfterOneNext_IterindexIsTheReturnedElement (SU42); * Surface_SetAtIndexGeneric_WrongElementType_DoesNotWriteIntoTheNeighbours (Release-only; Inconclusive in Debug, where the Debug.Assert fires). Verification ------------ * Oracle suite (OpenBugs / HighMemory excluded, as CI): 277 passed on net10.0 and net8.0 (Windows). * Surface_KnownGaps: 986 / 986 carved cases still diverge across 42 ids (net10.0 and net8.0); MaSurface_KnownGaps: 10,123 across 37 ids. * Linux x64 (WSL, the working tree overlaid on a journey4 clone): Surface and MaSurface pass, both gap replays stay open at the same counts. PowerFast and Binary_DivModPower fail there, as they do on the parent commits (pre-existing, not this tier). * The 71 surface / numpy.ma surface [OpenBugs] repros all fail as documented (net10.0 Release). Trap recorded in CLAUDE.md: a corpus regenerated with --uncarved and copied into bin/.../Fuzz/corpus stays there after the carved regeneration (copy-if-newer keeps the newer bin file), so the gate replays the uncarved file - copy the carved corpus and gap files into both bin/Release/{net10.0,net8.0} after an uncarved replay.
…netGhActionsCicd - two projects that run NumPy, pandas, SciPy, scikit-learn, Matplotlib and PyTorch over NumSharp arrays through NumSharp.Interop.pythonnet: a console app that embeds the Python on the local machine (virtual environments included), and its asserted MSTest twin with the GitHub Actions workflow that runs it on Windows, Linux and macOS
WHAT
examples/NumSharpPythonnetLocalExample (console app, net10.0)
- Program.cs starts the engine, prints which Python it embedded and the versions of the six
libraries, runs the six examples and prints where Matplotlib saved its figure.
- PythonHost.cs finds a Python and owns the engine. Discovery order: the interpreter that
NUMSHARP_PYTHONNET_PYTHON names (binding), then .venv in the current directory or beside the
project file, then python3 / python on PATH (py on Windows). It runs the interpreter once to
read sys.version_info, sys.executable, sys.prefix, sys.base_prefix, INSTSONAME and LIBDIR as one
JSON object (ASCII-safe for any code page), checks the version against
PythonEngine.Min/MaxSupportedVersion, and finds the shared library of the BASE install
(pythonXY.dll, LIBDIR/INSTSONAME, the macOS framework binary, lib/libpythonX.Y.so).
- Libraries/{NumPy,Pandas,SciPy,ScikitLearn,Matplotlib,PyTorch}.cs are written like the twelve
tutorials of examples/NumSharp.Interop.pythonnet.Examples: a dynamic Python namespace, one
NDScope for each example, Py.GIL() around the Python parts, no output, and the outcome of each
line in the comment beside it.
- requirements.txt pins numpy 2.4.2, pandas 3.0.5, scipy 1.16.3, scikit-learn 1.9.1,
matplotlib 3.11.2, torch 2.13.0 and adds PyTorch's CPU-only index.
examples/NumSharpPythonnetGhActionsCicd (MSTest 3.8.3, net10.0, 18 tests)
- The same code as the six examples, with each comment turned into an assertion:
NumPyTests (3), PandasTests (4), SciPyTests (3), ScikitLearnTests (3), MatplotlibTests (2),
PyTorchTests (3).
- PythonEnvironment.cs ([AssemblyInitialize] / [AssemblyCleanup]) embeds exactly the interpreter
that NUMSHARP_PYTHONNET_PYTHON names. Unset under CI=true is a FAILURE with the line to add to
the workflow; unset on a developer machine is Inconclusive. It imports the six libraries before
the first test and writes a version table to GITHUB_STEP_SUMMARY. [assembly: DoNotParallelize]
because the MSTest template turns parallel test methods on.
- PythonTest.cs gives each test a new Python namespace and fails a test whose
NDArrayPythonInterop.LiveExports / LiveImports do not return to their value before the test
(it pumps both collectors for up to 15 seconds).
.github/workflows/example-pythonnet.yml
- ubuntu-latest, windows-latest, macos-latest; fail-fast off; triggers: workflow_dispatch, and
push / pull_request on master / main limited by a paths filter to the example, the interop
package and the workflow file.
- setup-dotnet 10.0.x, setup-python 3.12 with a pip cache keyed on the example's
requirements.txt, pip install into the job's interpreter, dotnet build, dotnet test with
NUMSHARP_PYTHONNET_PYTHON = steps.python.outputs.python-path, upload of TestResults (the .trx
report and the figure that the Matplotlib test attached).
- -p:WarningLevel=0 on the build step: inside this repository the project builds NumSharp from
source, and that build prints 557 warnings (measured, from-scratch Release build).
Both project files refer to src/NumSharp.Interop.pythonnet when that path exists and restore the
NuGet package NumSharp.Interop.pythonnet otherwise, so a copied folder needs no edit. Both projects
are in SciSharp.NumSharp.sln under "examples". Pointers were added to the interop package README,
the tutorials README and the "See also" list of docs/interop/pythonnet-numpy.md. .claude/CLAUDE.md
records the workflow and the facts below in "CI Pipeline".
WHY THESE DESIGN CHOICES (each one measured in this session)
- PythonEngine.ProgramName is set for EVERY interpreter, a virtual environment or not.
Windows, no program name: sys.executable is the .NET host, and a venv is not adopted (numpy was
imported from the base install). Linux, no program name: the engine takes its paths from the
first python3 on PATH; with another Python there (the system 3.12.3 against a 3.12.14 library)
the process died with a segmentation fault. With the name set, both platforms give the right
sys.executable, sys.prefix and packages whatever PATH holds. test/NumSharp.Tests.Interop's
PythonSession sets it for virtual environments only; this goes one step further.
- pythonnet 3.0.5's Runtime.Shutdown runs 87 full PyGC_Collect passes (3 + 20 + 20 attempts of two
passes each, plus one; the loop does not stop early: read from the v3.0.5 source). With the six
libraries imported one pass takes about 0.1 s, so the shutdown took 7.3 s. The Local example
runs gc.collect(); gc.freeze() first: 0.2 s, exit code 0, and the interop's export sweep still
brings LiveExports to 0. The test fixture keeps the plain shutdown (speed does not matter
there, and it is the sequence the interop suite runs on all three systems).
- torch.optim.SGD lazily imports about 800 more modules (torch._dynamo, sympy): 1.3 to 2.5 s in
plain Python, and the first run of that test took 19 s in the test host. The autograd example
updates the weights by hand: the same result (loss 1.2e-31), 0.05 s, and autograd stays visible.
- torch.__version__ is an instance of a str subclass: PyObject.As<string>() throws
("'TorchVersion' value cannot be converted to System.String"); ToString() is Python's str().
- The .NET app host (<app>.exe, testhost.exe) is not long-path aware on Windows. A module file
past 260 characters failed to import as "No module named
sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode" while python.exe and
"dotnet <app>.dll" imported it. Both READMEs say so.
- The pandas view example uses a Series that pandas owns alone. Under Copy-on-Write a write to a
frame does not reach a column that was taken from it before (the first version asserted that it
does, and failed).
- macOS, not run here: the macOS arm64 wheels of torch 2.13.0 and scikit-learn 1.9.1 each
contain a libomp.dylib (listed from the wheels). LLVM's runtime aborts when a second copy
starts (OMP: Error #15) unless KMP_DUPLICATE_LIB_OK is set, and sklearn/__init__.py sets it at
import (line 56 of 1.9.1). Both hosts import all six libraries before any computation.
VERIFIED
Windows 11 x64, CPython 3.12.12, pythonnet 3.0.5, the six pins:
- 18/18 tests, three runs in a row, Release; 18/18 in Debug; 0 warnings for both projects at the
default warning level.
- The Local example runs to completion in 5.6 s (exit 0) and writes a correct 400 x 300 figure.
- Fixture policies: with the variable unset the tests report Inconclusive; unset with CI=true,
every test fails with the message; a missing interpreter and a numpy-only environment fail
with the cause named.
- The leak check fails a test that parks a NumPy view in builtins ("LiveExports 0 -> 1").
- Discovery: .venv in the current directory, PATH, no Python, the Microsoft Store alias, a
named interpreter that does not exist.
Linux (WSL Ubuntu 24.04), GitHub's own Python 3.12.14 build (actions/python-versions) with the
libraries installed by pip as the workflow does (torch 2.13.0+cpu, 0 nvidia packages):
- The workflow's build and test commands with CI=true: 18/18, the job summary written, the
.trx and the PNG under TestResults; also 18/18 with another python3 first on PATH.
- The Local example by its README (python3 -m venv .venv, pip install, dotnet run): exit 0,
3.7 s for a second run; every discovery branch, including "started from another directory".
Copied out of the repository, both projects restore NumSharp.Interop.pythonnet 0.70.0,
NumSharp 0.70.0 and pythonnet 3.0.5 from nuget.org: dotnet run exits 0 and dotnet test is 18/18.
The pins resolve as binary wheels for Python 3.11, 3.12 and 3.13 on win_amd64,
manylinux_2_28 x86_64 and macOS arm64 (uv pip compile); without the CPU index the Linux
resolution adds 19 nvidia / cuda / triton packages.
NOT VERIFIED
The workflow has not run on GitHub: this commit is local. The macOS leg in particular was not run
anywhere. Its interpreter discovery is the logic the interop suite already uses on macOS runners;
the two-libomp fact above is the known risk.
…nstructor (arrays, matrices, "1 2; 3 4" literals, sequences), T / H / I / A / A1, the row / column / scalar rule of indexing, * as the matrix product and ** as the matrix power, the two-axis reductions and every member ndarray hands down, held to NumPy 2.4.2 by a 26,027-case zero-excuse oracle tier (values, result class, view / copy / owner, error type and text); np.asmatrix and np.bmat return NDMatrix; index-array and boolean-mask assignment take NumPy's value rules library-wide
WHY
NumSharp had no numpy.matrix: np.asmatrix returned a plain 2-D NDArray and np.bmat a plain array, so ported code
that relies on matrix semantics (m * m is the matrix product, m[0] keeps two axes, m.sum(0) is a row, m.I, m.H,
np.matrix("1 2; 3 4")) either did not compile or silently computed something else. The coverage inventory listed
the 83 numpy.matrix rows as missing.
NAMING (how numpy.matrix points to NDMatrix)
- The CLR type is NumSharp.NDMatrix (the house ND* prefix, as numpy.ma.MaskedArray is NDMaskedArray):
`public partial class NDMatrix : NDArray, IEnumerable` in src/NumSharp.Core/Matrixlib/.
- The NumPy names reach it: np.matrix(data, dtype=None, copy=True) (Matrixlib/np.matrix.cs, four overloads:
NDArray / string / Array / object), np.asmatrix(...) and np.bmat(...) return NDMatrix.
- coverage/object_surfaces.py maps the object surface: ObjectSurface("matrix", "Matrix objects",
"NumSharp.NDMatrix"), so every numpy.matrix.<member> row of the inventory is credited by the NDMatrix member of
the same name: 80 of 83 available (ctypes, dump and dumps are missing, as on ndarray).
DESIGN
- Matrix semantics follow the STATIC type. NumPy finds the subclass at run time; here the compiler binds the
members and operators declared on NDMatrix. The same object through an NDArray reference is a plain 2-D array
(* is elementwise there, an integer index drops an axis). Reason: every kernel and helper of the library is
written against NDArray, and run-time dispatch would change that code's meaning for a caller who passed a matrix
in. Consequence, documented as a divergence: the np.* functions return NDArray where NumPy returns a matrix
(np.sum(m, 0), np.transpose(m), np.linalg.inv(m)); use the member (m.sum(0), m.T, m.I) or np.asmatrix(result),
an O(1) view. NDMatrix.Clone() HIDES NDArray.Clone() for the same reason (as an override it made np.positive(m)
return a matrix while np.copy(m) did not).
- A 0-d NDMatrix is a scalar, not a matrix. The 15 C# scalar types convert implicitly to a 0-d NDMatrix, so
m * 2, m + 1.5 and m[mask] = 0 compile and bind the (NDMatrix, NDMatrix) operators; * with a 0-d operand scales.
- Result forms (NDMatrix.cs): ViewOf (a.view(matrix), never an owner), From (a result NumPy allocates as a
matrix: the matrix takes the result's storage over and owns when it did), FromWrapped (a ufunc result that
__array_wrap__ returns: a view of the hidden result, never an owner), FromIndexing (matrix.__getitem__'s rule;
an index-array result is a view of the gathered array, NumPy's wrap_out_array; a 0-d integer array index gives
an owning K-order copy, HAS_SCALAR_ARRAY). All apply the finalize rule Shape.AsMatrix() (View/Shape.Matrix.cs):
0-d -> (1,1), 1-d -> (1,N), more than two axes longer than 1 -> ValueError "shape too large to be a matrix.".
WHAT IS IN THE CLASS (69 member names, 18 operator symbols, 9 indexers, 15 scalar conversions)
- Construction (NDMatrix.Construction.cs): the four branches of matrix.__new__ in NumPy's order. A literal is
_convert_from_string plus a port of ast.literal_eval for one token (the four number bases, "_", floats,
imaginary numbers, signs, parentheses, True / False; SyntaxError and "malformed node" errors; new house
exception Exceptions/SyntaxError.cs). A sequence goes through PolySequence (NumPy's array coercion); with a
dtype each element is stored by PyArray_Pack (ReductionInitial.TryPack): a Python int out of range raises
OverflowError, a complex number into a real dtype raises TypeError.
- T, H, I, A, A1 (and getT / getH / getI / getA / getA1), mT, real, imag, shape.
- Indexing (NDMatrix.Indexing.cs): a 1-d result is a COLUMN when len(index) > 1 and isscalar(index[1]), else a
row; other ranks are returned as they are (m[newaxis] is a 3-d matrix object, as in NumPy). m[int, int] is a
0-d NDArray view. Index errors carry NumPy's texts.
- Assignment: ndarray's assignment gets the view of a subclass through its own __getitem__ for an index of
integers and slices, so the value is broadcast into the row or COLUMN: m[:, 0] = [7, 8, 9] raises "could not
broadcast input array from shape (3,) into shape (3,1)". One element stores through the dtype's setitem.
- * and ** (NDMatrix.Operators.cs): m * x is np.dot(m, asmatrix(x)) (a 1-d x becomes a ROW), x * m is
np.dot(x, m); __mul__ / __rmul__ / __imul__ / __pow__ / __ipow__ / __rpow__. Every binary operator has five
forms (matrix/matrix, matrix/array, array/matrix, matrix/object, object/matrix). The other operators are
ndarray's ufuncs, wrapped back into a matrix; reflected comparisons evaluate with the matrix as the first ufunc
operand (the order of NumPy's broadcast error text).
- Scaling is np.dot's scalar route (cblas_matrixproduct): the dtype is promote_types of both operands (not the
weak-scalar ufunc rule); for float32 / float64 / complex128 the result is a new C-contiguous owner and each
element is 0 + scalar * x (?axpy into zeros), so a product of -0.0 is +0.0, except a ONE-element result (the
direct product); every other dtype is the multiply ufunc, wrapped. A plain 0-d NDArray on the LEFT stands for a
NumPy scalar and is the elementwise ufunc. Three routes with one result: (1) a C-contiguous float operand of the
loop dtype runs ONE direct pass over raw memory (ScaleContiguous: the polynomial package's PolyAffineDouble /
PolyAffineSingle, multiply then add, never an FMA; GC.KeepAlive on the operands after the kernel); (2) another
operand of the loop dtype runs one fused np.evaluate pass from the measured crossover on (4,096 elements for a
float view, 512 for complex128); (3) the rest runs the multiply ufunc, then adds the zero in place with the same
raw kernel (a complex buffer as twice as many float64 values).
- Reductions (NDMatrix.Reductions.cs): sum / prod / mean / var / std / any / all / max / min keep two axes
(keepdims then _collapse), argmax / argmin / ptp use _align; (int axis, ...) returns NDMatrix, (int? axis = null,
...) returns NDArray (the 0-d scalar without an axis). A matrix reduction is always NumPy's ARRAY form (a
float16 mean rounds twice; std(dtype=int) raises the sqrt cast error also without an axis). With out= the loop
is promote_types(out.dtype, operand dtype) without the small-integer widening (reducelike_promote_and_resolve),
stored with an unsafe cast; var / std hand out to their LAST sum only.
- Methods ndarray hands down, each returning NumPy's class and ownership (NDMatrix.Methods.cs): reshape, ravel,
flatten, squeeze, transpose, swapaxes, diagonal, trace, take, repeat, compress, choose, clip, round (PyArray_
Round's class rules: a float matrix rounded to decimals != 0 is a plain array in NumPy), conj / conjugate,
astype, copy, view, cumsum, cumprod, dot, argsort, argpartition, byteswap, getfield, to_device; repr / str
("matrix([[1, 2], ...])" through NDArray.ReprClassName) and iteration over rows.
LIBRARY FIXES THIS NEEDED (they change NDArray)
- BREAKING: np.asmatrix and np.bmat return NDMatrix (they returned NDArray). `var m = np.asmatrix(a); m * m` is
now the matrix product and m[0] keeps two axes; type the variable NDArray or use .A for the array semantics.
np.asmatrix also returns a NEW view where it returned a 2-D operand itself (the array-object oracle's gap VF15,
retired: view_flags.jsonl regenerated, 7,294 gate + 610 carved cases, 46 gap ids). np.r_'s "r" / "c" directive
result object is an NDMatrix (typed NDArray by the indexer).
- BREAKING: NumPy's MapIter value rule for every index-array assignment (new NDArray.PrepareAssignedValue,
Selection/NDArray.Indexing.AssignedValue.cs; three call sites in the setter and the mask setter). A value with
more axes than the selection loses its leading axes when the trailing ones hold every element (a (1, N) matrix
row fits an (N,) selection; it threw), and a value that does not fit raises ValueError "shape mismatch: value
array of shape (1,4) could not be broadcast to indexing result of shape (2,)" BEFORE anything is written (it was
IncorrectShapeException with another text).
- BREAKING: a boolean mask of the array's own shape follows array_assign_boolean_subscript
(Backends/Default/Indexing/Default.BooleanMask.cs): a value with two or more axes is TypeError "NumPy boolean
array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions"; a 1-d value whose length
is neither 1 nor the true count is ValueError "NumPy boolean array indexing assignment cannot assign 3 input
values to the 6 output values where the mask is true" (also for a true count of 0). A prefix mask takes the
MapIter rule.
- np.dot(1-D, 2-D) and np.matmul with a 1-D operand return an OWNER (the inserted axis is dropped in place,
DefaultEngine.DropInsertedAxis; they returned a squeezed view). np.dot(N-D, 1-D) reports a mismatch in np.dot's
words ("shapes (3,4) and (5,) not aligned: 4 (dim 1) != 5 (dim 0)").
- DefaultEngine.VarStdNumPy gains outDtype / sumOnly for the matrix out= route (no behavior change for np.var).
- NDArray.Equals(object) no longer leaks its comparison array (found by the leak gate through NDMatrix.Equals).
ORACLE (test/oracle/gen_matrix_oracle.py -> corpus/matrix.jsonl; needs numpy==2.4.2; deterministic)
- 26,027 gate cases over 87 op keys ("matrix", "asmatrix", "matrix.<member>", Python's dunder names for the
operators; params.sym / params.via say which C# form the replay calls). 5,339 of them are NumPy errors, held to
type and text. Sections: construction (arrays of every rank, matrices, 62 literals, nested sequences, dtype
conversions), properties, getitem / setitem (integers, slices, new axes, ellipses, index arrays, masks, 0-d
arrays; values of every shape), * and the elementwise operators (matrix / array / sequence / Python scalar /
NumPy scalar on either side), **, the reductions (axis, dtype=, out= of seven dtypes, ddof), the methods, text
and iteration; 14 source layouts (C, F, transposed, stepped, reversed, a C view at an offset, read-only,
broadcast, a row, a column, 1 x 1, two empty shapes, square) and the dtypes.
- Three harness additions: an operand descriptor with "matrix": true rebuilds as an NDMatrix over the same storage;
expected.cls (matrix / ndarray / scalar: 18,221 / 349 / 1,866 cases) holds NumSharp to the class NumPy returned;
the memory facet of the array-object oracle (five flags, byte strides, shares / is each operand) is recorded for
every array result, so a matrix that owns where NumPy's is a view of the hidden ufunc result fails.
- Zero excuses (valueclass "matrix": MisalignedRegistry returns null). MinCases floor 24,700.
- Three replays, one per float scaling route: Matrix (production; asserts the direct pass ran),
Matrix_FusedScaling (NDMatrix.DisableDirectScale + FusedScaleMinElementsTestOverride = 2),
Matrix_TwoPassScaling (both single-pass routes off).
- Files: OpRegistry.Matrix.cs (the op-name -> NDMatrix call map), FuzzCorpusTests.Matrix.cs, FuzzCorpus.cs,
FuzzCorpusTests.Kinds.cs (class comparison), OpRegistry.cs / .Kinds.cs (routing), MisalignedRegistry.cs.
THE CARVE (corpus/gaps/matrix_gaps.jsonl, 172 cells, 11 ids; exact in both directions: every carved case
diverges, no kept case does - checked against an uncarved replay's failure dump)
The first replay had 2,441 divergences; fixes in NDMatrix and in the array library closed 2,269. Every one of the
172 left is an NDArray member or np.* function BELOW the class:
MX01 56 a[:, index_array]: NumPy's transposed MapIter view (F-contiguous); NumSharp a C-ordered array
MX02 36 np.cumsum / np.cumprod result layout for an F, transposed or broadcast-bool operand
MX03 15 ndarray.imag of a real F-contiguous array is C-ordered
MX04 5 ndarray.round of a broadcast operand: result layout
MX05 2 np.conjugate of a transposed strided / reversed complex128 view writes C order (reached through m.H)
MX06 16 reshape with -1 on a broadcast view throws NotSupportedException
MX07 8 a two-entry fancy assignment into a C-contiguous view at an offset throws IndexOutOfRangeException
MX08 15 np.prod in a cast loop: flat dtype=bool throws, flat complex128 / float16 loops return NaN or garbage
MX09 4 complex128 np.prod over a reversed view: the sign of a zero part
MX10 3 np.dot of an N-D operand with an M-D one computes in float64: complex128 loses its imaginary parts
MX11 12 np.repeat of a C-contiguous view at an offset reads from the parent's start
Recorded, not fixed. Seven have repros through public calls in test/NumSharp.Tests/OpenBugs.Matrix.cs (all
failing, as [OpenBugs]); MX01, MX06, MX09 and MX11 are pinned by older files. Matrix_KnownGaps ([OpenBugs]) reports
172 / 172 still open.
TESTS
- test/NumSharp.Tests/Matrixlib/NDMatrix.Tests.cs (42): NumPy's test_defmatrix.py ported, plus the design pins
(static typing, the scalar stand-in, ownership of each result form) and Scaling_EveryRoute_GivesTheUfuncBits (the
three scaling routes against np.add(np.multiply(x, s, dtype: loop), 0) at 70 x 70, bit for bit, with the route
counters). Two planted bugs (the direct pass adding -0.0; the complex add-zero pass covering half the buffer)
are caught.
- Eight older tests pinned the old IncorrectShapeException / texts of setter value mismatches and now hold NumPy's
ValueError / TypeError texts (BooleanMasking.MatrixTests, BooleanMasking.UsingTests, CombinedIndexing.MatrixTests,
Indexing.KernelRoute.Tests, np.asmatrix.Tests), each checked against NumPy 2.4.2.
- A route-equality probe (not committed): 30,378 comparisons of the three scaling routes with the ufunc
reference over float32 / float64 / complex128, nine layouts (a C-contiguous np.unstack child at an offset
included), both operand orders, scalars of every dtype that keeps the loop (0-d views included), special
values, 2 to 300,300 elements: 0 value / shape / dtype differences; 12 differ only in the payload of a NaN that
meets a NaN scalar (NumPy's comes from the BLAS ?axpy kernel, which no route models).
LEAK AND COVERAGE GATES
- LeakSurfaceCoverageTests: NDMatrix joins ObjectOwners and OperatorOwners (MatrixOperatorKeys maps the C#
operators to the corpus's dunder keys); a member is credited by its corpus key matrix.<name>.
LeakCatalogue.Matrix.cs measures the members no corpus row reaches (Clone, Equals, GetHashCode, GetEnumerator,
the indexers, argpartition, choose, getfield, to_device, the conversions, !).
UndisposedIntermediateTests.Properties.cs reads every property over three matrix targets.
- coverage/oracle_map.json: the prefix rule "matrix" -> numpy.matrix.{rest} and 37 explicit keys;
coverage/test_object_surfaces.py: test_matrix_members_map_to_ndmatrix.
PERFORMANCE (NPY/NS, both sides on two pinned P-cores, best of 9 rounds, every C# cell warmed in four rounds;
97 cells: 33 operations x 4 / 64 / 512 square, plus three literals)
- geomean 2.26. Construction 2.6-18x, m[i] / m[:, j] 5x, m + m 1.8-5x, m * 2.0 2.6x / 1.9x / 25x, m.sum()
2.9-6.2x, m * m at 4x4 6.6x, m.I at 4x4 8.6x, literals 8-13x.
- 31 cells are under 1.5x, each a cost of the array library below the class: two index arrays (m[rows, cols]
0.33-0.69x, its assignment 0.18-0.27x: the fancy delegate route); m > scalar from 4,096 elements (0.25-0.52x:
the scalar comparison is a scalar loop on journey4; vectorized on branch strings, plan X24 / X25); products at
64x64 without the OpenBLAS backend (m * m 0.58x, m ** 3 0.50x, m.I 1.12x); axis reductions (m.sum(0) 1.1-1.4x,
m.mean(0) and m.argmax(1) at 512 1.0-1.1x, m.cumsum(0) 1.2-1.3x); m[mask] (0.84x at 512); members at the
NDArray allocation floor (m.copy() 1.1-1.3x, m[i, j] and m.A 1.4-1.5x).
- A C# literal operand costs 0.2-1.2 microseconds more than a held 0-d operand (m * 2.0 at 256 elements: 1.5
against 0.58): the conversion's 0-d temporary is left to the finalizer, as for a * 2.0 on a plain array (plan
X26, open).
- Measuring trap recorded in the code and CLAUDE.md: a harness that warmed a cell once read 4.3 microseconds for
a fused np.evaluate call that takes 1.4 at tier 1, which first put the fused threshold four times too high.
DIVERGENCES FROM NUMPY (documented in the class remarks and .claude/CLAUDE.md)
- The static typing above (np.* functions return NDArray).
- A C# literal keeps its C# type (matrix(int8) * 2 is int32; NumPy's Python int gives int64).
- NumPy raises for m * np.array(2); NumSharp cannot tell that 0-d array from a NumPy scalar and scales.
- The typed setters take a matrix or a number (__setitem__(index, object) takes any value); no tuple axes.
- The object address in ast's error texts is zero; complex128 scaling by a complex scalar is the ufunc product,
which can differ from OpenBLAS zaxpy in the last bit.
DOCS
.claude/CLAUDE.md (a new "np.matrix ... NDMatrix" section, the creation list, the Core Source Files row, the
OpenBugs files entry, "The matrix tier" oracle bullet), test/NumSharp.Tests.Oracle/Fuzz/README.md ("The matrix
tier": schema additions, the three replays, the gap table, the rules for redrawing the carve).
VERIFIED (Release, after the last change)
- NumSharp.Tests (TestCategory!=OpenBugs&TestCategory!=HighMemory): 18,443 passed on net10.0, 18,442 on net8.0,
0 failed. One earlier full net8.0 run of the same sources reported a single failure that did not recur in six
more runs and was not identified (that run's output was not saved); the matrix and indexing classes passed 15
loops of 953 tests.
- NumSharp.Tests.Oracle: 280 passed, 1 skipped (RandomApiSoak) on both frameworks; Matrix_KnownGaps 172 / 172 open.
- NumSharp.Tests.Interop: 736 passed on both frameworks.
- python -m pytest coverage: 33 passed; coverage/generate_coverage.py --check: current.
…rojects, and two OpenBLAS example projects join them - NumSharpOpenBlasDefaultExample (one package reference, no configuration) and NumSharpOpenBlasAdvancedExample (a pinned OpenBLAS build, one thread, pinned kernels, results that reproduce bit for bit and equal NumPy 2.4.4's)
WHAT
1. The project files of the two pythonnet examples are simplified (owner's request).
examples/NumSharpPythonnetLocalExample and examples/NumSharpPythonnetGhActionsCicd had two
custom MSBuild properties (NumSharpInteropVersion, NumSharpInteropProject), a conditional
ProjectReference / PackageReference pair, and SignAssembly / IsPackable lines. Each file is now
exactly what `dotnet new` and `dotnet add package NumSharp.Interop.pythonnet` write: the
template's properties and one plain <PackageReference ... Version="0.70.0" />. The reference
line was written by `dotnet add package` itself.
Consequences:
- Both projects build against the PUBLISHED package, in the repository too. A build takes 2
to 5 seconds instead of 40 to 80, and prints 0 warnings, so the workflow's build step lost
its -p:WarningLevel=0.
- .github/workflows/example-pythonnet.yml no longer starts on changes under
src/NumSharp.Interop.pythonnet: a source change cannot change the result of a project that
restores the released package. It starts on changes to the example or to itself.
- The READMEs say what to copy in terms of the two commands, not of a reference switch.
- Inside the repository Directory.Build.props still strong-names the two assemblies (no
opt-out line any more); that produces no warning.
2. examples/NumSharpOpenBlasDefaultExample (console app, net10.0, new).
The project file is `dotnet new console` + `dotnet add package NumSharp.Interop.OpenBLAS`.
- Program.cs: OpenBlasEngine.Enable() as the first statement, then a status report
(OpenBlasEngine.Info, IsBundledLibrary, LapackAvailable).
- Products.cs: np.dot / np.matmul / np.inner / np.outer / np.vdot, a stack of matrices,
float32 and complex128, np.einsum / np.tensordot / np.linalg.multi_dot / matrix_power, and
an integer product that stays on NumSharp's own kernels. Silent; the outcome of each line
is in the comment beside it.
- LinearAlgebra.cs: solve, inv, det, slogdet, cholesky, qr, eigh, svd, lstsq, pinv,
matrix_rank, cond, norm, eig. It returns the largest residual of the identities it shows
(4.3e-15 on Windows and on Linux), which Program.cs prints.
- Speed.cs: one 1024 x 1024 float64 product by OpenBLAS and, after OpenBlasEngine.Disable(),
by NumSharp's own kernels, with the largest difference between the two results.
3. examples/NumSharpOpenBlasAdvancedExample (console app, net10.0, new).
- The project file pins the OpenBLAS build as an attribute of the same regular reference:
<PackageReference Include="NumSharp.Interop.OpenBLAS" Version="0.70.0"
OpenBlasVersion="0.3.31.188.0" />. That is the scipy-openblas release that NumPy 2.4.4
pins. A comment lists the other settings (OpenBlasSha256, OpenBlasPath,
OpenBlasDistribution, OpenBlasFeed, OpenBlasDelivery); the README explains each.
- Program.cs names no type of the package and sets NUMSHARP_OPENBLAS_BUNDLE_AUTOINSTALL=0
first, so that the first load of the library is the call in Backend.Start.
- Backend.cs: OpenBlasEngine.Enable(threads: 1, coreType: OpenBlasEngine.CoreType) with a
fallback for a processor that cannot run the Haswell kernels and for a library that was
loaded earlier; a report that reads the build's openblas.source.json marker.
- Reproducibility.cs: the SHA-256 of one float32 (512,784) @ (784,512) product with 1 thread
(twice), 2 threads, all threads, and NumSharp's own kernels.
4. Both OpenBLAS projects are in SciSharp.NumSharp.sln under "examples". Pointers were added to
src/NumSharp.Interop.OpenBLAS/README.md and to the "See also" list of
docs/website-src/docs/interop/openblas.md. .claude/CLAUDE.md records the regular-csproj rule,
the two new examples and the measured facts below.
FINDINGS ABOUT THE RELEASED PACKAGE (measured against NumSharp.Interop.OpenBLAS 0.70.0; reported,
not fixed here - the examples and their READMEs work around them and say so)
A. A package reference alone does not install the backend.
The backend is installed by a [ModuleInitializer], and the runtime runs it when it first
prepares a method that names a type of the package. NumSharp.Core never loads the assembly.
A program that references the package and calls only np.* functions:
- has TensorEngine.Blas == null before and after np.dot (the assembly is not even loaded),
- computes dot / matmul / solve / inv / det / slogdet / matrix_power(-1) on managed code,
- gets OpenBlasMissingBackendException from cholesky, qr, svd, svdvals, eig, eigvals, eigh,
eigvalsh, lstsq, pinv, matrix_rank, cond and norm with ord 2 / 'nuc'. The text of that
exception tells the user to reference the package, which the project already does.
The quick start of the docs works because its method names OpenBlasEngine.Info: the
initializer runs at the ENTRY of the first method that names the package (probed: the backend
is installed before that method's first statement; same result with tiered compilation off,
PGO off, QuickJit off, ReadyToRun off and JitStress=1). tools/verify_package_consumer.sh's
consumer names the type as well, so no gate sees the gap.
The default example therefore starts with OpenBlasEngine.Enable(), and its README has a
section "Name the package before you compute".
B. On glibc Linux the build-time pin cannot stage scipy-openblas 0.3.33 or later.
The fetch task selects the wheel with name.Contains("-" + tag + "."), where tag is the first
component of the platform tag of the manifest's wheel (manylinux2014_x86_64). From 0.3.33.0.0
the glibc wheels are named manylinux_2_27_x86_64.manylinux_2_28_x86_64, so the build fails:
NumSharp.Interop.OpenBLAS: expected exactly one scipy-openblas64 0.3.33.112.0 wheel for
platform tag 'manylinux2014_x86_64', found 0.
(measured on Ubuntu 24.04 for 0.3.33.112.0, the OpenBLAS of NumPy 2.5.0 / 2.5.1, and for
0.3.34.0.0). The same versions build, load and compute on Windows (0.3.33.112.0 and
0.3.34.237.0 were run: BLAS and LAPACK). 0.3.29.0.0 fails on glibc Linux for the mirror-image
reason: its compound tag lists manylinux_2_17 first. Versions 0.3.29.265.1 to 0.3.31.188.0
have a file with the expected tag for all eight RIDs, which is why the example pins
0.3.31.188.0.
C. Smaller facts that the examples and READMEs encode:
- Kernel request (coreType): applied when the call is the first load; accepted when the
loaded kernels are the same; InvalidOperationException when the library is loaded with
other kernels; PlatformNotSupportedException when the processor cannot run them ("SkylakeX"
on this machine); an UNKNOWN name raises nothing - OpenBLAS ignores it (read CoreName).
- OPENBLAS_CORETYPE and OPENBLAS_NUM_THREADS in the environment of the process work with no
code (Info showed "threads 3 ... Sandybridge").
- A build output holds the libraries of all eight RIDs: 201 MB. `dotnet publish -r <rid>
--self-contained false` gives 23 MB on Windows and 29 MB on Linux; the pinned library and
its marker are then beside the application, and the program runs from there.
- The released NumSharp package ships the NDW012 analyzer: example code that creates arrays
outside an NDScope gets a warning for each one. The examples open one scope per method.
- The download of a pin is 7 to 10 MB (the wheel), not the 20 MB of the extracted library.
PARITY, MEASURED (one Core i9-13900K; Windows 11 and WSL Ubuntu 24.04)
- The bundled library is byte-identical with the file inside the NumPy 2.4.2 wheel
(sha256 74a408729250596b0973e69fdd954eea07a70ff527a1dbaccf9ae21247b80377), and the pinned
0.3.31.188.0 library with the file inside the NumPy 2.4.4 wheel
(63c857e738469261263c764a36be9436ebdeaa272e340a828f42047a97131080). On Linux NumPy's copy has
other bytes (auditwheel rewrites the ELF header) and carries the hash of the staged file in
its name (libscipy_openblas64_-32a4b2a6.so).
- The float32 product of the advanced example (np.random.seed(7) inputs, which NumSharp's
generator reproduces) has the same SHA-256 in NumSharp and in NumPy:
threads pinned 0.3.31.188.0 = NumPy 2.4.4 bundled 0.3.31.22.0 = NumPy 2.4.2
1 d02ca290eb70d865 d02ca290eb70d865
2 67cdb5ec364fa7df 67cdb5ec364fa7df
4 e3433600e14af1cd
24 d16b7257b4864151 cc2acd3eb7be1ab1
NumSharp's own kernels give b5560c0b1370343c. The 1- and 2-thread hashes are the same on
Windows and on Linux (NumPy 2.4.4 on Linux was compared at 1 and 2 threads). Every hash
repeats from run to run.
- NumPy's pins, read from requirements/ci_requirements.txt at each tag: 2.2.6 / 2.3.0 ->
0.3.29.0.0; 2.3.2 to 2.3.4 -> 0.3.30.0.1; 2.3.5 / 2.4.0 / 2.4.1 -> 0.3.30.0.7; 2.4.2 ->
0.3.31.22.0; 2.4.3 -> 0.3.31.126.1; 2.4.4 -> 0.3.31.188.0; 2.5.0 / 2.5.1 -> 0.3.33.112.0.
VERIFIED
Windows 11 x64:
- All four projects build from scratch with 0 warnings and 0 errors (Debug and Release).
- pythonnet: 18/18 tests (Release with CI=true, and Debug); 18 Inconclusive without the
variable; the local example exits 0.
- OpenBLAS default: exit 0; 1024 x 1024 float64 matmul 8.3 ms against 89.8 to 190 ms for
NumSharp's own kernels across runs on a loaded machine; exits 1 with the package's reason
when no library can be loaded (NUMSHARP_OPENBLAS_USE_BUNDLED=0); works with the automatic
start turned off in the environment.
- OpenBLAS advanced: the build downloads, caches and stages the pin; "bundled False", "pin
scipy-openblas64 0.3.31.188.0, required: True", "kernels Haswell (pinned)"; exits 1 with
OpenBlasRequiredOverrideException's text when the pinned library is removed from the
output; the published (-r win-x64) copy runs.
Linux (WSL Ubuntu 24.04 x64), from copies of the four folders only (no repository):
- pythonnet: the workflow's build and test commands with GitHub's own Python 3.12.14 build:
0 warnings, 18/18, job summary and artifacts written; the local example exits 0.
- OpenBLAS default and advanced: build with 0 warnings and run to completion; the advanced
build stages the manylinux2014 wheel with its vendored libgfortran / libquadmath; the
published (-r linux-x64) copy runs from another directory.
NOT VERIFIED
- macOS and ARM: none of the four projects was run there. On ARM the advanced example takes its
PlatformNotSupportedException branch (no Haswell kernels) and pins the thread count only.
- The workflow has still not run on GitHub: journey4 is not pushed.
…he scalar-broadcast comparison kernels of branch strings (d0a6f4c) brought to journey4 unchanged: m > 0.75 on a 512x512 float64 matrix 0.25x -> 1.0x NumPy and on 64x64 0.52x -> 1.2x, a float64 array against a 0-d operand 6.8x faster at 100K elements; with the loop-bound fix the same files carry - the comparison and predicate SIMD loops no longer read one element past their inputs and write one bool past their result Why === The numpy.matrix benchmark (201e9c7) left the row "m > scalar" under NumPy: m > 0.75 at 4x4 / 64x64 / 512x512 was 1.39x / 0.52x / 0.25x. The class adds almost nothing to that call. The time was in the array library: on journey4 the comparison of an array with a one-element operand (ExecutionPath SimdScalarRight / SimdScalarLeft) was a scalar loop, 0.7 ns an element for float64 (0.36 for the integers, 1.4 for bool), where the array-against-array kernel is vectorized. Branch strings had already replaced those loops in d0a6f4c (plan api-parameter-parity X24 + X25). What was brought, and what was not ================================== Three files are now BYTE-IDENTICAL on journey4 and strings (git blob ids in brackets), so the later merge of strings does not conflict in them: - src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.Comparison.cs [dfc0ef0] - src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.Unary.Predicate.cs [66db492] - test/NumSharp.Tests/Backends/Kernels/ComparisonBroadcastKernelTests.cs [647ae48] (new on journey4; the same file strings adds) Both kernel files were changed on strings by d0a6f4c only, and not at all on journey4 since the two branches split (d70cbb0), so "the file as strings has it" is exactly "journey4's file plus the kernel half of d0a6f4c". NOT brought: the rest of d0a6f4c. That is the NEP 50 weak-scalar pack of the comparisons (X24), the weak-int range check of the binary ufuncs (X2 / X21), the numpy.ma strong-scalar rule, the NDExpr change, and the 15,050-case weak_scalar oracle tier. They are a breaking change of results, they sit on top of the string-dtype commits of that branch, and journey4 has four oracle tier families that strings does not have yet (relayout, surface, ma surface, matrix). They arrive with the merge. What the kernels do =================== - array op scalar / scalar op array compare a whole vector per step with the scalar broadcast into a vector once (EmitComparisonBroadcastSimdLoop, CanUseComparisonBroadcastSimd). The condition is that the array already has the comparison dtype: the vector body loads the elements as they are. - The 4x-unrolled bodies store their four masks PACKED into result bytes on AVX2 hosts (EmitMasksToBools / EmitPackedMasksToBools, NumPy's npyv_pack_b8_b16 / _b32 / _b64). 1-byte lanes store mask & 1 as one vector. The array-against-array kernel (SimdFull) uses the same stores. - The single-mask store reads the lane bits as 64 bits (a Vector512 mask of 1-byte lanes has 64; the old 32-bit local could not hold them). The loop bound (silent memory corruption on journey4 until this commit) ====================================================================== The comparison SimdFull kernel and the contiguous predicate kernel (isnan, isinf, isfinite, isposinf, isneginf, signbit) ran their 4x-unrolled and one-vector loops while i <= n - step + 1. So they ran one body too many when n % step == step - 1: one element read past both inputs and one bool written past the result. A fresh result's extra byte landed past its allocation (the pool's buckets have exact sizes); an out= view's extra byte landed in the next slot of its parent buffer. The loops now run while i <= n - step. Proof on journey4: with the old two kernel files and the new test file, three of its six tests fail: ComparisonKernels_NeverWritePastTheResult "Boolean Equal SimdFull n=31 wrote result slot 31", found 0x01 PredicateKernel_NeverWritesPastTheResult "IsNan Single n=7 wrote result slot 7", found 0x00 Comparisons_KeepTheParentBufferPastAnOutView "Boolean n=31 array: slot 31 past the out view changed" With the new kernels all six pass. A value-only gate (the oracle, the unit tests) cannot see a write past the result, which is why this stayed hidden. Measured (NPY/NS, higher is better; two pinned P-cores, Release) ================================================================ The matrix benchmark's row, re-run with the harness of 201e9c7 (97 cells, two rounds a side, the minimum; "before" is that commit's last run): before (us) after (us) cell NumPy NumSharp NumPy NumSharp NPY/NS m > 0.75 @4 1.68 1.21 1.89 0.94 1.39x -> 2.02x m > 0.75 @64 2.34 4.54 2.57 2.13 0.52x -> 1.20x m > 0.75 @512 46.7 185.4 46.3 45.9 0.25x -> 1.01x The whole benchmark: geomean 2.26x -> 2.35x, cells under 1.5x 31 -> 29. A second harness (87 cells, best of three interleaved rounds a side; "before" is journey4's head, measured in the same harness): float64, n x n NumPy us before us after us after a > held 0-d @4 0.83 0.44 0.43 1.95x a > held 0-d @64 1.37 3.39 0.96 1.43x a > held 0-d @512 41.5 187.8 40.1 1.03x m > held 0-d @64 2.49 3.70 1.16 2.14x greater(a, s, out) @64 1.25 3.60 1.09 1.14x a > b (two arrays) @512 94.9 118.7 100.1 0.95x a > s, s a 0-d of the array's dtype, 100K elements (us) dtype NumPy before after dtype NumPy before after float64 11.07 70.5 10.31 int16 5.10 35.6 5.39 float32 6.11 70.2 6.04 uint16 6.26 35.6 5.57 int64 34.2 36.2 18.3 int8 3.87 35.5 3.63 uint64 33.7 53.1 18.2 uint8 4.17 35.6 4.05 int32 7.81 35.6 9.03 bool 4.16 139.3 3.58 uint32 9.90 52.9 7.16 the same at 10M (ms): float64 6.05 / 8.08 / 4.46, float32 4.27 / 7.13 / 2.17, int32 4.49 / 4.23 / 2.22, int16 3.12 / 3.61 / 1.13, int8 2.63 / 3.58 / 0.71, bool 2.66 / 13.99 / 0.67. Over the 87 cells: geomean 0.52x -> 1.62x, minimum 0.03x -> 0.82x. At 1K every dtype is 1.9-2.4x, at 10M 1.4-4.0x. At 100K both libraries run the same packed loop at the speed of the cache (0.86-1.9x). Why two cells of the row stay under 1.5x ======================================== 1. 64x64: the C# literal. m > 0.75 converts 0.75 into a 0-d matrix that no one disposes. With a held operand the cell takes 1.16 us (2.14x NumPy); the literal adds 0.9 us. Measured over one second of calls: a dropped "NDArray s = 0.75" costs 1.13 us (0.32 when it is disposed; a dropped object with an empty finalizer costs 0.18), and (a > 0.75) at 4x4 takes 1.75 us against 0.47 with a held operand. This is plan item X26 (open, library-wide). It needs an operand that is not a finalizable object, which is a design decision and not part of this change. 2. 512x512: memory. The kernel already runs at the cost of READING its input. Called directly on aligned buffers it takes 0.36 us for 4,096 float64 elements and 18.5 us for 196,608; a hand-written loop that only loads and compares (it writes nothing) takes 0.37 and 17.7. A 512x512 float64 matrix is 2 MB, the size of this CPU's L2 cache (i9-13900K, P-core): there the read-only loop takes 30 us, the kernel 39 and NumPy 40-48. In the same hand-written loop (38.1 us) a software prefetch changes nothing (37.7 us) and a non-temporal result store gains 6 % (35.9 us), at the cost of every later read of the mask. So 1.5x NumPy is not reachable at that size. What still runs the scalar loop on journey4 (until strings merges) ================================================================= The vector loop needs the array to have the comparison dtype, and journey4 still promotes a 0-d operand strongly (X24 is not here). So a literal that promotes the comparison stays slow. At 100K elements: float32 > 50.5 (double) 72.4 us float32 > 50.5f 7.9 us uint8 > 50 (int) 39.3 us uint8 > (byte)50 3.9 us int16 > 50 (int) 36.7 us int16 > (short)50 4.7 us int32 > 50L (long) 39.1 us int32 > 50 7.3 us A float64 array takes the vector loop with an int, float or double literal (10.8 us), an int32 / int64 array with an int literal. Finding, not acted on: buffer alignment is a 20-35 % lottery ============================================================ A buffer whose address is 16 mod 32 makes every second 32-byte load cross a cache line. The kernel called directly on 100K elements: float32 4.8 us aligned and 6.4 us not, float64 9.3 and 12.5, int32 6.0 and 8.2. For 1- and 2-byte lanes the RESULT's alignment costs as much (int16 at 262,144 elements: 9.0 and 12.8 us). Pooled buffers are 16-byte aligned, so each array wins or loses by its address. Through the public operator, the same 100K float64 values take 10.1 us in a view that starts on a 32-byte boundary and 13.3 us in a view that starts 16 bytes later (float32: 5.8 and 7.3). It applies to every SIMD kernel of the library, so the fix belongs in the allocator (a 64-byte aligned allocation), not in this kernel. The 2 MB matrix of the row is not affected: the operating system returns a large block 64-byte aligned. Validation ========== All in a private worktree of 07aa1d1 with the four files of this commit, Release: - NumSharp.Tests (CI filter): net10.0 18,449 passed, 0 failed, 11 skipped; net8.0 18,448 passed, 0 failed, 11 skipped. - NumSharp.Tests.Oracle (the NumPy 2.4.2 gate): 280 passed, 0 failed, 1 skipped (RandomApiSoak, which needs its environment variable) on net10.0 and on net8.0. - ComparisonBroadcastKernelTests: 6 of 6. With the old kernels: 3 fail (above). - Other SIMD configurations, 231 comparison / predicate tests each, all pass: DOTNET_EnableAVX=0 (128-bit vectors and no BMI2: the mask-store branch an arm64 host takes), DOTNET_EnableAVX2=0, DOTNET_EnableHWIntrinsic=0 (no vectors). A probe confirmed that the switches change the kernels' vector width (256 / 128 / 0). The only vector method the new code resolves by itself, Vector{N}.BitwiseAnd, is the cross-platform one on every host. - NOT run: an arm64 host and a 512-bit host (this machine has neither). Docs ==== .claude/CLAUDE.md, the numpy.matrix paragraph: the row's new numbers, what was brought and what stays on strings, the two floors, the literal's measured cost, the alignment finding. Merge note for strings -> journey4 ================================== - The three files above are identical on both sides: no conflict. - d0a6f4c's other files merge as they would have before this commit. - .claude/CLAUDE.md: this commit edits only the numpy.matrix paragraph, which strings does not have. The paragraph's sentences about what "stays on strings" and about literals that promote become stale at the merge.
…pauses - a second run of the one-second probe of 19a0057 with pause and lock-contention counters; the NDScope numbers of that commit's paragraph are replaced by the range of both runs 19a0057 measured the 0-d temporary of a C# literal operand (plan X26, open) and wrote in .claude/CLAUDE.md that a scope around the call "does not make it cheap either", with the numbers of ONE run (2.27 us in an NDScope, 2.43 without, 1.45 with a held operand). A second run of the same probe read 1.86 / 2.30 / 1.38. The paragraph now gives both runs as ranges, and says what the scope does: it removes the collector's pauses, not the scalar. The second run also printed GC.GetTotalPauseDuration and Monitor.LockContentionCount per case (each case is about one second of calls, Release, two pinned P-cores): case mean ns gen0/gen1 GC pause lock contentions object with a finalizer, dropped 253 5 / 4 182 ms 0 NDArray.Scalar(0.75), disposed 338 90 / 0 109 ms 0 NDArray s = 0.75, dropped 1133 27 / 26 454 ms 8 NDMatrix s = 0.75, dropped 1120 27 / 26 435 ms 5 (a > held) 4x4 475 59 / 0 82 ms 0 (a > 0.75) 4x4 1682 34 / 33 394 ms 11 (m > held) 4x4 764 59 / 0 81 ms 0 (m > 0.75) 4x4 2025 37 / 36 401 ms 2 (a64 > held) 64x64 982 249 / 0 48 ms 0 (a64 > 0.75) 64x64 1871 132 / 131 282 ms 0 (m64 > held) 64x64 1381 180 / 0 53 ms 0 (m64 > 0.75) 64x64 2296 108 / 107 284 ms 1 (m64 > 0.75) in an NDScope 1863 131 / 0 69 ms 0 So the cost is not a lock and not the scalar pool: a dropped scalar makes every collection a generation-1 collection of about 17 ms (the finalizable array and its memory block's disposer are promoted with the storage object, the boxed slice, the closure and the delegate they reference), against about 1.4 ms a collection when the operand is held or disposed. The thread is paused for 0.28 to 0.45 s of each second. The scalar pool running dry (about 25,000 extra slots a second) is a symptom of the same lag. This also explains why a best-of-rounds harness reads the literal's cost 0.2 to 0.9 us low: it reports a round with few collections. No code change.
… element past the view - np.zeros(163840, np.uint8)[1::2].copy() ended the process with an access violation, because seven SIMD deinterleave loops loaded one element beyond a view that ends at its buffer's last element; the second load now starts one element early and shifts (same speed), and a fenced-memory test puts the source of all 15 x 15 casts against inaccessible pages THE DEFECT (in the library since 2026-06-20, commits 32353c3 / 4a77991 / f21876c) The strided cast kernels for one- and two-byte elements deinterleave a step of two (a[::2]) with two vector loads per group of N results: N elements at s[2i] and N elements at s[2i+N]. Together they cover s[2i .. 2i+2N-1]. The group needs s[2i], s[2i+2], ..., s[2i+2N-2]. The loads therefore read ONE element past the last element of the group. For the last group of a row that element is one past the view's last element. When the view's last element is its buffer's last element (a[1::2] of an even-length buffer, a[::2] of an odd-length one), it is outside the buffer. Values were right (the element is masked away), so no value test and no oracle case could see it. It became a crash when the buffer ended where mapped memory ends: var z = np.zeros(new Shape(40 * 4096), NPTypeCode.Byte); // 163,840 bytes var c = z["1::2"].copy(); // AccessViolationException, the process ends A zeroed array of 128 KB or more is its own VirtualAlloc region (OsVirtualMemory). 40 pages are two and a half allocation granules, so nothing can be mapped behind the buffer. NumPy returns 81,920 zeros. Seven loops had the pattern: DirectILKernelGenerator.Cast.SubwordCopy.cs SubwordCopyStrided1B, SubwordCopyStrided2B (same-type copies of bool / int8 / uint8 / int16 / uint16 / char / float16, and same-width casts such as uint8 -> int8) DirectILKernelGenerator.Cast.SubwordNarrow.cs SubwordNarrowInt2to1, SubwordNarrowBool2to1 (2-byte -> 1-byte / bool) DirectILKernelGenerator.Cast.SubwordWiden.cs SubwordWidenSigned1to2, SubwordWidenUnsigned1to2 (1-byte -> 2-byte) DirectILKernelGenerator.Cast.ToBool.cs StridedHalfToBool (float16 -> bool) The 4- and 8-byte kernels (Cast.WordCopy.cs) bound their loop for this already (i + 9 <= n, i + 5 <= n). THE FIX The second load starts one element EARLY, at s[2i+N-1]. Its lanes then hold the wanted elements in their HIGH half (the unwanted s[2i+N-1+2k] below, the wanted s[2i+N+2k] above), so a logical right shift by one element width replaces the mask of that load. The instruction count is the same, and the load ends exactly on s[2i+2N-2]. Speed, old loop against new loop in one process, interleaved, two pinned P-cores, (1000, 1000) -> (1000, 500): 1-byte 26.60 us -> 26.80 us (1.008) 2-byte 87.73 us -> 87.83 us (1.001) A loop bound of one element less would have sent the last 16 or 32 results of every row whose length is a multiple of the group through the scalar tail, which is the common case (rows of 512). HOW IT WAS FOUND The comparison engine of the next commits stages stepped operands through these kernels. Its tests place every operand against an inaccessible page; the first run ended in SubwordCopyStrided1B. TESTS test/NumSharp.Tests/Utilities/FencedMemory.cs (new, test utility) Native memory with an inaccessible page directly below its first byte and directly above its last one. The region is reserved whole and only its data pages are committed, so a read or a write one byte outside ends the process. Place(bytes, atEnd) puts a block against either page. Windows (VirtualAlloc); a test that needs it reports Inconclusive elsewhere. test/NumSharp.Tests/Backends/Kernels/StridedCastFenceTests.cs (new, 3 tests) - OneDimensionalViews_OfAFencedBuffer_CopyAndCastWithoutLeavingIt: for every dtype, a buffer against the low or the high page, the views [:], [::2], [1::2], [::3], [1::3], [2::3], [::-1], [::-2], [-2::-2], [::-3] at 18 lengths around each kernel's vector group; each view is copied and cast to all 15 dtypes. - TwoDimensionalViews_OfAFencedBuffer_CopyAndCastWithoutLeavingIt: stepped and reversed columns and rows and the transpose of a (3, n) buffer. - SteppedViewEndingAtTheLastElementOfAZeroedRegion_IsCopied: the np.zeros call above for the seven sub-word dtypes (copy, narrow, widen, same-width cast). Values are checked too: the copy against the base's elements, each cast against the same cast of a contiguous copy. A violation is not a failed assertion; the test host ends with the kernel's stack. Do the tests see each loop? Each of the seven fixes was reverted alone and the class run again: SubwordCopy 1-byte test host crashed in SubwordCopyStrided1B SubwordCopy 2-byte test host crashed in SubwordCopyStrided2B SubwordNarrow loop 1 test host crashed in SubwordNarrowInt2to1 SubwordNarrow loop 2 test host crashed in SubwordNarrowBool2to1 SubwordWiden loop 1 test host crashed in SubwordWidenSigned1to2 SubwordWiden loop 2 test host crashed in SubwordWidenUnsigned1to2 half -> bool test host crashed in StridedHalfToBool With all seven fixed the class passes, and no other cast kernel leaves its source on any of the 225 dtype pairs. VERIFICATION OF THE WHOLE TREE (with the two following commits; the three were developed and tested together) NumSharp.Tests net10.0 and net8.0, filter TestCategory!=OpenBugs&TestCategory!=HighMemory: 0 failures NumSharp.Tests.Oracle net10.0 and net8.0: 0 failures The whole unit suite and the whole oracle replay under NUMSHARP_DEBUG_GUARD_PAGES=1 (every pooled buffer ends at an inaccessible page): no crash. Only the tests that count pool takes and returns fail there, because the mode bypasses the pool. DOCUMENTATION .claude/CLAUDE.md: "A strided copy or cast never reads outside its view" (after the ILKernel coverage table) and the sentence at the end of the Cast row. BRANCH NOTE The four kernel files are byte-identical on journey4 and on strings before this commit, so the change merges without a conflict.
…rgument - the typed overload installed the Shape as given, so the Shape of a view, a transposed or a broadcast array put that array's strides over this array's buffer (strides (16,8,1) over 40 elements read up to element 75: other heap memory, or an access violation); it now passes newShape.AsClean() on, which is what the untyped overload always amounted to
THE DEFECT
NumPy's reshape takes a shape TUPLE. The untyped NDArray.reshape(Shape) (ReshapeCore, the port of
_reshape_with_copy_arg) reads the dimensions of its argument and nothing else.
The typed overload NDArray<T>.reshape(Shape) / reshape(ref Shape) did not go through ReshapeCore. It relabelled
its storage with the Shape as given: UnmanagedStorage.Reshape(ref Shape) -> Shape.Reshape(Shape), which builds the
result from the argument's dimensions AND strides. A Shape taken from another array carries that array's layout:
var view = np.arange(80).reshape(5, 2, 8)[":, :, :4"]; // dims (5,2,4), strides (16,8,1), buffer of 80
NDArray<bool> m = np.zeros(new Shape(40), NPTypeCode.Boolean).AsGeneric<bool>();
var r = m.reshape(view.Shape);
// before: dims (5,2,4), strides (16,8,1), bufferSize 40 -> element [4,1,3] is read at offset 75 of 40
// now: dims (5,2,4), strides (8,4,1), bufferSize 40 (NumPy: np.zeros(40, bool).reshape((5, 2, 4)))
The result addressed memory outside its own buffer: wrong values read from whatever lies behind it, writes into
it, and an access violation when the buffer ends where mapped memory ends. The Shapes of a stepped view, a
reversed view (negative stride and offset), a transposed array (F strides) and a broadcast array (stride 0) were
installed the same way.
A comparison returns NDArray<bool>, so the natural port of NumPy's `(a % 3 == 0).reshape(x.shape)` hit it. An
existing test did exactly that: Indexing_KernelRoute_Tests.Where_NarrowRows_3D_LeadingAxisWalkedPerBlock (added
2026-09-03) built its where= mask with `.reshape(sv.Shape)`, sv being x[":, :, :4"]. Its assertion compared
np.add(..., where: mask) with np.where(mask, ...): both sides read the same mask outside its buffer, so the test
passed. Under NUMSHARP_DEBUG_GUARD_PAGES=1 it ended the test host in the masked 2-D kernel, which is how this was
found.
THE FIX (src/NumSharp.Core/Generics/NDArray`1.ReShape.cs)
reshape(ref Shape newShape) relabels with newShape.AsClean(): a Shape that already is a plain dimension list
(offset 0, the C strides of its dimensions, a buffer size equal to its element count) is passed on as it is, so
a.reshape(2, 3) and a.reshape(new Shape(2, 3)) allocate what they did; any other layout is re-laid out in C order
first. The -1 inference and NumPy's size-mismatch text are unchanged (pinned by the new tests).
NOT CHANGED, ON PURPOSE
- reshape_unsafe(Shape), typed and untyped: the unchecked relabel that installs the Shape it is given.
Broadcasting uses it for exactly that.
- The typed overloads still COPY a non-contiguous source where the untyped route returns NumPy's strided view
(_attempt_nocopy_reshape). That is a view / copy difference, not an addressing error. Routing the typed
overloads through ReshapeCore would close it; ReshapeCore lives in Creation/NDArray.ReShape.cs, which branch
strings also edits, so it is left for after that merge.
TESTS (test/NumSharp.Tests/Manipulation/TypedReshapeViewShapeTests.cs, new, 3 tests; expected values are NumPy
2.4.2's: np.arange(40).reshape((5, 2, 4))[i, j, k] == (i*2 + j)*4 + k)
- TypedReshape_WithTheShapeOfAnotherArraysView_UsesItsDimensionsOnly: the Shapes of a stepped view, a column
view, a reversed view, a transposed array and a broadcast array. The result has dims (5,2,4), strides (8,4,1),
offset 0, bufferSize 40, the untyped overload's strides, and NumPy's 40 values.
- TypedReshape_WithAViewsShape_IsAViewOfItsOwnBuffer: a write through the result shows in the source; a
comparison's NDArray<bool> reshaped with a view's Shape counts NumPy's 14 True values.
- TypedReshape_WithAPlainShape_IsUnchanged: a plain Shape, the -1 inference, NumPy's error text.
Before the fix the first test reads strides (16,8,2) / (16,8,1) / ... where (8,4,1) is expected.
Where_NarrowRows_3D_LeadingAxisWalkedPerBlock now runs over a mask inside its buffer and passes under guard pages.
VERIFICATION (the tree of this commit and its two neighbours, which were developed together)
NumSharp.Tests net10.0 and net8.0, filter TestCategory!=OpenBugs&TestCategory!=HighMemory: 0 failures
(no test depended on the old relabel)
NumSharp.Tests.Oracle net10.0 and net8.0: 0 failures
Under NUMSHARP_DEBUG_GUARD_PAGES=1 the whole unit suite now runs to its end without a crash (it stopped at the
test above before), and so does the whole oracle replay.
DOCUMENTATION
.claude/CLAUDE.md, "Critical: View Semantics": "NDArray<T>.reshape(Shape) reads the DIMENSIONS of its argument".
BRANCH NOTE
Generics/NDArray`1.ReShape.cs is byte-identical on journey4 and on strings before this commit.
…>, >= runs the vector kernels - a ComparisonRun turns any chunk (stepped, reversed, another dtype, a stepped out) into calls of the contiguous SIMD kernels, a two-level route runs a matrix against a row, a column or an outer product without an iterator, and complex128 and int64-against-uint64 get vector kernels: 1,279 benchmark cells geomean 1.15x -> 2.05x NumPy, cells under 1.0x 445 -> 43, the lowest cell 0.03x -> 0.81x WHY Commit 19a0057 gave array-against-scalar comparisons a vector loop. Every other variation of the six comparisons still ran one element at a time: a matrix against a row or a column, a stepped or reversed operand, two dtypes, a stepped or float64 out=, complex128, int64 against uint64. On a 1,279-cell benchmark 445 cells were slower than NumPy and 88 of them more than twice as slow (the lowest 0.03x). WHAT NUMPY DOES (numpy/_core/src/umath/loops_comparison.dispatch.c.src, PyUFunc_SimpleBinaryComparisonTypeResolver) Its comparison loops are vectorized for three operand forms only: two contiguous inputs, or one input a single value, each with a contiguous bool output. A stepped or reversed input, an input of another dtype (cast through a buffer first) and a stepped output run its scalar loop at 0.6 - 1 ns an element; so do complex128 and float16. int64 against uint64 has exact loops. WHAT THIS COMMIT DOES 1. ComparisonRun (src/NumSharp.Core/Backends/Kernels/ComparisonRun.cs, new) One cached instance per (operator, left dtype, right dtype). It takes one inner-loop chunk (three pointers, three byte strides, a count: NumPy's inner-loop contract) and runs it through the three contiguous vector kernels (SimdFull, SimdScalarRight, SimdScalarLeft), whatever the strides and dtypes are: direct an input that already is a contiguous run of the dtype the kernel compares in, or one value, is read in place. A long run that starts off a vector boundary gets a short first kernel call (AlignmentPeel: the input stream for lanes of two bytes or more, the output for one-byte lanes). staged any other input is copied and converted 8 KB at a time into a stack block, and the kernel reads the block while it is in L1. The house SIMD copy removes a step of two or a reversal, the new strided move any other stride, the house SIMD cast converts. flipped inputs that run backwards in the kernel's dtype are read forwards from their last element and the bool results are stored reversed (one byte an element moved, not 8 or 16). scattered a bool output that is not contiguous receives each block through a strided byte store. adopted an array of another dtype against ONE value is compared in the array's own dtype (point 3). A chunk under 64 elements runs the per-element kernel the iterator routes always used. 2. Three callers - DefaultEngine.TryTrivialContiguousComparisonOp -> Run: two contiguous arrays of one shape, or an array and one value, of any two dtypes; also two one-element operands of different rank. - DefaultEngine.TryRowsComparisonOp (DefaultEngine.CompareOp.Rows.cs, new) -> RunRows: an iteration of R runs of N elements, without an iterator. The structure is read from the strides with NumPy's coalescing rule (stride[d] == stride[d+1] * extent[d+1]). It declines an F-ordered result, an iteration of three or more levels, and many rows under 16 elements whose inputs are contiguous or one value (the iterator's 2-D block kernel loops those inside one call). Measured: an iterator costs 428 ns to build and dispose and 23 ns for each chunk it hands out (40 float64 values); a kernel call per run costs 7.8 ns. - NDIterRef.TryExecuteComparisonRun (NDIter.Execution.ComparisonRun.cs, new) -> one Chunk call per inner-loop chunk: every other iteration, and every out= call (DefaultEngine.ExecuteComparisonUfuncInto; an out of another dtype is filled through the iterator's Boolean buffer). where= keeps its masked route (1.5 - 11x NumPy). 3. Scalar adoption is exact, never a cast (ComparisonRun.ScalarAdoption) NumPy's comparison is convert(a) op convert(v) in the promoted dtype. It has the same answer as a op a' for ONE value a' of the array's dtype, or the same answer for every element: integer array, integer value a' = v inside the array dtype's range; outside it the chunk is decided integer array <= 32 bits, float t a > t is a > floor(t), a >= t is a >= ceil(t), a < t is a < ceil(t), a <= t is a <= floor(t); == can hold only for an integral t in range float16 / float32 array, wider t > and <= use the largest array-dtype value not above t, >= and < the smallest not below it; == can hold only when t is an array-dtype value a NaN value decides the chunk (False; True for !=) So float32_array > 0.1 still compares against the double 0.1. Not adopted: a 64-bit integer array against a float, complex, decimal, char. 4. New kernels (src/NumSharp.Core/Backends/Kernels/Direct/) - DirectILKernelGenerator.Comparison.Complex.cs: complex128, four elements a step. Two loads are split into real and imaginary lanes, compared by NumPy's CLT rule (a NaN in an imaginary part makes the ordered operators False), and the mask bits become bytes (pdep, or integer arithmetic without BMI2). 0.66 ns an element for <, NumPy 0.8 - 1.45. - DirectILKernelGenerator.Comparison.MixedSign.cs: int64 against uint64, either order, as machine compares (a negative lane is decided by its sign, the others compare unsigned), eight elements a step. The emitted loop compared in System.Decimal. - DirectILKernelGenerator.StridedMove.cs: an unrolled IL strided move for elements of 1 / 2 / 4 / 8 / 16 bytes. DirectILKernelGenerator.Comparison.cs hooks the two comparison kernels into GenerateComparisonKernel. NDPolySeries.cs gains PolySeriesView.RawStrided (a run of raw memory at any byte stride, for the staging copy). FIXED ON THE WAY - np.less(int64, uint64, out=o) compared in float64 (inexact past 2^53). Every route is exact now. - Two one-element operands of different rank ((1,) < 0-d) took 1.03 us through an iterator; 0.44 us now. RESULTS ARE UNCHANGED Every route gives the bytes of the per-element comparison at NumPy's comparison dtype. TESTS (test/NumSharp.Tests/Backends/Kernels/ComparisonRunTests.cs, new, 19 tests) - every dtype pair x operator x 1-D layout (contiguous, steps of 2 and 3, reversed, negative steps, one value, views that end at the buffer's last element) against an element reference written from NumPy's rules; - two- and three-dimensional layouts (row, column, outer product, C against F, reversed and stepped rows and columns, partial broadcasts of a 3-D operand); - the four out= forms (a stepped out's skipped slots are not written), long runs off a vector boundary, an out that is an input, int64 against uint64 on every route, result order, one-element operands of different rank; - scalar adoption over every uint8 / int8 value, every float16 bit pattern and the float32 values around each threshold, both sides, six operators; - Run and RunRows on raw memory with sentinel margins, and with every operand against an inaccessible page (FencedMemory): no byte outside an operand is read or written; - the complex, mixed-sign and strided-move kernels and the bit helpers; - the run against the routes it replaced (ComparisonRun.Disabled). The class takes about 4.5 s. MUTATION PASS: 35 planted defects (kernels, adoption, staging, stores, the rows route, dispatch). 33 fail one test or more. The two survivors give the same results: a range test at the boundary (the restated comparison is all False as well), and a branch of the bit-pattern step that its caller never reaches. The pass also showed that the first value generator of the tests drew ONE value for pools of seven entries (a step of 7); it is a multiplicative hash now, and the scratch sweep that shared the flaw was run again. SWEEP: a scratch program against an independent reference (15 x 15 dtype pairs x 6 operators x layouts x out= forms x 2-D and 3-D broadcasts): 373,543,380 checks, 0 failures, with AVX2, with DOTNET_EnableAVX2=0 and with DOTNET_EnableHWIntrinsic=0. The unit classes also pass with EnableAVX=0, EnableBMI2=0 and EnableSSE41=0. SUITES NumSharp.Tests net10.0 18,474 passed, net8.0 18,473 passed, 0 failed (CI filter) NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed Both under NUMSHARP_DEBUG_GUARD_PAGES=1: no crash (the pool-accounting tests fail there by design). BENCHMARK (NPY/NS; two pinned P-cores; tier-up forced with DOTNET_TC_CallCountingDelayMs=0; per-cell minimum over three NumSharp runs and two NumPy runs; in brackets the build before this commit, measured the same way) 1,279 cells: A contiguous (13 dtypes x 6 operators x array / value forms x 1K / 100K / 10M), B 20 layouts, C 12 dtype pairs, D out= and where= forms, E C# literal operands, F tiny and edge operands. all cells geomean 2.05x (1.15x) under 1.5x: 463 (904) under 1.0x: 43 (445) under 0.5x: 0 (88) lowest 0.81x (0.03x) A 1.81x (1.62x) B 2.12x (0.75x) C 3.06x (1.20x) D 2.24x (1.75x) E 1.47x (0.73x) F 1.43x (1.37x) float64, 100K elements, NumPy us | before -> after: a matrix against a row 37.1 | 75.8 -> 14.6 (0.49x -> 2.55x) against a column 38.1 | 107.6 -> 11.2 (0.35x -> 3.41x) an outer product 58.3 | 107.9 -> 10.6 (0.54x -> 5.49x) reversed columns 133.2 | 113.1 -> 25.4 (1.18x -> 5.25x) a reversed 1-D pair 63.4 | 107.6 -> 18.5 (0.59x -> 3.42x) a stepped array against a value 63.9 | 109.5 -> 30.9 (0.58x -> 2.06x) a fully broadcast operand 11.6 | 107.7 -> 10.5 (0.11x -> 1.10x) C against F 78.2 | 117.8 -> 61.3 (0.66x -> 1.28x) other dtypes, 100K: int32 against a column 20.6 | 108.5 -> 8.0 (0.19x -> 2.58x) uint8 reversed columns 85.0 | 112.1 -> 12.5 (0.76x -> 6.79x) complex128 < 143.1 | 256.0 -> 90.7 (0.56x -> 1.58x) complex128 < against a value 158.1 | 273.6 -> 73.6 (0.58x -> 2.15x) two dtypes, 100K: float32 array, float64 value 58.1 | 70.0 -> 5.95 (0.83x -> 9.77x) int32 array, float64 array 63.8 | 70.4 -> 24.0 (0.91x -> 2.65x) uint8 array, int32 value 45.5 | 35.3 -> 3.7 (1.29x -> 12.2x) int64 against uint64 (1K) 1.21 | 4.36 -> 0.71 (0.28x -> 1.72x) out=, 100K: a stepped bool out 63.7 | 104.7 -> 38.1 (0.61x -> 1.67x) a float64 out 107.1 | 95.2 -> 56.9 (1.13x -> 1.88x) C# literals, 100K: float32 > 50.5 6.8 | 70.3 -> 7.1 (0.10x -> 0.95x) uint8 > 50 4.5 | 45.3 -> 4.7 (0.10x -> 0.96x) int32 > 50.5 59.4 | 70.4 -> 8.1 (0.84x -> 7.3x) float16 > 50.5 261.8 | 275.2 -> 15.7 (0.95x -> 16.7x) uint64 > 50 34.2 | 89.1 -> 19.6 (0.38x -> 1.74x) The numpy.matrix benchmark (97 cells) reads geomean 2.30x; m > 0.75: 1.99x / 1.29x / 1.20x at 4 / 64 / 512. THE CELLS STILL UNDER 1.5x ARE MEASURED FLOORS (every cell under 1.0x is between 0.81x and 0.99x) - Contiguous operands of one dtype at 100K elements, and their literal, F-ordered and offset forms (26 of the 43 cells under 1.0x; most of the 163 contiguous 100K cells under 1.5x). Both libraries run a vector loop at the L2 transfer bound and the buffer's address decides the rest. Kernel only, ns an element, input offset / output offset in bytes past a 64-byte boundary: float64 two arrays 0.141 at 0/0 and 0.250 at 16/0; float32 0.074 and 0.126; uint8 against a value 0.026 at 0/0 and 0.036 at 16/16. A prefix aligns ONE stream. Pooled buffers are 16-byte aligned, so each array wins or loses by its address; NumPy's are too (ctypes.data % 64 is uniform over 0 / 16 / 32 / 48 up to 800 KB). OPEN LEVER, not in this commit: 64-byte aligned pool buffers from one page up (SizeBucketedBufferPool is the only source of Native blocks, so the change is contained there). - Two stepped float64 / int64 operands, and complex128 == / !=, at 100K (10 cells, 0.85 - 0.97x): 3.2 MB of source, above this CPU's 2 MB L2. - Six 1K cells at parity (0.90 - 0.99x): two stepped float64 / int64 operands (two staging passes), bool C against F. - The empty result (0,3) < (3,) (0.85x): branch strings has an empty-result guard that arrives with the merge. - Between 1.0x and 1.5x (420 cells): contiguous at 100K (the same transfer bound), contiguous at 1K and tiny operands (the NDArray allocation floor: about 0.35 us of a 0.45 - 0.55 us call), C against F, literal operands at 1K (the literal's 0-d temporary, plan X26). MERGING strings The edits stay out of the hunks strings changed in DefaultEngine.CompareOp.cs and repeat its comparison-type lines of DefaultEngine.UfuncOut.cs byte for byte. git merge-tree reports the same conflict list with and without this commit (the ten files that conflicted before). DOCUMENTATION .claude/CLAUDE.md: "The comparison engine" under Comparison & Logic, "Memory-safety gates: fenced memory and guard pages" under Build & Test, and a row in the Core Source Files table.
…ta science library and their C# ports through a Python.* tree; the first pair is scikit-learn supervised learning (split, scale, fit, predict, score), and the port prints the same seven lines as its Python program on Windows and on Linux
WHAT THIS IS
A new example project for the "pythonic" way to drive a Python library from C# (.claude/skills/pythonic):
a root `static class Python` whose nested classes follow the Python module paths, so that a call reads in
C# as it reads in Python and the compiler checks it. The two existing pythonnet example projects
(NumSharpPythonnetLocalExample, NumSharpPythonnetGhActionsCicd) show the other style, `dynamic` and a
codec. This project shows the style for a bridge.
Each example is a PAIR:
- a real Python program, kept in the folder as the reference (ScikitLearn/classification.py);
- its C# port (ScikitLearn/Classification.cs), statement by statement. The Python lines stand as a
comment above the C# lines that port them.
The two programs print the same lines. verify.py runs each pair and compares the outputs.
This commit holds ONE pair, for review. The owner asked for one library and one feature first. The plan
for the other five pairs is at the end of this message.
THE FIRST PAIR: scikit-learn, supervised learning
The program loads the Wisconsin breast cancer table (569 tumors, 30 numbers each), takes the ten "mean"
columns, keeps a stratified quarter apart (train_test_split), scales with a StandardScaler that learns from
the training part only, fits a LogisticRegression, and reports accuracy, area under the ROC curve and the
confusion matrix on the test part. Then it answers a question with array code: which probability limit
finds every malignant tumor of the test, and how many benign tumors does that limit refer?
tumors 569 with 30 numbers each: 212 malignant, 357 benign
accuracy 0.9161
roc auc 0.9759
[[46 7]
[ 5 85]]
missed 7 of 53 malignant tumors at the limit 0.5
limit 0.9528 finds all of them and refers 28 of 90 benign tumors
Three rules give every line of the port:
1. A library call is a member of the tree with the same name:
Python model = LogisticRegression(max_iter=1000); model.fit(X_train_scaled, y_train)
C# using PyObject model = Python.sklearn.linear_model.LogisticRegression.New(max_iter: 1000);
Python.sklearn.linear_model.LogisticRegression.fit(model, X_train_scaled, y_train);
A keyword argument is a C# named argument with Python's name (test_size: 0.25, stratify: y_np,
random_state: 0). A method of a Python object is an unbound member that takes the object first, as
Python itself can write it. A constructor is `New`.
2. A NumPy line is a NumSharp line with the same operators: `y_test == 0`, `a & b`, `~mask`, `p[mask]`,
`.max()`, np.count_nonzero. Only a slice is spelled as a string: X[:, :10] is X[":, :10"].
3. An array changes sides in one visible call: nd.ToNumpy() (NumPy views NumSharp's memory, no copy,
the strided column view included) and pyobj.ToNDArray() (one copy of a Python result). A name that
ends in `_np` is NumPy's side of the array that NumSharp holds under the name without the ending.
Two things are C#'s and have no line in the Python program: PythonRuntime.Run(() => { ... }) holds the GIL
for the Python part and returns .NET values only, and every PyObject is a `using` local (no call is chained
on the result of another one). One NDScope releases the NumSharp arrays of a port.
THE FILES (examples/Pythonic)
Python.cs the root `static partial class Python` and the seven rules of the tree
ScikitLearn/Python.sklearn.cs the scikit-learn branch, twelve members:
Python.sklearn.datasets.load_breast_cancer(return_X_y)
Python.sklearn.model_selection.train_test_split(X, y, test_size, random_state, stratify)
Python.sklearn.preprocessing.StandardScaler.New / fit_transform / transform
Python.sklearn.linear_model.LogisticRegression.New(max_iter) / fit / predict / predict_proba
Python.sklearn.metrics.accuracy_score_t / roc_auc_score_t / confusion_matrix
ScikitLearn/classification.py the Python program
ScikitLearn/Classification.cs its port
PythonRuntime.cs the backend under the tree: Initialize (idempotent, one engine for each
process), Run / Run<T> (the one place that takes the GIL), Import (cached,
process-owned handles, the loser of a race disposes), Shutdown; and
PythonInteropException, the base type for bridge errors
PythonInstall.cs finds the Python to embed: NUMSHARP_PYTHONNET_PYTHON (binding), then a
.venv in the current directory or beside the project file, then PATH
Program.cs `dotnet run` (every port under a heading), `dotnet run -- <name>` (one
port, the reference's lines and no more), and the lifetime check
verify.py builds once, runs each */*.py and its port, compares line by line; the
exit code is the number of ports whose output is different
requirements.txt numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1
README.md, .gitignore, Pythonic.csproj
CONVENTIONS OF THE TREE (the ones that the skill does not spell out, decided here)
- The members carry Python's parameter names in Python's spelling (test_size, random_state, max_iter,
return_X_y), so a C# named argument reads like the Python keyword.
- A keyword parameter is nullable, and `null` means "do not send it": Python's own default applies and
the tree never restates a default that a later library version can change.
- A member returns a PyObject that the caller owns. A `_t` member returns a .NET value in its place
(accuracy_score_t returns a double).
- fit() returns void: Python returns the model itself from fit, and the member drops that second
reference, so a caller cannot leak it.
- Strict disposal inside a member: the bound function, the keyword dictionary and each boxed argument
are `using` locals (`using (PyObject value = new PyInt(seed)) kwargs["random_state"] = value;`).
- One file for each library branch (`static partial class Python`). The module classes are lower case,
as Python has them, which raises CS8981. A `#pragma warning disable CS8981` at the top of the branch
file silences it, because the project file must stay in the regular form (no NoWarn).
- The tree holds the calls that its programs use and no more.
- No member moves array data. Arrays cross at the call site with the verbs of NumSharp.Interop.pythonnet.
The codec is NOT registered: it is process-global, and a bridge must not set it for other code.
THE BACKEND
PythonRuntime is library-agnostic. It keeps what the measurements of 2026-10-09 found for the Local
example: PythonEngine.ProgramName is set for every interpreter (it is what makes CPython adopt a virtual
environment; without it, Windows reports the .NET program as sys.executable and Linux takes its paths from
the first python3 on PATH), sys.prefix is compared with the selected environment after the start, and the
shutdown runs gc.collect() and gc.freeze() first (pythonnet 3.0.5 runs 87 full collections otherwise) and
sets RuntimeData.FormatterType to NoopFormatter. A missing library is a PythonInteropException whose
message names the interpreter in use and the pip command; Program.cs prints it without a stack trace and
ends with exit code 1.
THE LIFETIME CHECK
After the ports, Program.cs reads NDArrayPythonInterop.LiveExports and LiveImports. Both must be 0: each
port disposed every Python object it made, so no NumPy array views NumSharp memory any more and no NumSharp
array views Python memory. Otherwise the program prints the two counts and ends with exit code 2. The check
has an effect: a scratch copy of the port with one `using` removed (means_np) ended with exit code 2 and
"A port left memory shared with Python: 1 NumSharp buffers, 0 Python buffers."
THE PROJECT FILE
Pythonic.csproj is what `dotnet new console` and `dotnet add package NumSharp.Interop.pythonnet` write, and
nothing more (the owner's rule for copy-ready examples of 2026-10-10). So the project builds against the
PUBLISHED NumSharp 0.70.0 packages and uses released API only. It is in SciSharp.NumSharp.sln under the
`examples` solution folder.
VERIFIED
Windows 11 x64, Python 3.12.12, numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1, pythonnet 3.0.5, .NET SDK 10:
- dotnet build, Debug and Release: 0 warnings, 0 errors (inside the repository, with its
Directory.Build.props).
- `python ScikitLearn/classification.py` and `dotnet run -- classification` print the same seven lines.
Unrounded, the accuracy (0.916083916083916), the area under the curve (0.9758909853249477) and the limit
(0.9527624954311276) are equal with all their digits.
- `dotnet run`: the heading and the seven lines, exit code 0. `dotnet run --project examples/Pythonic`
from the repository root finds the .venv beside the project file.
- `python verify.py`: "ScikitLearn/classification.py 7 lines same output", exit code 0.
- An unknown port name: a message that lists the ports, exit code 1.
- NUMSHARP_PYTHONNET_PYTHON that names a file that does not exist: "... cannot be used: not found",
exit code 1, no other Python is tried.
- An environment without scikit-learn: "Python cannot import 'sklearn.datasets' (No module named
'sklearn'). The interpreter in use is ... Install the libraries of this project into it: ...", exit 1.
- A base install in place of a virtual environment works.
- Wall time of one run: 2.6 to 3.8 seconds.
Ubuntu 24.04 x64 on WSL 2, Python 3.12.14 (GitHub's build), the folder copied OUTSIDE the repository
(so without Directory.Build.props: the project stands alone):
- the two README commands create the environment: 282 MB (258 MB on Windows);
- dotnet build: 0 warnings, 0 errors;
- the port prints the same seven lines as the Python program, verify.py reports "same output", and the
Linux lines are equal to the Windows lines.
NOT VERIFIED: macOS. No workflow runs this project yet.
FACTS ABOUT NumSharp 0.70.0 THAT THE PORT RELIES ON (measured)
- NDArray.ToString() prints a 2-D integer array exactly as NumPy does ("[[46 7]\n [ 5 85]]").
- np.count_nonzero returns long. `~mask` on a Boolean array is the logical not. `mask & other`,
`p[mask]` and `(double)nd.max()` work as in NumPy. nd.shape is long[].
- nd[":, 1"] of a (143, 2) float64 array is a strided view (16-byte strides), and ToNumpy() exports it
as a view.
- np.sum of that strided view gives 87.20879206270375 where NumPy gives 87.2087920627038. A port must
print a NumSharp float result with a fixed number of digits. The programs of this folder print counts,
comparisons and a maximum, which have no such last digit.
- NDArray<T> is in NumSharp.Generic. The port declares its masks as plain NDArray.
HELD BACK UNTIL THE OWNER ACCEPTS THE SHAPE
No paragraph in .claude/CLAUDE.md, and no link from the website, from the package README or from the
sibling example READMEs.
THE PLAN FOR THE OTHER FIVE PAIRS (not started)
The three libraries are scikit-learn, SciPy and Matplotlib: NumPy is what NumSharp ports, and pandas and
PyTorch already have adapters in NumSharp.Interop.pythonnet. Two features for each library:
scikit-learn 2 Model selection: a Pipeline (scaler and classifier) searched with GridSearchCV under
cross-validation; best_params_, best_score_ and cv_results_ come back as arrays. The
pipeline and its parameter grid stay Python text (the snippet runner, `evalStmts`).
SciPy 1 scipy.optimize: curve_fit and minimize on measurements that NumSharp makes; the objective
is a C# function that Python calls back (the "a callback does not hold the GIL" rule).
SciPy 2 scipy.stats: a t-test and a linear regression; the result objects become `_t` members
that return C# tuples.
Matplotlib 1 A line chart: subplots, plot, labels, legend, savefig on the Agg backend; a style context
(`with plt.style.context(...)`) becomes an IDisposable scope. Both programs print the
figure size and a hash of the RGBA canvas.
Matplotlib 2 A histogram and a scatter plot in one figure; the canvas comes back as an NDArray view
without a copy.
Shared additions when a pair first needs them: the snippet runner (evalStmts), a context-manager scope,
Python.builtins members, more pins in requirements.txt.
…e buffer pool hands out 64-byte aligned buffers from 4096 bytes up (the C allocator aligns to 16, so half of all arrays started in the middle of a 32-byte vector and every second vector load of a SIMD kernel crossed a cache line): two-array comparisons of 100,000 elements run 12-25 % faster (float64 20.0 -> 15.1 us), 82 of 614 benchmark cells at that size gain 10 % or more, the comparison benchmark reads 2.08x NumPy with 23 cells under 1.0x (2.05x and 43) WHY The comparison engine of e2ef7f3 left its contiguous two-array cells at 100,000 elements between 0.81x and 1.3x of NumPy, and the reason was the buffer's address. Kernel only, ns an element, input offset / output offset in bytes past a 64-byte boundary: float64 two arrays 0.141 at 0/0 0.250 at 16/0 float32 two arrays 0.074 0.126 uint8 against a value 0.026 0.036 at 16/16 A kernel can align ONE of its streams with a short first call (ComparisonRun.AlignmentPeel does). Only the allocator can align all of them. Pooled buffers came straight from the C allocator, which aligns to 16 bytes, so each array won or lost by its address. NumPy's buffers are 16-byte aligned as well: `a.ctypes.data % 64` is uniform over 0 / 16 / 32 / 48 for arrays up to 800 KB (measured, NumPy 2.4.2 on this host). WHAT CHANGES (src/NumSharp.Core/Backends/Unmanaged/Pooling/SizeBucketedBufferPool.cs) A buffer of AlignedMinBytes (4096, one page) or more starts on a BufferAlignment (64) byte boundary, whether it is fresh (Take), zeroed (TakeZeroed), reused from a bucket, or outside the pooled window (64 MiB and more). - AllocAligned allocates a block AlignedOverheadBytes (80) larger, hands out the first boundary at least a header past its start, and stores the block's own pointer and the requested size in the 16 bytes in front of the buffer. FreeFresh reads the pointer back and frees the block. - Below one page nothing changes: the buffer is the allocator's own block. Such a buffer stays in the L1 cache while a kernel works on it, and 80 bytes would be a large part of it. - A header, not NativeMemory.AlignedAlloc: the zeroed path must stay calloc. For a large block the C runtime hands out pages that the operating system zeroes when they are first written, which is what makes np.zeros of a huge array cost microseconds. There is no aligned calloc, and an aligned allocation followed by a clear would touch every page. With the header both paths end in the same NativeMemory.Free on every operating system; writing the header touches the block's first page only. - A cache line, not a page: a page boundary costs up to 4 KB a buffer, and buffers that all start at one page offset put every stream of a many-operand kernel in the same cache set at each step. A cache line is enough for the kernels measured: the distance between an output's and an input's offset inside a page does not change their speed (bool == value over 100,000 elements: 2.55 - 2.70 us at twelve distances from 0 to 4032 bytes; uint8 < uint8: 3.23 - 3.40 us). THE RULE IT ADDS Return(ptr, bytes) must get the size the buffer was taken with. The size already selected the bucket; it now also tells whether the buffer has a header. A Debug.Assert compares it with the size in the header. A pool pointer must never be freed directly. Every Native block comes from Take / TakeZeroed (two sites in UnmanagedMemoryBlock), and the other callers (np.copyto, the reduction iterator, the exact min / max engine, the polynomial arena and Vandermonde scratch) pair Take and Return with one size each, so the change is contained in the pool. MEASURED (two pinned P-cores, tier-up forced; per-cell minimum of three runs of each build) The comparison benchmark, 1,279 cells, aligned / unaligned time: 100,000 elements geomean 0.967 82 of 614 cells 10 % or more faster, 4 slower by 10 % or more (two one-byte offset views, bool == True with its literal's temporary, complex128 > a value) 1,000 elements geomean 0.997 10,000,000 geomean 0.986 in interleaved runs, no cell moves 10 % Two arrays of 100,000 elements, us (and NPY/NS): float64 20.0 -> 15.1 (0.98x -> 1.30x) float32 10.6 -> 8.5 (1.01x -> 1.25x) int32 11.4 -> 9.4 (1.19x -> 1.43x) int16 6.4 -> 5.7 (1.14x -> 1.29x) An array against a value moved by 0 - 4 % (the prefix aligned its one stream already), one-byte dtypes by 1 - 6 %. The whole benchmark against NumPy: geomean 2.08x (2.05x), under 1.5x: 451 (463), under 1.0x: 23 (43), the lowest cell 0.86x (0.81x); section A (contiguous operands) has no cell under 1.05x. The numpy.matrix benchmark (97 cells, three interleaved runs of each build): geomean 0.986 of the unaligned time, no cell 10 % slower; m.cumsum(0) at 64 x 64 7.9 -> 4.5 us, an int32 matrix times 2 at 512 x 512 142 -> 129 us. THE 23 CELLS STILL UNDER 1.0x (0.86x - 1.00x) - two stepped float64 / int64 operands at 100,000 elements (8 cells): 3.2 MB of source, above this CPU's 2 MB L2; - a C# literal operand at 100,000 elements (9 cells): the literal's 0-d temporary costs about 1 us of a 4.6 us call (plan X26); the same comparison with a held 0-d operand is 1.07x or more; - four cells at 1,000 elements at parity (bool C against F, two stepped int64 operands); - np.less(a, b) of two float64 arrays at 100,000 (0.98x: both libraries at 15.3 - 15.6 us); - the empty result (0,3) < (3,) (branch strings has a guard that arrives with the merge). TESTS (test/NumSharp.Tests/Backends/Unmanaged/PoolBufferAlignmentTests.cs, new, 4 tests) - TakeAndTakeZeroed_FromOnePageUp_AreAligned: eleven sizes from 4096 to 64 MiB + 1, several buffers of a size alive at once, first and last byte written and read back, a zeroed buffer is zero, a buffer taken again after its return is aligned. - BelowOnePage_BuffersAreTheAllocatorsOwn: sizes 1 to 4095 are usable from the first byte to the last. - Arrays_FromOnePageUp_StartOnACacheLine: np.empty, np.zeros, np.arange, a + b, a < b, astype and copy. - Clear_FreesAlignedBuffers. Under NUMSHARP_DEBUG_GUARD_PAGES=1 the two alignment tests report Inconclusive (that mode places a buffer's END at a page boundary instead). SUITES NumSharp.Tests net10.0 18,478 passed, net8.0 18,477 passed, 0 failed (CI filter) NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed NumSharp.Tests in a DEBUG build, net10.0: 18,478 passed. The Debug.Assert on the returned size is live there, so every Return of the suite gave the size it took. ALSO ComparisonRun.cs: the remarks of AlignmentPeel said that NumPy's large arrays are 64-byte aligned on Windows. The measurement above shows that this holds from about 1.6 MB only; the text now says what the prefix is still for (a view that starts inside its buffer, memory the pool did not allocate). DOCUMENTATION .claude/CLAUDE.md: a row in Key Design Decisions; "The pool aligns its buffers" and the rewritten "Floors" in the comparison section, with the benchmark numbers of this build; the alignment finding of the matrix section marked as acted on.
…- an explicit 0-d ARRAY (np.array(2), np.asarray(2)) is an ndarray to matrix.__mul__, so it raises "shapes (2,3) and (1,1) not aligned" unless the matrix has one column, and np.array(2) * m takes np.dot's scalar route; every computed 0-d value (m.sum(), np.dot(v, v), np.linalg.norm(a), a[0, 0], NDArray.Scalar) still scales WHY 201e9c7 documented one divergence of NDMatrix as [Misaligned]: m * np.array(2) returned the scaled matrix, where NumPy 2.4.2 raises ValueError. The owner decided for NumPy's behavior, with the condition that nothing else is hurt. There is no mathematics in NumPy's error, only the order of two type tests (numpy/matrixlib/defmatrix.py): def __mul__(self, other): if isinstance(other, (N.ndarray, list, tuple)): return N.dot(self, asmatrix(other)) # np.array(2) lands here: asmatrix makes it [[2]] if isscalar(other) or not hasattr(other, '__rmul__'): return N.dot(self, other) # 2, np.float64(2), m.sum() land here A 0-d ndarray passes the first test, becomes a 1 x 1 matrix, and the matrix product of (2,3) with (1,1) has unequal inner dimensions. For a matrix with one column the same call works and returns the scaled column. On the left, np.array(2) * m is matrix.__rmul__ = N.dot(other, self), where np.dot takes a 0-d array as a scalar. THE DOUBLE-CHECK THAT SHAPED THE DESIGN NumSharp has one 0-d type for both roles (a NumPy scalar and a 0-d ndarray). A rule that reads the flags cannot tell them apart. Probe of 39 ways to get a 0-d NDArray, with what NumPy returns from the same call: - writeable + owning, NumPy returns an ARRAY: np.array(2.5), np.asarray(2.5), NDArray.Scalar(2.5), np.zeros(()), z.copy(), z.astype(float32); - writeable + owning, NumPy returns a SCALAR: np.dot(v, v), np.vdot, np.inner, np.linalg.det, np.linalg.norm, np.polyval(v, 2.0), np.add(z, 1.0), z + z, np.sqrt(z), np.multiply(z, z), np.negative(z), np.float_power(z, 2) (13 of the 39); - read-only, NumPy returns a SCALAR: sum, mean, max, prod, std, median, average, ptp, trace, trapezoid, nansum; - writeable + not owning: a[1, 2], m[1, 2], v[2] (NumPy: scalars) and x.reshape(()), np.squeeze(x) (NumPy: arrays). So "writeable means array" would make m * np.linalg.norm(a) and m * np.dot(v, v) raise, where NumPy scales. WHAT CHANGED - New per-object mark NDArray.IsExplicitZeroDimArray (Backends/NDArray.ZeroDimArray.cs), set by the two explicit spellings only: np.array<T>(T scalar) and np.asarray<T>(T data). np.array(nd) and np.asarray(nd) of a marked array carry it to the new object (NumPy returns a 0-d ndarray from both). UNMARKED MEANS SCALAR, so nothing that scaled before stops scaling except those spellings. The mark belongs to the object, not to its storage: a view, a .copy(), an .astype() or an operation result does not have it. Branch strings marks C# literals the same way (NDArray.WasPythonLiteral), in another file, so the merge does not conflict. - NDMatrix.Product reads the mark: a marked plain 0-d operand on the RIGHT of a matrix is From(np.dot(m, ViewOf(a0))), the 1 x 1 product (NumPy's alignment text, or the scaled column); on the LEFT it is Scale(a0, m), np.dot's scalar route (a new C-contiguous owner, 0 + s * x for the float dtypes; the multiply ufunc, wrapped, for the others). A 0-d NDMatrix (the stand-in a C# number converts to) is a scalar as before. - Eight internal call sites built COMPUTED scalars with np.array(x): the results of np.all / np.any (np.all.cs, np.any.cs, Default.LogicalReduction.cs) and a temporary in np.histogram.cs. They now use NDArray.Scalar(x), the same function without the mark. Found by planting [Obsolete(error: true)] on the two overloads and reading the compiler's list: np.asarray<T>(T) has no internal caller. - Not marked on purpose: np.zeros / np.ones / np.empty / np.full with a 0-d shape. NumSharp's own kernels allocate their 0-d results with those functions, so marking there would turn computed scalars into arrays. A 0-d array built that way, or by x.reshape(()) / np.squeeze(x), still scales a matrix where NumPy raises ([Misaligned], documented in the class remarks and in .claude/CLAUDE.md). BREAKING m * np.array(2) and m * np.asarray(2) raise unless m has one column (they returned the scaled matrix since 201e9c7, which is not released). To scale, write m * 2, m * NDArray.Scalar(2.0) or m * (NDMatrix)2. ORACLE (test/oracle/gen_matrix_oracle.py -> matrix.jsonl, NumPy 2.4.2) 735 new cases, zero excuses, zero carves: m * a0 and a0 * m for the 13 dtypes x 5 scalar dtypes x the shapes (2,3), (3,1), (1,3), (1,1), seven source layouts, the object and method forms, and the sign of a zero through the left route (float32 / float64 / complex128 with -0.5, 0.5, -0.0, 0.0). NumPy's answers: 213 ValueErrors ("shapes (2,3) and (1,1) not aligned: 3 (dim 1) != 1 (dim 0)"), 148 products on the right, 373 results on the left (owners for the float dtypes, views of the ufunc result for the others). params.array0d lists the operand positions the replay marks (OpRegistry.Matrix.cs); a 0-d operand without it is a NumPy scalar, as before. The values are non-zero on the right route, because np.dot's size-1 branch is a known gap of the array library for the sign of a zero (pinned in OpenBugs.Reductions.cs). Tier: 26,761 gate cases over 87 op keys (was 26,027), 172 carved cases unchanged; the three replays (production, fused, two-pass) pass on both frameworks. MinCases floor 24,700 -> 25,400. TESTS NDMatrixTests.ExplicitZeroDimArray_IsAnArrayOperand_NotAScalar: the four spellings that raise (np.array, np.asarray, np.array(np.array(...)), the object overload), the one-column product, the left route (owner, +0.0), six kinds of 0-d value that still scale (a C# number, NDArray.Scalar, a reduction result, an element, a norm, a vector dot), and where the mark is and is not. ALSO FOUND (no change): the unit test that failed once on net8.0 A hunt (the net8.0 suite in a loop from a copy of the build output of 201e9c7) failed in its third run: AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy, "Expected sw.ElapsedMilliseconds to be less than 32L ... but found 33L". It is a wall-clock limit (10 np.nonzero calls over 1M elements in under 32 ms; 4-6 ms on an idle host) that a concurrent build pushed over. A host-load signal, not a product defect; the test's own remarks say so. VERIFIED (Release) - NumSharp.Tests (TestCategory!=OpenBugs&TestCategory!=HighMemory): 18,479 passed on net10.0, 18,478 on net8.0, 0 failed. - NumSharp.Tests.Oracle: 280 passed, 1 skipped (RandomApiSoak) on both frameworks.
…asses are C# classes with instance methods (var scaler = new StandardScaler(); scaler.fit_transform(X_train)), a module is imported with using static, arrays are NumSharp arrays on both ends of every call, one PythonScope replaces every using, and print($"...{x:.4f}") takes Python's own format specs
WHY
The owner's review of the first pair (b10ac7e), in his words: ".cs with pythonic should be using classes
and not statics necessarily which will allow scaler = StandardScaler() / X_train_scaled =
scaler.fit_transform(X_train) / X_test_scaled = scaler.transform(X_test). We need these fixed because the
Python.* are very long in this example", and "Can we also make our pythonic code look more like actual
python?". The first version wrote every object member as an unbound static member behind its full module
path, and every Python object as a `using PyObject` local:
using PyObject scaler = Python.sklearn.preprocessing.StandardScaler.New();
using PyObject X_train_scaled = Python.sklearn.preprocessing.StandardScaler.fit_transform(scaler, X_train);
using PyObject X_test_scaled = Python.sklearn.preprocessing.StandardScaler.transform(scaler, X_test);
WHAT A PORT LOOKS LIKE NOW (ScikitLearn/Classification.cs, beside ScikitLearn/classification.py)
# Python // C#
from sklearn.preprocessing import StandardScaler using static Python.sklearn.preprocessing;
X, y = load_breast_cancer(return_X_y=True) var (X, y) = load_breast_cancer(return_X_y: true);
means = X[:, :10] var means = X[":, :10"];
scaler = StandardScaler() var scaler = new StandardScaler();
X_train_scaled = scaler.fit_transform(X_train) var X_train_scaled = scaler.fit_transform(X_train);
model = LogisticRegression(max_iter=1000) var model = new LogisticRegression(max_iter: 1000);
model.fit(X_train_scaled, y_train) model.fit(X_train_scaled, y_train);
limit = p_benign[malignant].max() var limit = p_benign[malignant].max();
print(f"accuracy {accuracy_score(y_test, y_pred):.4f}") print($"accuracy {accuracy_score(y_test, y_pred):.4f}");
The port has the statements of the Python program in the same order, with the same names and the same
comments. C# writes `var`, `new`, `:` for the `=` of a keyword, `$` for the `f` of an f-string, `true`, and
a slice as a string. One line has no line in Python:
using var scope = new PythonScope(NDScope.Open());
No PyObject, no `_np` twin name, no ToNumpy() / ToNDArray() and no GIL block is left in a port.
PythonRuntime.Run(() => { ... }) is gone.
THE DESIGN
1. A module is a static class; `using static` is its import. The directive brings in the functions AND the
nested classes of the module, as `from module import ...` does. The `using` lines stand below the
file-scoped `namespace Pythonic;` line, where the name `Python` is the class of the project (above that
line it is pythonnet's namespace and the directive needs `Pythonic.Python...`). An alias imports one
class (`using StandardScaler = Pythonic.Python.sklearn.preprocessing.StandardScaler;`), and the full
path works without a directive (`Python.sklearn.metrics.accuracy_score(y, p)`). All three were compiled
and run.
2. A Python class is a C# class that derives from PythonObject (new file). The base owns the one Python
reference, registers itself with the PythonScope of its thread, gives Python's `str(obj)` from
ToString() ("LogisticRegression(max_iter=1000)"), and disposes under the GIL. `fit` returns `this`, so
`new LogisticRegression(max_iter: 1000).fit(X, y)` chains as in Python.
3. A member takes and returns .NET values. Each member is three lines: take the GIL
(PythonRuntime.GIL(), new; re-entrant), Call(...), convert the result. Three private helpers of the root
class do the work (Python.cs):
- Call(target, name, args, kwargs): the one place where .NET arguments become Python objects (an
NDArray is ToNumpy(), a view; a tree object is the reference it holds; bool, integers, double, float,
string are boxed) and where those temporaries are released. A keyword that is null is not sent.
- Create(module, type, kwargs): the constructor call for a base initializer (it takes the GIL itself).
- AsArray(result): `result.AsNDArray(allowReadonly: true)`, a NumSharp VIEW of the memory that NumPy
allocated. So nothing is copied in either direction. In the first version a result was one copy.
The `_t` suffix is gone: accuracy_score returns a double.
load_breast_cancer(return_X_y) has the pair form only and refuses `false` (Python returns another type
there, and a C# method has one return type).
4. PythonScope (new file) is the lifetime of a port: `new PythonScope(NDScope.Open())`. NumSharp's NDScope
collects the arrays, the views of Python memory included; the PythonScope collects the PythonObjects of
its thread and closes both, objects first. The NDScope is opened in the port's own line ON PURPOSE:
NumSharp's analyzer exempts a method that calls NDScope.Open() and reports NDW012 for the arrays of a
method that does not. A PythonScope that opened the NDScope inside its constructor produced 3 NDW012
warnings in the port; the alternatives were [NDScopedCovered] on each port (it also silences a port that
forgot the scope), two scope lines, or an .editorconfig that switches the rule off.
5. print and f-strings (new file Python.builtins.cs). `print(fstring)` takes an
[InterpolatedStringHandler] type, so the compiler passes each hole to it. A hole with a spec goes to
Python's own format(), a double without a spec to Python's str(); integers, text and bool are written
in C# (the digits are the same), and an array by NumSharp, which prints as NumPy does. So the specs are
Python's: .4f, >10.3f, +.2f, 08.3f, the thousands comma, _, .1%, .3e, .3g, d, x, #x, b, o, .2s.
`print(object)` writes one value, `print()` an empty line. There is no print(a, b): a `$"..."` that
is not written directly as the argument is formatted by .NET, and a two-argument form would take that
path without a warning. `str` and `format` are members too. C#'s alignment syntax ({x,10}) does not
compile against the handler.
6. Python.gc.collect (new file Python.gc.cs), which the scope uses.
FOUND AND SOLVED ON THE WAY (each one measured)
- Python garbage holds the training table. After the first run the check at the end reported "1 NumSharp
buffers, 1 Python buffers", also after a 5 second wait. LogisticRegression.fit leaves four unreachable
objects in a reference cycle (a tuple, a function, scipy.optimize._differentiable_functions.
_ScalarHessWrapper and a bound method); the tuple is the `args` of the optimizer and refers to X, which
is now a NumPy view of NumSharp memory. Python frees a cycle only when its collector runs. So after its
release the scope compares NDArrayPythonInterop.LiveExports with the count at its start and, while more
are alive, runs gc.collect(0), then (1), then (2). Generation 0 is enough here and took 0.8 ms; a full
collection took 23 ms. The first version did not see this, because the arrays that it passed to fit were
Python's own.
- A view of Python memory is given back on another thread. NumSharp.Interop.pythonnet 0.70.0 queues the
release of an imported buffer and a thread-pool thread drains the queue under the GIL (or the next
conversion call does). Right after a scope closed, 7 to 9 of its 11 imported buffers were still counted,
and 300 ms later only the one that Python's garbage pinned. So the check in Program.cs waits up to 5
seconds for both counts to reach 0 before it judges.
- A caught Python error kept the arguments of the failed call shared. After one failed
train_test_split(X, y[":5"]) two NumSharp buffers stayed exported: after the scope closed, after a .NET
collection with finalizers, after another Python call and after a full Python collection. They were
released only by a call of pythonnet's Finalizer.Instance.Collect() (its Threshold property reads 200).
The frames of the traceback hold the arguments, and the exception holds the traceback. Disposing the
exception's Type, Value and Traceback did not help. Python's own traceback.clear_frames did: Call
catches PythonException, empties the frames (ReleaseFrames) and rethrows the same exception. The exports
read 0 right after a failed function call, a failed method call and a failed format; the message, the
type name, the formatted traceback (13 lines) and the Python lines of the stack trace are still there.
- NumSharp 0.70.0 prints arrays as NumPy does: 19 arrays (integers, floats of mixed magnitude, float32,
bool, nan and inf, 3-D, a 2000-element summary, a wrapped line, 0-d, empty) gave the same text as
NumPy's str(). That is why print(array) needs no Python call.
- Facts of the published interop that the tree's documentation states, each probed: AsNDArray of a
read-only NumPy array is a non-writeable view; of a 0-d array a 0-d view; of a NumPy scalar a
non-writeable 0-d view; of a reversed float32 array a view; of a complex64 array NotSupportedException
("ToNDArray widens it to Complex (complex128) as a copy"); of an int, a str, a list, a dict or None
NotSupportedException ("the object does not export a PEP 3118 buffer"). copy() plus NDScope.Detach
takes an array out of a scope.
VERIFIED
Windows 11 x64, Python 3.12.12, numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1, pythonnet 3.0.5, against the
published NumSharp.Interop.pythonnet 0.70.0:
- dotnet build from a clean folder, Debug and Release: 0 warnings, 0 errors.
- `dotnet run -- classification` prints the seven lines of `python ScikitLearn/classification.py`;
verify.py: "7 lines same output", exit code 0. Three runs: exit code 0 each (the check at the end
passes), 2.6 to 2.7 seconds.
- print against real Python: a scratch port of 26 lines (every spec above, floats 1.0 / 1e-05 / 1e+16 /
1e15 / inf / nan / -0.0 / 5e-324, integers up to 2**64 - 1, bool, None, text, a 1-D array, a 0-d array
with and without a spec, an estimator, str() and format()) printed the same 26 lines as its Python twin.
- A port without its scope line: 3 NDW012 warnings at build, and "A port left memory shared with Python:
1 NumSharp buffers, 11 Python buffers", exit code 2, at run.
- Misuse: a tree object after its scope gives ObjectDisposedException and "<released StandardScaler>" from
ToString(); a scope closed out of order gives InvalidOperationException; a second Dispose does nothing;
a scope on a second thread works and leaves nothing shared.
- Python errors arrive as PythonException with Python's text ("Found input variables with inconsistent
numbers of samples: [569, 5]", sklearn's InvalidParameterError, "Unknown format code 'f' for object of
type 'str'"), and a hole of a type without a Python form gives NotSupportedException.
- Error paths of the program are unchanged: unknown port (exit 1), an interpreter that does not exist
(exit 1), an environment without scikit-learn (the message with the pip command, exit 1).
Ubuntu 24.04 x64 on WSL 2, Python 3.12.14, the folder copied outside the repository: 0 build warnings, the
port prints the lines of the Python program, exit code 0, verify.py "same output", and the Linux lines are
equal to the Windows lines.
NOT VERIFIED: macOS. No workflow runs the project. The 26-line print test ran on Windows only (the
formatting is done by Python, so the platform should not matter).
BREAKING (inside the example; nothing outside it used the first form)
The tree of b10ac7e is replaced: the unbound members (StandardScaler.New / fit_transform(scaler, X)), the
`_t` members, PythonRuntime.Run and the PyObject-typed signatures do not exist any more.
OPEN
- The skill file .claude/skills/pythonic/SKILL.md still teaches the unbound static form that this commit
replaces. It is not changed here: the owner decides.
- The plan for the other five pairs (b10ac7e's message) stands, and each pair now takes this form. For
them the tree needs, when a pair first uses it: text parameters typed as the f-string type, a `with`
block as an IDisposable scope, a C# function as a Python callback, attributes of fitted objects.
…70 rules for a port, the tree, arrays, the GIL, lifetime, errors, print, the backend, verification and the project, each one met by examples/Pythonic at ebcc245, with the measurement behind it and a list of what is not specified yet WHY The owner asked: "Kindly write down a new PYTHONIC_SPEC.md". The form of pythonic code changed in ebcc245 (classes with instance methods, `using static` imports, NumSharp arrays on both ends of a call, one scope line, print with Python's format specs). Until now that form was written down only in the README, which is a tour, and in commit messages. The skill file .claude/skills/pythonic/SKILL.md still describes the form that the owner's review rejected. WHAT IT IS examples/Pythonic/PYTHONIC_SPEC.md, version 2 (version 1 is the form of b10ac7e). Each rule has an identifier, so that a review can name it: PORT-1..12 the port: a pair is two files, one method, the namespace line first and the `using` lines below it, one C# form for each Python import form, the scope line as the first statement, one C# statement for each Python statement, Python's names, `var`, Python's comments, a translation table, what a port must not contain, the same printed lines TREE-1..13 the tree: the root class, one file for each package, a module is a static class, a function is a static method, Python's parameter names with `null` = not sent, .NET values in and out (a type table), one return type, a Python class is a sealed class over PythonObject, the three steps of a member, the three helpers and the argument types of Call, only fixed calls, the documentation of a member, accessibility ARR-1..7 arrays: NDArray on the C# side, an argument is a view, a result is a view, a copy only when a view is not possible, no element loops, no codec, NumSharp computes on Python's memory GIL-1..5 each member takes the GIL through PythonRuntime.GIL(); a port never sees it LIFE-1..9 a tree object owns one reference; the scope line; NDScope.Open() stands in the method (the analyzer); the close order; Python's collector for garbage that holds an export; one thread, innermost first; what leaves a scope; the two counts after a scope; the shutdown ERR-1..4 a Python error is pythonnet's PythonException, unchanged; Call empties the frames of the traceback; a setup problem is PythonInteropException; the tree's own refusals PRINT-1..6 the three forms of print; which side writes which hole; the `$"..."` directly as the argument; no C# alignment; str and format BACK-1..6 one engine, the interpreter search, ProgramName, the import cache, the shutdown VER-1..5 the same lines (verify.py), the lifetime check and the exit codes, no warning, a new format or conversion path is tested against Python, where a pair ran PROJ-1..3 the regular project file, the pinned requirements, the folder Section 1 states five goals. Section 13 lists thirteen constructs that this version does not specify (an attribute of an object, a dict or list argument, a `with` block, a C# function that Python calls, a text parameter with a format spec, a keyword that is None on purpose, operators and indexers, a second form of one function, control flow, a catch by the Python type, Python source as text, await, a shared tree), each with a proposal that is marked as a proposal. Section 14 has the checklists for a new member, a new pair and a review. Appendix A gives the nine measurements behind the rules. VERIFIED WHILE WRITING IT (scratch ports against the published 0.70.0 package and real Python; nothing in the project's code changed) - PORT-4, the import forms. All of them were compiled and run, and each printed what its Python twin printed: `using static Python.pkg.mod;`; a class alias (`using C = Python.pkg.mod.C;`); a module alias (`using metrics = Python.sklearn.metrics;`, for `from sklearn import metrics`); `using static Python;` and then `sklearn.datasets.load_breast_cancer(...)`, for `import sklearn.datasets`; and `using xp = NumSharp.np;`, for `import numpy as xp`. A lower-case alias gives the warning CS8981, which the rule states. - ARR-2. A reversed array (y["::-1"]), a stepped array (y["::3"]) and one column of a table as arguments of accuracy_score, confusion_matrix and roc_auc_score gave Python's results (0.4798, the matrix [[64 148] [148 209]], 0.061684). - PRINT-5. `print($"[{12.5,10}]")` does not compile: error CS1739, "The best overload for 'AppendFormatted' does not have a parameter named 'alignment'". It does not fall back to the print(object) overload and .NET's formatting. - PRINT-3. A C# float hole prints the exact value of the float: 0.1f gives 0.10000000149011612, as float(np.float32(0.1)) does in Python. - A script checked that each rule identifier that the text refers to exists (70 rules, 9 appendix items) and that each file that the spec and the README name exists. The README links the specification and lists it in its file table. The project builds with 0 warnings and verify.py reports "same output", as before. NOT DONE The skill file is not changed. Whether it gets this form, or points to this document, is the owner's decision. The same holds for the bridges in the sibling repository OptunaSharp, which are written to version 1.
…ed tile by tile, and a matrix against one short row several rows per kernel call - the F operand's rows are no longer gathered one strided row at a time (a new AVX2 kernel copies a tile of several rows, reading each column's rows as one contiguous piece): float64 (250, 400) 63.5 -> 35.3 us (NumPy 78.2), uint8 32.9 -> 9.3 us (NumPy 46.5), a (1000, 1000) matrix against its transpose 1111 -> 548 us (NumPy 1104), a (25000, 4) matrix against a row 85 -> 12.8 us (NumPy 140); 1,279 benchmark cells geomean 2.08x -> 2.12x NumPy, 54 new cells 1.17x -> 3.61x WHY After e2ef7f3 (the comparison engine) and 5b15137 (the aligned pool) 451 of 1,279 benchmark cells were under 1.5x NumPy. Most of them are floors both libraries share (memory transfer at 100,000 elements, the array allocation at 1,000). Three classes had a lever that was not built: - a C-ordered operand against an F-ordered one (24 cells, 0.89 - 1.54x), which is also `m == m.T`; - a one-byte matrix against a row (1.2 - 2.0x), and any matrix against a row of a few elements; - two stepped operands at 1,000 elements (1.0 - 1.4x). This commit builds the first two. The third was probed and has no cheap lever (see FLOORS). 1. TILES: AN OPERAND WHOSE ROWS ARE ADJACENT ELEMENTS An F-ordered matrix read in C order has rows whose elements are one COLUMN apart. A row of N elements touches N cache lines, and N pages once a column is 4 KB long. The rows route (ComparisonRun.RunRows) staged such a row with one strided move a row: 0.3 - 0.6 ns an element for a (250, 400) matrix, 0.65 - 1.3 for (1000, 1000), 3.8 - 8.6 for (3162, 3162). NumPy's scalar loop pays the same. The elements of one column ARE adjacent, so a tile of B rows reads B contiguous elements of each column and deals them out to B rows of a stack block. src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.TransposeTile.cs (new) TransposeTile(src, srcColumnStride, dst, dstRowStride, rows, cols, itemSize): dst[b * dstRowStride + w * itemSize] = src[w * srcColumnStride + b * itemSize] AVX2 groups of 2 x 4 elements of 8 bytes, 4 x 8 of 4 bytes, 8 x 16 of 2 bytes, 16 x 32 of one byte. - Each 256-bit register is built from two columns: the first column's piece in the low lane, the piece of the column `far` further in the high lane (vinserti128 with a memory operand, which uses no shuffle port). The in-lane transposition then gives rows of twice the columns and needs no lane-crossing shuffle. - Integer unpacks for every width. Current Intel cores run them on two ports and the floating-point ones (vunpcklpd, vunpcklps) on one: the 2-byte group runs at 0.13 cycles an element where its 24 shuffles on one port would take 0.19, and the 4-byte kernel went from 0.13 to 0.08 ns an element with them. - The 8-byte loop takes two row groups a step (0.18 -> 0.13 ns an element). - A tile that is not a whole number of groups ends with a group that OVERLAPS the one before it, so there is no scalar tail. That needs a destination that is not the source. - Below one group, for 16-byte elements and without AVX2 the tile is moved by the strided move kernel along its longer side (a plain loop without runtime IL generation). L1 speed, ns an element: 0.13 / 0.08 / 0.05 / 0.03 for 8 / 4 / 2 / 1 bytes (a strided move: 0.2; a memcpy of the same bytes: 0.05 / 0.026 / 0.013 / 0.007). It reads exactly the tile's elements and writes exactly the tile's elements. src/NumSharp.Core/Backends/Kernels/ComparisonRun.cs RunTiles (called by RunRows before its row-by-row loop) and TileOperand. - An operand is tiled when its successive runs are one element apart, in either direction, and the elements of one run are further apart (IsTiled). Rows that go downwards in memory are read from the tile's last row and stored from the block's last row upwards. - The tile is at most 8 KB of the kernel dtype. It is a whole row wide when 8 rows of that width fit, else the row is cut evenly; then it is as many rows high as fit. Widths and heights are multiples of the vector group. From a row SPAN of 4 MB (elements a row times the column stride) the fewest rows are 16 (4- and 8-byte elements) or 32: each column piece is then a cache miss, and further up a TLB miss. - The last tile of each direction is moved back to end on the operand's edge and computes some results a second time. RunRows never has an output that is an input (its contract). - The operand beside the tiled one: contiguous rows are read in place, or converted into the other block (one cast for the tile when the rows follow each other); one value a row is read in place; a second tiled operand gets its own block. - One kernel call for a tile when every operand's rows of the tile follow each other and the output's do; otherwise one call a row. - Declined (the row-by-row loop runs): 16-byte elements with rows over 64 elements, and conversions to complex128. Two F operands never arrive (the result is F-ordered: the iterator route), nor do out= calls (they arrive as chunks). Geometry, measured through the public operator (one process, the configurations in turn, five passes, minimum; us a call): stage of 8 KB against 16 KB (400, 250) uint8 8.9 / 13.6 (250, 400) int16 11.7 / 14.4 (250, 400) float64 40.5 / 36.8 (1000, 1000) float64 791 / 880 fewest rows 4 / 8 / 16 (250, 400) float64 40.5 / 35.3 / 43.2 float32 19.2 / 21.2 / 27.0 (1000, 1000) float64 791 / 641 / 753 float32 414 / 340 / 334 rows 8 / 16 / 32 on big ones (1448, 1448) float32 1550 / 1355 / 1456 int16 625 / 449 / 514 (3162, 3162) float64 17330 / 15261 / 17221, float32 11993 / 8726 / 8855, int16 9350 / 5545 / 4686, uint8 (16 / 32 / 64 rows) 3557 / 2926 / 2643 16-byte elements: a tile of them only saves the calls of the row-by-row loop (a 16-byte element is one move whatever the route). Rows of 3 to 40 complex128 elements gain 1.4 - 2.9x, rows of 400 lose (0.93x), and a conversion to complex128 through the house cast is slower than the per-element kernel (0.70x). Hence the rule. 2. A MATRIX AGAINST ONE ROW A kernel call costs about 8 ns whatever it compares, and a row that ends in a partial vector pays a scalar tail: together most of a short row (400 one-byte elements take 10 ns). ComparisonRun.RunRepeatedRow / RepeatsRow When the matrix's rows follow each other in memory, there are 16 rows or more, a row is at most 2,048 bytes and neither operand is converted, the row is repeated in an 8 KB stack block (filled by doubling: 12 copies for a row of two elements) and a block of rows is compared per kernel call. Kernel time of the repeated form against a call a row: 0.31x for (25, 40) of one byte, 0.52x for (250, 400) of one byte, 0.86x for (1000, 100) of eight bytes (800 bytes a row), 0.95x for (250, 400) of four bytes (1,600), 1.05x for (250, 400) of eight bytes (3,200: the limit is under it). With 7 rows the copy costs about 10 % more than the calls it saves. DefaultEngine.CompareOp.Rows.cs TryRowsComparisonOp sent many rows of fewer than 16 elements to the iterator's 2-D block kernel (1.5 ns a row plus 1 ns an element). A matrix against one row now stays here whatever the row's length. RESULTS ARE UNCHANGED Both routes give the bytes the row-by-row loop gives. Test switches: ComparisonRun.TilesDisabled, ComparisonRun.RepeatedRowDisabled. Counters: ComparisonRun.TileRuns, ComparisonRun.RepeatedRowRuns. TESTS test/NumSharp.Tests/Backends/Kernels/TransposeTileTests.cs (new, 3 tests) - EveryTileShape_IsCopiedExactly: 285 tile shapes (heights and widths on both sides of every group size) x 5 widths x 3 column strides (columns adjacent, with a gap, in descending order) x rows stored upwards and downwards; the destination holds the transposed tile and its sentinel margins are untouched. - Tile_BetweenInaccessiblePages_TouchesNoByteOutsideIt: the same shapes with the source and the destination against inaccessible pages (FencedMemory). - Groups_FollowTheInstructionSet_AndDegenerateTilesAreSafe. test/NumSharp.Tests/Backends/Kernels/ComparisonRunTests.cs (5 more tests, 24 in all; new helper RawTiled) - TiledOperands_MatchAnElementReference: 19 dtype pairs; an F operand on either side, with reversed rows, with reversed columns, two F slices, a column against an F operand; shapes from (2, 2) to (9, 600). - TilesAndRepeatedRows_AgreeWithTheRowByRowRoute: the routes against the loop they replace and against C-ordered copies up to 4.4 million elements (rows cut into several tiles, the taller tiles); the counters prove which route ran, and that long complex rows, a conversion to complex128 and two F operands are declined. - MatrixAgainstOneRow_MatchesAnElementReference: 8 dtype pairs, rows of 2 to 100 elements, the row on either side, and matrices whose rows do NOT follow each other (every second row, reversed rows). - RunRows_Tiles_OnRawMemory_TouchNoByteOutsideTheirOperands: a tiled operand on either side, each form of the operand beside it, gaps after the column pieces, downward rows and columns, every operand and the output against inaccessible pages. - RunRows_RepeatedRow_OnRawMemory_TouchesNoByteOutsideItsOperands: the same for a matrix against one row, with the exact number of calls that must take the route. The class takes about 6 s (4.5 s before). MUTATION PASS: 32 planted defects in the kernel and the two routes. 31 fail: 26 by an assertion, 5 by ending the test host (they read or write outside a buffer). The survivor gives the same results: it drops `width == count` from the test whether an in-place operand's rows are joined, which the output's stride already implies. A trap of the script: a test host that ends in the middle of a run still prints a summary line for the tests that had finished ("Passed! - Failed: 0, Passed: 1, Total: 1"), so five out-of-bounds mutants first read as survivors. The script now compares the summary's total with a clean run's. SWEEP: the scratch program against an independent reference, with a section for these forms over all 15 x 15 dtype pairs and six operators: 453,687,480 checks, 0 failures, with AVX2, with DOTNET_EnableAVX2=0 and with DOTNET_EnableHWIntrinsic=0 (373,543,380 before the section). The two test classes also pass with EnableAVX2=0, EnableAVX=0, EnableHWIntrinsic=0 and under NUMSHARP_DEBUG_GUARD_PAGES=1. SUITES (Release, CI filter TestCategory!=OpenBugs&TestCategory!=HighMemory) On 5b15137 with this change: NumSharp.Tests net10.0 18,486 passed, net8.0 18,485 passed, 0 failed NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed On 8aba70c with this change (see BASE): NumSharp.Tests net10.0 18,487 passed, net8.0 18,486 passed, 0 failed NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed The first net10.0 run there had one failure: AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy, a wall-clock limit on np.nonzero (115 ms where it allows 32). It passed in three runs alone and in a second run of the whole suite; the host had other work at that time. This commit does not touch np.nonzero. BENCHMARK (NPY/NS; two pinned P-cores; tier-up forced; per-cell minimum over three NumSharp runs and two NumPy runs) The 1,279 cells of sections A - F, the build of 5b15137 -> this commit: geomean 2.08x -> 2.12x under 1.5x: 451 -> 425 under 1.0x: 23 -> 22 lowest 0.86x -> 0.87x section B (layouts) 2.16x -> 2.27x; the other sections did not move A new section G (54 cells: F against C, a matrix against its transpose, C against F at 1,000,000 elements, a matrix of rows of four elements against a row), this executable with the routes off -> on: geomean 1.17x -> 3.61x under 1.5x: 48 -> 0 NumPy us | routes off -> on (one executable; CMP_OLD_ROUTES=1 switches the routes off): C against F, 1,000 elements float64 2.28 | 1.65 -> 0.84 (1.38x -> 2.72x) bool 1.87 | 2.11 -> 0.67 (0.89x -> 2.80x) C against F, 100,000 float64 78.2 | 63.5 -> 35.3 (1.23x -> 2.21x) float32 51.4 | 50.3 -> 21.2 (1.02x -> 2.42x) int32 54.0 | 52.7 -> 23.8 (1.03x -> 2.27x) int64 83.1 | 67.1 -> 39.6 (1.24x -> 2.10x) uint8 46.5 | 32.9 -> 9.3 (1.41x -> 5.01x) C against F, 1,000,000 float64 1281 | 1290 -> 680 (0.99x -> 1.88x) uint8 616 | 656 -> 151 (0.94x -> 4.08x) m < m.T, 316 x 316 float64 74.2 | 63.7 -> 36.6 (1.16x -> 2.03x) uint8 46.8 | 35.1 -> 8.5 (1.33x -> 5.50x) m < m.T, 1000 x 1000 float64 1104 | 1111 -> 548 (0.99x -> 2.02x) a matrix against a row, 100,000 uint8 8.97 | 7.23 -> 3.84 (1.24x -> 2.33x) float32 20.3 | 8.46 -> 6.82 (2.40x -> 2.97x) (250, 4) against a row float64 2.69 | 2.14 -> 0.72 (1.26x -> 3.75x) (25000, 4) against a row float64 140 | 85.1 -> 12.8 (1.65x -> 10.99x) uint8 122 | 64.5 -> 3.95 (1.89x -> 30.8x) The other cells did not move: geomean 0.981 of the earlier build's time. Four cells read 10 % slower (a contiguous array against a value, code this commit does not touch); in interleaved runs of the two builds they read 0.97 - 1.02. FLOORS THAT REMAIN (22 cells under 1.0x, 0.87 - 1.00x; 403 between 1.0x and 1.5x) - Two stepped float64 / int64 operands at 100,000 elements (8 cells): 3.2 MB of source, above this CPU's 2 MB L2. - A C# literal operand at 100,000 elements (9 cells): the literal's 0-d temporary, about 1 us of a 4.6 us call (plan X26). - Two stepped operands at 1,000 elements (3 cells under 1.0x; 1.3 - 1.5x for 4-byte elements). Probed: the two staging copies are 0.42 - 0.48 us of the 0.61 us the run takes (0.2 ns an element for each stream; a direct two-load deinterleave written for the probe measured 0.18), and the call's fixed part is 0.65 us where contiguous operands take 0.43. No cheap lever was found. - np.less(a, b, out) of two float64 arrays at 100,000 (0.97x) and the empty result (0,3) < (3,) (0.87x; branch strings has a guard). - Between 1.0x and 1.5x: contiguous operands at 100,000 elements in any order (the L2 transfer bound), contiguous and tiny operands at 1,000 elements (the array allocation: about 0.35 us of a 0.45 - 0.55 us call), a one-byte matrix against a column at 100,000 (1.3x: 250 calls with a value each), literal operands at 1,000 (plan X26). MEASUREMENT NOTES - The host's clock read 2.8 GHz during this work (a chain of dependent 64-bit multiplies took 1.06 ns a step; 3.4 GHz during the engine's work). Cycle counts above use the measured clock. - The host was not quiet (about 17 % background load). Routes were compared in one process, in turn, and effects under 1.3x were not acted on. MERGING strings ComparisonRun.cs, DefaultEngine.CompareOp.Rows.cs, the two test files and the kernel file exist on journey4 only (they were added by e2ef7f3 and by this commit). git merge-tree reports the same 13 conflicting files with and without this commit (10 at 5b15137; np.asarray.cs, np.all.cs and np.any.cs joined with daa5882, another session's commit). BASE Written and measured on 5b15137; moved onto 8aba70c (three commits of other sessions: daa5882 fix(matrix), ebcc245 and 8aba70c for examples/Pythonic) and the unit suite and the oracle replay run again there. DOCUMENTATION .claude/CLAUDE.md, "The comparison engine": the two forms of a rows iteration, "Tiles", "A matrix against one row", the kernel, the gates and the mutation-script trap, the benchmark lines, the floors, the clock note; the Core Source Files row.
…on that receives a numpy.matrix through its NDArray parameter returns NumPy's class and shape (np.sum(m, 0) is a 1 x N matrix, np.negative(m) and np.linalg.inv(m) are matrices, np.ravel(m) a plain array, np.stack((m, m)) raises), while the operators, the indexers and the NDArray members keep the plain-array meaning; 363 rule statements and 203 plain-array regions, a 4,645-case oracle tier, and an audit that replays 311,725 corpus cases with matrix operands and finds no library method that reaches a matrix rule
WHAT CHANGED FOR A CALLER
Before, the matrix semantics followed the static type everywhere: np.sum(m, 0) of an NDMatrix returned a plain
(N,) array, where NumPy returns a 1 x N matrix. Now a public np.* function looks at the OBJECT. The result is an
NDMatrix typed NDArray; a cast gives the matrix members back, no copy is made.
NDMatrix m = np.matrix("1 2 3; 4 5 6");
NDArray a = m; // the same object
np.sum(a, 0) // matrix([[5, 7, 9]]) was array([5, 7, 9])
np.negative(a), np.transpose(a) // matrices were plain arrays
np.ravel(a), np.where(a > 2, a, 0) // plain arrays (NumPy's asarray), unchanged
np.stack(new[] { a, a }) // ValueError: shape too large to be a matrix.
np.sum(a, 0, keepdims: true) // TypeError: matrix.sum() got an unexpected keyword argument 'keepdims'
a * a, a[0], a.sum(0) // operators, indexers, members: plain, unchanged
BREAKING: code that passed an NDMatrix to an np.* function and relied on the plain result now gets NumPy's
matrix result (a different shape for reductions and 1-d results). Pass m.A for the plain answer.
THE DESIGN (src/NumSharp.Core/Matrixlib/NDMatrixBoundary.cs)
NumPy decides per function, by three mechanisms, and each has a rule here:
* Wrapped __array_wrap__ or allocation through the operand's class: ufuncs, np.transpose, np.concatenate,
np.dot, np.linalg.inv. A 1-d result becomes a 1 x N row; more than two long axes raise.
* Reduced the fromnumeric wrappers call the operand's METHOD: a scalar without an axis, a row for axis 0, a
column for axis 1; keepdims: true raises the matrix method's TypeError.
* Plain asarray inside: np.cross, np.einsum, np.linalg.norm, every np.fft function.
* WrappedAll / WrappedPair / WrappedTriple / WrappedFirstTwoOfFour for several results (np.split, np.unstack,
np.divmod, np.frexp, np.modf, np.linalg.qr / eig / eigh / svd / lstsq), and WrappedIf for np.block, whose
class follows NumPy's ROUTE (repeated concatenation: a matrix; the single-allocation route past 2*512*512
elements: a new ndarray).
One statement per public overload:
if (NDMatrixBoundary.Sees(a))
return NDMatrixBoundary.Wrapped((a, axis), static s => flip(s.a, s.axis));
Sees is true only for a matrix object at depth zero. A rule raises a per-thread depth counter around a SECOND call
of the function, so that call and everything below it compute with plain-array semantics; the rule then shapes the
result. The arguments travel in a tuple to a static lambda: a capturing lambda would allocate a closure at method
entry on every call, plain arrays included.
A function WITHOUT a rule opens a region (using var plain = NDMatrixBoundary.PlainFor(a, ...)): its result is the
plain array NumPy's asarray-based functions return, and the safe answer where NumPy's is not implemented. An
object that KEEPS an array for later calls stores the plain view (PlainOf): numpy.ma (the masked array's data and
mask, AsData, MaskOperand) and the samplers' converted parameters (RandomParam.Convert). The NDArray members that
are implemented by np.* functions, the NDMatrix members and the engine methods that call back up into np.* open
regions too (PlainFor(this) / Inside()).
Facts NumPy's source decides, each probed on 2.4.2: np.linalg.solve and lstsq take their class from b; np.clip
from the array OR a bound; np.iscomplex is a matrix for a complex matrix and a plain array otherwise; np.r_ /
np.c_ / np.block cannot give a matrix entry a third axis (np.block([[[m]]]) raises "axis -3 is out of bounds for
array of dimension 2", np.r_['0,3', m, m] is a 4 x 3 matrix); np.average over a tuple of axes keeps every reduced
axis; np.apply_along_axis and np.piecewise return a matrix; np.real of a complex matrix is a matrix over the real
lane; power(in, in) and reshape(ref Shape) carry the rule of their by-value overloads.
THE CONTRACT IS MEASURED
No method of the library may reach the matrix rule of a public function at depth zero: library code is written
for plain results. FuzzCorpusTests.MatrixOperand_LibraryNeverReachesAMatrixRule turns that into a measurement. It
replays every corpus case that has a two-dimensional operand (311,725: the ordinary tiers, the numpy.ma tiers, the
index tiers) with those operands typed as matrices while NDMatrixBoundary.AuditInternalCallers is on, and requires
the set of library callers to stay empty. The audit reads the stack by NAME (the public function's frame can be
inlined). It found 57 call paths when it was written:
* NDArray members and the ~ operator that call np.* with `this`;
* engine methods that call back up (IsClose, NonZero / FlatNonZero / Argwhere, TryClipFlat, the bool add loop,
the variance helpers, Vdot, np.evaluate's arg stream, the float16 sum fallback);
* public functions without a rule (np.place, np.putmask, np.block, np.r_, np.bmat, np.iscomplex, fftshift,
hfft, the average with several axes);
* numpy.polynomial (mapdomain, the array arithmetic of PolyNumber, the calculus scale) and numpy.random (the
ufunc steps of uniform, hypergeometric and negative_binomial).
Each is closed by a region, by a rule for the calling function, or by PlainOf.
THE ORACLE TIER (test/oracle/gen_matrix_operand_oracle.py)
Existing corpus cases that a plain NumPy call reproduces byte for byte are re-emitted with one two-dimensional
operand typed as a matrix, and NumPy's answer is recomputed: class, dtype, shape, values, or the error type and
text. 4,645 answers over 260 functions, 680 of them errors.
matrix_operand.jsonl 3,885 + matrix_operand_host.jsonl 420 (RunHostLibmCorpus)
gaps/matrix_operand_gaps.jsonl 330 + gaps/matrix_operand_host_gaps.jsonl 10, carved exactly (every carved case
diverges, no kept case does):
MO01 24 np.isclose returns NDArray<bool>, which cannot be a matrix object
MO02 32 np.count_nonzero with an axis: NumPy hands keepdims to matrix.sum and raises
MO03 211 the median / percentile / quantile family: NumPy's keepdims and ravel paths meet matrix methods
MO04 46 np.roll, np.ediff1d, np.vdot, np.union1d, a refused np.append: ravel() of a matrix stays 1 x N
MO05 27 np.polyval, np.i0, narrow-float np.sinc, np.trapezoid with a matrix x: `*` is the matrix product
MO02 to MO05 are NumPy artifacts (matrix semantics leak into NumPy's own Python code); NumSharp returns the
plain result shaped by the function's rule.
The class is compared without a registry call. MisalignedRegistry returns null for a shape, threw, error-text or
expected-error divergence of a case whose layout starts with matrix@, and the two NEP50 0-d blankets skip the
tier. Operator keys replay through the function form (OpRegistry.MatrixOperand.cs).
Functions outside the truth table (tuple results, nested operands, non-deterministic arrangements, LAPACK) took
their rule from NumPy's source and two direct probes; they are pinned by unit tests with the probed class and
shape.
LIMITS ([Misaligned], pinned)
* a typed return cannot carry the class (np.isclose; the typed np.all / np.any overloads);
* np.gradient and np.meshgrid return plain arrays in their result structs (MO06 / MO07, OpenBugs.MatrixOperand.cs);
* numpy.ma has no masked_matrix (NumPy's ma.array(m).sum(axis=0) is 1 x N; here the plain (N,));
* the numpy.polynomial package computes plain (NumPy's mapdomain of a matrix is a matrix);
* a result is always a matrix VIEW of the plain result: no flags.owndata parity where NumPy allocates the
subclass;
* artifacts not reproduced: np.linalg.vector_norm(m) (an array of |x| in NumPy), np.average(m, axis=1,
weights=ndarray) (an N x N matrix in NumPy), np.apply_along_axis when func1d depends on the slice's rank
(NumPy hands it a 1 x N matrix), np.lexsort(m).
TESTS
NumSharp.Tests net10.0 18,505 passed / net8.0 18,504 passed (TestCategory!=OpenBugs&!=HighMemory)
NumSharp.Tests.Oracle net10.0 283 passed / net8.0 283 passed
NumSharp.Tests.Interop net10.0 741 passed (OpenBLAS backend)
new: Matrixlib/NDMatrixBoundary.Tests.cs (24), OpenBugs.MatrixOperand.cs (2 open gaps),
NumSharp.Tests.Interop/MatrixOperandLapackTests.cs (5), FuzzCorpusTests.MatrixOperand.cs (2 tiers, 2 gap
replays, the audit)
NDMatrix.Tests: Semantics_FollowTheStaticType became
Semantics_OperatorsAndMembersFollowTheStaticType_FunctionsFollowTheObject.
One run of the net10.0 unit suite failed AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy (108 ms
against a 32 ms wall-clock limit while other builds ran); it passes 3 of 3 times alone and in every later run.
PERF (two pinned P-cores, best of 15 batches, a host with other load: runs differ by about 3 %)
Plain arrays pay a type test per call. 44 cells (22 functions at 4 and 1,024 float64 elements), the tree before
against this one, each measured twice: geomean new / old 1.02, inside the noise but for np.asarray (2.1 -> 2.8
ns) and np.diagonal (176 -> 187 ns: two regions, each a try / finally).
A matrix operand pays the rule, about 270 to 350 ns a call (the scope that reclaims the hidden plain result, the
second call, the matrix view): np.add of 2 x 2 matrices 681 ns (plain arrays 413 ns, the NDMatrix operator 687
ns). Against NumPy's own matrix calls, 32 cells: geomean 3.0x, min 1.1x (m.sum(0) at 2 x 2 and the managed
np.dot at 32 x 32); the np.* cells 2.2x to 12x.
TRAPS FOUND
* NumPy 2.4.2's np.stack / np.dstack with a matrix operand raise and CORRUPT NumPy's heap (a later call dies in
PyDataMem_UserFREE). The generator makes every matrix call in a worker process and retires it after such an
error.
* CheckError reports "NumPy raises but NumSharp produced a result" as a VALUE divergence, so a source tier's
value excuse hid error-parity gaps; expected-error cases are strict for this tier.
* A function with a rule that is called by a function with a REGION loses its rule (the depth is raised). A
region therefore goes only on a function that has no rule on any overload.
* System.Collections.IList of a rectangular CLR array throws on a single index: the nested-operand scan
(np.block, np.r_) reads only one-dimensional arrays of references.
* An indexer's [CallerMemberName] is "Item" while its stack frame is get_Item: np.r_ passes the name.
* The benchmark project (benchmark/NumSharp.Benchmark.CSharp) does not build: CS0121 on np.amin / np.amax /
np.min / np.max in MinMaxBenchmarks.cs. It fails the same way on the parent commit; not touched here.
FILES
src/NumSharp.Core/Matrixlib/NDMatrixBoundary.cs (new); one statement in 275 source files of NumSharp.Core;
Matrixlib/NDMatrix.cs and NDMatrix.Tests.cs documentation; .claude/CLAUDE.md (the np.matrix section, the tier,
the source-file row, the OpenBugs list); test/NumSharp.Tests.Oracle/Fuzz/README.md (the tier's section).
…try under New APIs (construction, T / H / I / A / A1, the matrix product and power, the two-axis reductions, np.* functions that return NumPy's class and shape for a matrix operand) and three breaking-change lines (np.asmatrix / np.bmat return NDMatrix, the index-array assignment rule, the boolean-mask assignment rule) The release notes were swept at 7310f3f, before the numpy.matrix work. Its three commits (201e9c7 the class, daa5882 the explicit 0-d array, 51bd91f the np.* functions) had no line. New APIs & Modules: one feature bullet with a sub-list, the hashes on the parent line (the sub-bullets are facets of one feature, CHANGELOG_STYLE section 6). The oracle counts are the two tiers': 26,761 cases for the class (matrix.jsonl) and 4,645 NumPy answers for the functions with a matrix operand (matrix_operand*.jsonl with the carved gaps). Breaking Changes, each with old -> new and the migration: * np.asmatrix / np.bmat return NDMatrix (was a plain 2-D NDArray): `*` is the matrix product, an index keeps two axes, and np.* functions return NumPy's matrix results for it (np.sum(m, 0) has shape (1, N)); `.A` gives the plain array. Note for a reader of the earlier plan: typing the variable NDArray keeps the operators and members plain, but since 51bd91f it does not change what an np.* function returns, so the line names `.A` only. * An index-array assignment follows NumPy's value rule: a value that does not fit raises ValueError with NumPy's "shape mismatch" text before anything is written (was IncorrectShapeException), and a (1, N) value fits an (N,) selection (it threw). * A boolean-mask assignment takes a 0-d or 1-d value: a 2-d value raises NumPy's TypeError, a 1-d value whose length is neither 1 nor the number of true elements NumPy's ValueError. Not added: a Testing & Tooling line (the tiers are named in the feature bullet; the section's corpus total is a sweep-time figure and was left as swept), and a line for `m * np.array(2)` (a fix inside a feature that ships for the first time in this release, folded as a hash).
…heir matrix rule adds - TypeError for keepdims with a numpy.matrix operand (46 overloads with a Reduced rule) and ValueError "shape too large to be a matrix." for a result that cannot be a matrix (256 overloads with a Wrapped rule) 51bd91f gave the public np.* functions NumPy's rule for a numpy.matrix operand. Two of those rules can raise where the function did not raise before, and the XML documentation of the overloads did not say so: * a Reduced rule (np.sum, np.prod, np.mean, np.std, np.var, np.any, np.all, np.max, np.min, np.average, the nan twins) raises TypeError for keepdims: true, because NumPy calls the matrix METHOD, which has no such keyword ("matrix.sum() got an unexpected keyword argument 'keepdims'"); * a Wrapped rule (the ufuncs, the shape functions, np.concatenate, np.dot, np.linalg.inv ...) raises ValueError when the result has more than two axes longer than 1 ("shape too large to be a matrix."), for example np.stack((m, m)) or a ufunc of a matrix with a 3-d array. 302 <exception> lines in 142 files, each added after the last line of the overload's existing /// block. The lines were placed by a script that reads which rule the overload's body applies: * Reduced with a method name, a KeywordError throw or np.average's matrix helper -> the TypeError line; a Reduced rule whose method is null (the argmax family, the median family, np.trapezoid) accepts keepdims and gets no line; * Wrapped / WrappedAll / WrappedPair / WrappedTriple / WrappedFirstTwoOfFour / WrappedIf -> the ValueError line. Not documented: 4 overloads that have a rule and no /// block at all (np.mod, np.remainder, np.fmod with an object operand in Math/np.math.cs, and one np.ptp overload). A lone <exception> line would be their whole documentation; they need a full block, which is a separate change. Comments only: no code changed, NumSharp.Core builds for net8.0 and net10.0.
…and 39 pairs chosen by GitHub measurements (Matplotlib, SciPy and scikit-learn confirmed; Pillow, OpenCV and TensorFlow new), and 52 planned rules for the C# forms that the plan needs, each tried in a scratch port against a Python program on Windows and on Ubuntu, or marked open
The owner's review of the plan that came with version 2 (five more pairs: scikit-learn model selection
with the pipeline as Python text, scipy.optimize, scipy.stats, a Matplotlib line chart, a Matplotlib
histogram and scatter chart): "These are not great or enough, figure more, also use github advanced
search to find separate top 3 python libraries". This commit is the answer, in the specification. It
changes two documents and no code. No pair is implemented: the owner reviews the plan first.
WHAT IS NEW IN examples/Pythonic/PYTHONIC_SPEC.md (562 -> about 1,300 lines)
- Section 13, "The pairs". Nine rules for a pair (PAIR-1 to PAIR-9: one question with real data, a
feature that many programs use, NumSharp does a part of the work, the pair prints its proof, no
person / display / network, the same lines on each run, short, it states what it adds, plain Python).
The table of the six libraries. One table of pairs for each library: scikit-learn 7 (SK-1 is the pair
that exists), SciPy 11, Matplotlib 5, Pillow 5, OpenCV 6, TensorFlow with Keras 5. Each row has the
feature with its counts, the program as a question with its data, and what the pair adds to the
approach. Then the order of the work, a comparison with the plan of five pairs, and what the plan
needs from the project.
- Section 14, "The planned rules" (it replaces "Not specified yet"). PLAN-1 to PLAN-52: the statements
of a port (literals, loops, slices, operators, builtins, with, lambdas, the standard modules), the
parameters of a member (overloads, an optional positional parameter, a keyword of several forms, text,
functions, a tuple of one item), the results (a table of 14 kinds), the objects (attributes,
iterators, callable objects, objects and constants of a module, a module and its class under one
name, an object that NumPy converts, operators and indexes, one name for different classes, lists and
grids of objects, a function that writes into the array of the caller), and arrays / print / backend /
lifetime. Each has a state: "tried (C-n)" with the round of Appendix C, or "open". Section 14.6 has
nine open questions and a table of the rules of sections 3 to 12 that change when a planned rule
becomes a rule.
- Appendix B, "The measurements of GitHub": the method (code search for an import phrase or a name,
the dependents page, the stars, the topic search for the pool of candidates), the limits of the
counts, the table of 36 libraries, the ten topics, the counts of more than 200 names of the six
libraries, and the packages.
- Appendix C, "The scratch ports": six rounds (C-1 to C-6) and 16 findings (C-7 to C-22).
- Section 15 is the checklists (it was 14), with the steps for the plan and for a scratch port.
- The header table: version 3, the plan, the planned rules, the history. Sections 1 to 12 keep each
rule. Two sentences of PORT-10 now point to section 14.
README.md of the example: three sentences that point to the plan.
THE MEASUREMENT (2026-10-10, gh api, the legacy code search: "phrase" language:Python -> total_count)
Python files that import the library / repositories that depend on it:
numpy 47.4 M / 3.56 M, pandas 18.2 M / 2.79 M, torch 18.0 M / 0.85 M (not in the plan: NumSharp is
the NumPy of a port; pandas and PyTorch have their own conversions in the interop package)
matplotlib 9.88 M / 1.86 M, scipy 6.72 M / 1.42 M, sklearn 6.08 M / 1.35 M (the three of version 2:
confirmed)
PIL 3.82 M / 2.46 M, cv2 3.75 M / not available, tensorflow 2.97 M / 0.54 M (the separate three)
then transformers 2.95 M (PyTorch, downloads models), sympy 1.30 M, networkx 1.00 M, seaborn 0.92 M,
plotly 0.85 M, keras 0.80 M (inside TensorFlow), skimage 0.38 M, statsmodels 0.31 M, xgboost 0.17 M.
requests, tqdm, PyYAML and Beautiful Soup have high counts and exchange no arrays.
The features of a library are ranked the same way, for example scipy.stats 1.64 M, scipy.sparse 1.47 M,
scipy.optimize 1.31 M; plt.figure 3.41 M, plt.subplots 3.22 M; Image.open 2.61 M, Image.fromarray
1.01 M; cv2.cvtColor 1.48 M, cv2.imread 1.16 M; keras.layers 0.97 M, tf.keras 0.73 M. The five modules
of scikit-learn with the highest counts (metrics, model_selection, preprocessing, datasets,
linear_model) are all in the pair that exists.
Limits that the appendix states: the search cuts a phrase at each point and does not tell capital
letters; a name of several libraries counts for all (marked); a name with a variable in it is a lower
bound (marked); the table is not a full list of GitHub, only of the candidates that were named. After
about 150 requests GitHub answered about one request in one minute.
WHAT WAS TRIED (scratch ports: a copy of the project with more tree members, a port, and a Python
program with the same statements; "tried" = the same lines and exit code 0; not in the repository)
C-1 scikit-learn, SciPy 18 lines literals, attributes, dict results, an iterator, with,
C# functions that curve_fit and minimize call, named results
C-2 scikit-learn, SciPy 27 lines one name for two classes, None, range / enumerate / zip, a
callable object, an object at module level, a lambda for quad,
operators and an index on a sparse matrix, output=
C-3 Matplotlib, Pillow, cv2 17 lines the canvas as an NDArray view, hist, a grid of charts, a photo
to an array and back, OpenCV that draws into NumSharp memory
C-4 none 4 lines a line that Python prints by itself
C-5 Matplotlib, cv2 20 lines random numbers, computed slices, rows, @ and **, sorted,
a comprehension, bytes, an error by its type
C-6 TensorFlow 2.21, Keras 17 lines a Keras model, GradientTape, model.summary()
Rounds C-1, C-2, C-3 and C-5 also ran on Ubuntu 24.04 (WSL 2) with the same lines as on Windows.
Two Python scripts made the main calls of the 34 pairs that the rounds do not cover.
FINDINGS THAT CHANGED THE PLAN OR THE RULES
- A line that a Python library prints by itself is LOST when the output of the process goes to a file
or a pipe (2 of 4 lines, on Windows and on Ubuntu). The backend must send sys.stdout to the console
of .NET (PLAN-47) before the first pair whose library prints (model.summary(), verbose=1).
- OpenCV 5.0 (opencv-python-headless 5.0.0.93) has no cv2.CascadeClassifier and no detector files. The
face detector pair became a QR code and a template search (CV-6). OpenCV 4.14.0.94 still has it.
- A C# function that Python calls needs three lifetime rules (PLAN-49): the result goes back through
a Python list (27 arrays stayed exported with a return value), each call has a plain NDScope (15 s
-> 3 s), and a .NET exception stays out of Python (1 array stayed exported).
- A C# array converts to an NDArray by itself, so len(names) is CS0121 without a T[] overload.
- Two overloads for one optional positional argument bind plot(x, y, "r--") to the keyword `label`
without an error. One method with the optional parameter at Python's position does not.
- n / 2 with two C# integers truncates, Math.Round(2.675, 2) is 2.68 where Python gives 2.67, and
np.array([1, 2]) is int32 where NumPy gives int64.
- Environment.SetEnvironmentVariable does not reach TensorFlow; os.environ through Python does.
- np.random.seed(0) gives the same normal / uniform / randint / permutation numbers in NumSharp 0.70.0
and NumPy 2.4.2, so a pair can make its own noise.
- TensorFlow loads into the .NET process in 2.2 s and gives the same numbers after set_random_seed(0);
its environment has 1.9 GB (445 MB with the other five libraries).
- Matplotlib copies the data of a line: a chart that is not closed does not fail the lifetime check.
NOT VERIFIED
- No pair of the plan is implemented, and no planned rule is in the reference implementation
(examples/Pythonic at ebcc245). "Tried" is less than "verified": the scratch members have no
documentation and no review.
- The round with TensorFlow ran on Windows only. Nothing ran on macOS.
- Open, in section 14.6: TensorFlow in the plan or not; an error by its Python type (the exact-name
filter misses subclasses; proposal: NumSharp's ValueError / TypeError classes); with ... as name when
__enter__ returns another value; setters; integers (int / long); a name that needs a type; keywords
by position; a result that can be None; BigInteger; an object whose class Python decides at run time.
- The skill file .claude/skills/pythonic/SKILL.md still describes version 1 (the owner's decision).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NumSharp 0.71.0
🧭 Overview
np.polynomial.*- thenumpy.polynomialpackage, 172 of its 193 functions for all six bases, bit-exact.np.*- +48 functions.np.random.*- every NumPy bit generator, the wholeGeneratordistribution surface, and a legacyRandomStatethat is NumPy's own sampler code line for line, all byte-identical streams.np.ma.*- the masked-array module;np.emath.*- the automatic-domain math module.DTypeis NumPy's two-level dtype model -NDArray.dtypereturns anumpy.dtype-shaped descriptor, withnp.dtypes, byte order anddatetime64/timedelta64descriptors.max,min,ptp,nanmax,nanmin,argmaxand the fusednp.evaluatereductions match NumPy bit for bit, down to the sign of a ±0 tie and a NaN's payload, and most of them run faster.Tensor<T>/TensorSpan<T>.Detailed Breakdown
Show All
📦 New NuGet Packages
One optional companion package ships for the first time, co-versioned with NumSharp 0.71.0 (it depends on NumSharp).
NDArrayand System.Numerics.Tensors (Tensor<T>,TensorSpan<T>,ReadOnlyTensorSpan<T>); all 15 NumSharp dtypes cross as their own CLR type,Half,Complex,decimalandcharincluded.AsTensorSpan<T>()views anNDArrayof any non-negative-stride layout (contiguous, sliced, transposed, strided, broadcast) as aTensorSpan<T>over the same unmanaged buffer, through a disposable handle that keeps the buffer pinned;ToTensor<T>()copies into a denseTensor<T>.AsNDArray()views aTensor<T>as anNDArraywithout copying (writes go through to the tensor);ToNDArray()copies aTensor<T>,TensorSpan<T>orReadOnlyTensorSpan<T>.TensorPrimitivesdirectly, with no fill loop.📊 Dashboards & Docs
numpy.ma,numpy.emath,numpy.polynomial,numpy.char,numpy.strings, ...) and the members of its objects (nditer,dtype,finfo, the random classes): 1,137 of 2,519 APIs available in that expanded scope, the dashboard's default view.✨ New APIs & Modules
np.polynomial- thenumpy.polynomialpackage for all six bases (polynomial,chebyshev,legendre,laguerre,hermite,hermite_e) pluspolyutils: 172 of its 193 functions, bit-exact with NumPy 2.4.2 against generated oracles (139,882 cases, 0 excused);{p}below ispoly/cheb/leg/lag/herm/herme-57545200,38ed8da7,ac8608d9,38526155,e2ff9477,0c02740c(+0896cbf8,d494e83d).{p}val,{p}val2d,{p}val3d,{p}grid2d,{p}grid3d,{p}valnd- evaluation.{p}add,{p}sub,{p}mulx,{p}mul,{p}div,{p}pow,{p}fromroots- series arithmetic.{p}der,{p}int- calculus, with everym/k/lbnd/scl/axisform.{p}vander,{p}vander2d,{p}vander3d- Vandermonde matrices.{p}companion,{p}roots- companion matrices and roots (the roots compute through the OpenBLAS backend'seigvals).{p}trim,{p}line,{p}domain,{p}zero,{p}one,{p}x, the fiveX2poly/poly2Xpairs - helpers, module constants and basis conversions.as_series,trimseq,trimcoef,getdomain,mapparms,mapdomain-polyutils.chebpts/chebinterpolate,polyvalfromroots,format_floatand the six series classes.np.randombit-generator family - NumPy's engines, seeding and stream splitting, byte-identical to NumPy 2.4.2 and replayed overload by overload (493 members, 5 engines x 10 seeds) -c1e0feb5,65bdea7b(+0e39714b,eebe85a6).PCG64DXSM,Philox,SFC64- new bit generators;PCG64.advance,jumped,random_raw,state- engine controls;MT19937is now a NumPyBitGeneratortoo.SeedSequence.spawn,spawn_key,pool_size,state,BitGenerator.spawn,Generator.spawn,ISeedSequence- seeding and independent streams.default_rng- every NumPy seed form (integers,BigInteger, arrays,SeedSequence, a bit generator, aGenerator, aRandomState,nullfor fresh entropy).np.random.PCG64(42),np.random.Generator(...),np.random.SeedSequence(...)and the other classes - NumPy's module spellings;get_bit_generator,set_bit_generator.Generatordistributions - NumPy's wholenumpy.random.Generatorsurface on NumPy's modern algorithms, 29 samplers byte-identical todefault_rng-c0613b2f.beta,chisquare,f,noncentral_chisquare,noncentral_f,standard_t,standard_cauchy,vonmises- the gamma family and friends.pareto,weibull,power,laplace,gumbel,logistic,lognormal,rayleigh,wald,triangular- one transform per draw.binomial,negative_binomial,poisson,zipf,geometric,hypergeometric,logseries- discrete.multinomial,dirichlet,multivariate_normal(svd/eigh/cholesky),multivariate_hypergeometric(marginals/count) - multivariate.pareto/powerstay within 2 ULP /ceil(2/a)+3ULP of NumPy, whose C runtimeexpm1has no portable twin;multivariate_normalis byte-identical with the OpenBLAS backend and may pick the other eigenvector sign without it.RandomStateandGeneratortakeNDArrayparameters, broadcast againstsizeas NumPy does, byte-identical streams -39d446e0(+5f204884).integers,randint- array bounds andBigIntegerbounds, sog.integers(0, BigInteger.Pow(2, 64), dtype: np.uint64)ports verbatim -0e39714b.RandomStategains NumPy's form over any bit generator (np.random.RandomState(new PCG64(42))), the dictget_state(legacy: false)/set_state,tomaxint,ranfandsample-f87b4d04.np.ma- NumPy's masked-array module on a newNDMaskedArraytype, bit-exact vs NumPy 2.4.2 and gated by a 68,860-case differential corpus -5f1476f5(+ba15bad4,bf66bc0f,b33b6515,3fb1d897,e0d6a81c,67f8f96e,7a262afc).average,median,dot, the set operations,sort,unique,cov,corrcoef,convolve,correlate,polyfit,apply_along_axis,apply_over_axes- the extras.str/repr; only the structured/record-dtype functions are missing.np.emath- automatic-domainsqrt,log,log2,log10,logn,power,arccos,arcsin,arctanh(an out-of-domain input gives the complex result instead of NaN) -a3666a5f.np.*functions, NumPy 2.4.2 parity, each gated by the differential-fuzz oracle -hypot(correctly rounded: float32/float16 bit-exact, float64 bit-exact on ~91% of inputs and 1 ULP more accurate on the rest),divmod,fmod,remainder,float_power,heaviside,spacing,frexp,ldexp,gcd,lcm,fabs,signbit,fix- elementwise math ufuncs -5c667b09,df232348,44636747,6e0e5ea2,ade8d6b5,83e0e43e,a15f0ee4,68a84d3c,46f5f651.sinc,i0,real_if_close,unwrap- special functions and signal helpers -c3be0b7f,a0c6e2fe,877a36b2,b1937c51.bartlett,blackman,hamming,hanning,kaiser- window functions, bit-identical -0e9dbc42.trapezoid,gradient- numerical integration and differentiation, bit-exact -22d5f4d3(+f6037bc8,4898b40a).histogram,histogram_bin_edges,histogramdd,histogram2d- histograms -2693d83e(+0e80bc89).packbits,unpackbits,bitwise_count- bit packing and population count -3936fd0d,bab51edb.piecewise,putmask,put_along_axis- indexed and conditional assignment -76b2ab9d,86cc28b8,f417827e.vectorize,frompyfunc,apply_along_axis,apply_over_axes,broadcast_shapes- functional and shape helpers -b1841830,53b7a149.array_equiv,may_share_memory,shares_memory(withTooHardError) - array predicates -38f9a14e,4cd3b72f.logspace,geomspace- log-scale ranges, bit-exact -d48737ec(+3a367520).binary_repr,base_repr,typename- string helpers -c2f129b5,7cd38a97.maximum,minimum,fmax,fmin-out=/where=-e5afe84c.left_shift,right_shift,logical_and,logical_or,logical_xor,logical_not,modf-out=/where=/dtype=-3384a0b5.array_equal-equal_nan=-38f9a14e.indices,indices_sparse-long[]/Shapedimensions (np.indices(arr.shape)), andmgridpast 2^31 elements -07c7b94c.np.evaluate/NDExpr- the fused-expression DSL covers most of the ufunc surface and every reduction, NumPy-typed and bit-exact (29,563-case oracle tier) -6175a017,bd0091fb,e1400974,2f02d65d,eb1921e1,fbef1c4e,c5422cfc,dcbe7079.Maximum,Minimum,FMax,FMin,Fmod,CopySign,NextAfter,LogAddExp,LogAddExp2,Hypot,Heaviside,Gcd,Lcm,LeftShift,RightShift- binary nodes.Positive,Conjugate,Fabs,Spacing,SignBit,IsPosInf,IsNegInf,BitwiseCount,Rint,Real,Imag,Angle,Cast,Round(decimals)- unary nodes.LogicalAnd,LogicalOr,LogicalXor,Select,Clip, and the</>/<=/>=operators onNDExpr- logic and selection.Any,All,CountNonzero,NanSum,NanProd,Ptp,NanMin,NanMax,ArgMax,ArgMin,NanMean,Var,Std(ddof), the weightedAverage, axis forms andkeepdims- reductions.dtype=,casting=,order=,where=- keywords;expr.Compile()returns a reusableCompiledExpression.If,Switch,Mux,Saturate,NanTo,Coalesce,Relu,LeakyRelu,Elu,Sigmoid,Swish,Softplus,Gelu,Step,Nand,Implies,IsClose,Between,Bucketize,Lerpand 18 more - 38 decision and activation combinators that fuse like any node -d3057b39(+7aba9b4a).DTypesystem - NumPy's NEP 40-50 two-level dtype model (dtype classes and dtype instances), the descriptor every dtype parameter andNDArray.dtypenow use -896f02b2,c916c579,72e4d29c.np.dtypes(Float64DTypeand the other dtype classes),np.datetime_data- the class module.num,kind,char,itemsize,str,descr,byteorder,newbyteorder,isbuiltin,isnative,flags,type_num(NPY_TYPES) - the fullnumpy.dtypesurface.np.dtype(">i4")) anddatetime64/timedelta64descriptors that parse, print, promote and cast-check (no datetime storage yet).promote_types,result_type,can_cast,issubdtype,isdtype-DTypeoverloads with NEP 50 weak scalars.np.flat<T>(a)/a.flatiter.AsTyped<T>()- typed by-refC-order flat iteration -fa2e8521.np.ndenumerate<T>(a).AsRef(),np.ndindex(...).AsSpans()- allocation-free enumeration -afb0d5ce.nd.Unsafe.Span<T>(),Memory<T>(),Bytes(),Pointer<T>(),TryGetSpan<T>(),TryGetMemory<T>()- zero-copy BCL views of the buffer (they do not keep it alive: hold theNDArray);nd.AsEnumerable<T>()and the typed iterators'ToArray()/CopyTo(Span<T>)- safe copies -37a112be.Py.RegisterNumSharpCodec()(NumSharp.Interop.pythonnet) - the one-call codec registration, spelled on pythonnet's ownPy(a C# 14 extension member) -f18aaa23(+b648ee53).NumSharp.Collections-ConcurrentOrderedDictionary<TKey,TValue>,ConcurrentOrderedCompactDictionary<TKey,TValue>(about a third of the memory) and the lock-freeOrderedDictionary<TKey,TValue>: insertion-ordered, index-addressable dictionaries, plus a vendoredNumSharp.Collections.Concurrent.ConcurrentDictionary<TKey,TValue>with by-ref access -39e5ba17,47c12604,ba5c347d(+04677203,873900b1,478b5310).🧩 ndarray surface
ToArray<T>()- aTother than the dtype converts likeastype(it threwArrayTypeMismatchException), and every layout copies in one pass (~5.8x faster than before on the measured set) -73514657.ToMuliDimArray<T>()/(Array)nd- one pass over any layout; float16, complex128, aTother than the dtype (converted likeastype), 0-d and empty decimal arrays now work (they threw) -c243201a.ToJaggedArray<T>()- any rank (rank 6 came back wrong, deeper ranks threw), any layout (a transposed view gave wrong values, a column slice read out of bounds) and aTother than the dtype; 10.7x faster with 7.7x less memory -4231d2d0(+6f2ec473,ac89847a).⚡ Performance
Ratios are NumPy ÷ NumSharp - higher is better (
x2= twice NumPy's speed);xLOW->xHIGHspans the worst→best measured cell across sizes and dtypes.x0.70->x3.1-np.max/np.min/np.ptpon NumPy's exact reduction schedules, slab reductions folding eight reduced indices per pass (axis cells on non-C layouts were x0.28-x1.0; L3-bound 4M cells of 1- and 4-byte lanes still trail) -1d79ce61,f8f91b3d.x0.41->x4.7-np.nanmax/np.nanmin(float32/float64 axis cells were x0.15-x0.50, flat C now x2.89 / float32 x3.33; float16/complex128 non-C axes run NumPy's scalar loop at x0.41-x0.69) -de6333dd,268d91fd.x0.77->x106-np.argmax/np.argminalong an axis walk the strides instead of a per-output div/mod walk (axis 0: float64 x4.4-x8.8, integers x10-x16, bool x95-x106; memory-bound contiguous rows at 4M x0.77-x1.04), and the 64-bit flat argmax runs NumPy's single-pass tournament -ceac36f2.x0.89->x2.65-np.mean/np.sumalong an axis with a large output: the identity seed and the mean's division run as typed block passes (np.mean(a, axis=0)of 3x1M float32 was x0.29;mean(axis=1)over 3-wide rows stays below NumPy) -3f22d30a.x1.43->x1.83-np.convolvelong products without the OpenBLAS backend (1000x1000 float64 was ~x0.9, complex128 x0.25) -38526155.x1.15->x5.9- complex128divide,multiplyand integer-exponentpower, now on NumPy's own formulas -dc9caff5,338d260a,3e7d6b33.x1.26->x4.0-np.modfplain,out=and promoted calls (out=at 10M was x0.40) -3384a0b5.x1.0->x16-np.array_equal/np.array_equivfused early-exit compare (x15-x16 at N=1000, parity at 100K, instant on an early mismatch) -44604193.x1.05->x6.4- everynp.randombit generator fills through a bulk layer: all 105 engine x op x size cells beat NumPy -aa9905b7.x1.08->x5.58- legacyRandomStatesamplers (hypergeometric was x0.96, zipf x1.12, multinomial x0.97; the samplers bound by the C runtime'spow/log/expstay at x1.17-x1.49) -910a6480.x0.65->x17.6-np.evaluatecomparison, logical,whereand bool trees vectorize instead of running the whole tree scalar (a > 0.5was x0.16) -939d0636.x1.0->x22.2-np.evaluatemixed-dtype trees vectorize through lane groups (i4*2+f8x14.2 andf4*f8+f4x22.2 at 100K; 1K cells sit at the per-call floor) -3631fcc7.x0.86->x53-np.evaluatereductions stream block by block instead of materializing their child, andAny/Allstop at the deciding block (any(a > b, axis=0)was x0.37;count_nonzero(a > b)at 100K sits at the read-bandwidth ceiling) -ee639862,0f8819b7,129fbaba,78692d91,ceac36f2,0f78f6e2.x2.06- a smallnp.evaluatecall skips the iterator like NumPy's trivial loop:a*b+cat n=8 runs in 255 ns vs NumPy's 526 (out=: 75 ns) -30dbfe0d(+df799381).🎯 Parity & Fixes
np.max/np.min/np.ptp, flat and along an axis, run NumPy's own reduction schedules - the sign of a ±0 tie and a canonical-vs-payload NaN now match on every non-broadcast layout -1d79ce61(+129fbaba,f8f91b3d).np.nanmax/np.nanminare NumPy'sfmax.reduce/fmin.reducebit for bit: complex128 skips NaN (it propagated it), float16 on non-C layouts read the wrong elements, and an all-NaN float16 slice returns its first NaN -de6333dd(+268d91fd).divide(1 ULP off on ~33% of operands),multiply(NumPy's fusedsimd_cmul),powerwith an integer exponent (up to ~1e14 ULP off) andabs(NumPy's fusedsimd_cabsolute; 35.5% of values were off) -dc9caff5,338d260a,3e7d6b33,d1b89c11.percentile/quantile/medianbit-exact - NumPy's two-branch_lerpin each dtype's arithmetic,medianas the mean of the middle pair, and a view at a non-zero offset no longer reads the wrong data -d2c8d26e.np.round/np.aroundwithdecimals != 0portPyArray_Round- negative decimals threw, complex values stayed unrounded, float32 rounded in double, and the result dtypes follow NumPy -6db8238a.sum/prodof a size-0 or size-1 array widen the accumulator by NEP 50 like any other size -23ea2ad9.np.array_equal(a, a)with a NaN inareturnedTrue; NumPy returnsFalseunlessequal_nan-38f9a14e.53602da9.mod/floor_divide/fmodby zero give ±inf like NumPy (they gave NaN), and float16add/subtract/multiply/divideon theout=/where=/ strided paths compute in float32 like NumPy's HALF loops (they double-rounded through double) -df232348,b53a0cc1.maximum/minimumof float16 with float32/float64 keep the NaN's bits -ac89847a.np.linalg.eig/eigvalsof a float32 matrix with complex eigenvalues return NumPy's complex64 values (they kept double precision) -0c02740c.np.real/np.imag/.real/.imag,view(dtype)andgetfielddid not keep their owner's buffer alive, so after the owner was disposed (as insideeigvals) the next allocation overwrote them;resize(refcheck: true)now sees such views -59890501.np.convolve/np.correlatewithout the backend are byte-exact on float32 positions under 32 terms and complex128 positions under 8 (NumPy's scalarsdot/zdoturegimes), and complex products with inf/NaN follow NumPy (convolve([1+0j], [inf+0j])isnan+nanj) -38526155,058f4f27.np.arctan2(complex)corrupted the heap and crashed the process; it raises NumPy'sTypeError-dc2fc075.cbrt,floor,ceil,trunc,deg2rad,rad2deg,floor_divideandmodraises NumPy'sTypeError(it raisedNotSupportedException, and a zero-size complex input returned an empty array);IncorrectTypeExceptionnow derives fromTypeError-9a1eeb7e,2e205a1f.np.modfaccepts int, float16 and bool input like NumPy (it rejected them), float16 byte-exact -3384a0b5.np.traceof uint8/uint16/char accumulates in uint64 like NumPy;ndarray.flatof a 0-d array has shape(1,);np.take,ndarray.item()andlen()of a 0-d array raise NumPy's verbatim texts -e84fd2f1.np.argsortof an empty array with sort axis >= 2 overran the heap -74656f13.flags.owndata), like NumPy -32162b64.!(logical not) on a non-contiguous array returned wrong values -76b2ab9d.np.choose,tofile,tobytes,ogrid,poly1d,polyfit,multivariate_normal,apply_along_axis/apply_over_axes,flatiterslice/fancy access, cross-dtypeSetDataand the float16sum/meantemporaries;NDArray.ReplaceDatacould free another array's buffer -1dd23bbe,6f2ec473,7e60327d,e84fd2f1.Generatorparity (it shipped in 0.70.0) -1855ea9e.integerswith ahighpast the dtype returned constant data (integers(0, 257, dtype: np.uint8)was all zeros); it raises NumPy's error.lock, as in NumPy.choicewithout asizereturns one element (it returned shape(1,));permutedfollows memory order and casts intoout=safely; a-0.0scale, shape or range is rejected; a()size keeps float32.RandomStateparity -c1e0feb5,f87b4d04.RandomState()took its seed fromEnvironment.TickCount, so instances made in the same millisecond were identical; it uses OS entropy.choiceignoredreplace=Falseand searched the probability CDF from the wrong side;randint(dtype: bool)crashed.legacy-distributions.cline for line:gamma(shape<1),binomial,negative_binomial,poisson,f,pareto,power,standard_cauchy,multinomial,hypergeometric,zipf,geometricandvonmisesused other algorithms.SeedSequence- a typednullseed array seeded the empty sequence (the same stream on every run; now fresh entropy, NumPy'sNone), andSeedSequence(new[] { -1 })silently seeded0xFFFFFFFFinstead of raising -0e39714b,65bdea7b.result_typeof four or more operands dropped every other pair (result_type(int8, int8, float64, int8)was int8; it is float64) -896f02b2.np.evaluate- flat and axisSum/Mean/Prod/Min/Maxfollow NumPy's own schedules bit for bit (they matched the value only), a flatMeanof an integer child is NumPy's buffered float64 sum, NEP 50 typing gaps are closed, and two closures over one lambda no longer share a kernel -53602da9,e1400974,129fbaba,df799381.nd["1:3"]) re-parsed its pattern on every call because the staticRegexcache thrashed - 9-18 µs -> 0.6-2.8 µs per parse -28fa9d79.🧰 Testing & Tooling
numpy.polynomialunit (139,882 cases),np.ma(68,860),np.evaluate(29,563),np.emath,ndarrayinstance members, and complex128 / Decimal coverage, with a dtype-spread gate (at least 4 dtypes per op) and an applicability gate -57545200,67f8f96e,7a262afc,e84fd2f1,55a2a6a7,192f456c,1dd23bbe.np.randomoverload against NumPy 2.4.2 by its exact C# signature (493 members, 5 engines x 10 seeds, the full post-call state), with coverage gates and a nightly soak under 10 fresh seeds -20a5b60d,1c7ac693,8f70af7c,ecdab2fb.fd058f03.💥 Breaking Changes
DTypereplacesSystem.Typeas the dtype currency -c916c579,68a02258.NDArray.dtypereturnsDType(wasType), andnp.float64,np.int32and the rest areDTypedescriptors (wereType);DType->Typeis explicit (a.dtype.type), anda.dtype.Nameisa.dtype.name.DType: theType/NPTypeCodetwins are gone (aType,NPTypeCodeor dtype string still converts implicitly), andlinspace/nancumsum'stypeCode:isdtype:.result_type,promote_types,find_common_type,common_type, ...) returnsDType, andisdtyperejectsissubdtype's vocabulary ("floating"), as NumPy does.finfo.dtype/iinfo.dtypereturnDType(wereNPTypeCode), and the pythonnet dtype maps takeDType-ea080d8e."Safe"raisesValueError) -896f02b2.np.int32/np.int64report the local NumPy's LLP64 type numbers and chars (7/'l'and9/'q'; they were hard-coded to Linux's5/'i'and7/'l') -72e4d29c.ValueError(wasNumSharpException); it derives fromArgumentException, socatch (NumSharpException)no longer catches it -f319b657.np.maximum/np.minimum/np.fmax/np.fmintakeoutas the third positional argument (NumPy's order), notdtype; passdtype:by name -e5afe84c.np.logical_and/logical_or/logical_xor/logical_notreturnNDArray(wasNDArray<bool>), andnp.modf's tuple fieldIntergralisIntegral-3384a0b5.ndarray.shape = ...raises NumPy'sAttributeErrorwhen the layout cannot be reshaped in place (it silently copied); usereshape-6f2ec473.np.nanmax/np.nanminof an empty reduction raise NumPy's "zero-size array to reduction operation fmax which has no identity" (they returned a 0-d NaN) -de6333dd.new MT19937(42)seeds like NumPy'snp.random.MT19937(42)(throughSeedSequence), not likeRandomState(42);mt._legacy_seeding(42)is the old stream.MT19937's non-NumPy members (NextUInt32,NextDouble,NextInt,NextLong,Next,NextBytes,Seed,SeedByArray,SetState,Key,Pos,Clone) are gone, andPcg64StateDataisPCG64.State-c1e0feb5.RandomStatestreams ofgamma(shape<1),binomial,negative_binomial,poisson,f,pareto,power,standard_cauchy,multinomial,hypergeometric,zipf,geometric,vonmises(andgumbel/logistic/rayleighat scale 0) change to NumPy's values;binomial(int n)isbinomial(long n); inputs NumPy accepts (dirichlet([]), a NaN alpha,negative_binomial(p: 0)) no longer throw -f87b4d04.long:randintdefaults to int64 (was int32),random_integers/permutation/choice/multinomialreturn int64,NumPyRandom.Seedis auint(wasint), andSeedSequence.generate_statereturns anNDArray(wasArray) -eebe85a6.Generator(bit_generator)(wasbitGenerator),PCG64(seed)and the other bit generators (wasseedSeq, now anyISeedSequence), anddefault_rng(seed)(wasbitGenerator/generator) - named-argument calls must be renamed -eebe85a6,1c7ac693.NumPyRandom.uniform(NDArray low, NDArray high, DType dtype)is removed; two-array calls bind NumPy's broadcasting overload (float64;.astype()the result for another dtype) -39d446e0.