C++ search#210
Draft
ms609 wants to merge 918 commits into
Draft
Conversation
ms609
marked this pull request as draft
March 25, 2026 14:21
ms609
added a commit
that referenced
this pull request
Mar 28, 2026
ms609
added a commit
that referenced
this pull request
Mar 28, 2026
ms609
added a commit
that referenced
this pull request
May 18, 2026
In R CMD check, R runs as a non-interactive subprocess with captured stdout. R_FlushConsole() calls fflush() on that pipe; when the buffer fills the call blocks indefinitely, causing the 6 h GHA timeout seen on every ubuntu runner for PR #210. Gate the \r-overwrite progress line and the flush behind R_Interactive (FALSE in batch/check contexts). Interactive sessions are unchanged. At verbosity >= 2 in batch mode, emit plain \n-terminated lines so diagnostic logs still carry progress detail without the flush risk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ms609
added a commit
that referenced
this pull request
May 18, 2026
In R CMD check, R runs as a non-interactive subprocess with captured stdout. R_FlushConsole() calls fflush() on that pipe; when the buffer fills the call blocks indefinitely, causing the 6 h GHA timeout seen on every ubuntu runner for PR #210. Gate the \r-overwrite progress line and the flush behind R_Interactive (FALSE in batch/check contexts). Interactive sessions are unchanged. At verbosity >= 2 in batch mode, emit plain \n-terminated lines so diagnostic logs still carry progress detail without the flush risk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 19, 2026
WideSample() now selects topologically spread trees by solving the Max-Min Diversity Problem with the MaxMin package, dispatching by a quality tier auto-chosen from length(trees): - exact node-packing IP (highs) for very small sets, - DropAdd-TS while the dense distance matrix is affordable, - matrix-free anchored Gonzalez (on-demand distance-column oracle) beyond that, so the largest tree sets never materialise an N x N matrix. - Imports: MaxMin (Remotes: ms609/MaxMin) instead of FurthestPoint; the GonzalezColumn / DropAddTS / ExactMaxMin calls dispatch to MaxMin. - Add Porumbel (2011) and Sayyady & Fathi (2016) references. - dev/smoke_40k.R: 40,000-tree matrix-free smoke test. - test-WideSample: 16/16 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Until available on CRAN
Resize n0/n1 only when a new character has more states than current buffer capacity (once per character rather than once per taxon × split). Also compute max_state in the existing char-column pass (no extra loop). Adds dev/benchmarks/bench_quartet_concordance.R for timing at small/medium/large scales and profvis flame-graph support. Benchmarks (Windows, R 4.5): small (<1ms), medium (14ms), large (186ms) per call at 25/100/300 taxa respectively. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…n test (#247) * T-303: document css_search HSJ/XFORM safety; add sectorial HSJ regression test build_reduced_dataset() omits hierarchy_blocks/tip_labels/n_orig_chars/ hsj_alpha/sankoff_* fields, so rss_search/xss_search are already guarded to fall back under HSJ/XFORM (commit e5ff294, same class as the T-275 guard). css_search needs no guard: it never builds a reduced dataset — it runs tbr_search() with a sector_mask against the full ds, so score_tree() dispatches hsj_score()/Sankoff with complete data and the sector-internal heuristic is correct for every scoring mode. Documented this with an in-code comment so it is not re-flagged. Approach (b) (copy the fields into the reduced dataset) is not tractable: the HTU pseudo-tip is a Fitch from_above state-set with no valid HSJ tip_labels or Sankoff tip_costs representation, so the reduced dataset cannot be made correct for those modes without new from-above machinery in both scoring kernels. Adds a Tier-2 regression test driving the full HSJ + sectorial pipeline (rss/xss guarded, css on full ds) and asserting it completes, stays self-consistent, and is deterministic across identical-seed runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * paramrename * -progress * Suggest `highs` for MaxMin solver --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rift/sector opt-in + findings (#273) * docs(profiling): settle union-of-finals exactness gate (empirical) Fresh rooted union-of-finals is NOT an exact EW-Fitch insertion cost, under any rooting or on binary data. Three-way empirical verdict over 6066 reconnection candidates vs TreeLength full-rescore ground truth: - Per-child edge set E(D)=combine(prelim(D),up(D)): EXACT (6066/6066). - Union of TRUE edge sets E(A)|E(D): sound LOWER bound (under-counts, never exact). - Union of the DEPLOYED simplified final_ (fitch_indirect_length): OVER-counts (unsafe as a lower bound), because uppass_node's simplified rule never unions so final_ is a strict subset of prelim. Fresh finals => formula limitation, not staleness. The code comment claiming it under-counts is wrong for the deployed path. => "root like TNT -> cheap exact union-of-finals" refactor is a mirage. Adds dev/profiling/exactness-gate.md (verdict) + exactness-gate.R (harness). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(drift,spr): opt-in exact directional EW scorer + scan-error census The DRIFT phase (ts_drift.cpp) and standalone spr_search (ts_search.cpp) still scored pure-EW reconnection candidates with the union-of-finals approximation (fitch_indirect_length_bounded, reading the incrementally-maintained tree.final_), never migrated to the exact directional edge set that main TBR adopted in 2b299e4. Severity (dev/profiling/drift-exactness-gate.md): no inexact score is ever leaked (every accept is drift_full_rescore'd / full_rescore-verified), but on the deployed incremental path the union error is TWO-SIDED (over- AND under-counts up to ~100 steps, not the strict over-count the exactness gate measured with fresh finals). argmin selection then suffers an optimizer's curse (picks moves the scan is optimistic about), so drift admits moves outside its AFD envelope (env_violation 52-62% on Zhu2013/Zanol2014) and spr_search converges prematurely (union 810 vs exact 639). Fix (opt-in, union stays default; NA/IW untouched, gate !has_na && !use_iw): - TS_DRIFT_EXACT routes drift's SPR + reroot EW scans to fitch_indirect_length_cached over a per-clip compute_insertion_edge_sets, mirroring tbr_search. - TS_SPR_EXACT does the same for spr_search. - TS_DRIFT_SCANCHK: two-sided scan-error + decision census (select_flip, env_violation, applied-move predicted-vs-full_rescore; cf. TS_IW_SCANCHK). Validation: exact path applied_mismatch == 0 across all applied moves incl. reroots; default (flags-off) path byte-identical (drift/SPR/char-ordering tests pass). Local matched-wall pilot: frontier CROSSOVER -- exact wins the auto/ wall-race regime on all 3, union wins the thorough/saturated regime on 2/3 -> land opt-in, do not flip the default. spr_search is an unambiguous reach win. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(hamilton): drift-exactness matched-wall gate build + array scripts Dedicated build (lib-driftexact, does not clobber the shared cpp-search lib) fetching claude/happy-jennings-e7a165, and a 3-task array (one per dataset) running dev/profiling/drift-exactness-gate-bench.R at 30 seeds x nCycles {8,16,32,64,128,256}, union vs TS_DRIFT_EXACT, to resolve the matched-wall plateau-vs-climb crossover the local pilot showed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(hamilton): resolve gate deps from shared lib (R_LIBS_USER) lib-driftexact holds only the branch's TreeSearch; point R_LIBS_USER at the shared TreeSearch/lib so Rcpp/TreeTools/ape resolve at build+run time while the harness loads our TreeSearch via lib.loc=GATE_LIB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): close matched-wall gate with 30-seed Hamilton result Hamilton array (17870087, 30 seeds x nCycles {8..256}, 3 datasets) confirms the 8-seed pilot's matched-wall CROSSOVER with clean monotone gaps: exact wins the auto/wall-race regime on all three (~2.7-3x faster, no time-to-optimum regression), union wins the thorough/saturated regime on 2/3 (Zhu2013, Zanol2014 - exact plateaus, union over-diversifies deeper); Dikow2009 exact dominates every budget. best-of-30 equal on all three (reliability effect, not reachability). Verdict: LAND OPT-IN (TS_DRIFT_EXACT), do not flip the drift default. Adds the per-dataset raw CSVs and the frontier reanalysis script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): soften saturated-tail claim (nCycles-cap censoring) The nc<=256 gate compared a censored exact tail (exact ~3x faster, so frozen at its nc256 score with unspent wall) against union's fuller curve; exact was still descending nc128->256, not plateaued. So "union wins thorough on 2/3" is not established. auto/wall-race win stands. Extension (exact+union nc {512,1024} on Zhu2013+Zanol2014) running to settle at matched wall. Opt-in landing unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): resolve saturated tail — crossover is real (nc1024 check) Extended exact+union to nc {512,1024} on Zhu2013/Zanol2014 (30 seeds) to remove the nCycles-cap censoring. Exact does NOT plateau (Zhu 624.80->624.40) but descends slower than union; at genuinely matched wall union stays ahead through the saturated tail (Zhu floor 624.0 vs exact ~624.4; Zanol narrower ~0.13). So the auto/thorough crossover is real, not an artifact: exact wins the wall-race regime on all three, union wins the thorough regime on 2/3. Verdict CLOSED: land TS_DRIFT_EXACT opt-in, do not flip the drift default. Adds the extension CSVs and the combined-frontier reanalysis script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): sector-godrift isolation gate harness + Hamilton array (route B) Full "thorough" search with driftCycles=0 (main-tree drift off; verified gd=0 => 0 drift calls, gd=25 => all-sectoral) isolates the in-sector godrift scorer. Three arms none/union/exact, maxReplicates knob, matched-wall frontier. Array adds mbank_X30754 (180t) as the larger/harder case where sector drift is most active. Reuses lib-driftexact (no rebuild). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(spr): make exact directional the default EW scorer for spr_search spr_search is a pure hill-climb with no exact-TBR phase to launder union's over-count, so union converges catastrophically short (multistart best-of-25: Zhu2013 650 vs exact 625; Zanol2014 1292 vs 1265 — systematic, not variance), and exact is also faster per start. No crossover, unlike drift. Flip default to exact; TS_SPR_UNION restores the old scorer (kill switch). Tests green (SPR/spr-nni-opt/spr-state-restore/drift). Reach today = standalone ts_spr_search (sprFirst OFF in presets), but this makes an SPR warmup viable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): route-3 marginal-value experiment (drift vs ratchet at matched wall) From a shared per-seed local optimum, compare where a wall-second is best spent: incumbent ratchet vs exact-drift vs union-drift (control). If exact-drift's anytime frontier beats ratchet's over some wall range and dominates union-drift, drift earns a place in wall-limited recipes (route 3) -> worth re-tuning. mbank_X30754 (180t) is the large/hard anchor. Reuses lib-driftexact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): route-3 follow-ons — +replicate gap-closer + training-corpus recipe test arm A (+replicate): is a wall-second better on exact-drift than on the project's real incumbent lever, more RAS+TBR replicates? Same datasets as route3 for frontier comparison. arm B' (recipe): whole-search thorough on TRAINING-split MorphoBank (EW-recoded, one key/task, sector drift off), arms none/union/exact main-tree drift, to see if the drift scorer choice moves a wall-limited recipe. Both reuse lib-driftexact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): route-3 recipe verdict — NO-GO (marginal win, ensemble-redundant) Marginal value (from a shared local-opt): exact-drift is the best single continuation, beating ratchet + union-drift + replicate. But the whole-search recipe test on 8 training-split MorphoBank keys (EW, sector-drift off, sequestered) shows main-tree drift is REDUNDANT in the multistart+ratchet+ sectorial+TBR ensemble at both saturated (none ties + faster) and tight-budget ends (exact beats none only 1/4 keys). So the isolated win doesn't survive -> no recipe surgery; exact-drift stays opt-in (TS_DRIFT_EXACT), D preset-refactor not worth it. Adds companion note + raw CSVs + analysis. (spr_search exact default stands as the real landed win from this line.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… Scheme 1) Exact incremental replacement for the per-clip from-scratch compute_insertion_edge_sets recompute, gated to the plain directional EW/IW TBR regime (TS_L3B_INCREMENTAL, default OFF). Patch-from-full "Scheme 1": maintain a pristine per-pass intact-tree base; per clip patch only the changed frontier (worklist incremental up-pass, value-based attenuation stop), score, then restore base over the changed-node list. Shares ts_fitch_combine with the from-scratch path so patched words are byte-identical. Chose Scheme 1 over the design doc's Euler "Config C": the 482t evidence that reopened the lever (fp_ref) supports Scheme 1; the Euler descend-delta was killed at small N and never re-measured; the clip loop already restores per clip (Scheme-1-native) and keeps the shuffle => bit-identical C-vs-A. Advisor concurred. Rationale + results in dev/profiling/l3b-land.md. Correctness: per-clip oracle (edge_set == from-scratch, full array) mismatch=0 across random EW (64 configs), real multi-state/ambiguous panel + project5432 subsamples, EW+IW, single-rooting and do_reroot; C-vs-A bit-identical; tbr completeness oracles clean. do_reroot supported (not a footprint penalty). Wall (raw tbr_search, real project5432, bit-identical trajectory): wash at <=75t, 1.10x@120t, 1.39x@240t, 1.36-1.38x@482t (both unrooted) — the predicted large-N ceiling. Open: end-to-end MaximizeParsimony at shipped ratchet on Hamilton (sector share is gated out, so end-to-end is diluted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full MaximizeParsimony C-vs-A on project5432 (EW): 1.01x wash at both 120t and 240t, despite raw tbr_search being 1.39x at 240t. Coverage confirmed (l3b fires in driven+ratchet; sector ~30% gated out) and the per-clip oracle is clean in the ratchet upweight/collapsed context. The raw-TBR win is diluted to nothing end-to-end by the sector exclusion + per-pass base-recompute amortization across ratchet's short passes. Verdict: exact/correct large-N raw-TBR lever, NOT a net mission-wall lever as built -> land default-OFF; the unlock is incremental base update after accept. Replaces the earlier ~1.2x end-to-end estimate with measured numbers; marks the Hamilton 482t full-search run not-worth-running until the base-update follow-on lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ran the cheap discriminator (MP 240t, TS_L3B_STATS+ORACLE): in-search fp_changed = 0.244, essentially the clean 240t footprint (0.243) -> the per-clip patch win is INTACT, refuting "in-search fp elevated -> win structurally small". So the wash is sector-exclusion (predicts ~1.24x) + the per-pass O(n_node) base recompute, which MP's short ratchet passes amortize poorly. Confirms base-incremental-update as the plausible unlock; notes it is itself a correctness-critical kernel (TBR-accept, two dirty paths + reroot, harder than the clip patch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… honest verdict Replaces the per-pass full compute_insertion_edge_sets base recompute with an O(footprint) in-place update after each accepted SPR move (update_base_after_spr_move): the move decomposes into a remove-leg at nz + a single-site insert-leg at above; nx gets a fresh up so A1-path up is recomputed (not assumed invariant). Gated SPR-only (TBR-reroot/reroot/first-pass fall back to full recompute); pass-top recompute skipped when l3b_base_current; buffers synced over the changed set; kill switch TS_L3B_NOBASEINCR. Base oracle (base == from-scratch after each SPR accept) clean across EW+IW tbr_oracle (to 120t x18) and full ratchet MaximizeParsimony on real Zhu2013/Zanol2014; first-version root-child seed bug caught by the oracle and fixed. Measured (project5432, EW): base-incr removes the per-pass recompute, converting plain-l3b's small MP regression (0.99x @240t) into a clean wash (1.00x), and preserves raw tbr_search 1.39x@482t / 0.97x@37t. But it yields NO net MP win => the base recompute was NOT the bottleneck. The MP wash is structural: short ratchet-recovery passes (few clips/pass) + sector ~30% excluded. Verdict unchanged and now complete: exact large-N raw-TBR lever, not a mission-wall lever; land default-OFF. STOP (advisor gate: fixed cost removed, still no win). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The L3b incremental edge-set patch is EXACT (bit-identical trajectory) and 1.36-1.38x faster on raw large-N TBR (dev/profiling/l3b-land.md); end-to-end MaximizeParsimony is a clean wash (neutral, with the base-incremental follow-on). Per objective (ii) be-as-fast-as-possible, take the free large-N win: default-ON for n_tip>=150; small trees (where the raw-TBR win washes out) pay zero overhead. Gate: l3b_active = !TS_L3B_OFF && (TS_L3B_INCREMENTAL || n_tip>=150) && use_directional && ... TS_L3B_OFF kill-switch (full disable) TS_L3B_INCREMENTAL force-on regardless of size (A/B + small-tree testing) Verified on the merged+flipped build (CCACHE_DISABLE --preclean; header changed): force-ON == force-OFF byte-identical on the panel (Zhu/Zanol x2 seeds: score+n_evaluated+passes), l3b fires (patch_clips=3909), n_tip<150 default inert (== pre-#6 #7 baseline). The n_tip>=150 exact path was validated at 482t by the chip's per-clip oracle + C-vs-A. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-seed assertion (seed 1001) that zero-mode and upweight-mode ratchet produce different search statistics coincidentally held equal on macOS after d8c5999 (homoplasy character ordering) shifted the RNG-seeded trajectory under libc++'s std::sort tie-break. On a 15x10 binary matrix the two modes coincide on ~3-7% of seeds; which seeds coincide is platform dependent, so a single fixed seed is fragile. Assert instead that the modes diverge on >=1 of several seeds (early break), so an incidental single-seed coincidence no longer fails while a genuine collapse (upweight silently == zero) still does. The deterministic regime-distinctness guarantee is already covered by test-ts-na-evcache.R. Validated green on a clean committed-HEAD build (CCACHE_DISABLE=1 --preclean): all 9 blocks pass, no crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…liverable ratchetPerturbMaxMoves 5->auto/scaled corpus anytime A/B (Hamilton job 17872154, 25 training matrices x 3 arms, validation sequestered) => NO broad wall benefit: 14/25 tie exactly; clean large/xlarge reach the optimum in the same replicate (rep2hit ratio 1.000); narrow reach regression on a degenerate 4062t giant; marginal noisy +reach on one hard multi-basin matrix. Verdict: keep the default at 5; the kick is an opt-in hard-tail lever, not a default. Rewrite dev/plans/2026-07-11-...: retire the FALSIFIED clade-generation thesis (restore238 shows TS's TBR rebuilds 1943 from within ~8-12 TBR moves), reframe to cold-start basin capture, correct the numbers, fold in the kick result. Adds kick_anytime.R (harness), kick_anytime_hamilton.sh (SLURM array), kick_analyze.R (analysis), kick_anytime_FINDINGS.md (write-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…from 1943) Mission A angle #2 (push the reweighting kick as a basin-HOPPING schedule to cross the last steps past the ~1945 TS floor on project5432). Mechanism: src/ts_ratchet.cpp is a MONOTONE strict-descent ILS (accept-only-if- strictly-better, else reset-to-best), so strength/schedule/multi-kick all stay inside a monotone frame; the only escape lever is the acceptance frame (accept-worse: drift, or reweighting with accept-worse between cycles). Gate (buildless, dev/benchmarks/basin_hop_gate.R): ordered-char re-score by label (self-check L(1943 floor)==1943) + TBRDist(exact=FALSE, hard lower bounds) to the 1943 floor. Every near-optimal TS tree from EVERY generation method (kick 1945 / strongperturb / cold / random / largefirst / invweight, 1945-1958) is tbr_min >=97 TBR from 1943 (min 97, best kick only 2 score-steps above 1943). Corroborated by basin_5432 (descent 0/5 by d~20) and geom_full (~109). No accept-worse schedule walks ~100 rearrangements to a narrow distant basin, so the afd-sweep probe and the accept-worse build were NOT run. Scope: refutes route #2 only (escape the floor). Angle #1 (cold-start constructor) and #3 (structured breadth) are untouched - they generate a start rather than walk from a floor. Kick is not useless (CID 0.22 vs cold 0.38: a better LOCAL basin). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gating [focus-test: ts-ratchet-opt] R-CMD-check gates the macOS `full` job behind the ubuntu/windows `core` wave (full needs core needs sense-check), so a flake elsewhere stops the macOS run from ever building. focus-test runs ONE testthat file on ONE OS (default macOS-latest) with no full R CMD check -- via the Actions tab (workflow_dispatch) or a [focus-test] / [focus-test: <filter>] marker in a commit message. This commit's own [focus-test: ts-ratchet-opt] marker triggers a macOS run of the de-flaked ratchet test as a live check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto-memory is now organised as campaign hubs indexed at the top of MEMORY.md; record the write-side rule (advance a campaign by updating its hub the same turn) on the always-loaded project-instructions channel so future agents follow it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o diameter build)
Zero-code precondition for a TNT-style dss (diameter-limited local sector) build.
The dss benefit (local sector -> cheaper escape) is driven by TIP-COUNT, which the
existing sectorMaxSize knob controls; rev5's missiongate sweep never varied it small
(held maxS>=0.65n). This probe sweeps small maxS {12,20,30} x ras=3 from the fixed
TNT T0, vs a large-sector anchor and ratchet, on the missiongate protocol.
Local pilot (Zanol s1, Zhu s1, n=3) + Zanol reach test (n=16 restarts, up to r=256):
- steps/Mcand APPEARS to favour small sectors (-2.4 vs ratchet -1.5) but this is a
candidate-COUNT artifact: small-sector candidates cost ~2x ratchet's per wall
(per-pick reduced-dataset build + 3x RAS overhead), so on the WALL axis (the true
mission axis) ratchet dominates the hard target (-29 vs -24 steps/s, reaches deeper).
- REACH CEILING: small sectors PLATEAU at -12 on Zanol at ANY budget; ratchet reaches
the -13 target. Whole-tree-barrier prior (rev2): a sector <= maxS tips cannot emit
the split(s) for the -13 reorganization.
=> dss/diameter-limiting is strictly MORE local -> shallower reach -> cannot help.
No C++ build. Confirms + extends rev5 (sectorial Pareto-dominated by ratchet) to the
small-maxS regime it never measured. Hamilton n=15 confirmation scripts included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-a source dir gone on Hamilton) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…losed Hamilton array 17898332 (build 17898331, HEAD 023b038), 12 cells x n=15 (3 TNT-T0 seeds x 5 reps), corrected wall+reach aggregate. VERDICT 0/4: no small-sector config reaches ratchet's median depth at <= its wall. On the hard Zanol target ratchet dominates EVERY axis incl the biased steps/Mcand (-1.90 vs best smallmax -1.78). The n=3 local-pilot "lead" (maxS=12 r=1 = -2.39 st/Mcand; a Zhu wall win) was noise + metric bias: at n=15 it falls to -1.19 and the Zhu win vanishes (ratchet med -3 vs smallmax -1). steps/Mcand is a BIASED axis here: small-sector candidates cost ~2x ratchet's per wall (reduced-dataset build + 3x RAS overhead over fewer candidates), so it flatters small sectors. Adjudicated on WALL + reach instead. => dss/diameter-limiting is strictly MORE local -> shallower/less-competitive -> cannot help. No C++ build. Confirms + EXTENDS rev5 (sectorial Pareto-dominated by ratchet) to the small-maxS regime rev5 never measured. Also refutes rev5's handoff nuance (small maxS + ras=3 vs single ratchet start on Zhu/Giles): smallmax median <= ratchet median on all cells. Full writeup: DSS_FINDINGS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… retested The selectem_large anchor used auto picks (~3/round), under-provisioning rev5's selectem (explicit pk 5-20): my Zhu selectem_large median -1 vs rev5's -6. So it is NOT a faithful rev5 reproduction and cannot speak to rev5's handoff (large-band selectem being lower-VARIANCE than a single ratchet start on Zhu/Giles — a variance question never measured here). Retract "refutes rev5's handoff"; that stays OPEN as rev5 left it. Core verdict UNAFFECTED: the smallmax arm used auto picks too, but for small bands that gives MANY picks (~16/round at maxS=12) = over-provisioned, so its 0/4 loss vs ratchet (esc_med-vs-ratchet-at-wall, same pooled seeds) is decisive. Also de-emphasised the cross-seed-pooled reach flag as non-load-bearing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TS_PACK_LOCAL packs each 64-char block to its own applicable-state alphabet
rather than the global one, cutting Fitch planes scanned per candidate on
low-state morphological data (~1.1x small -> ~1.5x char-rich, climbing with
nChar). Score-EXACT per tree; now default-ON (set TS_PACK_LOCAL=0 to force off).
- ts_data.cpp/.h: default-ON flip; add CharBlock::plane_state[] mapping each
packed plane back to its global applicable label (identity in the unpacked
layout), populated in build_dataset by inverting loc[] through state_remap.
- ts_rcpp.cpp: ts_na_debug_char reconstruction consumes plane_state[] so packed
output prints the global state label, not the plane index.
- test-ts-pack-local.R (new): score-exactness (TreeLength ON vs OFF) + a
hand-built non-contiguous-alphabet fixture that exercises plane_state[] -- the
only gate that does, since the scorer is index-agnostic.
- reeval pack_{gate,reach,mpt_gate}.R: OFF-arm switched to TS_PACK_LOCAL=0
(unset now means ON).
Validation: pack_mpt_gate Sansom2010 EW/IW/NA byte-identical ON vs OFF (score +
ntree + candidates_evaluated, 3 seeds; NA reproduced the 66-tree MPT set); full
testthat default-ON 0 failures (11337 pass); corpus 18/18 reach-neutral.
ts_na_debug_char is package-internal (no exported surface changed) and
PlotCharacter is pure-R, so no user-visible reconstruction changes. The packed
ambiguous-set content differs (local vs global alphabet) -- correct, not a
regression: phantom states never appear in an MPR, proven by identical
score+candidates_evaluated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give the 4-wide flat reroot batch kernels (fitch_indirect_cached_flat_x4 / indirect_iw_cached_flat_x4 + NA) the same next-row software prefetch the scalar reroot path already has. The x4 batches issued 4 cold demand loads to scattered vroot_cache rows per batch with no hint; the in-batch MLP can't hide the batch-boundary head latency (cutoff early-break is a control dependency). EW x4 prefetches kept_ei[ki..ki+3] (next batch). IW x4 tracks a parallel iw_pf cursor into kept_ei -- valid because its skip predicate is identical to the kept_ei construction (use_collapsed == !collapsed.empty(), ts_tbr.cpp:1532), so kept_ei is the same candidate sequence in the same order. Purely additive and flag-gated (opt-in TS_TBR_PREFETCH=1), so flag-OFF is byte-identical to prior committed code by construction; a prefetch hint cannot change a score anyway. Byte-identity validated: ts_tbr_diagnostics ON vs OFF on project970, EW + IW, identical score + n_candidates_evaluated (38.9M/44.8M) + n_clips + final edge. Local Windows/MinGW A/B flat (project970 60.0->60.2, project510 57.1->57.0, no regression) -- expected, since the memory-latency win only surfaces in the EPYC L3-pressure regime. Landed default-OFF as a validated building block per the lever-6 precedent; the win/wash is settled on Hamilton EPYC raw-TBR + full MaximizeParsimony wall before any default flip. See dev/profiling/kernel-prefetch-x4.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the +1 slack on the EW/NA integer bail cutoff (best_candidate - divided_length + 1 -> + 0) at the EW sites: spr_scan_plain_ew template, SPR general, reroot x4, reroot scalar. The scorers bail on extra_steps >= cutoff; the +1 let an equal-length, non-improving candidate accumulate its full extra before rejection (scanning every block on the near-optimal plateau), whereas +0 bails it the moment its running extra reaches the current minimum. Byte-identical: acceptance is strict < everywhere, so a bailed candidate (extra >= cutoff = best-divided) is rejected identically, a true improver never reaches the tighter cutoff, and an equal candidate bails one block earlier but is still rejected. The +1 was not protecting acceptEqual (decided post-scan on the recorded best_candidate), and the drift path already ships slack 0. Gated by a cached tbr_cutoff_slack() (default 0; TS_TBR_LOOSE_CUTOFF=1 reverts to the historical +1 for the A/B). Left the +1 on spr_scan_plain_iw (cutoff dead on the IW path -- IW bounds on the float best_candidate), the IW-float cutoff, and the INT_MAX init. Validated byte-identical (score + n_candidates_evaluated + n_clips + final edge) tight vs loose on project970 EW + IW + Sansom2010 NA. Chosen by the post-packing lever-hunt workflow with 3/3 adversarial verifiers confirming exactness + non-wash. Local ns A/B is flat (blocks-scanned is hardware-independent, so representative): the exact-equal-full-scan population is small in practice (dedup skips equivalent rerootings; worse candidates already bail early). Landed default-on regardless -- byte-identical, guaranteed non-negative, the correct tighter bound; may pay in the raw-TBR @482t plateau regime, to be checked on the EPYC A/B. See dev/profiling/kernel-cutoff-tighten.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ened it) Whole-program -mavx2 is 8-9% faster per candidate than the package -msse2 build (project970 60.6->55.2, project2668 34.7->31.9) because the target(avx2) reduce helper is a non-inlined call boundary and packing (ns 2-4) made that a big fraction of the block body. Pre-packing this was -1.2% (T-P5f); packing reopened it. The portable source capture (target(avx2) scorer clone + cpu_has_avx2 dispatch) was flat locally and reverted: cpu_has_avx2() is almost certainly false on this MinGW/Windows box (__builtin_cpu_supports unreliable there), so the shipped Windows binary likely runs SSE2 -- a separate portable win. On Linux/EPYC the runtime check works, so the baseline already uses the AVX2 helpers and the clone (boundary kill) is the right lever there. Resolution deferred to a Hamilton EPYC -msse2-vs-mavx2 A/B (after the 5432 packbuild job frees the lib). See dev/profiling/kernel-avx2-reduce-gate.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0% end-to-end EPYC A/B (job 17908514): the shipped -msse2 build pays a per-block AVX2-call boundary (cpu_has_avx2 true on Linux, but the target(avx2) helper can't inline into the -msse2 scorer). -mavx2 removes it: 1.67x/1.38x/1.26x per candidate on project970/510/2668, scaling with block count. Windows masked this (cpu_has_avx2 false there -> inline SSE2 -> gap looked like 9%). But end-to-end (job 17908602, fixed-seed maxReplicates=4, identical trajectory): project970 1.01x, project510 1.10x. Amdahl: scoring is only ~3% (970) / ~34% (510) of MaximizeParsimony wall; the rest is non-scoring machinery (ratchet/ sectorial/tree-ops). The per-candidate win does not propagate -- same dilution that sank lever-6, and why packing (shrinks total_words pervasively) beat AVX2 (reduce compute only). Banked as a -march=native Hamilton deployment build (dev/profiling/tsmarch_build.sh -> /nobackup/pjjg18/tsmarch/lib): free ~1-10% for EPYC runs, machine-specific, NOT for CRAN (breaks non-AVX2 CPUs; release keeps -msse2 + runtime dispatch). The portable target(avx2) clone is not worth it (small, dataset-dependent end-to-end). Confirms the mission: time-to-optimum is bounded by search machinery/stopping, not the per-move kernel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t=off Verified the -march=native deployment lib gives byte-identical EW+IW scores vs -msse2 (project970, 8 trees, IW k=10, 12dp). Add -ffp-contract=off so byte-identity is guaranteed (not just empirical): keeps the AVX2-inline win (boundary-kill, not FMA) but forbids FMA fusion of the IW/profile float accumulators, which could otherwise shift IW ties -> trajectory drift in a race. tsmarch/lib is thus a transparent speedup, safe for IW-tie-sensitive race runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…full-breadth control Expands the @param targetHits one-liner into explicit guidance for the two goals a user brings to a search: a small value for a single tree one can be reasonably confident is an MPT, a large value + high maxReplicates to represent the full range of MPTs. States the mechanism (targetHits is shared by every strategy preset; presets differ in per-replicate effort, not stop policy) and the honest caveat that full-breadth completeness cannot be auto-detected. Regenerates man/MaximizeParsimony.Rd. Outcome of the Lever-B over-search closure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…stop curves)
Hamilton harnesses behind the Lever-B closure: lever_b_curve.R captures the per-rep pool-vs-rep breadth curve (uncapped pool + never-stop = distinct-MPT count); lever_b_{stop,disc,regime}.sh drive the mode-(i) targetHits sweep, the discovery-rate probe, and the mode-(ii) intermediate-regime probe. Referenced by the lever-b-oversearch memory.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Goloboff-1996 union-construct screen refutation: lower-bound proof note (unsound with collapsed uppass finals) and the gate script that exercises it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kernel speed-baseline / reslice-plan / goloboff-gates notes plus the disc_nstates and corpus_state_heterogeneity characterisation scripts (+ small .rds caches). The TS_AUDIT_PROBE bail-depth instrumentation from the same original commit is intentionally OMITTED here: origin/cpp-search rewrote ts_fitch.cpp (+264 lines, spr-drift/L3b) so the probe's call-sites no longer map; it remains preserved on branch claude/mission-b-kernel-goloboff-6f5c0e for a future manual port if needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…#2/#3) + harnesses project5432 Mission A: all cold-start-basin-capture angles refuted by measurement. - #1 constructor CLOSED five ways: per-split detectability (anti-detect), absolute synapomorphy count (chance), joint clique recovery (36th pctile, count-matched), scaffold-sufficiency (all-deep-fixed 1/12), clique-START routability (0/60, worse than RAS), clique-proportional reweight (~uniform, wrong-skew). - #2 basin-hop schedule CLOSED (all near-optimal TS trees >=97 TBR from 1943). - #3 structured breadth CLOSED (random resample lands farther, 0/400). Capstone: the 1943 deep backbone is an emergent aggregate (~4/183 clean-supporting chars), so no per-char/compatibility/reweight start-generator can bias toward it; reach is search-side only (not a hard limit -- TNT ~1/3). Adds dev/benchmarks harnesses. Updates the deliverable's open-angles + status so the next 1939 phase does not re-run these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e skips them The mission brief still listed constructor/schedule/breadth as 'none yet refuted' — the exact re-investigation trap. Now points to the closure record + capstone (emergent-aggregate backbone => search-side only, not a constructor). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t-in) Recovered from a stale GitHub Desktop auto-stash. POOL_RESEED (TS_POOL_RESEED) reseeds a fraction of replicates from a retained best-score pool tree instead of a fresh Wagner start, giving the diverse equal-length pool its own sectorial re-solve pass. Terminal fuse (TS_TERMINAL_FUSE=1) runs a final fuse over a few best-score anchors after the loop converges, recombining complementary clades the in-loop fuse never reached. Both default-off; deployed behaviour is byte-identical. Also carries dev/profiling driver scripts used during that exploration and a stray citation-syntax fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…result Commit the score-verified best-known artifacts for the Mission-A epitome instance project5432, so they can never go missing again (the lesson of the retracted "1939" trace phantom): - dev/benchmarks/hardtail/project5432_best_1942_tnt.tre (1942, verified) - dev/benchmarks/hardtail/project5432_ts_reach_1943.tre (1943, verified) - dev/benchmarks/hardtail/README.md (regime, provenance, config, re-run footnote) Result (Hamilton array 17943743, 16 seeds x ~48h): TreeSearch's aggressive perturbation swarm reaches 1943 (6/16 seeds) — the new best TS-generated score, one step below the prior TS-side best of 1944. Recorded observationally: block-1 cold still floors ~1945-1947; sustained iteration under the perturbation stack descends to 1943. Reframing: 1943 is a SHARED TS/TNT floor (28h heavy TNT also floored at 1943 across all seeds; 1942 = one lucky regen; 1939 = phantom). The residual 1943->1942 gap is a rarity/luck-and-speed gap, not a TS pathology. Updates the Mission-A plan doc's dated result-note + artifact pointers; CLOSED sections left intact. Swarm ran on Hamilton lib TreeSearch 2.0.0 (lags cpp-search); a current-engine re-run to settle whether TS reaches 1942 is held, not launched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual testing underway; shiny app in particular has some usability issues.