Repository navigation
1.4.0: traceback functions vote, function list for every prompt; phase-1 priors measured; fresh jsoup holdout confirms S1 - #144
Merged
Conversation
…old locked before any run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pt, NM_DIAG variants, prior prototypes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut confirms S1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Session 18, phases 1 and 3. Every idea prototyped offline on SWE-bench dev first (both halves, bar fixed in advance: file Acc@1 or func Acc@5 ≥ +0.02 in dev-fast and dev-rest). Lite/Verified untouched.
Kept (engine)
File "…", line N, in name/ pytestpath.py:N: in name) resolves to the indexed file (longest unique suffix) and the function it names (or the innermost one containing the line); test frames skipped; RRF weight 2 indefinition_order(so MCPfunctions_to_lookgets it too).packet --jsonlistsdefinitionsfor every prompt (short issues were automatic misses).File level unchanged on every dev instance (0.307/0.524/0.618/0.689).
Rejected with numbers (contributions-log §8.14–8.16)
commit-history prior (BugLocator: file@1 0.271→0.169), repro-code identifiers, test→module (fires on 0), RM3, doc→code bridge (ripgrep 0.500→0.375, axios 0.833→0.583), owner/unstemmed definition variants (
NM_DIAG), dense short-list (near miss: file@1 +0.017/+0.024, +10.6 s/issue).Holdout
Fresh concept-holdout5 (jsoup 1.18.1, Java, gold committed before any run): list R@3 v1.2.0 0.417 → v1.3.0/this 0.750, BM25 0.667 — S1 confirmed. Packet 0.667 / 0.235.
15 gold sets with this code: all equal to the numbers on record (dev 0.938, large 0.675, holdout 0.700, holdout-c 0.589, holdout-lang 0.589, holdout-ml 0.478, holdout-ml2 0.632, holdout-cfg 0.767, holdout-web 0.917/0.643, concept 0.679/0.392, ripgrep 0.500/0.156, click 0.958/0.342, cobra 0.833/0.357, axios 0.667/0.528) + jsoup 0.667/0.235.
Also: research tooling (
priors.py,chunk_prf.py,chunk_ablate.py,dense_shortlist.py,doc_bridge.py,diag_eval.py,paper_tables.py), harness--no-bm25,llm_localize.py --model/--functions(ready; all five Baseten keys return HTTP 402 — no credit).🤖 Generated with Claude Code