Repository navigation
Python test-selection recall: -m spawns, getattr f-string dispatch, typing gaps, builtin protocol (stacked on #1873, #1878) - #1883
Merged
Conversation
pytest fills a test's parameter with whatever the fixture of that name returned or yielded, and no call site spells that call, so the parameter stayed untyped: every method called on it was ambiguous_unknown, and everything derived from it (a local assigned from one of those calls, a with block) went with it. Across ten public suites 1,529 such calls were unresolved; a suite whose fixtures are annotated already resolved its own. value-flow.dl now treats the runner's call as one more argument reaching a parameter: param_arg_type from the fixture's value type (its return, its yield, or its declared return). The fixture serving a parameter comes from a syntax-only copy of the runner's lookup (class, own module, nearest conftest): the existing py_fixture_injection reads module_member_method for star-imported fixtures, which is resolution, and its nearest-conftest MAX would then be a cyclic aggregate. Star-imported and pytest_plugins fixtures still reach their tests through the injection edge; they only stay untyped. Measured with a behavioural oracle (break each of 600 sampled functions, record which test files fail) on ten held-in repositories: impact --tests recall 0.668 -> 0.698, every-failing-file-selected 46.7% -> 50.5%, precision unchanged; the three repositories carrying the pattern move, the seven without it are byte-identical. Engine suite 43/43 with identical case results; query cases 310/310. New case fixture-value-types-the-parameter fails on the base engine on its derived-local check. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…(T, wrapper) returns wrapper functools.update_wrapper(wrapper, wrapped) hands back `wrapper`, and typing.cast(T, x) hands back `x`; both are documented, neither module is staged in a client-only run. A decorator written `return update_wrapper(wrapper, f)`, or typed as `return t.cast(F, update_wrapper(wrapper, f))`, was therefore opaque: every method it decorates was decorator_replaced_target and no call through the attribute reached anything. That shape occurs in five of sixteen public repositories measured. A catalogue, py_returns_arg(DottedName, Position) in builtins.dl, lists the two functions and the argument each returns; call_chain.dl follows such calls (to any depth) to the returned name, matched through the import that binds the callee: a module import, its alias (`import typing as t` is MODULE_IMPORT_ALIAS, which the copy.copy rule beside it also misses), or a from-import. On ten held-in repositories, path's found rate on runtime-proven chains goes 0.720 -> 0.744 (the repository with the shape: 0.52 -> 0.76); test selection and the other verbs unchanged. Engine suite 43/43 with identical case results; query cases 313/313; case decorator-returning-update-wrapper fails on the base engine on both decorator checks. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…y the import walk impact's `at import` rung names every test file importing a module whose BODY reaches the change through calls that run, because a module that raises while being imported fails every file importing it. A module body's calls inside `if __name__ == "__main__":` are real edges, but they run only when the file is executed as a script, never on import. Walking them named every importer of a module that ends in a demo block. The fact export now records guard_only(module, callee) where every call site the module body has to that callee lies inside a top-level main guard, and up_running does not take those edges. Ordinary reachability is unchanged; only the import hop is. Measured with the mutation oracle on ten held-in repositories, the at-import rung was the second-noisiest (precision 0.16 over 720 selected test files), and 637 of those 720 were for targets that never run on import. On the repository carrying the shape its at-import selections drop 713 -> 305 and test-selection precision 0.335 -> 0.373 with recall unchanged; the control repository whose at-import selections were all correct is unchanged. Query cases 315/315 python, 264/264 typescript, 333/333 java; case main-guard-does-not-run-on-import fails on the base engine on the guarded check. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…_"`) is a guard too `if sys.platform != "win32" and __name__ == "__main__":` is false on import whatever the other conjunct is, so the block cannot run then; an `or` could, and is not read as a guard. The first guard rule missed the conjunction, which on the repository carrying the shape was where almost all of the remaining false at-import routes came from. There: at-import selections 305 -> 55, of which 53 fail under the mutation oracle; test- selection precision 0.373 -> 0.446. Recall 0.810 -> 0.794: a few failing files had been selected only through an at-import route that cannot run, right by accident; their real route is one the graph does not see. Control repository unchanged. IMPACT_VERSION 65, so fact caches written under 64 are not reused. Case extended with the conjunctive spelling; query cases green. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…hat builds the object, not from every constructor impact takes a hop from a protocol member (a dunder no call site names) to whoever constructs its type: you do not build a context manager without entering it. That holds for __enter__/__exit__, __bool__, __hash__, __call__ (precision 0.61-0.80 under the mutation oracle), and not for showing a value (__repr__, __str__, __format__), pickling or copying it (__getstate__, __setstate__, __reduce__, ...) or deleting from it (__delitem__, __delattr__, __del__): those run only when some code asks, and a constructor deep inside other code says nothing about that. Measured on ten held-in repositories, __repr__ alone was 124 selected test files at precision 0.05, across eight of them. For those members the hop is now taken only from test code that constructs the type: a test that builds the object is the one that asks for its repr. Dropping the hop outright removed 403 false files but left 13 targets with a failing test and an empty answer; this form keeps every answer non-empty. Ten held-in repositories (this commit with the two main-guard ones): test-selection precision 0.408 -> 0.454, recall 0.698 -> 0.694, empty answers unchanged at 84/535. Case on-demand-dunder-hop-from-tests fails on the base engine on the __repr__ check; its control (__eq__ keeps every constructor) passes on both. Query cases 318/318 python, 264/264 typescript, 333/333 java. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…setter and deleter The engine emitted a PROPERTY_READ edge for reading a @Property and nothing for writing or deleting one, though `obj.x = v` runs `@x.setter` and `del obj.x` runs `@x.deleter` exactly as a read runs the getter. A setter therefore had no caller: an impact on it named no test, and everything the setter calls was cut off from whoever assigns. Setters occur in five of sixteen public repositories measured; the torture fixture documented the gap as a known miss. type_property_accessor finds the setter / deleter on the class that wins the name in the receiver's MRO (they share the getter's binding, so mro_lookup's single answer is not enough); property_write_edge keys on the attribute access's STORE / DEL context; the edge is exported as call kind PROPERTY_WRITE (schema vocabulary for Python, the kind TypeScript already uses for its accessors), mapped to the `property` rung. Torture: the CPython oracle confirms the new edge (agree 587 -> 588, missing 47 -> 46, extra unchanged); f43's known-miss marker is removed with its docstring rewritten at the same line count; goldens updated. Engine suite 43/43. Query cases 322/322 python, 264/264 typescript, 333/333 java; case property-setter-is-a-call fails on the base engine on all three write/delete checks. All six commits together, ten held-in repositories: test-selection recall 0.668 -> 0.700, precision 0.403 -> 0.452, impact source-caller recall 0.673 -> 0.686, path found 0.720 -> 0.752, context recall@5 0.789 -> 0.799; no repository regresses beyond 0.003 on any of them. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…operand of a 3+ way union counts Two gaps in "Optional[X] IS X", the rule that types a value annotated Optional[X] or X | None: - It existed for parameters, returns and fields but not for a LOCAL variable's annotation, so `ctx: Optional[Context] = ...` and `x: Foo | None = ...` left every call on the local unresolved while the same annotation on a parameter resolved. - `|` is left-associative: `A | B | None` is `(A | B) | None`, so A and B sit two levels below the annotation, under a nested union, and the depth-1 element rule saw only that nested union (which names no type) and None. Every union of three or more operands lost its members, on parameters, returns, fields and locals alike. binding_declared_nominal gains the Optional/union clause; union_operand walks nested PEP 604 unions and feeds their operands to type_ref_element, resolved and by name. Containers are untouched (union_operand is rooted at an Optional/union kind only). Optional/union locals occur in 15 of 16 public repositories measured (96 written with `|`, 50 with Optional[]/Union[]). Ten held-in repositories, against the previous commit: path found 0.752 -> 0.756, impact source-caller recall 0.686 -> 0.687 (one repository 0.795 -> 0.807); the new callers are a union's dispatch set and are labelled `one of a set`, resolved-caller precision unchanged at 0.979. Engine suite 43/43, torture unchanged; query cases 325/325; case optional-and-union-annotations fails on the base engine on both the local and the three-operand checks. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… a test file imports the conftests above it The import facts named a Python module only by its path from the repository root, so in a src layout `src/app/core.py` was `src.app.core` and no `import app.core` matched it: a src-layout repository had almost no test-to-source imports, which the import walk (a function that runs while a module is imported breaks every file that imports it) and the loads-the-change filter both read. A module is now also named from its package root, the first directory above it that is not a package. pytest imports every conftest.py from the rootdir down to a test file's directory before the file, so what a conftest imports is imported for each test file beneath it; a test file now imports those conftests. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…it resolves to the .py The project linker keys modules by qualified name, and a package that ships `impl.pyi` beside `impl.py` declares that name twice. The later file won the map, and `.pyi` sorts after `.py`, so `from .impl import *` and `import pkg` resolved to the stub. The engine refuses a stub declaration as an edge target, so every call into such a package ended at the library boundary and its tests reached nothing in it. A stub no longer replaces a non-stub module of the same name. A stub with no `.py` beside it is still the import's target. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
The spawn link (a test that starts a process whose arguments name an indexed file) only read PATHS. `subprocess.run([sys.executable, "-m", "pkg", ...])` names a MODULE: Python runs pkg/__main__.py (or pkg/mod.py for `-m pkg.mod`) as __main__, so everything the package's entry reaches was untested from that test's point of view, and a CLI suite that drives its tool this way selected nothing for an edit to the tool's internals. ax_spawn now resolves the name after each `-m` like an import: from the repository root or a source root (a directory that is no package and no test directory), package __main__ first, and only to a file this graph indexed. `-m coverage run -m pkg` names two modules; coverage is not indexed and links nothing. A string that only mentions `-m pkg` starts no process and stays unlinked. IMPACT_VERSION 66 -> 67 (spawns_fact changed). Measured with the mutation oracle on sixteen public repositories: the one whose CLI tests spawn `-m <tool>` moves test-selection recall 0.422 -> 0.468 (+175 failing test files selected, none lost, 121 added files that do not fail); a second repository's `python -m <pkg>` spawn links but its targets sit behind a protocol call, so it does not move; no other repository spawns `-m`. Case test-runs-a-module-with-dash-m fails 3 of 4 checks on the base. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…members of self's constructed subclasses
Visitor and parser dispatch by a computed name -- `return getattr(self,
f"visit_{type(node).__name__}", None)` in a base class, then `f = self.get_visitor(node);
f(node)` -- had no edges at all, so every visit_* body was cut off from whoever drives the
visitor. Three gaps stacked:
- the parser gave an f-string an empty literal value. It now carries the f-string's
constant lead (the text before its first `{`), which every string it can produce starts
with; nothing else reads an FSTRING literal.
- the existing prefix rule (`getattr(obj, "as_" + x)`) read only a `+` concatenation and only
obj's own MRO. It now also reads an f-string lead, and on `self` it widens over the
CONSTRUCTED subclasses, the same RTA bound self.m() has; a decorated member resolves to
the wrapper its decorator returned.
- a method returning such a lookup hands the set back (method_returns_method), as it would a
bare name.
The set is the program's own dispatch table, so the dispatch cap no longer refuses it
(call_computed_name_family): a visitor with sixty members is sixty targets by
construction, and the cap had dropped the whole site.
Mutation oracle, sixteen public repositories (ten tuning, six held out): the visitor-heavy
tuning repository moves test-selection recall 0.474 -> 0.878 (+169 failing test files
selected, none lost; 203 added files do not fail, the same 0.45 precision as its base);
one held-out repository +2; every other repository unchanged. Case
getattr-fstring-prefix-dispatch fails 3 of 4 checks on the base (2 of 4 with the f-string
rule but without the cap exemption). value-callee-is-unknown's computed-name check moves
to a name with no written part (`getattr(self, event)`), which stays an unknown: the
f-string form it used now resolves, as its `+` control already did.
Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…elf without an annotation with_target_declared typed a `with ... as` target from __enter__'s declared return or a literal `Self`. An unannotated `def __enter__(self): return self` -- the commonest way to write it -- typed nothing, so `with Session() as s: s.request(...)` was an untyped receiver and everything the session does was cut off from the code that opens one. Five such __enter__ definitions in three of sixteen public repositories. The written return is now read: no annotation and a return value that is `self` binds the target to the context manager's own type. An __enter__ that returns something else is untouched (control in the case). Mutation oracle, sixteen repositories: one tuning repository +22 failing test files selected (14 added files do not fail), none lost; all others unchanged. Case with-enter-returns-self fails its first check on the base. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… its targets from the annotation `for name, field in self.fields.items()` with `fields: dict[str, Field]` bound nothing: no element rule looked through a view call, and an items() loop has a TUPLE target that no rule bound. About 80 loops over an annotated mapping's view in eleven of sixteen public repositories. mapping_annotation_of finds the container annotation an expression carries (a field through self or a typed object, a parameter, a callee's declared return, a single-write local holding one); the slots are read POSITIONALLY (key at generic position 0, value at 1) rather than through type_ref_element, which merges both. values()/keys() yield one slot; items() binds each name of a tuple target to its own slot. The view names are catalogued in builtins.dl. Mutation oracle, sixteen repositories: one tuning repository +11 failing test files selected (7 added do not fail), none lost; path found on tuning 0.756 -> 0.764. Case dict-view-loop-types-the-value fails 3 of 4 checks on the base; its control (a key slot is never the value type) passes on both. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… argument the runner fills
`@pytest.mark.parametrize("cls", [String, Integer])` calls the test once per value, and
`@pytest.fixture(params=(A, B))` hands `request.param` each one in turn; neither is a
written call, so the parameter stayed untyped. Parametrize values that are classes,
instances or functions occur in fourteen of sixteen public repositories.
framework-behavior/dispatch.dl reads the rows: the decorator's values written in place, or
a display held by a single-write module name (also through `from m import NAME`); one
argument name takes the whole row, several take the row's k-th element (the existing
argnames position, now asked for every direct argument); `pytest.param(...)` rows count.
value-flow.dl types the parameter from each value (instance, class object, or function),
and `request.param` inside a params fixture likewise; a fixture that returns a class makes
the requesting parameter that class. The pytest names are catalogued in knobs.dl.
Mutation oracle, sixteen repositories: one held-out repository +17 failing test files
selected, another +1; 154 added files do not fail (the converter fixture there types every
test that takes it; 52 of the 169 added ran the target). Nothing lost. Case
parametrize-values-type-the-argument fails 3 of 4 on the base; control: string values
naming a class type nothing.
Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…s run the argument's dunder `assert repr(v) == "<...>"` reached nothing of v's: the call resolves to the builtin (C code) and the dunder it runs had no caller. BUILTIN_PROTOCOL edges now go from the argument expression to the dunder repr, str, len, hash, bool, iter, next, abs, format and reversed run on the argument's type -- only for the bare builtin (a client `repr` is not it), with exactly one argument (`str(b, "utf-8")` decodes), and with CPython's fallbacks (str -> __repr__, bool -> __len__ when the class has no slot of its own). New call kind BUILTIN_PROTOCOL in the schema vocabulary, tiered by target count like the other protocol edges; the names are catalogued in builtins.dl. Mutation oracle, sixteen repositories: +13 failing test files selected across five repositories and NO added file that does not fail; path found 0.764 -> 0.776 on tuning and 0.600 -> 0.607 held out. Case builtin-runs-the-dunder fails 3 of 4 on the base; control: str() on a class with its own __str__ does not run its __repr__. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…rgument does `def getter(val, field_obj=field_obj): return field_obj.deserialize(val)` -- the closure idiom that pins a loop variable -- is called with one argument, so field_obj only ever holds its default, and no call site passes it: the parameter was untyped and the call on it unresolved. The default is a value the def statement writes once (the parser links it, param_default_expr); value-flow.dl now lets it reach the parameter exactly as a call-site argument does -- an instance types it, a class object makes it that class, a function makes it callable as that function -- alongside whatever callers pass. A parameter defaulted to a name from an enclosing scope appears in fifteen of sixteen public repositories (sentinels and constants included, which type nothing callable). Mutation oracle, sixteen repositories: one tuning repository +13 failing test files selected (16 added files do not fail), none lost, every other repository unchanged. Case parameter-default-types-it fails 2 of 3 on the base; control: an argument passed at a call site still reaches the parameter. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked. This branch starts from the local Python branch that will be pushed as #1873's update (8 Python fixes on
apps/integration-0.1.9) plus #1878's parser commit cherry-picked, so the measurements match. Review only the last 7 commits; rebase once #1873 and #1878 land.Python test-selection recall for misses that are not library hand-backs (library callbacks are handled separately).
Population (first broken hop per missed failing test file, 3,397 misses, 16 repos)
-m modulespawnswith ... as, mapping views__rich_console__-style protocol over 30 classes (capped)repr,len,str, ...)Commits (each with a
tests/cases/python/case that fails on the base, plus a control)-m pkgspawn runs pkg/main.py[sys.executable, "-m", ...]getattr(self, f"visit_{...}")returned and called; prefix family exempt from the cap__enter__returning selfwith X() as yfor k, v in self.m.items()withm: dict[K, V]params=values@parametrize("cls", [A, B]),request.paramrepr(x),len(x),str(x)(fallbacks) ...def g(v, f=f)closure idiom,cb=defaultTotals (base = the stacked candidate; 16 repos, 60 mutated targets each)
+423 failing test files selected, 0 lost in any repository; 515 added files do not fail (152 of them in one held-out repo whose converter fixture now types every test that takes it).
Validation
--oracleper-case checks fail identically on the base (locks absent for cases 15-43 in the external oracle, 08 stale).Not done here, measured
mod.__file__spawns where the module comes fromimportlib.import_module(f"...{request.param}"): 489 misses, one held-out repository only, so not taken.type()/setattr, attrs-style generated methods.