Skip to content

Python semantic layer: semantic_layer() and measure collection - #249

Draft
jat255 wants to merge 10 commits into
jat255/wwmt-schema-fixturefrom
jat255/wwmt-layer-collection
Draft

Python semantic layer: semantic_layer() and measure collection#249
jat255 wants to merge 10 commits into
jat255/wwmt-schema-fixturefrom
jat255/wwmt-layer-collection

Conversation

@jat255

@jat255 jat255 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Third of four stacked PRs building the Python semantic layer (M3). Based on #248.

semantic_layer() collects measures from inline decorated functions, bare records, lists, module objects, .py file paths, and directories of them. Duplicate names, non-measure items, and missing paths all fail at construction.

The layer keeps two mappings, keyed differently on purpose. measures is keyed by measure name and source_text by Python function name. The two differ whenever @measure(name=...) renames a measure, so one map would lose the other. Both are wrapped in MappingProxyType, because a plain dict on a frozen dataclass is still mutable in place. Only text is kept: the worker session reads measure definitions and never receives a callable, which is the security property pkg-r/R/measures.R states in its header.

The R package has no equivalent of loading files by path, because read_measures() sources every measure file into one shared environment and helpers resolve for free. Python modules do not work that way, so a measure file has to import its helpers. Making that true took more than the docstring line originally planned.

While loading a file by path, its parent directory goes on sys.path and comes off again in a finally, and only if this code put it there. It is appended rather than inserted at the front. Both forms let a sibling import resolve, but prepending also lets a json.py sitting beside the measures shadow the standard library for everything imported afterwards, which is the usual script footgun and hard to diagnose.

Appending removes that hazard and leaves a smaller one, so loading a directory first checks every .py file in it and refuses when a name would collide. The check consults importlib.util.find_spec() as well as sys.modules, because the case most likely to bite is a package that is installed but not yet imported: without the spec lookup, a duckdb.py next to the measures passes every other test and then the sibling's import duckdb silently resolves to the installed package, leaving the author's helper unreachable with no error. sys.modules is still consulted alongside it, for a helper another measure directory already loaded under its bare name, which has no discoverable spec.

The check, the sys.path change, and module execution are serialized under a reentrant lock. Reentrant because the lock is held while user code runs, and a measure file that constructs another layer during its own import would otherwise deadlock its own thread.

Rejected: giving loaded files a synthetic package context so relative imports work. It does not help absolute imports, and relative imports inside a directory that is not a package read strangely.

Not in this PR: injection resolution and the public exports, which are the PR above it.

_collect()'s list/tuple branch merged harvested source with
sources.update(), so within a nested list the last definition of a
Python name won; semantic_layer() itself uses setdefault, so the first
wins. Use the same rule in both places so source_text is independent
of how measures are nested.
@jat255 jat255 changed the title jat255/wwmt layer collection Python semantic layer: semantic_layer() and measure collection Sep 2, 2026
@jat255 jat255 added the py Affects the Python implementation label Sep 2, 2026
@jat255
jat255 force-pushed the jat255/wwmt-layer-collection branch from 433b1ce to fa3ebf0 Compare September 2, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

py Affects the Python implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant