hrw4u: Adds procedures (macros) and libraries#12939
Open
zwoop wants to merge 2 commits intoapache:masterfrom
Open
hrw4u: Adds procedures (macros) and libraries#12939zwoop wants to merge 2 commits intoapache:masterfrom
zwoop wants to merge 2 commits intoapache:masterfrom
Conversation
Extends the hrw4u grammar and compiler with a procedure system for defining reusable, parameterized blocks of rules. Procedures use namespaced names (e.g. `local::set-cache`) with `$param` substitution, and can be defined inline or loaded from external `.hrw4u` library files via `use` directives. A `--output=hrw4u` flatten mode expands all procedure calls into a self-contained source file. This commit also adds LSP hover/completion support for procedures, comprehensive test coverage, validation for circular imports and arity mismatches, and documentation in the admin guide.
Contributor
Author
|
[approve ci osx] |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the hrw4u DSL and toolchain with a procedure/macro system (namespaced ns::proc), including use-based library loading, parameter substitution/defaults, and a flattening mode that expands procedures into self-contained hrw4u source. It also updates the LSP to surface procedure information, expands test coverage with new fixtures, and documents the feature in the admin guide.
Changes:
- Add
usedirectives +proceduredeclarations to the grammar and implement procedure expansion and--output=hrw4uflattening in the compiler. - Improve developer ergonomics via humanized parse error messages and LSP hover/definition support for procedures.
- Add procedure-focused tests/fixtures and update documentation and build/package metadata.
Reviewed changes
Copilot reviewed 104 out of 104 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/hrw4u/tests/utils.py | Adds procedure-specific test helpers (procedure output/flatten/roundtrip/failure runners). |
| tools/hrw4u/tests/test_units.py | Updates unit tests; adds coverage for humanize_error_message() and parse error messaging. |
| tools/hrw4u/tests/test_procedures.py | New integration test suite for procedure expansion + flattening. |
| tools/hrw4u/tests/data/procedures/wrong-namespace.fail.input.txt | New failure fixture for namespace enforcement. |
| tools/hrw4u/tests/data/procedures/wrong-namespace.fail.error.txt | Expected error substring for wrong namespace fixture. |
| tools/hrw4u/tests/data/procedures/wrong-arity.fail.input.txt | New failure fixture for arity mismatch. |
| tools/hrw4u/tests/data/procedures/wrong-arity.fail.error.txt | Expected error substring for arity mismatch. |
| tools/hrw4u/tests/data/procedures/unknown-proc.fail.input.txt | New failure fixture for calling a non-imported procedure. |
| tools/hrw4u/tests/data/procedures/unknown-proc.fail.error.txt | Expected error substring for unknown procedure. |
| tools/hrw4u/tests/data/procedures/transitive.output.txt | Expected HRW output for transitive use/call case. |
| tools/hrw4u/tests/data/procedures/transitive.input.txt | Input using transitive procedure dependency. |
| tools/hrw4u/tests/data/procedures/transitive.ast.txt | Expected AST for transitive case. |
| tools/hrw4u/tests/data/procedures/top-level-only.fail.input.txt | New fixture enforcing use directives must be top-level. |
| tools/hrw4u/tests/data/procedures/top-level-only.fail.error.txt | Expected error substring for top-level-only rule. |
| tools/hrw4u/tests/data/procedures/string-param.output.txt | Expected HRW output for string parameter substitution. |
| tools/hrw4u/tests/data/procedures/string-param.input.txt | Input calling a procedure with a string parameter. |
| tools/hrw4u/tests/data/procedures/string-param.ast.txt | Expected AST for string parameter case. |
| tools/hrw4u/tests/data/procedures/reexport.output.txt | Expected HRW output for re-export-only library case. |
| tools/hrw4u/tests/data/procedures/reexport.input.txt | Input demonstrating use re-export behavior. |
| tools/hrw4u/tests/data/procedures/reexport.ast.txt | Expected AST for re-export case. |
| tools/hrw4u/tests/data/procedures/procs/test/wrong-namespace.hrw4u | Library file defining a mismatched namespace procedure (for validation). |
| tools/hrw4u/tests/data/procedures/procs/test/set-origin.hrw4u | Library file defining a parameterized procedure. |
| tools/hrw4u/tests/data/procedures/procs/test/set-cache.hrw4u | Library file defining a procedure with a default parameter. |
| tools/hrw4u/tests/data/procedures/procs/test/mixed-body.hrw4u | Library file mixing conditionals + statements inside a procedure. |
| tools/hrw4u/tests/data/procedures/procs/test/classify-request.hrw4u | Library file with elif/else in a procedure body. |
| tools/hrw4u/tests/data/procedures/procs/test/add-debug-header.hrw4u | Library file for simple parameter substitution. |
| tools/hrw4u/tests/data/procedures/procs/test/TagAndOrigin.hrw4u | Library file with multiple procedure declarations. |
| tools/hrw4u/tests/data/procedures/procs/reexport/debug.hrw4u | Library file demonstrating re-export via use. |
| tools/hrw4u/tests/data/procedures/procs/circular/B.hrw4u | Library file for circular import detection (B imports A). |
| tools/hrw4u/tests/data/procedures/procs/circular/A.hrw4u | Library file for circular import detection (A imports B). |
| tools/hrw4u/tests/data/procedures/procs/caller/Wrap.hrw4u | Library file for transitive import + call. |
| tools/hrw4u/tests/data/procedures/procs/base/Stamp.hrw4u | Library file providing base procedure for transitive test. |
| tools/hrw4u/tests/data/procedures/proc-after-section.fail.input.txt | Fixture enforcing procedure declarations must precede sections. |
| tools/hrw4u/tests/data/procedures/proc-after-section.fail.error.txt | Expected error substring for late procedure declarations. |
| tools/hrw4u/tests/data/procedures/override-param.output.txt | Expected HRW output when overriding a default parameter. |
| tools/hrw4u/tests/data/procedures/override-param.input.txt | Input overriding a default parameter. |
| tools/hrw4u/tests/data/procedures/override-param.ast.txt | Expected AST for override param case. |
| tools/hrw4u/tests/data/procedures/multi-use.output.txt | Expected HRW output for multiple use directives. |
| tools/hrw4u/tests/data/procedures/multi-use.input.txt | Input for multi-use case. |
| tools/hrw4u/tests/data/procedures/multi-use.ast.txt | Expected AST for multi-use case. |
| tools/hrw4u/tests/data/procedures/multi-section-mixed.output.txt | Expected HRW output for multi-section procedure expansion. |
| tools/hrw4u/tests/data/procedures/multi-section-mixed.input.txt | Input defining a local procedure and calling it from multiple sections. |
| tools/hrw4u/tests/data/procedures/multi-section-mixed.flatten.txt | Expected flattened hrw4u output for multi-section case. |
| tools/hrw4u/tests/data/procedures/multi-section-mixed.ast.txt | Expected AST for multi-section case. |
| tools/hrw4u/tests/data/procedures/multi-proc.output.txt | Expected HRW output for multiple procedures from a single library file. |
| tools/hrw4u/tests/data/procedures/multi-proc.input.txt | Input using multiple procedures in one section. |
| tools/hrw4u/tests/data/procedures/multi-proc.ast.txt | Expected AST for multi-proc case. |
| tools/hrw4u/tests/data/procedures/mixed-body.output.txt | Expected HRW output for procedure body with multiple rulesets. |
| tools/hrw4u/tests/data/procedures/mixed-body.input.txt | Input for mixed-body procedure call. |
| tools/hrw4u/tests/data/procedures/mixed-body.flatten.txt | Expected flattened hrw4u output for mixed-body case. |
| tools/hrw4u/tests/data/procedures/mixed-body.ast.txt | Expected AST for mixed-body case. |
| tools/hrw4u/tests/data/procedures/local-with-params.output.txt | Expected HRW output for local procedure with params. |
| tools/hrw4u/tests/data/procedures/local-with-params.input.txt | Input defining and calling a local procedure with params. |
| tools/hrw4u/tests/data/procedures/local-with-params.ast.txt | Expected AST for local-with-params case. |
| tools/hrw4u/tests/data/procedures/local-proc.output.txt | Expected HRW output for local procedure with no params. |
| tools/hrw4u/tests/data/procedures/local-proc.input.txt | Input defining and calling a local no-arg procedure. |
| tools/hrw4u/tests/data/procedures/local-proc.ast.txt | Expected AST for local-proc case. |
| tools/hrw4u/tests/data/procedures/local-multi-section.output.txt | Expected HRW output when reusing a local procedure across sections. |
| tools/hrw4u/tests/data/procedures/local-multi-section.input.txt | Input for local multi-section expansion. |
| tools/hrw4u/tests/data/procedures/local-multi-section.ast.txt | Expected AST for local multi-section case. |
| tools/hrw4u/tests/data/procedures/local-mixed-body.output.txt | Expected HRW output for a local procedure containing a conditional. |
| tools/hrw4u/tests/data/procedures/local-mixed-body.input.txt | Input for local mixed-body case. |
| tools/hrw4u/tests/data/procedures/local-mixed-body.flatten.txt | Expected flattened hrw4u output for local mixed-body case. |
| tools/hrw4u/tests/data/procedures/local-mixed-body.ast.txt | Expected AST for local mixed-body case. |
| tools/hrw4u/tests/data/procedures/local-and-use.output.txt | Expected HRW output mixing external use and local procedures. |
| tools/hrw4u/tests/data/procedures/local-and-use.input.txt | Input for mixing use with local procedures. |
| tools/hrw4u/tests/data/procedures/local-and-use.ast.txt | Expected AST for mixing use and local procedures. |
| tools/hrw4u/tests/data/procedures/in-conditional.output.txt | Expected HRW output for procedure call inside an if block. |
| tools/hrw4u/tests/data/procedures/in-conditional.input.txt | Input calling a procedure inside a conditional. |
| tools/hrw4u/tests/data/procedures/in-conditional.flatten.txt | Expected flattened hrw4u output for in-conditional case. |
| tools/hrw4u/tests/data/procedures/in-conditional.ast.txt | Expected AST for in-conditional case. |
| tools/hrw4u/tests/data/procedures/elif-in-proc.output.txt | Expected HRW output for elif/else inside procedure bodies. |
| tools/hrw4u/tests/data/procedures/elif-in-proc.input.txt | Input for elif inside procedure body. |
| tools/hrw4u/tests/data/procedures/elif-in-proc.flatten.txt | Expected flattened hrw4u output for elif in proc. |
| tools/hrw4u/tests/data/procedures/elif-in-proc.ast.txt | Expected AST for elif in proc. |
| tools/hrw4u/tests/data/procedures/duplicate-proc.fail.input.txt | Fixture for duplicate procedure declaration detection. |
| tools/hrw4u/tests/data/procedures/duplicate-proc.fail.error.txt | Expected error substring for duplicate declarations. |
| tools/hrw4u/tests/data/procedures/default-param.output.txt | Expected HRW output using default parameter value. |
| tools/hrw4u/tests/data/procedures/default-param.input.txt | Input invoking a procedure with default parameters. |
| tools/hrw4u/tests/data/procedures/default-param.ast.txt | Expected AST for default param call. |
| tools/hrw4u/tests/data/procedures/circular-use.fail.input.txt | Fixture for circular use dependency. |
| tools/hrw4u/tests/data/procedures/circular-use.fail.error.txt | Expected error substring for circular use. |
| tools/hrw4u/tests/data/procedures/basic-call.output.txt | Expected HRW output for a basic external procedure call. |
| tools/hrw4u/tests/data/procedures/basic-call.input.txt | Input for basic external procedure call. |
| tools/hrw4u/tests/data/procedures/basic-call.ast.txt | Expected AST for basic call. |
| tools/hrw4u/src/visitor_base.py | Refactors base visitor utilities and shared helpers used by both visitors. |
| tools/hrw4u/src/visitor.py | Implements procedure parsing/loading/expansion and flattening; updates symbol resolver debug wiring and value handling. |
| tools/hrw4u/src/symbols_base.py | Allows injecting a shared debug logger into symbol resolvers. |
| tools/hrw4u/src/symbols.py | Plumbs shared debug logger into SymbolResolver. |
| tools/hrw4u/src/procedures.py | New helper for resolving use specs to .hrw4u library file paths. |
| tools/hrw4u/src/lsp/strings.py | Typing cleanup and removal of section pre-validation helper. |
| tools/hrw4u/src/lsp/hover.py | Refactors hover generation and adds table-driven lookup helpers. |
| tools/hrw4u/src/hrw_visitor.py | Aligns with base visitor API changes and shared debug logger injection. |
| tools/hrw4u/src/hrw_symbols.py | Plumbs shared debug logger into inverse resolver. |
| tools/hrw4u/src/errors.py | Adds humanize_error_message() and error limit support in collected parsing. |
| tools/hrw4u/src/common.py | Adds --max-errors, visitor kwarg plumbing, and --output=hrw4u flatten dispatch. |
| tools/hrw4u/scripts/testcase.py | Enables procedure search path discovery for test runner outputs. |
| tools/hrw4u/scripts/hrw4u-lsp | Adds procedure-aware hover/definition features and initialization option for procedures path. |
| tools/hrw4u/scripts/hrw4u | Adds --procedures-path and --output (including hrw4u flatten) CLI options. |
| tools/hrw4u/pyproject.toml | Bumps hrw4u version and registers the new procedures pytest marker. |
| tools/hrw4u/grammar/hrw4u.g4 | Adds use/procedure syntax, qualified identifiers, and $param references. |
| tools/hrw4u/Makefile | Includes new src/procedures.py in build inputs. |
| doc/admin-guide/configuration/hrw4u.en.rst | Documents procedure syntax, library usage, defaults, and CLI flags. |
| ci/rat-regex.txt | Adds .hrw4u to RAT exclusion regex list. |
- Validate that required params precede optional params in procedure declarations; raises a clear error at declaration time. - Extract source line from input stream in _get_value_text errors so the error output includes the offending line and caret position. - Use SystemDefaults.INDENT_SPACES instead of hardcoded 4-space literals in flatten() and _flatten_conditional(). - Remove dead _collect_errors() in TestParseErrorMessages; _first_error() is the working equivalent used by all tests.
Contributor
Author
|
If you wonder why there are so many files, it's all the tests. TDD for the win. |
bryancall
approved these changes
Mar 12, 2026
Contributor
bryancall
left a comment
bryancall
left a comment
There was a problem hiding this comment.
Reviewed the full diff. Thorough TDD, clean grammar extension for procedures/macros, comprehensive validation (circular imports, arity, duplicates). The vast majority of the 104 changed files are test fixtures. LSP hover/definition support is a nice touch. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the hrw4u grammar and compiler with a procedure system for defining reusable, parameterized blocks of rules. Procedures use namespaced names (e.g.
local::set-cache) with$paramsubstitution, and can be defined inline or loaded from external.hrw4ulibrary files viausedirectives. A--output=hrw4uflatten mode expands all procedure calls into a self-contained source file. This commit also adds LSP hover/completion support for procedures, comprehensive test coverage, validation for circular imports and arity mismatches, and documentation in the admin guide.