Repository navigation
Fix string function edge cases reported in #382 - #408
luisleo526 wants to merge 10 commits into
Conversation
Preserve UTF-16 units across concatenation, search and string-array joins; retain complete literal bytes in input and metadata consumers; preserve string result types and lexical bindings. Add native regression coverage and document the fixture's measured and derived expectations. Unicode semantic refinements and exact-head verification remain pending.
Share immutable integer proofs with argument validation and retain dynamic runtime boundary checks. Add positional, named, alias, missing and runtime controls. Diagnostic catalog generation and exact-head verification pending.
Carry the generator-written PF-E2094 entry and semantic pin, with its authored presentation. The temporary PF-E2093 seed is excluded.
Use generated Unicode 14 tables from checksum-bound source data, extend casing context across decimal runs, and preserve whole matches with shifted backreferences. Add parser, provenance, context and regex-compiler checks. This checkpoint still requires exact-head native and member verification.
Use the empty default separator and shared numeric formatting while preserving integer digits and receiver/separator evaluation order. Reject proven builtin boolean-array joins after authored-method resolution, and validate named string-length arguments through the declared signature.
Compute the expected keyword column from the call argument region while retaining the colliding assignment identifier and strict diagnostic checks.
Include the timestamp tests and PR skip-reason reporting from main.
Keep Unicode generator path refusal checks on protected directories. The join default is the empty string. Check explicit and default separator bindings for both string and numeric arrays, and retain numeric formatting and string return checks after receiver binding replaced direct indexing. Keep the operand-order assertion on the actual array receiver read after the outer left-operand binding, followed by its element reads.
Reject every hidden path component instead of naming individual directories, so all previously protected hidden paths remain protected. Keep the .env suffix and symlink refusals unchanged. Cover each protected path class once.
There was a problem hiding this comment.
Graphify reviewed this change.
No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.
Not checked: tests were not run; formal verification proved 0 of 43 changed function(s) (4 sampled, not proven, 39 not verified).
Formal verification. PR-changed functions: 4/43 verified (0 proven, 4 may-equivalent, 0 distinguished) · 39 not verified (2 vacuous, 19 unsupported, 18 over cap).
Not verified on this run: \_top\_level\_branch\_needs\_lexical\_scope (unsupported), \_visit\_FuncCall (unsupported), \_error (unsupported), \_generate\_cpp (unsupported), \_resolve\_known (unsupported), \_runtime\_ctor\_arg\_for\_reset (unsupported), \_string\_setting\_value (vacuous: never exercised), emit\_settings\_members (vacuous: never exercised), \_emit\_foreign\_security\_lookups (unsupported), \_emit\_foreign\_security\_registration (unsupported), and 11 more; 18 changed function(s) beyond the cap and 0 skipped when the time budget ran out.
Graphify review — findings
Adds a batch of string-handling fixes to codegen and the analyzer: surrogate-aware +/join, Unicode-14 str.upper/str.lower, Unicode-character str.match, and string-typed na/array joins that survive helper and method forwarding. It also changes runtime results for str.pos, which now returns integer na when the substring is absent (previously -1), str.substring, which now treats na endpoints canonically and refuses a provably negative constant start at analysis time, and str.trim, which strips more whitespace characters; these changes need a C++ regenerate and relink. Authored variables named like bar fields now resolve through their lexical declarations, so a string binding passed to array.from() no longer reads the numeric bar field. Top-level branches that declare over a builtin get their own child scope, which restores the builtin when the branch ends. str.length now validates named arguments against its declared parameter, and builtin array.join refuses boolean arrays during translation. String input defaults, options and titles keep embedded NUL bytes through getters, metadata and receipts, while the public C setters keep their NUL-terminated contract.
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 6888 functions depend on the 2864 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
Analyzer— 120 callers, 15 callees - worse:
CodeGen— 75 callers, 14 callees - worse:
_generate()— 11 callers, 25 callees - worse:
LoopTypeSources— 15 callers, 11 callees - worse:
_Fold— 4 callers, 8 callees - new:
parse_ucd()— 4 callers, 7 callees - new:
_analyze()— 4 callers, 6 callees - new:
validate_v6_constant_arguments()— 2 callers, 11 callees - …and 5 more — each is listed as a finding
Verification — 6888 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 6051 function(s) in the blast radius were not formally verified this run
Cross-repo impact
Cross-repo impact
This change impacts 2 symbol(s) across 1 peer repo(s) — blast radius over derived cross-repo edges:
- pineforge-4pass/pineforge-hpo —
python/pineforge_hpo/transpile.py(via import_moniker, heuristic),tests/python/test_transpile.py(via import_moniker, heuristic)
1 peer repo(s) in this group are not yet indexed — cross-repo impact may be incomplete.
Cross-repo linkage is derived (SCIP-moniker matches are sound; name/contract matches are heuristic) and ADVISORY — it never affects this check's conclusion.
Test selection
Test selection
540 of 540 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_action_pins.py— full-run-safetytests/test_allow_entry_in_qualifier.py— impact, full-run-safetytests/test_allow_entry_in_request_const.py— impact, full-run-safetytests/test_analyzer.py— impact, full-run-safetytests/test_analyzer_matrix_inference.py— impact, full-run-safetytests/test_analyzer_ta_return_types.py— impact, full-run-safetytests/test_anchor_tf_change.py— impact, full-run-safetytests/test_anchor_time.py— impact, full-run-safetytests/test_anchor_vwap.py— impact, full-run-safetytests/test_array_bounds_insert_fill_slice.py— impact, full-run-safetytests/test_array_checked_access.py— impact, full-run-safetytests/test_array_empty_aggregates.py— impact, full-run-safetytests/test_array_history.py— impact, full-run-safetytests/test_array_join_domains.py— impact, changed-test, full-run-safetytests/test_array_missing_integer_index.py— impact, full-run-safetytests/test_array_new_color_fill_and_na_size.py— impact, full-run-safetytests/test_array_new_defaults_and_order.py— impact, full-run-safetytests/test_array_reference_semantics.py— impact, full-run-safetytests/test_array_retained_alias.py— impact, full-run-safetytests/test_array_slice_view.py— impact, full-run-safetytests/test_array_sort_indices.py— impact, full-run-safetytests/test_array_terminal_returns.py— impact, changed-test, full-run-safetytests/test_arrseg_coded_run_stops.py— impact, full-run-safetytests/test_arrseg_review_fixes.py— impact, full-run-safetytests/test_binop_operand_order.py— impact, changed-test, full-run-safetytests/test_block_shadow_primitive_types.py— impact, full-run-safetytests/test_block_var_plain_siblings.py— impact, full-run-safetytests/test_bool_na_refusal.py— impact, full-run-safetytests/test_bool_na_refusals_2.py— impact, full-run-safetytests/test_bool_na_values.py— impact, full-run-safetytests/test_bool_string_inference.py— impact, full-run-safetytests/test_builtin_name_shadow.py— impact, changed-test, full-run-safetytests/test_builtin_named_locals_compile.py— impact, full-run-safetytests/test_builtin_named_locals_price_refusal.py— impact, test-dependency, full-run-safetytests/test_builtin_named_locals_receipt.py— impact, full-run-safetytests/test_builtin_named_locals_scope.py— impact, full-run-safetytests/test_builtin_named_parameters.py— impact, full-run-safetytests/test_builtin_named_parameters_reference.py— full-run-safetytests/test_calc_on_order_fills_codegen.py— impact, full-run-safetytests/test_calendar_lambda_locals.py— impact, full-run-safetytests/test_call_argument_order.py— impact, full-run-safetytests/test_callable_compound_type_flow.py— impact, full-run-safetytests/test_callable_local_histories.py— impact, full-run-safetytests/test_callable_series_param_types.py— impact, full-run-safetytests/test_callable_string_series_type.py— impact, full-run-safetytests/test_callable_ta_forwarded_length.py— impact, full-run-safetytests/test_callable_var_first_reach.py— impact, full-run-safetytests/test_checked_settings.py— impact, full-run-safetytests/test_ci_engine_main_touched.py— full-run-safetytests/test_clone_name_allocation.py— impact, full-run-safety- … and 490 more
non-code file(s) changed (
CHANGELOG.md,pineforge_codegen/diagnostics_catalog.json,tests/fixtures/diagnostic_codes_pin.json,tests/fixtures/string_pos_na/README.md,tests/fixtures/string_pos_na/strategy.pine) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
CHANGELOG.md,pineforge_codegen/codegen/_unicode_case_data.py,pineforge_codegen/codegen/emit_top.py,pineforge_codegen/codegen/input.py,pineforge_codegen/codegen/tv_string_units.py…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
README.md§ How it works (lines 516-542): references changed symbols__init__
Formal verification
Could not verify: Could not verify \_top\_level\_branch\_needs\_lexical\_scope.
The verifier did not have enough to check \_top\_level\_branch\_needs\_lexical\_scope, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_visit\_FuncCall.
The verifier did not have enough to check \_visit\_FuncCall, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_error.
The verifier did not have enough to check \_error, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
No difference found (not proven): No behavior difference found in fixed\_builtin\_result (not a proof).
The verifier ran both versions of fixed\_builtin\_result on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in \_type\_spec\_from\_expr (not a proof).
The verifier ran both versions of \_type\_spec\_from\_expr on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: the synth_receiver tier. A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_generate\_cpp.
The verifier did not have enough to check \_generate\_cpp, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_resolve\_known.
The verifier did not have enough to check \_resolve\_known, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_runtime\_ctor\_arg\_for\_reset.
The verifier did not have enough to check \_runtime\_ctor\_arg\_for\_reset, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
No difference found (not proven): No behavior difference found in \_string\_setting\_arg (not a proof).
The verifier ran both versions of \_string\_setting\_arg on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_string\_setting\_value.
The verifier did not have enough to check \_string\_setting\_value, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 40 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ValueError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify emit\_settings\_members.
The verifier did not have enough to check emit\_settings\_members, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 234 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_emit\_foreign\_security\_lookups.
The verifier did not have enough to check \_emit\_foreign\_security\_lookups, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_emit\_foreign\_security\_registration.
The verifier did not have enough to check \_emit\_foreign\_security\_registration, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_emit\_includes.
The verifier did not have enough to check \_emit\_includes, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_settle\_tv\_string\_units.
The verifier did not have enough to check \_settle\_tv\_string\_units, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_input\_default\_expression.
The verifier did not have enough to check \_input\_default\_expression, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_input\_key\_literal.
The verifier did not have enough to check \_input\_key\_literal, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_\_init\_\_.
The verifier did not have enough to check \_\_init\_\_, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_build\_security\_expr.
The verifier did not have enough to check \_build\_security\_expr, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_emit\_security\_linear\_helper\_call\_scoped.
The verifier did not have enough to check \_emit\_security\_linear\_helper\_call\_scoped, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_emit\_security\_rebind\_assignment.
The verifier did not have enough to check \_emit\_security\_rebind\_assignment, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
No difference found (not proven): No behavior difference found in \_emit\_security\_selection\_stmt (not a proof).
The verifier ran both versions of \_emit\_security\_selection\_stmt on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: the synth_receiver tier. A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_security\_bind\_helper\_args.
The verifier did not have enough to check \_security\_bind\_helper\_args, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_security\_integer\_na\_operand.
The verifier did not have enough to check \_security\_integer\_na\_operand, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
Could not verify: Could not verify \_security\_linear\_helper\_storage\_frame.
The verifier did not have enough to check \_security\_linear\_helper\_storage\_frame, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)
· 8 grounded finding(s) anchored inline below; 5 more finding(s) on lines outside this diff (see the check run).
| return None | ||
|
|
||
|
|
||
| def validate_v6_constant_arguments( |
There was a problem hiding this comment.
validate_v6_constant_arguments()
fans out to 11 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
| class _NumericValueQuery(_CounterQuery): | ||
| """One value query for storage and counters, including integral proofs.""" | ||
| class IntegerConstants: |
There was a problem hiding this comment.
IntegerConstants
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return tuple(result) | ||
|
|
||
|
|
||
| def parse_ucd(unicode_data: str, special_casing: str, derived_properties: str) -> CaseData: |
There was a problem hiding this comment.
parse_ucd()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| FILENAME = "join_domains.pine" | ||
|
|
||
|
|
||
| def _analyze(body): |
There was a problem hiding this comment.
_analyze()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert context.symbols.resolve("observed").pine_type == PineType.STRING | ||
|
|
||
|
|
||
| def test_namespace_receiver_type_query_does_not_register_ta_again(): |
There was a problem hiding this comment.
test_namespace_receiver_type_query_does_not_register_ta_again()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert _refusal(body).phase == Phase.ANALYZER | ||
|
|
||
|
|
||
| def test_codegen_checks_join_domain_when_analysis_is_bypassed(): |
There was a problem hiding this comment.
test_codegen_checks_join_domain_when_analysis_is_bypassed()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @pytest.mark.parametrize("value", [ | ||
| "\0", "x\0" + "789", "x\0y\0z", 'é"\\\0中😀', "a\x01b", "60", "", | ||
| ], ids=["nul", "digits", "multiple", "utf8-escapes", "other-control", "ascii", "empty"]) | ||
| def test_length_aware_literal_consumers_keep_the_authored_bytes(value): |
There was a problem hiding this comment.
test_length_aware_literal_consumers_keep_the_authored_bytes()
fans out to 10 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert diagnostic.location == SourceLocation("substring.pine", 3, column, column + 1) | ||
|
|
||
|
|
||
| def test_direct_analyzer_checks_constant_negative_substring_arguments(): |
There was a problem hiding this comment.
test_direct_analyzer_checks_constant_negative_substring_arguments()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Addresses #382's string-function defects:
String-order comparisons keep their refusal.
The normal test suite gains reported-case regressions, including compilation of the literal-na split example (
tests/test_string_split_empty_separator.py::test_a_literal_na_separator_has_the_string_parameter_type). The other reported value cases are covered in the string-units, split, NUL, case and regular-expression test modules.array.joinwithout a separator now joins with an empty string; it previously used a comma. TradingView accepts the one-argument form. Its default separator value is being confirmed against TradingView before this merges.The Unicode case-data generator refuses every hidden path component, every
.envpath and symlinks before it reads input.Part of #382.