Skip to content

Fix string function edge cases reported in #382 - #408

Open
luisleo526 wants to merge 10 commits into
mainfrom
fix/issue-382-string-function-parity
Open

luisleo526 wants to merge 10 commits into
mainfrom
fix/issue-382-string-function-parity

Conversation

@luisleo526

Copy link
Copy Markdown
Contributor

Addresses #382's string-function defects:

  • UTF-16 units and empty splitting;
  • non-ASCII case conversion and regular-expression matching;
  • missing search positions;
  • NUL-bearing literals and their consumers;
  • trimming;
  • substring bounds and missing values;
  • literal-na split argument typing.

String-order comparisons keep their refusal.

The normal test suite gains reported-case regressions, including compilation of the literal-na split example (tests/test_string_split_empty_separator.py::test_a_literal_na_separator_has_the_string_parameter_type). The other reported value cases are covered in the string-units, split, NUL, case and regular-expression test modules.

array.join without a separator now joins with an empty string; it previously used a comma. TradingView accepts the one-argument form. Its default separator value is being confirmed against TradingView before this merges.

The Unicode case-data generator refuses every hidden path component, every .env path and symlinks before it reads input.

Part of #382.

Preserve UTF-16 units across concatenation, search and string-array joins;
retain complete literal bytes in input and metadata consumers; preserve
string result types and lexical bindings. Add native regression coverage
and document the fixture's measured and derived expectations.

Unicode semantic refinements and exact-head verification remain pending.
Share immutable integer proofs with argument validation and retain dynamic
runtime boundary checks. Add positional, named, alias, missing and runtime
controls. Diagnostic catalog generation and exact-head verification pending.
Carry the generator-written PF-E2094 entry and semantic pin, with its
authored presentation. The temporary PF-E2093 seed is excluded.
Use generated Unicode 14 tables from checksum-bound source data, extend
casing context across decimal runs, and preserve whole matches with shifted
backreferences. Add parser, provenance, context and regex-compiler checks.

This checkpoint still requires exact-head native and member verification.
Use the empty default separator and shared numeric formatting while
preserving integer digits and receiver/separator evaluation order.
Reject proven builtin boolean-array joins after authored-method resolution,
and validate named string-length arguments through the declared signature.
Compute the expected keyword column from the call argument region while
retaining the colliding assignment identifier and strict diagnostic checks.
Include the timestamp tests and PR skip-reason reporting from main.
Keep Unicode generator path refusal checks on protected directories.

The join default is the empty string. Check explicit and default separator
bindings for both string and numeric arrays, and retain numeric formatting
and string return checks after receiver binding replaced direct indexing.
Keep the operand-order assertion on the actual array receiver read after
the outer left-operand binding, followed by its element reads.
Reject every hidden path component instead of naming individual directories,
so all previously protected hidden paths remain protected. Keep the .env
suffix and symlink refusals unchanged. Cover each protected path class once.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.

Not checked: tests were not run; formal verification proved 0 of 43 changed function(s) (4 sampled, not proven, 39 not verified).

Formal verification. PR-changed functions: 4/43 verified (0 proven, 4 may-equivalent, 0 distinguished) · 39 not verified (2 vacuous, 19 unsupported, 18 over cap).

Not verified on this run: \_top\_level\_branch\_needs\_lexical\_scope (unsupported), \_visit\_FuncCall (unsupported), \_error (unsupported), \_generate\_cpp (unsupported), \_resolve\_known (unsupported), \_runtime\_ctor\_arg\_for\_reset (unsupported), \_string\_setting\_value (vacuous: never exercised), emit\_settings\_members (vacuous: never exercised), \_emit\_foreign\_security\_lookups (unsupported), \_emit\_foreign\_security\_registration (unsupported), and 11 more; 18 changed function(s) beyond the cap and 0 skipped when the time budget ran out.


Graphify review — findings

Adds a batch of string-handling fixes to codegen and the analyzer: surrogate-aware +/join, Unicode-14 str.upper/str.lower, Unicode-character str.match, and string-typed na/array joins that survive helper and method forwarding. It also changes runtime results for str.pos, which now returns integer na when the substring is absent (previously -1), str.substring, which now treats na endpoints canonically and refuses a provably negative constant start at analysis time, and str.trim, which strips more whitespace characters; these changes need a C++ regenerate and relink. Authored variables named like bar fields now resolve through their lexical declarations, so a string binding passed to array.from() no longer reads the numeric bar field. Top-level branches that declare over a builtin get their own child scope, which restores the builtin when the branch ends. str.length now validates named arguments against its declared parameter, and builtin array.join refuses boolean arrays during translation. String input defaults, options and titles keep embedded NUL bytes through getters, metadata and receipts, while the public C setters keep their NUL-terminated contract.

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 6888 functions depend on the 2864 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: Analyzer — 120 callers, 15 callees
  • worse: CodeGen — 75 callers, 14 callees
  • worse: _generate() — 11 callers, 25 callees
  • worse: LoopTypeSources — 15 callers, 11 callees
  • worse: _Fold — 4 callers, 8 callees
  • new: parse_ucd() — 4 callers, 7 callees
  • new: _analyze() — 4 callers, 6 callees
  • new: validate_v6_constant_arguments() — 2 callers, 11 callees
  • …and 5 more — each is listed as a finding

Verification — 6888 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 6051 function(s) in the blast radius were not formally verified this run

Cross-repo impact

Cross-repo impact

This change impacts 2 symbol(s) across 1 peer repo(s) — blast radius over derived cross-repo edges:

  • pineforge-4pass/pineforge-hpo — python/pineforge_hpo/transpile.py (via import_moniker, heuristic), tests/python/test_transpile.py (via import_moniker, heuristic)

1 peer repo(s) in this group are not yet indexed — cross-repo impact may be incomplete.

Cross-repo linkage is derived (SCIP-moniker matches are sound; name/contract matches are heuristic) and ADVISORY — it never affects this check's conclusion.

Test selection

Test selection

540 of 540 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/test_action_pins.py — full-run-safety
  • tests/test_allow_entry_in_qualifier.py — impact, full-run-safety
  • tests/test_allow_entry_in_request_const.py — impact, full-run-safety
  • tests/test_analyzer.py — impact, full-run-safety
  • tests/test_analyzer_matrix_inference.py — impact, full-run-safety
  • tests/test_analyzer_ta_return_types.py — impact, full-run-safety
  • tests/test_anchor_tf_change.py — impact, full-run-safety
  • tests/test_anchor_time.py — impact, full-run-safety
  • tests/test_anchor_vwap.py — impact, full-run-safety
  • tests/test_array_bounds_insert_fill_slice.py — impact, full-run-safety
  • tests/test_array_checked_access.py — impact, full-run-safety
  • tests/test_array_empty_aggregates.py — impact, full-run-safety
  • tests/test_array_history.py — impact, full-run-safety
  • tests/test_array_join_domains.py — impact, changed-test, full-run-safety
  • tests/test_array_missing_integer_index.py — impact, full-run-safety
  • tests/test_array_new_color_fill_and_na_size.py — impact, full-run-safety
  • tests/test_array_new_defaults_and_order.py — impact, full-run-safety
  • tests/test_array_reference_semantics.py — impact, full-run-safety
  • tests/test_array_retained_alias.py — impact, full-run-safety
  • tests/test_array_slice_view.py — impact, full-run-safety
  • tests/test_array_sort_indices.py — impact, full-run-safety
  • tests/test_array_terminal_returns.py — impact, changed-test, full-run-safety
  • tests/test_arrseg_coded_run_stops.py — impact, full-run-safety
  • tests/test_arrseg_review_fixes.py — impact, full-run-safety
  • tests/test_binop_operand_order.py — impact, changed-test, full-run-safety
  • tests/test_block_shadow_primitive_types.py — impact, full-run-safety
  • tests/test_block_var_plain_siblings.py — impact, full-run-safety
  • tests/test_bool_na_refusal.py — impact, full-run-safety
  • tests/test_bool_na_refusals_2.py — impact, full-run-safety
  • tests/test_bool_na_values.py — impact, full-run-safety
  • tests/test_bool_string_inference.py — impact, full-run-safety
  • tests/test_builtin_name_shadow.py — impact, changed-test, full-run-safety
  • tests/test_builtin_named_locals_compile.py — impact, full-run-safety
  • tests/test_builtin_named_locals_price_refusal.py — impact, test-dependency, full-run-safety
  • tests/test_builtin_named_locals_receipt.py — impact, full-run-safety
  • tests/test_builtin_named_locals_scope.py — impact, full-run-safety
  • tests/test_builtin_named_parameters.py — impact, full-run-safety
  • tests/test_builtin_named_parameters_reference.py — full-run-safety
  • tests/test_calc_on_order_fills_codegen.py — impact, full-run-safety
  • tests/test_calendar_lambda_locals.py — impact, full-run-safety
  • tests/test_call_argument_order.py — impact, full-run-safety
  • tests/test_callable_compound_type_flow.py — impact, full-run-safety
  • tests/test_callable_local_histories.py — impact, full-run-safety
  • tests/test_callable_series_param_types.py — impact, full-run-safety
  • tests/test_callable_string_series_type.py — impact, full-run-safety
  • tests/test_callable_ta_forwarded_length.py — impact, full-run-safety
  • tests/test_callable_var_first_reach.py — impact, full-run-safety
  • tests/test_checked_settings.py — impact, full-run-safety
  • tests/test_ci_engine_main_touched.py — full-run-safety
  • tests/test_clone_name_allocation.py — impact, full-run-safety
  • … and 490 more

non-code file(s) changed (CHANGELOG.md, pineforge_codegen/diagnostics_catalog.json, tests/fixtures/diagnostic_codes_pin.json, tests/fixtures/string_pos_na/README.md, tests/fixtures/string_pos_na/strategy.pine) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (CHANGELOG.md, pineforge_codegen/codegen/_unicode_case_data.py, pineforge_codegen/codegen/emit_top.py, pineforge_codegen/codegen/input.py, pineforge_codegen/codegen/tv_string_units.py …) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

Formal verification

Could not verify: Could not verify \_top\_level\_branch\_needs\_lexical\_scope.

The verifier did not have enough to check \_top\_level\_branch\_needs\_lexical\_scope, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_visit\_FuncCall.

The verifier did not have enough to check \_visit\_FuncCall, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_error.

The verifier did not have enough to check \_error, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

No difference found (not proven): No behavior difference found in fixed\_builtin\_result (not a proof).

The verifier ran both versions of fixed\_builtin\_result on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

No difference found (not proven): No behavior difference found in \_type\_spec\_from\_expr (not a proof).

The verifier ran both versions of \_type\_spec\_from\_expr on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: the synth_receiver tier. A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

Could not verify: Could not verify \_generate\_cpp.

The verifier did not have enough to check \_generate\_cpp, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_resolve\_known.

The verifier did not have enough to check \_resolve\_known, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_runtime\_ctor\_arg\_for\_reset.

The verifier did not have enough to check \_runtime\_ctor\_arg\_for\_reset, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

No difference found (not proven): No behavior difference found in \_string\_setting\_arg (not a proof).

The verifier ran both versions of \_string\_setting\_arg on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

Could not verify: Could not verify \_string\_setting\_value.

The verifier did not have enough to check \_string\_setting\_value, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 40 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ValueError — names the real obstacle, not a sampling gap)

Could not verify: Could not verify emit\_settings\_members.

The verifier did not have enough to check emit\_settings\_members, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 234 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)

Could not verify: Could not verify \_emit\_foreign\_security\_lookups.

The verifier did not have enough to check \_emit\_foreign\_security\_lookups, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_emit\_foreign\_security\_registration.

The verifier did not have enough to check \_emit\_foreign\_security\_registration, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_emit\_includes.

The verifier did not have enough to check \_emit\_includes, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_settle\_tv\_string\_units.

The verifier did not have enough to check \_settle\_tv\_string\_units, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_input\_default\_expression.

The verifier did not have enough to check \_input\_default\_expression, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_input\_key\_literal.

The verifier did not have enough to check \_input\_key\_literal, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_\_init\_\_.

The verifier did not have enough to check \_\_init\_\_, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_build\_security\_expr.

The verifier did not have enough to check \_build\_security\_expr, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_emit\_security\_linear\_helper\_call\_scoped.

The verifier did not have enough to check \_emit\_security\_linear\_helper\_call\_scoped, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_emit\_security\_rebind\_assignment.

The verifier did not have enough to check \_emit\_security\_rebind\_assignment, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

No difference found (not proven): No behavior difference found in \_emit\_security\_selection\_stmt (not a proof).

The verifier ran both versions of \_emit\_security\_selection\_stmt on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: the synth_receiver tier. A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

Could not verify: Could not verify \_security\_bind\_helper\_args.

The verifier did not have enough to check \_security\_bind\_helper\_args, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_security\_integer\_na\_operand.

The verifier did not have enough to check \_security\_integer\_na\_operand, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

Could not verify: Could not verify \_security\_linear\_helper\_storage\_frame.

The verifier did not have enough to check \_security\_linear\_helper\_storage\_frame, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: method — first parameter is `self`, which needs a constructed instance (not synthesizable)

· 8 grounded finding(s) anchored inline below; 5 more finding(s) on lines outside this diff (see the check run).

return None


def validate_v6_constant_arguments(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — validate_v6_constant_arguments()

fans out to 11 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.


class _NumericValueQuery(_CounterQuery):
"""One value query for storage and counters, including integral proofs."""
class IntegerConstants:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — IntegerConstants

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

return tuple(result)


def parse_ucd(unicode_data: str, special_casing: str, derived_properties: str) -> CaseData:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — parse_ucd()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

FILENAME = "join_domains.pine"


def _analyze(body):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _analyze()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

assert context.symbols.resolve("observed").pine_type == PineType.STRING


def test_namespace_receiver_type_query_does_not_register_ta_again():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — test_namespace_receiver_type_query_does_not_register_ta_again()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

assert _refusal(body).phase == Phase.ANALYZER


def test_codegen_checks_join_domain_when_analysis_is_bypassed():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — test_codegen_checks_join_domain_when_analysis_is_bypassed()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@pytest.mark.parametrize("value", [
"\0", "x\0" + "789", "x\0y\0z", 'é"\\\0中😀', "a\x01b", "60", "",
], ids=["nul", "digits", "multiple", "utf8-escapes", "other-control", "ascii", "empty"])
def test_length_aware_literal_consumers_keep_the_authored_bytes(value):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — test_length_aware_literal_consumers_keep_the_authored_bytes()

fans out to 10 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

assert diagnostic.location == SourceLocation("substring.pine", 3, column, column + 1)


def test_direct_analyzer_checks_constant_negative_substring_arguments():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — test_direct_analyzer_checks_constant_negative_substring_arguments()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant