Skip to content

Stream NC1 lines and template tokens to avoid allocating on hostile input - #37

Merged
mahach666 merged 3 commits into
mainfrom
fix/issue-6-stream-nc1-and-template
Nov 14, 2026
Merged

mahach666 merged 3 commits into
mainfrom
fix/issue-6-stream-nc1-and-template

Conversation

@mahach666

Copy link
Copy Markdown
Contributor

Stream DSTV lines and template tokens instead of holding the entire input as lists of lines or tokens, addressing issue #6.

  • Updates \TemplateLexer.Lex\ to yield tokens via \IEnumerable.
  • Updates \TemplateParser\ to consume tokens with a 2-token sliding buffer (_current, _next).
  • Replaces eager \SplitLines\ in \Nc1Reader\ with a streaming
    ef struct LineReader, parsing blocks and verifying ST/body lines incrementally.

Closes #6

@mahach666 mahach666 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not merge yet: template token streaming improves early rejection, but the NC1 path still buffers an entire malformed block. The case below remains an unaddressed part of the hostile-input allocation problem in #6.

Comment thread src/TdbIO/Internal/Nc1Reader.cs Resolved
…d blocks

Rejects invalid AK/BO lines immediately without buffering remaining lines of a bad block. Adds early-rejection allocation test.

@mahach666 mahach666 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original whole-block allocation issue is fixed: the 500,000-empty-line regression now passes on net8.0/net10.0. However, the newly added allocation test is flaky in the normal parallel suite; the attached failure occurred during this recheck.

Comment thread tests/TdbIO.Tests/Nc1Tests.cs Resolved
@mahach666
mahach666 merged commit 63c4266 into main Nov 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DSTV and template readers hold the whole input as lines or tokens

1 participant