Skip to content

Pipe: throttle TsFile parser retries under memory pressure#18141

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-parser-retry
Open

Pipe: throttle TsFile parser retries under memory pressure#18141
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-parser-retry

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Background

PR #18051 fixed a real self-blocking risk by releasing the TsFile parser when downstream tablet consumption hits PipeRuntimeOutOfMemoryCriticalException. That fallback is still necessary: with many pipes, parser-side memory and parsed tablet memory can otherwise hold each other and stop progress.

However, in scenarios such as 10 pipes with device-level pattern matching, a transient consumer-side OOM currently closes the parser immediately and retries the whole TsFile event later. This turns short memory pressure into repeated local TsFile reparsing, which is very expensive.

Changes

  • Reserve a TsFile parser memory permit before parsing, counted with the existing tablet/tsfile memory thresholds via pipe_tsfile_parser_memory, so too many TsFile parsers will not run concurrently under memory pressure.
  • Keep the current parser and retry the current parsed tablet locally for a bounded short window when the consumer reports OOM.
  • Fall back to releasing the parser and retrying the whole TsFile event when hard memory pressure is detected or the retry budget/time is exhausted.
  • Release the parser memory reservation on normal close and reference-resource finalization.

Tests

  • mvn -pl iotdb-core/datanode -Dtest=TsFileInsertionEventParserTest#testConsumeTabletInsertionEventsWithRetryReleasesParserOnOutOfMemory+testConsumeTabletInsertionEventsWithRetryKeepsParserForTransientOutOfMemory -DforkCount=0 test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant