Add ImportWAL tool with WAL replay controls - #18516
Conversation
|
|
||
| /** Compile-time i18n constants for the WAL import tool (English). */ | ||
| public final class ImportWALMessages { | ||
|
|
There was a problem hiding this comment.
Centralizing the import tool messages in the English locale keeps all user-facing CLI and replay errors localizable and gives the Chinese locale a stable key set to mirror.
|
|
||
| /** WAL 导入工具的编译期国际化常量（中文）。 */ | ||
| public final class ImportWALMessages { | ||
|
|
There was a problem hiding this comment.
The Chinese message class mirrors the English constants so with-zh-locale compiles the same ImportWAL behavior while presenting localized prompts and errors.
| return false; | ||
| } | ||
| try { | ||
| // An active WAL has no end marker until its writer closes. Reaching EOF exactly between |
There was a problem hiding this comment.
An active WAL may end cleanly at an entry boundary without an end marker. Checking available bytes before deserialization avoids misclassifying that boundary as corruption, while partial bytes still follow the existing corruption path. This is covered by the ImportWAL WAL reader tests.
| final String host = commandLine.getOptionValue("host", DEFAULT_HOST); | ||
| final int port = parsePort(commandLine.getOptionValue("port", String.valueOf(DEFAULT_PORT))); | ||
| final String username = commandLine.getOptionValue("username", DEFAULT_USER); | ||
| final WALReplayer.ReplayDecisionController replayDecisionController = |
There was a problem hiding this comment.
One controller is created for the entire import and shared by every worker. This makes an execute-all or skip-all answer consistent across WAL directories instead of applying only to whichever worker happened to prompt first.
| } | ||
| final long startNanos = System.nanoTime(); | ||
| final ReplayStatistics statistics = createReplayStatistics(walFiles); | ||
| final List<List<Path>> walDirectories = groupWALFilesByDirectory(walFiles); |
There was a problem hiding this comment.
Grouping WAL files by parent directory provides the concurrency boundary: files within one directory retain WAL-version order, while independent directories can be assigned to separate workers.
| } | ||
| offset = reader.getWALCurrentReadOffset(); | ||
| } | ||
| if (reader.isFileCorrupted()) { |
There was a problem hiding this comment.
The reader loop checks its corruption flag after replaying all available entries. This turns a truncated or malformed tail into a reported replay failure while allowing a clean active-WAL EOF to complete normally.
| replayInsert(insertNode); | ||
| return true; | ||
| } | ||
| if (entry.getValue() instanceof DeleteDataNode deleteDataNode) { |
There was a problem hiding this comment.
Tree-model deletes are routed through the decision prompt so operators can execute, skip, apply the choice to all subsequent deletes, or terminate before mutating the target database.
| replayTreeDelete(deleteDataNode); | ||
| return true; | ||
| } | ||
| if (entry.getValue() instanceof RelationalDeleteDataNode |
There was a problem hiding this comment.
Relational deletes and ObjectNode entries remain unsupported for execution. The prompt therefore permits only skip, skip-all, or terminate, preventing an unsupported WAL operation from being silently applied.
| this.console = console; | ||
| } | ||
|
|
||
| // The controller is shared by parallel workers so an "all" choice applies to the whole |
There was a problem hiding this comment.
The synchronized controller serializes console prompts and stores all decisions shared by parallel workers. This preserves deterministic operator control while multiple WAL directories are replayed concurrently.
| verify(treeSession, never()).deleteData(any(), anyLong(), anyLong()); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
These tests cover the new tree-delete decision outcomes, including execute/skip handling and the all-choice result paths, so the replay contract is verified independently from the interactive console.
| exit 1 | ||
| fi | ||
|
|
||
| JVM_OPTS="-Dsun.jnu.encoding=UTF-8 -Dfile.encoding=UTF-8" |
There was a problem hiding this comment.
The distribution launcher uses the standard IoTDB lib/* runtime and the same JVM encoding settings as other tools, making ImportWAL available from a packaged Linux installation without an extra classpath setup.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18516 +/- ##
============================================
+ Coverage 43.74% 44.05% +0.31%
Complexity 374 374
============================================
Files 5401 5403 +2
Lines 387084 389280 +2196
Branches 50482 50895 +413
============================================
+ Hits 169321 171489 +2168
- Misses 217763 217791 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
Summary
import-waltool for replaying tree/table model WAL files and MemTable snapshots into a target IoTDB instancethread_numparallel replay by WAL directory andon_successsource WAL handlingObjectNodeentriesValidation
mvn test -pl iotdb-core/datanode -Dtest=ImportWALTest -DfailIfNoTests=falsemvn test-compile -pl iotdb-core/datanode -Pwith-zh-locale -DskipTestsgit diff --check