Fix schema region NPE during DataNode shutdown by jt2594838 · Pull Request #18821 · apache/iotdb

jt2594838 · 2026-10-10T11:38:21Z

Description

During DataNode shutdown, SchemaEngine can clear a schema region while its validation lock is still registered. An in-flight schema write request then dereferences the missing region and returns INTERNAL_SERVER_ERROR (305).

Check region availability before accessing the schema region and return the retryable NO_AVAILABLE_REGION_GROUP (906) status when shutdown has already cleared it. The original fix covers internal timeseries creation quota validation. The same guard is applied to the highly similar AlterTimeSeries validation, Ratis CreateLogicalView validation, and CreateOrUpdateTableDevice quota validation paths. MTree path lookups now translate a cleared database root into the existing PathNotExistException, and table-device counting handles the cleared in-memory tree without dereferencing it. Errors from schema writes that obtained a region before SchemaEngine teardown are also returned as retryable while clear is in progress. Existing validation and consensus writes remain unchanged when the region is available; SimpleConsensus logical-view creation retains its direct consensus-write behavior.

Validation

Reproduced the original single-device and batch failures before the fix: expected status 906, received 305.
Reproduced the three similar missing-region paths before the fix: expected status 906, received 305.
mvn test -pl iotdb-core/datanode -am -Dtest=MTreeBelowSGMemoryImplTest,MTreeBelowSGCachedImplTest,RegionWriteExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -Drat.skip=true: passed, 13 tests with no failures, errors, or skips.
mvn test -pl iotdb-core/datanode -am -Dtest=RegionWriteExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -Drat.skip=true: passed, 8 tests, including the after-lookup teardown interleaving.
Full DataNode module build: passed.
Spotless and git diff --check: passed. No full cluster shutdown stress test was performed.

This PR has:

been self-reviewed.
added comments explaining the shutdown lifecycle constraint.
updated the prevalidation method documentation.
added unit tests covering the changed behavior.

Key changed classes: RegionWriteExecutor, SchemaEngine, MTreeBelowSGMemoryImpl, MTreeBelowSGCachedImpl and their tests, plus the locale message changes from the original fix.

jt2594838 · 2026-10-10T11:38:57Z


 public final class DataNodeQueryMessages {
+  public static final String MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A =
+      "Schema region is unavailable. Please retry later.";


Adds the retryable missing-region message to the English message catalog so the new validation result participates in the existing compile-time localization mechanism.

jt2594838 · 2026-10-10T11:38:57Z


 public final class DataNodeQueryMessages {
+  public static final String MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A =
+      "SchemaRegion 不可用，请稍后重试。";


Adds the same message key in Chinese to preserve locale parity. DataNode and ConfigNode compiled successfully with the Chinese profile; the full reactor stopped later at distribution because SNAPSHOT ZIP dependencies were unavailable.

jt2594838 · 2026-10-10T11:38:57Z

+        final String message =
+            DataNodeQueryMessages.MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A;
+        return RegionExecutionResult.create(
+            false, message, RpcUtils.getStatus(TSStatusCode.NO_AVAILABLE_REGION_GROUP, message));


Shutdown can clear the schema region while its validation lock is still registered. The previous quota check dereferenced null, and the outer catch returned INTERNAL_SERVER_ERROR because the lock still existed. Checking availability here returns NO_AVAILABLE_REGION_GROUP before dereferencing the region and covers the shared timeseries prevalidation callers. The method documentation now describes all three checks. Existing live-region validation and consensus writes are preserved.

jt2594838 · 2026-10-10T11:38:57Z

+  @Test
+  public void testInternalCreateTimeSeriesAfterSchemaRegionCleared() throws Exception {
+    // A shutdown can clear the region while its validation lock remains registered.
+    checkInternalCreateWithRegionAvailability(false, false);


Reproduces the single-device internal creation failure by retaining the validation lock while returning a missing region. Before the fix this test received status 305 instead of the retryable 906; it now passes.

jt2594838 · 2026-10-10T11:38:57Z

+  @Test
+  public void testInternalCreateMultiTimeSeriesAfterSchemaRegionCleared() throws Exception {
+    // Batch and Pipe auto-creation must return a retryable status for the same shutdown window.
+    checkInternalCreateWithRegionAvailability(true, false);


Covers batch internal creation against the same missing-region state so both auto-creation entry points have regression coverage. The helper also exercises Pipe-wrapped requests.

jt2594838 · 2026-10-10T11:38:57Z

+  public void testInternalCreateWithAvailableSchemaRegion() throws Exception {
+    // The guard must preserve quota checks and consensus writes for both live-region requests.
+    checkInternalCreateWithRegionAvailability(false, true);
+    checkInternalCreateWithRegionAvailability(true, true);


Checks both single-device and batch requests with an available region to ensure that the early rejection does not suppress valid quota checks or consensus writes.

jt2594838 · 2026-10-10T11:38:57Z

+    } finally {
+      IoTDBDescriptor.getInstance()
+          .getConfig()
+          .setSchemaRegionConsensusProtocolClass(originalProtocol);


The shared fixture runs SimpleConsensus and Ratis with ordinary and Pipe requests, using a real validation lock and mocked region/consensus services. Assertions check the returned status, lock release, and either the expected write or the absence of consensus interaction; configuration is restored in finally. All four tests in this class passed. This is deterministic executor-level coverage, not a full cluster shutdown stress test. The added imports support this fixture and its request variants.

jt2594838 · 2026-10-10T12:07:57Z

+      // Shutdown may clear SchemaEngine while the region's validation lock still exists.
+      // Reject the request as retryable before dereferencing the missing region.
+      if (schemaRegion == null) {
+        return createSchemaRegionUnavailableResult();


This reuses the centralized retryable result for internal timeseries creation after the original guard. Keeping one result builder ensures shutdown races consistently return NO_AVAILABLE_REGION_GROUP (906) with the localized message across all schema-write prechecks; live-region quota and type validation remain unchanged.

jt2594838 · 2026-10-10T12:07:59Z

        AlterTimeSeriesNode node, WritePlanNodeExecutionContext context, boolean receivedFromPipe) {
      ISchemaRegion schemaRegion =
          schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
+      if (schemaRegion == null) {


AlterTimeSeries fetches its measurement before consensus, so a cleared SchemaEngine entry could otherwise be dereferenced while the validation lock still exists. Returning the same retryable 906 result here preserves the shutdown behavior established for timeseries creation; the available-region path remains unchanged.

jt2594838 · 2026-10-10T12:08:00Z

      final ISchemaRegion schemaRegion =
          schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
      if (CONFIG.getSchemaRegionConsensusProtocolClass().equals(ConsensusFactory.RATIS_CONSENSUS)) {
+        if (schemaRegion == null) {


Ratis validates logical-view targets locally before writing consensus state. Guarding the missing region prevents a shutdown-time null dereference and makes the request retryable; SimpleConsensus continues to use its existing direct write path.

jt2594838 · 2026-10-10T12:08:02Z

        final boolean receivedFromPipe) {
      final ISchemaRegion schemaRegion =
          schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
+      if (schemaRegion == null) {


Table-device quota validation also reads the schema region before consensus. This guard covers the same shutdown window as the timeseries path and returns 906 before checkSchemaQuota is called, while preserving normal validation for an available region.

jt2594838 · 2026-10-10T12:08:04Z

          : PlanVisitor.super.visitCreateOrUpdateTableDevice(node, context);
    }

+    private RegionExecutionResult createSchemaRegionUnavailableResult() {


Centralizing the unavailable-region result keeps all guarded schema-write paths aligned on the retryable status and localized message. The helper also documents why a still-registered validation lock must not be treated as proof that the region is usable.

jt2594838 · 2026-10-10T12:08:06Z

 public class RegionWriteExecutorTest {

+  @Test
+  public void testAlterTimeSeriesRegionAvailability() throws Exception {


This regression test exercises AlterTimeSeries with the validation lock retained while the schema region is absent. It verifies the new retryable 906 result and the unchanged successful path for ordinary and Pipe requests under both consensus protocols.

jt2594838 · 2026-10-10T12:08:08Z

+  }
+
+  @Test
+  public void testCreateLogicalViewRegionAvailability() throws Exception {


This test covers both logical-view consensus modes: Ratis performs local target validation and must reject a missing region with 906, while SimpleConsensus keeps its direct write behavior. The matrix also checks available regions, Pipe wrapping, and lock cleanup.

jt2594838 · 2026-10-10T12:08:09Z

+  }
+
+  @Test
+  public void testCreateOrUpdateTableDeviceRegionAvailability() throws Exception {


This regression test covers table-device quota validation during the same region-cleared window. It verifies retryable failure without consensus writes, plus successful quota validation and writes when the region is available.

jt2594838 · 2026-10-10T12:08:11Z

+            Collections.singletonList(new Object[0])));
+  }
+
+  private void checkSchemaWriteWithRegionAvailability(final WritePlanNode node) throws Exception {


The shared test matrix keeps the similar-path checks consistent across missing and available regions, both consensus protocols, and ordinary/Pipe requests. It also verifies the validation lock is released and that consensus is skipped only for the unavailable-region case.

codecov · 2026-10-10T13:02:07Z

Codecov Report

❌ Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.90%. Comparing base (40e7ce6) to head (ef05aa2).
⚠️ Report is 24 commits behind head on master.

Files with missing lines	Patch %	Lines
...engine/execution/executor/RegionWriteExecutor.java	81.25%	3 Missing ⚠️
...org/apache/iotdb/db/schemaengine/SchemaEngine.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18821      +/-   ##
============================================
+ Coverage     45.57%   45.90%   +0.33%     
  Complexity      712      712              
============================================
  Files          5483     5501      +18     
  Lines        396291   398533    +2242     
  Branches      51567    51939     +372     
============================================
+ Hits         180590   182965    +2375     
+ Misses       215701   215568     -133

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Caideyipi · 2026-10-12T01:34:02Z

+      if (schemaRegion == null) {
+        return createSchemaRegionUnavailableResult();
+      }


[P2] Handle a region cleared after the availability check

This guard handles a region that is already absent from the map. An in-flight request can still obtain a non-null region and call fetchMeasurementPath() after shutdown clears that same object. In Memory mode, SchemaRegionMemoryImpl.clear() calls MTreeBelowSGMemoryImpl.clear(), which sets databaseMNode to null; getNodeByPath() subsequently dereferences it. That NullPointerException reaches the outer execute() catch, which still returns INTERNAL_SERVER_ERROR (305) if the validation lock remains registered.

Please coordinate this local validation with region teardown, or convert the failure to NO_AVAILABLE_REGION_GROUP (906) when the region is confirmed to have been cleared. The existing validation lock alone is insufficient because SchemaEngine.clear() does not acquire it.

The unavailable-region cases in the new tests start with getSchemaRegion() returning null. Please add a case that pauses after region lookup, clears the region, and resumes the request to cover this interleaving.

Addressed in 705d388. SchemaEngine now publishes its teardown state before clearing regions, RegionWriteExecutor returns retryable NO_AVAILABLE_REGION_GROUP for in-flight schema-write failures during teardown, and RegionWriteExecutorTest covers the after-lookup/clear interleaving.

jt2594838 · 2026-10-12T01:39:00Z

  public IMemMNode getNodeByPath(final PartialPath path) throws PathNotExistException {
    final String[] nodes = path.getNodes();
    IMemMNode cur = databaseMNode;
+    if (cur == null) {


SchemaRegion.clear() nulls the MTree database root. Converting that cleared state to the existing PathNotExistException lets checkDeviceNodeExists handle shutdown without an NPE in quota validation; live-tree path lookup behavior is unchanged.

jt2594838 · 2026-10-12T01:39:01Z

  // region table device management

  public int getTableDeviceNotExistNum(final String tableName, final List<Object[]> deviceIdList) {
+    if (databaseMNode == null) {


Table-device quota validation calls this count after the schema MTree may have been cleared. Treating every requested device as absent avoids dereferencing the missing root and keeps the count consistent with an empty tree.

jt2594838 · 2026-10-12T01:39:03Z

    ICachedMNode cur = databaseMNode;
    ICachedMNode next;
    try {
+      if (cur == null) {


After PBTree clear, the database root is null. This maps that state to PathNotExistException, and the guarded finally block avoids trying to unpin a null node. checkDeviceNodeExists can then handle the shutdown state without an NPE.

jt2594838 · 2026-10-12T01:39:05Z

 public class MTreeBelowSGMemoryImplTest {

+  @Test
+  public void testPathAndTableDeviceChecksAfterClear() throws Exception {


This regression test clears the memory MTree and verifies path lookup, device existence checks, and table-device counting handle the absent root without throwing an NPE.

jt2594838 · 2026-10-12T01:39:06Z

  }

+  @Test
+  public void testPathLookupAfterClear() throws Exception {


This regression test exercises path lookup and device existence checks after PBTree clear, covering both the missing-root exception and safe null-node unpinning.

jt2594838 · 2026-10-12T02:36:10Z

      }
+      // SchemaEngine clears regions before DataNodeRegionManager removes their validation locks.
+      // Failures from an in-flight schema write during that window must remain retryable.
+      if (groupId instanceof SchemaRegionId && schemaEngine.isClearing()) {


Maps exceptions from in-flight schema writes to retryable status while SchemaEngine teardown is active. The validation lock may still be registered after clear begins, so the general outer catch cannot infer region availability from that lock.

jt2594838 · 2026-10-12T02:36:26Z

  private SchemaMetricManager schemaMetricManager;

+  // Published so write handlers can identify teardown failures before region locks are removed.
+  private volatile boolean clearing;


Publishes teardown state across threads because write execution may observe clear() from another thread. init() resets it for a subsequent engine lifecycle, while clear() marks it before schema resources are torn down.

jt2594838 · 2026-10-12T02:36:37Z

+  }
+
+  @Test
+  public void testSchemaWriteWhenRegionIsClearedAfterLookup() throws Exception {


Covers the previously missing interleaving: lookup returns a region, its MTree is cleared, and local validation then throws while the lock still exists. The test checks that execution returns retryable 906 and never enters consensus.

jt2594838 · 2026-10-12T02:44:27Z

            : PlanVisitor.super.visitAlterTimeSeries(node, context);
      } catch (MetadataException e) {
+        if (schemaEngine.isClearing()) {
+          return createSchemaRegionUnavailableResult();


The MTree cleared-root guard reports PathNotExistException, which this validation method catches locally. Mapping MetadataException to the shared retryable result while SchemaEngine is clearing prevents this catch from returning a normal path-not-exist status during shutdown; ordinary metadata errors retain their existing behavior.

Fix schema region NPE during DataNode shutdown

126b14e

jt2594838 commented Oct 10, 2026

View reviewed changes

Handle missing schema regions in schema writes

df08230

jt2594838 commented Oct 10, 2026

View reviewed changes

Caideyipi reviewed Oct 12, 2026

View reviewed changes

Handle cleared MTree roots during lookup

f90ab64

jt2594838 commented Oct 12, 2026

View reviewed changes

Return retryable status while SchemaEngine clears regions

705d388

jt2594838 commented Oct 12, 2026

View reviewed changes

Return retryable status for alter during schema teardown

ef05aa2

jt2594838 commented Oct 12, 2026

View reviewed changes

jt2594838 merged commit 312c205 into master Oct 12, 2026
39 of 40 checks passed

jt2594838 deleted the fix_stop_schema_region_npe branch October 12, 2026 05:57

Conversation

jt2594838 commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jt2594838 commented Oct 10, 2026 •

edited

Loading

codecov Bot commented Oct 10, 2026 •

edited

Loading