[Bug](High) BE crashes in TabletSchema::to_schema_pb while creating rollup tablets after INSERT OVERWRITE

[MoanasDaddyXu]

Search before asking

• I searched the existing issues for TabletSchema::to_schema_pb, TabletSchema::to_key, create_initial_rowset, and rollup-related crashes. I did not find an issue with the same crash path.

Version

Reproduced on Apache Doris commit 40762c79bd014d66a843cf21ab2a1464e5bac092 in a non-cloud deployment with 1 FE and 3 BEs.

What's Wrong?

During a ROW binlog regression workload, creating a rollup after replacing a partition with INSERT OVERWRITE caused one BE to receive SIGSEGV at address 0x150 and exit.

The first Doris business frame is TabletSchema::to_schema_pb() at src/storage/tablet/tablet_schema.cpp:1437. The continuous path is:

create_tablet_callback
  -> StorageEngine::create_tablet
  -> TabletManager::create_tablet
  -> Tablet::create_initial_rowset
  -> BaseBetaRowsetWriter::init
  -> RowsetMeta::set_tablet_schema
  -> TabletSchema::to_key
  -> TabletSchema::to_schema_pb

The FE later reported that the rollup replicas assigned to the failed BE could not be created and cancelled the rollup job.

What You Expected?

After INSERT OVERWRITE, Doris should create the rollup tablets and their initial rowsets normally. If a tablet schema or create-tablet request is invalid, the BE should return a diagnostic error instead of dereferencing invalid memory and terminating the process.

How to Reproduce?

The following sequence was reconstructed from the retained audit and FE/BE logs. Object names have been normalized.

CREATE TABLE binlog_rollup_repro (
    event_date DATE NOT NULL,
    id BIGINT NOT NULL,
    value_text VARCHAR(32) NOT NULL
) ENGINE=OLAP
DUPLICATE KEY(event_date, id)
PARTITION BY RANGE(event_date) (
    PARTITION p202401 VALUES [('2024-01-01'), ('2024-02-01')),
    PARTITION p202402 VALUES [('2024-02-01'), ('2024-03-01'))
)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
    "replication_num" = "1",
    "binlog.enable" = "true",
    "binlog.format" = "ROW"
);

INSERT INTO binlog_rollup_repro VALUES
    ('2024-01-10', 1, 'old_p1'),
    ('2024-02-10', 2, 'keep_p2');

INSERT OVERWRITE TABLE binlog_rollup_repro PARTITION(p202401)
VALUES ('2024-01-15', 101, 'overwrite_p1');

ALTER TABLE binlog_rollup_repro
ADD ROLLUP rollup_event(id, event_date, value_text);

Observed sequence:

1. INSERT OVERWRITE created a temporary partition, loaded it, and replaced p202401.
2. ADD ROLLUP created a rollup job.
3. The FE sent create-tablet tasks for the rollup replicas.
4. The failing task selected the base tablet's data directory and entered initial-rowset creation.
5. The BE crashed before logging successful tablet creation.

This sequence has not yet been rerun in an isolated environment, so deterministic standalone reproduction still needs confirmation.

Regression Test Result

Sanitized continuous stack:

doris::signal::FailureSignalHandler(...)
PosixSignals::chained_handler(...)
JVM_handle_linux_signal
libc.so.6
doris::TabletSchema::to_schema_pb(doris::TabletSchemaPB*) const
  at src/storage/tablet/tablet_schema.cpp:1437
doris::TabletSchema::to_key() const
doris::RowsetMeta::set_tablet_schema(...)
doris::BaseBetaRowsetWriter::init(...)
  at src/storage/rowset/beta_rowset_writer.cpp:391
doris::BetaRowsetWriter::init(...)
  at src/storage/rowset/beta_rowset_writer.cpp:492
doris::RowsetFactory::create_rowset_writer(...)
doris::Tablet::create_rowset_writer(...)
  at src/storage/tablet/tablet.cpp:2241
doris::Tablet::create_initial_rowset(long)
  at src/storage/tablet/tablet.cpp:2222
doris::TabletManager::_internal_create_tablet_unlocked(...)
doris::TabletManager::create_tablet(...)
  at src/storage/tablet/tablet_manager.cpp:333
doris::StorageEngine::create_tablet(...)
  at src/storage/storage_engine.cpp:1428
doris::create_tablet_callback(...)
  at src/agent/task_worker_pool.cpp:1882
std::_Function_handler<...>::_M_invoke(...)
doris::ThreadPool::dispatch_thread()
  at src/util/threadpool.cpp:628
doris::Thread::supervise_thread(void*)
  at src/util/thread.cpp:461
start_thread
clone3

Resource evidence immediately before the crash showed that the process was well below its memory limit and the host still had substantial available memory, so this was not an OOM termination.

Initial Analysis

Confirmed:

• The direct crash point is TabletSchema::to_schema_pb() during initial-rowset creation for a rollup tablet.
• The create-tablet task reused the base tablet's data directory and never reached the successful-create log entry.
• The FE-side rollup failure and the BE-side create-tablet path refer to the same failed replicas.
• The running BE commit matches the public Apache Doris commit listed above.
• Memory pressure was not the immediate cause of the process exit.

Suspected:

• At this commit, line 1437 begins access to the TabletSchema member _cluster_key_uids. The near-null fault address 0x150 is consistent with an invalid or null TabletSchema object being dereferenced at a member offset.
• The rollup create request, the base-tablet schema reference, or RowsetWriterContext::tablet_schema may be missing, stale, or otherwise invalid after the partition-replacement path.
• These are hypotheses based on the stack and event ordering; core-variable inspection or isolated reproduction is still required to identify the exact invalid object and its origin.

Anything Else?

Tracking issue: #65265

Additional evidence still needed:

• Confirmation that the SQL sequence above reproduces deterministically on the same commit.
• A minimized test showing whether ROW binlog, INSERT OVERWRITE, and base-tablet reuse are all required.
• Core-variable inspection around TabletSchema::to_schema_pb(), TabletSchema::to_key(), and BaseBetaRowsetWriter::init().

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_2736850baa2c

Maintainer triage

This is a high-severity, source-confirmed null dereference in the ROW-binlog rollup create path. No labels are currently assigned; bug, binlog, and rollup/schema-change would be appropriate triage areas.

Verified mechanism at 40762c79bd014d66a843cf21ab2a1464e5bac092

1. RollupJobV2.createRollupReplica() copies the table's enabled ROW-binlog configuration into every rollup CreateReplicaTask, but passes null as the final rowBinlogMeta argument.
2. CreateReplicaTask.toThrift() always sets binlog_config when it is non-null, while setting row_binlog_schema only when ROW binlog is enabled and rowBinlogMeta != null. The resulting rollup request therefore has binlog.enable=true, binlog.format=ROW, and no row_binlog_schema.
3. On the BE, TabletMeta::create() / the constructor preserves the enabled ROW-binlog configuration, while the constructor initializes _row_binlog_schema only when that optional request field is present (lines 209-245). Thus _row_binlog_schema remains null.
4. Tablet::enable_row_binlog() checks only the config. Consequently create_initial_rowset() enters the ROW-binlog branch and passes the null row_binlog_tablet_schema() into the second rowset writer.
5. BaseBetaRowsetWriter::init() has only a DCHECK for the pointer, which is ineffective in a release build, and RowsetMeta::set_tablet_schema() immediately evaluates tablet_schema->to_key(). That produces the reported to_key() -> to_schema_pb() stack. The first access to _cluster_key_uids also explains the near-null 0x150 fault address.

Therefore, the bad runtime candidate is specifically RowsetWriterContext::tablet_schema for the row-binlog initial writer, not the ordinary rollup tablet schema. This source defect explains the supplied stack without requiring memory pressure or a stale base-tablet schema.

The same request construction and BE dereference were still present on upstream master at 4e3c1b84dd5936b8d751e0a95719350e8cd8e980 when checked.

Scope and remaining uncertainty

• INSERT OVERWRITE is unlikely to be a prerequisite: none of the request construction above depends on partition-replacement history. The minimal trigger is likely an enabled ROW-binlog table followed by creation of a rollup tablet. This still needs an isolated run to confirm the incident exercised exactly this deterministic path.
• Reusing the base tablet's data directory is expected for rollup/schema-change creation and is orthogonal to the null schema.
• The shared CreateReplicaTask behavior is also used by other non-base-index creation/recovery paths. Those callers should be audited for the same combination of enabled ROW-binlog config plus null rowBinlogMeta; their runtime impact has not been reproduced here.

Recommended next steps

1. Confirm the minimal reproduction on the reported commit by removing INSERT OVERWRITE and running only CREATE TABLE ... binlog.format=ROW followed by ADD ROLLUP. Retain the original sequence as a regression case as well.
2. Fix the FE request invariant after confirming the intended design: because ROW binlog metadata is base-index-only, rollup/non-base tablet requests should not advertise enabled ROW binlog unless they also carry a valid row-binlog schema.
3. Add BE request-boundary validation for the inconsistent combination (ROW enabled but row_binlog_schema absent) and return a diagnostic Status instead of relying on DCHECK. This is important for compatibility with an older FE sending the malformed request.
4. Add an FE unit assertion on the generated rollup TCreateTabletReq, a BE unit test for the malformed request, and a regression test that verifies the rollup completes and all BEs remain alive.

If the minimized run does not reproduce, please attach the serialized/logged TCreateTabletReq field presence for binlog_config and row_binlog_schema, together with the rollup/base tablet and index IDs. Core values for the two RowsetWriterContext::tablet_schema instances would then distinguish this path from unrelated corruption.