[Bug] Flink Connector DATETIME decoding regression: 26.3.0 returns 1970 timestamps, 25.0.0 returns correct values

[MxlIzumi]

Summary

When reading a DATETIME column from SelectDB 2.1.8-3 using the Doris Flink Connector over Thrift, version 26.3.0 produces an incorrect 1970 timestamp. Switching the test task to Connector 25.0.0 restores the expected value.

This is a report against a SelectDB distribution, not a claim that the same behavior has already been reproduced on stock Apache Doris 2.1.8.

Environment

• Flink: company distribution based on 1.20
• Server: SelectDB 2.1.8-3
• Artifact: org.apache.doris:flink-doris-connector-1.20
• Affected version tested: 26.3.0
• Working version tested: 25.0.0
• Read protocol: Thrift; FE arrow_flight_sql_port is -1, and Flight SQL was not used
• Date/time comparison below uses UTC+08:00
• Doris DATETIME mapped to Flink TIMESTAMP(0)

Observed behavior

Value	ctime / mtime
Direct query in Doris	2026-07-30 19:44:28
Flink Connector 26.3.0	1970-01-01 08:29:45.411868
Flink Connector 25.0.0 after switching versions	2026-07-30 19:44:28

The incorrect value is present in the raw TaskManager output, not just the platform's result viewer. Casting the Flink timestamp to STRING still gives the incorrect value. Other fields are read successfully.

SQL shape used for testing (anonymized)

The source is an existing Doris OLAP table with BIGINT, VARCHAR and DATETIME columns. Table names, identifiers and connection details below have been replaced with placeholders; this exact standalone example has not been independently rerun on stock Apache Doris.

CREATE TABLE doris_datetime_probe (
    id BIGINT,
    name STRING,
    ctime TIMESTAMP(0),
    mtime TIMESTAMP(0)
) WITH (
    'connector' = 'doris',
    'fenodes' = '<FE_HOST>:<HTTP_PORT>',
    'table.identifier' = 'test.datetime_probe',
    'username' = '<USER>',
    'password' = '<PASSWORD>'
);

CREATE TABLE datetime_probe_print (
    id BIGINT,
    name STRING,
    ctime STRING,
    mtime STRING
) WITH (
    'connector' = 'print'
);

INSERT INTO datetime_probe_print
SELECT id, name, CAST(ctime AS STRING), CAST(mtime AS STRING)
FROM doris_datetime_probe
WHERE id = 1;

Use a source row whose ctime and mtime are 2026-07-30 19:44:28, and compare the output between the two Connector versions. The id/name in the example are anonymized.

Source-code observations / suspected cause

The official published source packages show a relevant change in org.apache.doris.flink.serialization.RowBatch:

• In 25.0.0, getDateTime() delegates to longToLocalDateTime(long), which infers seconds / milliseconds / microseconds from the numeric magnitude. Its comment states: "Currently, the scale of doris's arrow datetimev2 is hardcoded to 6, and there is also a time zone problem in arrow, so use timestamp to convert first" (approximately lines 554–576).
• In 26.3.0, getDateTime() uses the Arrow timestamp metadata. For timestamps with timezone metadata it calls longToLocalDateTime(value, timestampType.getUnit(), DEFAULT_ZONE_ID); the MICROSECOND branch divides by 1,000,000 (approximately lines 702–741). For timestamps without timezone metadata it delegates to vector.getObject().

In UTC+08:00, the expected date corresponds to epoch seconds 1785411868. Interpreting that same integer as microseconds produces exactly 1970-01-01 08:29:45.411868.

This suggests a mismatch between the actual numeric unit and the declared Arrow unit, which the older magnitude-based conversion may have tolerated. We have NOT yet captured the raw Arrow value/unit/timezone or the loaded class location, so this is a hypothesis rather than a confirmed wire-level diagnosis. We also have not identified the first affected Connector release between 25.0.0 and 26.3.0.

Sources:

• 25.0.0 official sources
• 26.3.0 official base-module sources

Questions

1. Is this a known server/Connector compatibility issue or regression in handling legacy Arrow timestamp encoding?
2. Which Doris/SelectDB versions and read protocols are affected?
3. Is there a fix or recommended Connector/server combination? Is there a safe compatibility option rather than relying on timestamp magnitude heuristics?
4. Could the compatibility documentation mention any applicable restrictions? The broad Doris version range does not make this behavior apparent.

Happy to provide additional sanitized diagnostics. Please advise if this should be tracked in the Connector repository or with the SelectDB distribution instead.

[Doris-Breakwater]

Thanks for the unusually precise report. The code and the numeric result support a specific compatibility regression.

Initial assessment

Verified from the public sources: this is the interaction of an older Doris Thrift/Arrow schema bug with the stricter timestamp decoding introduced in Flink Connector 26.2.0. It is not consistent with a Flink result-viewer-only or CAST(... AS STRING) problem.

The server-side chain in the Apache Doris 2.1.8 source is:

1. DataTypeDateTimeV2::get_type_as_type_descriptor() discards the column's real scale and creates a default TYPE_DATETIMEV2 descriptor. That descriptor defaults to scale 6, so the generated Arrow field advertises MICROSECOND even for DATETIMEV2(0).
2. The value serializer still uses the real SerDe scale: scale 0 emits epoch seconds, scales 1-3 emit milliseconds, and scales 4-6 emit microseconds. Thus the schema and value unit disagree for scales 0-3.
3. Connector 25.0.0 hides that invalid wire representation by inferring the unit from numeric magnitude. Connector PR #676 removed the heuristic and began honoring Arrow unit metadata; the first released tag containing that change is 26.2.0, so 26.2.0 and 26.3.0 expose the server-side mismatch. 26.1.1 and earlier retain the heuristic.

The reported value is an exact fingerprint of this path: 2026-07-30 19:44:28 +08:00 is epoch second 1785411868; decoding 1785411868 as microseconds gives 1970-01-01 08:29:45.411868 +08:00.

Apache fixed the lost scale on branch 2.1 in PR #49244 by copying _scale into the Arrow type descriptor. In the public source tags I checked, 2.1.8-rc01 and 2.1.9-rc01 lack the fix, while 2.1.9-rc02, 2.1.10-rc01, and 2.1.11-rc01 contain it. This establishes that the stock Apache Doris 2.1.8 source has the same mechanism, although the report has not run a stock 2.1.8 binary reproduction.

Scope and compatibility

• The code-backed affected combination is the Thrift source path, a server build lacking the scale-preservation fix, DATETIMEV2 scale 0-3, and Connector 26.2.0 or newer.
• Scale 4-6 already uses microsecond values, so it is not affected by this specific unit mismatch.
• This report does not exercise Flight SQL, and its producer path has changed separately; I would not declare Flight SQL affected from this evidence.
• SelectDB 2.1.8-3 is a distribution-specific version. Its exact mapping cannot be inferred from the name. The observed value strongly indicates that its BE lacks the equivalent fix, but the exact build revision and raw Arrow field still need to be captured before treating that deployment detail as confirmed.

Recommended action

1. Preferred fix: upgrade the server/BE to a build containing the equivalent of Apache Doris PR branch-2.1: [fix](arrow) Fix Arrow serialization and deserialization of Date/Datetime/Array/Map/Struct/Bitmap/HLL/Decimal256 types #49244 (preserve DataTypeDateTimeV2::_scale when constructing the Arrow descriptor). FE and BE build revisions should both be checked; the BE emits the Arrow batch.
2. Temporary workaround: keep the already validated Connector 25.0.0 pin until the server is upgraded. Connector 26.1.1 is the last release before PR New SQL Optimizer #676 and has the same magnitude-based compatibility behavior, but it has not been validated in this environment, so it should not be presented as confirmed without a test.
3. I found no Connector 26.3.0 option that restores legacy magnitude decoding. Automatically reintroducing that heuristic is unsafe because magnitude is not an unambiguous timestamp-unit signal. If compatibility must be connector-side, an explicit legacy mode or a server capability/version handshake would be safer.
4. The connector compatibility documentation should state that 26.2.0+ requires a Doris/SelectDB build with correct Arrow DATETIMEV2 unit metadata, and should reference the server-side scale-preservation fix rather than only giving a broad Doris version range.

This is primarily an interoperability issue for apache/doris-flink-connector (the compatibility change is PR #676), while the producer-side remediation for this particular server build belongs with the SelectDB distribution. A regression test matrix should cover server-before/server-after PR #49244 against connector 26.1.1/26.2.0+, with DATETIMEV2(0), (3), and (6).

Diagnostics still needed to confirm the reported deployment

Please provide sanitized versions of:

• full FE and BE version/build commit output (not only 2.1.8-3) and SHOW CREATE TABLE for the real source columns, including their scales;
• for one affected Arrow cell: fieldVector.getClass().getName(), fieldVector.getField().getType() (unit and timezone), and ((TimeStampVector) fieldVector).get(rowIndex); the predicted tuple is a microsecond-declared timestamp carrying raw value 1785411868;
• the loaded RowBatch class location (for example, RowBatch.class.getProtectionDomain().getCodeSource().getLocation()) plus the deployed connector JAR checksum or dependency tree, to rule out a classpath collision;
• if practical, the same minimal table/row tested against a stock Apache Doris build before and after PR branch-2.1: [fix](arrow) Fix Arrow serialization and deserialization of Date/Datetime/Array/Map/Struct/Bitmap/HLL/Decimal256 types #49244.

No Doris execution profile is needed for this correctness issue. The Arrow field metadata and raw long value are the decisive evidence.

Breakwater-GitHub-Analysis-Slot: slot_bd1f9dd2153c

Thanks for opening this! I'd like to help investigate.

Could you share:

• Steps to reproduce
• Expected vs actual behavior
• Environment details

I'll take a look at the code and see if I can help.