Repository navigation
Conversation
Preserve the final byte of unterminated lines and recognize CR, LF and CRLF without reading beyond the configured buffer slice. Add JDK-backed regression tests for line contents, cursor positions and subsequent binary reads.
|
The low-level However, I could not connect this change to an end-to-end Paimon workload. Repository-wide caller tracing found no production caller of Closing for now because the current contribution lacks demonstrated end-to-end impact. This is not an implementation-correctness objection. Please provide an actual affected Paimon caller/workload and a regression at that boundary if there is one, and we can revisit the change. |
Purpose
DataInputDeserializer.readLine() drops the final byte of an unterminated line: reading "ab" returns "a". It also continues past a lone carriage return, consuming the following line or binary payload.
Read each byte before checking for the next one, recognize LF, CR and CRLF, and keep CRLF lookahead within the configured buffer slice. This updates the existing implementation to follow the JDK 8 DataInput contract. Other deserialization methods and serialized formats are unchanged.
Tests
mvn -o -B -ntp -pl paimon-common -am -DwildcardSuites=none -DfailIfNoTests=false -Dtest=DataInputDeserializerTest,DataOutputSerializerTest testWindows 11, JDK 8u451, Maven 3.8.9, baseline 1902379: 22 tests, 9 failures before the fix; the same 22 tests pass after it. No fast-build profile was used.
The regression tests compare returned lines, remaining bytes and cursor positions with JDK DataInputStream, including bounded slices and unsigned bytes. Mixed line/binary tests read int, long and UTF values written by JDK DataOutputStream. The existing DataOutputSerializerTest also passes.
Spotless and Enforcer passed. Both changed files passed Checkstyle 8.14 with the project configuration in a separate directory outside the target-path suppression. Local verification did not include Linux, full-module or Flink/Spark suites, or the verify-phase RAT check.
The publication branch is based on 0369648. Its selected parent/API/test-utils/common build inputs and reviewed files match the tested baseline above, and the patch is unchanged. Local tests were not rerun for this base change.