test: enable nested array cast coverage#4278
Conversation
|
@coderfender can you help re-trigger cancelled tests? |
|
@manuzhang , could you rebase with main whenever you get a chance ? There were some CI failures over the weekend. |
|
Happy to trigger CI once you get your main rebase ? For some reason, CI doesnt give me an option to retrigger the failed tests |
|
@coderfender This PR is already based on latest main branch. |
|
@manuzhang we are having issues with CI runners being fixed in #4285 :( |
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @manuzhang so the problem was with the test itself?
Enable the nested ArrayType cast matrix with bounded nested inputs, and keep DecimalType(38,18) out of the matrix for the known decimal-to-floating ULP mismatch. Co-authored-by: Codex <codex@openai.com>
manuzhang
force-pushed
the
codex/enable-nested-array-cast-test
branch
from
0bdb3f0 to
d6c6ae3
Compare
|
@coderfender @comphead please help review again. The tests passed now. |
|
This looks good @manuzhang . Let me review this shortly. Thank you for your patience |
|
@coderfender any updates? |
|
Hey sorry for the delay. The PR looks good to me . Let me take another look over the weekend and update the thread. Thank you for your patience |
coderfender
left a comment
coderfender
left a comment
There was a problem hiding this comment.
LGTM ! Thank you @manuzhang . I will let @comphead take another pass
| Row( | ||
| Seq[Any]( | ||
| values(i % values.length), | ||
| if (i % 3 == 0) null else values((i + 1) % values.length), |
There was a problem hiding this comment.
nit : may be add a comment to let user's know every 3rd element is null ?
Co-authored-by: Codex <codex@openai.com>
3 participants
Closes #3906
Summary
ArrayTypecast matrix inCometCastSuite.DecimalType(38,18)from the nested matrix for the same known decimal-to-floating ULP mismatch as the one-dimensional matrix.JavaConvertersusage withscala.jdk.CollectionConverters.Root Cause
The previous nested test used
generateArrays(100, ArrayType(dt)), which let Spark random data generation build very large nested arrays. Thearray<array<string>>toarray<array<boolean>>case produced a task result around 401 MiB and failed deserialization withunread block data. After bounding the nested inputs, the remaining failure was the existing decimal38-to-floating precision mismatch, so the nested matrix now mirrors the one-dimensional exclusion.Tests
make core./mvnw test -Pspark-4.1 -Dtest=none -Dsuites="org.apache.comet.CometCastSuite cast nested ArrayType to nested ArrayType"Co-authored-by: @codex