-
Notifications
You must be signed in to change notification settings - Fork 2k
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
DF's int to timestamp cast just reinterprets the nit to timestamp.
DataFusion: int value = timestamp value in target unit (microseconds)
Spark: int value = seconds since epoch, then converted to target unit
Example : casting 100 to TimestampMicrosecond:
- DataFusion -> 100 microseconds (0.0001 seconds)
- Spark -> 100 seconds = 100,000,000 microseconds
Describe the solution you'd like
Implement new spark compatible cast ScalarUDF and port the existing code from comet back to spark-expr crate
Describe alternatives you've considered
No response
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request