Skip to content

Support spark compatible cast from int -> timestamp #20554

@coderfender

Description

@coderfender

Is your feature request related to a problem or challenge?

DF's int to timestamp cast just reinterprets the nit to timestamp.

DataFusion: int value = timestamp value in target unit (microseconds)
Spark: int value = seconds since epoch, then converted to target unit

Example : casting 100 to TimestampMicrosecond:

  • DataFusion -> 100 microseconds (0.0001 seconds)
  • Spark -> 100 seconds = 100,000,000 microseconds

Describe the solution you'd like

Implement new spark compatible cast ScalarUDF and port the existing code from comet back to spark-expr crate

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions