fix(compute): detect temporal rounding overflow - #1127
Conversation
65f4e12 to
71e18ac
Compare
|
Since this is a draft, i'll hold off on further review until it's marked ready and the conflicts are resolved |
a61e0eb to
1c9b885
Compare
zeroshade
left a comment
There was a problem hiding this comment.
Found three blocking calendar-rounding issues: valid nanosecond results are rejected because helper boundaries are range-checked, wider temporal types are unnecessarily narrowed to nanoseconds, and long calendar periods still silently overflow through time.Duration midpoint arithmetic.
This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. If you think one of the findings is misapplied, please reply on the PR and a maintainer will weigh in.
More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
| if dateErr != nil { | ||
| return 0, dateErr | ||
| } | ||
| yearEnd, dateErr := checkedCalendarDate(nextYear, 1, tz) |
There was a problem hiding this comment.
Blocking: Restricting this intermediate period boundary to the nanosecond timestamp window rejects inputs whose selected result is representable. For timestamp[ns] 2262-01-05 rounded half-up to one year, the correct result is 2262-01-01, but this checks the unused 2263-01-01 boundary and returns overflow. Likewise, 2201-01-01 rounded to 100 years should return 2200-01-01, but checking the 2300 boundary fails. Main succeeds in both cases. Please permit calendar boundaries outside the output window and validate only the selected result with timeToNanos; cover year, quarter, and month boundaries.
| } | ||
|
|
||
| // convertToNanos converts a timestamp value to nanoseconds. | ||
| func convertToNanos(ts int64, unit arrow.TimeUnit) (int64, error) { |
There was a problem hiding this comment.
Blocking: Every calendar path is still narrowed to an int64 nanosecond intermediate, even when the input and correct result are representable in their original type. For example, flooring Date32(1500-06-15) to one year should return 1500-01-01, but this PR returns overflow. The same affects Date64 and second/millisecond/microsecond timestamps, whose ranges are much wider than timestamp nanoseconds. Please perform calendar arithmetic without narrowing to nanoseconds and range-check the final result in the input unit.
| if dateErr != nil { | ||
| return 0, dateErr | ||
| } | ||
| rounded = halfRoundPeriod(t, yearStart, yearEnd) |
There was a problem hiding this comment.
Blocking: halfRoundPeriod computes periodEnd.Sub(periodStart)/2; time.Time.Sub saturates at math.MaxInt64 for periods longer than about 292 years. Consequently, rounding 1948-01-01 to a 300-year multiple uses a midpoint in 1946 instead of the true 1950 midpoint and returns 2100 rather than 1800. This is pre-existing, but it is an unchecked calendar-rounding overflow directly within this PR’s scope. Please compute calendar midpoints without converting the entire period to time.Duration, with a regression test for a period over 292 years.
e7f637f to
e267708
Compare
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for the update. The earlier nanosecond-boundary and long-period midpoint examples are fixed, and the test suite passes locally. I found four remaining overflow/range cases that still need correction before this is safe across the supported temporal ranges.
|
|
||
| // Slow path: convert to nanoseconds for calendar origin or incompatible units | ||
| tsNanos := convertToNanos(ts, inputUnit) | ||
| tsNanos, err := convertToNanos(ts, inputUnit) |
There was a problem hiding this comment.
Blocking: The fixed-duration slow path still rejects representable wider-unit values when the requested rounding is finer than the input resolution. For example, flooring a second-resolution 1500-06-15 12:34:56 timestamp to one nanosecond is an exact no-op, but this conversion returns invalid: temporal rounding overflow. Please avoid narrowing when the input is already aligned to the requested interval, and add coverage outside the timestamp-nanosecond window.
| return time.Time{}, overflowError() | ||
| } | ||
|
|
||
| startSeconds := periodStart.Unix() |
There was a problem hiding this comment.
Blocking: time.Time.Unix() is undefined once a boundary lies outside the int64 seconds range, so this still rejects a representable selected result at the edge of timestamp[s]. With an input of January 2 in the year containing math.MaxInt64 seconds, half-up rounding to one year should select that year's representable January 1; the next-year boundary exceeds int64, these calls wrap, and checkedSubInt64 returns overflow. Please compute the midpoint without requiring both boundaries to fit Unix seconds and cover this edge.
| epochWeekStart = time.Date(epochWeekStart.Year(), epochWeekStart.Month(), epochWeekStart.Day(), 0, 0, 0, 0, tz) | ||
|
|
||
| daysSinceEpochWeek := int(startOfWeek.Sub(epochWeekStart).Hours() / 24) | ||
| daysSinceEpochWeek := int64(startOfWeek.Sub(epochWeekStart).Hours() / 24) |
There was a problem hiding this comment.
Blocking: This reintroduces the same approximately 292-year saturation through time.Time.Sub. Flooring a second-resolution 1500-06-15 12:34:56 timestamp to one week returns 1677-09-19 instead of 1500-06-10 because the day distance saturates before division. Please calculate calendar-day/week distance without time.Duration, with wide-range week regression coverage.
| } | ||
|
|
||
| // Convert back to the input unit, validating only the selected result. | ||
| roundedTimestamp, err := arrow.TimestampFromTime(rounded, inputUnit) |
There was a problem hiding this comment.
Blocking: This does not actually validate second-resolution results: arrow.TimestampFromTime returns val.Unix() without a range check for arrow.Second. Ceiling math.MaxInt64 seconds to the next year therefore succeeds with wrapped value -9223372036852412416 instead of returning overflow. Please explicitly validate second-unit conversion (or make TimestampFromTime do so) and add a boundary test.
Rationale for this change
Temporal rounding can overflow while converting timestamps to nanoseconds, calculating the rounding interval, or multiplying a rounded quotient. The current unchecked arithmetic can return an incorrect timestamp.
What changes are included in this PR?
Use checked arithmetic for temporal conversions, interval setup, rounding results, calendar conversion, and origin adjustments. Propagate overflow as an Arrow invalid-value error.
Are these changes tested?
go test ./arrow/computeAre there any user-facing changes?
Inputs and options that cannot be represented during temporal rounding now return an error instead of a wrapped result.