fix(sql): DATEDIFF / DATE_DIFF / TIMESTAMPDIFF follow their vendors; DATE_TRUNC(…, WEEK) is the ISO Monday

[fupelaqu]

What

DATEDIFF, DATE_DIFF and TIMESTAMPDIFF follow their vendors' definitions, and DATE_TRUNC(x, WEEK) is the ISO Monday.

• Direction, by argument layout. Every vendor computes end − start; the layout decides where the end date sits.
  • Dates first ⇒ first − second: DATEDIFF(a, b), DATEDIFF(a, b, unit) and DATE_DIFF(a, b[, unit]), as MySQL's DATEDIFF and BigQuery's DATE_DIFF(end_date, start_date, granularity) do.
  • Unit first ⇒ last − middle: DATEDIFF(unit, a, b), DATE_DIFF(unit, a, b) and TIMESTAMPDIFF(unit, a, b), as SQL Server, Snowflake, Redshift, DuckDB, Trino, Elasticsearch SQL and MySQL's TIMESTAMPDIFF do.
• Counting, by name.
  • DATEDIFF / DATE_DIFF count the calendar boundaries crossed, as SQL Server, Snowflake, Redshift, BigQuery and DuckDB's date_diff do.
  • TIMESTAMPDIFF counts the whole units elapsed between instants, with MySQL's month algorithm.
• Calendar. Weeks are ISO 8601, starting on Monday. Quarters start on Jan / Apr / Jul / Oct 1. All boundaries are in UTC.
• DATE_TRUNC(x, WEEK) is the ISO Monday 00:00 UTC in every venue. The script path used to move a date forward to the Sunday ending its week; GROUP BY already used Monday.
• One rendering for every venue: row level, WHERE, per group, the HAVING alias route, the materialized-view calculation channel and the ingest processor. A computed column now computes exactly what a query computes.
• A corrected premise. The earlier fix for MySQL's DATEDIFF sign assumed BigQuery's DATE_DIFF(start, end, unit) is end − start. BigQuery documents DATE_DIFF(end_date, start_date, granularity) = end_date − start_date. DateDiffSignSpec is rewritten accordingly.
• Docs and help:
  • the definition per layout and name, ISO weeks and UTC, with every example run;
  • the age / tenure examples rewritten: TIMESTAMPDIFF(YEAR, birthdate, CURRENT_DATE) and DATEDIFF(CURRENT_DATE, join_date, DAY);
  • a "Changed in 0.24.0" note and a migration note.

Measured (ES 8.18.3 through the gateway; oracles independent of the engine: vendor-documented examples and calendar arithmetic)

Check	main	this branch
DateFunctionExecutionSpec population (testkit, so CI runs every major): 4,896 statements = spellings × 8 units × edge pairs × 4 venues	2,188 answers differ from the definition, 189 errors	0 wrong, 0 errors
TIMESTAMPDIFF over COALESCE / CASE / CURRENT_DATE / ::DATETIME operands: 512 statements + 128 computed columns	—	0 wrong
DATE_TRUNC(…, WEEK) across venues	11/13 wrong	ISO Monday in 13/13
Vendor examples (BigQuery 559, SQL Server 1, DuckDB 2, MySQL 3 / −1)	—	all right
View channel and ingest processor vs search, same items	—	identical (840/840, 84/84); processor stores 2,016/2,016 right
Newly emitted Painless APIs, allow-lists of 6.8.23 / 7.17.29 / 8.18.3 / 9.0.3	—	all present
#409's HAVING population	—	unchanged, except 6 messages that now echo TIMESTAMPDIFF(…)
Parse / emission cost, 5 interleaved rounds against main	—	×1.016 / ×1.029, ranges overlap

The edge pairs:

• 1 s across a year / quarter / month / ISO-week / day / hour end;
• a partial span in every unit;
• reversed operands;
• the same instant;
• DATE against TIMESTAMP;
• literal, column and aggregate operands.

Suites: sql 1788, es6bridge 256, softclient4es8-sql-bridge 263, core 1148. ES 8.18.3 integration: 12 suites, 378/378. Lint is clean.

Not run locally, left to CI: ES 6.8 / 7.17 / 9.0, and the Scala 2.12 cross-compile.

⚠️ Breaking changes against 0.23.x (release notes)

• Direction. Every dates-first form (DATEDIFF(a, b[, unit]), DATE_DIFF(a, b[, unit])) returns a − b; 0.23 returned b − a. The unit-first forms keep b − a.
• Counting.
  • DATEDIFF / DATE_DIFF count calendar boundaries. From WEEK up they counted the whole units elapsed between the calendar dates: DATE_DIFF(MONTH, '1992-09-15', '1992-11-14') was 1 and is 2, and a WEEK boundary is a Monday.
  • HOUR / MINUTE / SECOND count boundaries. DAY still counts calendar days.
• DATE_TRUNC(x, WEEK) is the Monday. Outside GROUP BY it moved a date forward to the Sunday ending its week.
• Computed columns (SCRIPT AS) compute what a query computes. They counted elapsed time between instants for every unit (a DAY was 24 h), and a string literal operand failed CREATE TABLE.
• New in 0.24.0: TIMESTAMPDIFF, the QUARTER unit, HOUR / MINUTE / SECOND over a column or an aggregate, and literal operands.
• Deployed definitions keep their 0.23 results until re-created (measured). A computed column, an ingest pipeline and a materialized view keep their deployed script: inserts, ALTER TABLE … ADD COLUMN, SHOW CREATE, re-running SHOW CREATE PIPELINE's text, view refreshes and restarts all leave it as it was.
  • Re-creating from the same SQL adopts the new definition.
  • CREATE OR REPLACE MATERIALIZED VIEW with an unchanged text keeps the old results; DROP and CREATE the view to adopt the new definition.
  • SHOW CREATE shows the stored text, which now means the new definition.
  • To keep a 0.23 value, swap the dates: DATE_DIFF(b, a, unit). TIMESTAMPDIFF(YEAR, birthdate, CURRENT_DATE) is the age that DATE_DIFF(birthdate, CURRENT_DATE, YEAR) returned.
• TIMESTAMPDIFF renders as TIMESTAMPDIFF: in SHOW CREATE and in persisted view text.

Elasticsearch 6.8 (added after the first CI run)

The first CI run failed on ES 6.8 only. Two causes, both fixed, each in its own commit:

• Date functions read 6.8's date values. ES 6.8 hands scripts a JodaCompatibleZonedDateTime, which implements no java.time interface. That broke TIMESTAMPDIFF's month count, and DATEDIFF / DATE_DIFF over COALESCE(ts, d). The latter never worked on 6.8, v0.23.0 included. Such operands are now brought to a real ZonedDateTime, which reads the same instant on every major.
• The ES 6 Jest client can CREATE TABLE on 6.8. Every CREATE TABLE, and every PARTITION BY template, created through Jest failed on 6.8 with "Root mapping definition has unsupported parameters", v0.23.0 included. The fix: Jest now sends include_type_name=false on 6.8, as the REST client does. 6.7.x and 7+ are unchanged.
• Release note: Template.Create (softclient4es6-jest-client) gains a field. That is binary-incompatible, so downstream projects must rebuild.

Measured locally:

• ES 6.8.23, Jest and REST date specs: 10/10 each, 0 cancelled; the populations give 0 wrong.
• Jest specs pinned to 6.7.2 and 6.8.23 pass.
• ES 8.18.3: 379/379.

Follow-up

The web docs (the DATE_DIFF row and the age / seniority examples) and two blog posts carry the old definition; a separate web PR updates them.

🤖 Generated with Claude Code