Skip to content

fix(bigquery): apply DataFormatOptions to tabledata.list requests - #14889

Open
keshavdandeva wants to merge 2 commits into
mainfrom
fix/bigquery-tabledata-format-options
Open

keshavdandeva wants to merge 2 commits into
mainfrom
fix/bigquery-tabledata-format-options

Conversation

@keshavdandeva

Copy link
Copy Markdown
Contributor

b/579291912

BigQueryOptions.setDataFormatOptions(...) is only sent on jobs.query. Rows read through tabledata.list don't get it, so BigQuery returns TIMESTAMPs as float64 there while the client decodes them using the configured format. tabledata.list serves:

  • listTableData / Table.list
  • page 2+ of query()
  • Job.getQueryResults()

Impact:

  • useInt64Timestamp(true): getTimestampValue() / getTimestampInstant() throw NumberFormatException.
  • timestampFormatOptions(INT64): values are decoded as seconds, so dates come out wrong. This also happens on page 1.
  • ISO8601_STRING: rows from tabledata.list come back as float64.
  • float64 loses microseconds, e.g. 9999-12-31 23:59:59.999999 becomes +10000-01-01.

Changes

  • HttpBigQueryRpc: listTableData sends formatOptions.useInt64Timestamp / formatOptions.timestampOutputFormat, only when they differ from the defaults. Default requests are unchanged.
  • BigQueryImpl: rows are decoded as int64 when either useInt64Timestamp(true) or timestampFormatOptions(INT64) is set.

No public API changes.

Tests

  • HttpBigQueryRpcTest: defaults send no formatOptions, and configured options are sent.
  • BigQueryImplTest: INT64 values from listTableData decode as int64.
  • ITBigQueryTest#testLosslessMaxTimestampIntegration: now reads 2 pages (the second via tabledata.list) and adds an INT64 case.

All new and changed tests fail without the fix and pass with it.

@keshavdandeva
keshavdandeva requested review from a team as code owners November 2, 2026 14:24

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the BigQuery client to support and propagate DataFormatOptions (specifically TimestampFormatOptions) when listing table data. It modifies BigQueryImpl to pass the full DataFormatOptions object and updates HttpBigQueryRpc to set these format options on the underlying API request. Additionally, integration and unit tests are updated to cover these changes. A potential NullPointerException was identified in HttpBigQueryRpc.java when calling toString() on a potentially null timestampFormat object, which should be resolved by adding a null check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant