feat: default DECIMAL to native Arrow decimal128 on the wire (parity with Python/JDBC)

[msrathore-db]

Summary

Flip the Thrift backend's useArrowNativeDecimal default from false to true so DECIMAL columns are requested as native Arrow decimal128 (a compact fixed-width binary encoding) on the wire instead of UTF8 strings. This brings the Go driver to parity with the other first-party drivers:

• databricks-sql-python defaults _use_arrow_native_decimals=True (thrift_backend.py), sending decimalAsArrow=true.
• databricks-jdbc hardcodes setDecimalAsArrow(true) (DatabricksThriftServiceClient.java).
• databricks-sql-go was the outlier — defaulting to false, i.e. UTF8-string decimals on the wire.

The motivation is latency / payload size: native decimal128 is a compact fixed-width binary encoding, avoiding the larger variable-length UTF8 payloads and per-cell string parsing on decimal-heavy result sets.

What does not change

Scanning through database/sql is unchanged: a top-level DECIMAL is still returned as a lossless, scale-applied string. Go's driver.Value contract permits only int64/float64/bool/[]byte/string/time.Time/nil — there is no decimal type, so a string remains the faithful lossless scalar (Python returns Decimal and JDBC returns BigDecimal because their language APIs allow it; Go structurally cannot). The native decimal128 surfaces only through GetArrowBatches.

Verified end-to-end against a live warehouse:

Config	database/sql	GetArrowBatches (wire)
default (new)	string "19.99"	decimal(38,2) native
WithArrowNativeDecimal(false)	string "19.99"	utf8 (legacy)

Breaking change (GetArrowBatches only)

Consumers of the raw-Arrow GetArrowBatches path now receive DECIMAL columns as arrow.Decimal128 arrays rather than string arrays. To restore the previous string wire format, pass WithArrowNativeDecimal(false) or DSN useArrowNativeDecimal=false. The standard database/sql path is source- and value-compatible.

The kernel backend already sends Decimal128 on the wire and renders DECIMAL exactly, so it is unaffected.

Implementation notes

• ArrowConfig.WithDefaults() now sets UseArrowNativeDecimal = true.
• The useArrowNativeDecimal DSN carrier (UserConfig.UseArrowNativeDecimalDSN) becomes a *bool. This is necessary: the connector copies the carrier onto ArrowConfig unconditionally, so with a plain bool a silent sql.Open(dsn) would reset the new default back to false. As a pointer, nil = "unspecified → keep the native-on default", and a non-nil value = explicit override (so useArrowNativeDecimal=false still wins). DeepCopy deep-copies the pointer.

Test plan

• Full module suite green (go build, go vet, go test ./... — 1093 passing).
• Connector tests updated: native-on by default, explicit WithArrowNativeDecimal(false) override, DSN =false override, and silent DSN keeps the default.
• Three arrowbased scanner tests that replay the all_types.json fixture (whose decimal column is captured in the legacy string wire format) now explicitly opt to the string path, matching the sibling "Retrieve values" test.
• End-to-end validated against a live warehouse (table above).

This pull request and its description were written by Isaac.

[peco-review-bot]

✅ No issues identified by the review bot.