Skip to content

feat: migrate Arrow Go to v18 #530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mandukhai-Alimaa wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: migrate Arrow Go to v18 #530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .golangci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ linters:
- $gostd
- github.com/databricks/databricks-sql-go
- github.com/databricks/databricks-sql-kernel-bindings
- github.com/apache/arrow/go/v12
- github.com/apache/arrow-go/v18
- github.com/apache/thrift/lib/go/thrift
- github.com/coreos/go-oidc/v3/oidc
- github.com/golang-jwt/jwt/v5
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased

- **Breaking change:** Upgrade Arrow Go from v12 to v18.8.0. Arrow batch API consumers must migrate to `github.com/apache/arrow-go/v18`. The `database/sql` and Arrow IPC iterator APIs are unchanged.

- Upgrade go-oidc to v3.21.0 (go-jose v4.1.6) and x/oauth2 to v0.36.0 to address go-jose/v3 security findings (databricks/databricks-sql-go#526).
- Reduce kernel-backend JSON rendering allocations for nested Boolean and signed-integer values.
- Render most nested ARRAY / MAP / STRUCT scalars (numbers, booleans, and strings that need no escaping) without `encoding/json` on both backends, avoiding the per-value overhead of Go 1.27's json/v2-backed `encoding/json` (databricks/databricks-sql-go#527).
Expand Down
4 changes: 2 additions & 2 deletions doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,9 +444,9 @@ The driver exposes two public interfaces for working with record batches from th
}

type ArrowBatchIterator interface {
// Retrieve the next arrow.Record.
// Retrieve the next arrow.RecordBatch.
// Will return io.EOF if there are no more records
Next() (arrow.Record, error)
Next() (arrow.RecordBatch, error)

// Return true if the iterator contains more batches, false otherwise.
HasNext() bool
Expand Down
4 changes: 2 additions & 2 deletions examples/arrrowbatches/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"strconv"
"time"

"github.com/apache/arrow/go/v12/arrow"
"github.com/apache/arrow-go/v18/arrow"
dbsql "github.com/databricks/databricks-sql-go"
dbsqlrows "github.com/databricks/databricks-sql-go/rows"
"github.com/joho/godotenv"
Expand Down Expand Up @@ -124,7 +124,7 @@ func loopWithNext(db *sql.DB) {
}

var iBatch, nRows int
var b arrow.Record
var b arrow.RecordBatch
for b, err = batches.Next(); err == nil; b, err = batches.Next() {
log.Printf("batch %v: nRecords=%v\n", iBatch, b.NumRows())
iBatch += 1
Expand Down
4 changes: 2 additions & 2 deletions examples/ipcstreams/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"strconv"
"time"

"github.com/apache/arrow/go/v12/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/ipc"
dbsql "github.com/databricks/databricks-sql-go"
dbsqlrows "github.com/databricks/databricks-sql-go/rows"
"github.com/joho/godotenv"
Expand Down Expand Up @@ -97,7 +97,7 @@ func main() {

// Process records in the stream
for ipcReader.Next() {
record := ipcReader.Record()
record := ipcReader.RecordBatch()
recordCount++
log.Printf("Stream %d, Record %d: %d rows, %d columns",
streamCount, recordCount, record.NumRows(), record.NumCols())
Expand Down
31 changes: 11 additions & 20 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,48 +3,39 @@ module github.com/databricks/databricks-sql-go
go 1.27.3

require (
github.com/apache/arrow/go/v12 v12.0.1
github.com/apache/arrow-go/v18 v18.8.0
github.com/apache/thrift v0.24.0
github.com/coreos/go-oidc/v3 v3.21.0
github.com/golang-jwt/jwt/v5 v5.2.2
github.com/joho/godotenv v1.4.0
github.com/mattn/go-isatty v0.0.20
github.com/pierrec/lz4/v4 v4.1.15
github.com/stretchr/testify v1.8.1
github.com/mattn/go-isatty v0.0.24
github.com/pierrec/lz4/v4 v4.1.29
github.com/stretchr/testify v1.12.1
golang.org/x/oauth2 v0.36.0
golang.org/x/sync v0.22.0
gotest.tools/gotestsum v1.8.2
)

require (
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dnephin/pflag v1.0.7 // indirect
github.com/fatih/color v1.16.0 // indirect
github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/go-jose/go-jose/v4 v4.1.6 // indirect
github.com/goccy/go-json v0.9.11 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v2.0.8+incompatible // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
// Deliberate security-scan minimum; go mod tidy removes this unused requirement.
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
golang.org/x/mod v0.40.0 // indirect
golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/tools v0.49.0 // indirect
golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

require (
Expand Down
Loading