chore(migration): Migrate code from googleapis/python-bigquery-pandas into packages/pandas-gbq

[parthea]

See #10988.

This PR should be merged with a merge-commit, not a squash-commit, in order to preserve the git history.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request completes a significant structural refactoring by migrating the entire python-bigquery-pandas library into a new packages/pandas-gbq directory within the monorepo. This change consolidates the project's codebase, ensuring all source code, documentation, configuration, and tests are organized under the new package structure while preserving development practices and historical integrity.

Highlights

• Codebase Migration: The entire python-bigquery-pandas library has been migrated into a new packages/pandas-gbq directory, establishing a monorepo structure for the project.
• Configuration and Documentation Files: Numerous configuration files (e.g., .coveragerc, .flake8, .pre-commit-config.yaml, noxfile.py) and comprehensive documentation files (e.g., AUTHORS.md, CHANGELOG.md, CONTRIBUTING.rst, docs/) have been added to the new packages/pandas-gbq directory.
• Source Code and Tests Relocation: All Python source code (pandas_gbq/) and associated test files (tests/) have been moved to align with the new package structure under packages/pandas-gbq.
• Migration Script Update: The single-library.git-migrate-history.sh script was updated to include an additional --recurse-submodules flag for git clone and git filter-branch commands, ensuring proper handling of submodules during the migration process.

Changelog

• packages/pandas-gbq/.coveragerc
  • Configured code coverage settings for the new package.
• packages/pandas-gbq/.flake8
  • Configured flake8 linting rules for the new package.
• packages/pandas-gbq/.gitattributes
  • Defined Git attribute settings for file handling within the new package.
• packages/pandas-gbq/.gitignore
  • Specified files and directories to be ignored by Git for the new package.
• packages/pandas-gbq/.librarian/state.yaml
  • Configured librarian state for library management in the new package.
• packages/pandas-gbq/.pre-commit-config.yaml
  • Configured pre-commit hooks for code quality checks in the new package.
• packages/pandas-gbq/.readthedocs.yml
  • Configured Read the Docs for documentation building in the new package.
• packages/pandas-gbq/.repo-metadata.json
  • Defined repository metadata for the new package.
• packages/pandas-gbq/AUTHORS.md
  • Documented copyright holders and contribution policy for the new package.
• packages/pandas-gbq/CHANGELOG.md
  • Documented the project's release history, with the latest entry 0.34.0 detailing documentation updates, support for BigLake tables in pandas_gbq.sample, and compatibility with pandas 3.0 prerelease.
• packages/pandas-gbq/CODE_OF_CONDUCT.md
  • Established the project's code of conduct for the new package.
• packages/pandas-gbq/CONTRIBUTING.rst
  • Provided guidelines for contributing to the new package.
• packages/pandas-gbq/LICENSE.txt
  • Included the BSD 3-Clause License for the new package.
• packages/pandas-gbq/MANIFEST.in
  • Specified files to be included in the source distribution of the new package.
• packages/pandas-gbq/README.rst
  • Provided an overview and usage instructions for the new package.
• packages/pandas-gbq/ROADMAP.md
  • Outlined the future development plans and vision for the new package.
• packages/pandas-gbq/SECURITY.md
  • Documented the security policy and reporting procedure for the new package.
• packages/pandas-gbq/benchmark/README.md
  • Described the purpose and usage of benchmark scripts for the new package.
• packages/pandas-gbq/benchmark/read_gbq_large_results.py
  • Added a script for benchmarking large read_gbq results in the new package.
• packages/pandas-gbq/benchmark/read_gbq_small_results.py
  • Added a script for benchmarking small read_gbq results in the new package.
• packages/pandas-gbq/docs/Makefile
  • Provided commands for building documentation for the new package.
• packages/pandas-gbq/docs/_static/custom.css
  • Added custom styles for documentation in the new package.
• packages/pandas-gbq/docs/_static/js/authcodescripts.js
  • Added JavaScript for authentication code handling in the new package's docs.
• packages/pandas-gbq/docs/_static/style.css
  • Defined general styles for documentation in the new package.
• packages/pandas-gbq/docs/_templates/layout.html
  • Customized the documentation layout for the new package.
• packages/pandas-gbq/docs/api.rst
  • Documented the public API reference for the new package.
• packages/pandas-gbq/docs/conf.py
  • Configured Sphinx for documentation generation in the new package.
• packages/pandas-gbq/docs/contributing.rst
  • Detailed documentation on contributing to the new package.
• packages/pandas-gbq/docs/howto/authentication.rst
  • Provided instructions for authentication in the new package's documentation.
• packages/pandas-gbq/docs/howto/authentication_highly_constrained_environments.rst
  • Documented authentication for constrained environments in the new package's documentation.
• packages/pandas-gbq/docs/index.rst
  • Created the main index for the new package's documentation.
• packages/pandas-gbq/docs/install.rst
  • Provided installation instructions for the new package.
• packages/pandas-gbq/docs/intro.rst
  • Introduced the pandas-gbq package in its new location.
• packages/pandas-gbq/docs/oauth.rst
  • Documented OAuth flow for BigQuery sign-in in the new package's documentation.
• packages/pandas-gbq/docs/privacy.rst
  • Outlined the privacy policy for the new package.
• packages/pandas-gbq/docs/reading.rst
  • Documented how to read tables from BigQuery using the new package.
• packages/pandas-gbq/docs/requirements-docs.txt
  • Listed dependencies for building documentation for the new package.
• packages/pandas-gbq/docs/writing.rst
  • Documented how to write tables to BigQuery using the new package.
• packages/pandas-gbq/noxfile.py
  • Configured Nox sessions for testing and linting in the new package.
• packages/pandas-gbq/pandas_gbq/init.py
  • Initialized the pandas_gbq package in its new location.
• packages/pandas-gbq/pandas_gbq/_versions_helpers.py
  • Provided helper functions for version extraction in the new package.
• packages/pandas-gbq/pandas_gbq/auth.py
  • Implemented authentication logic for the new package.
• packages/pandas-gbq/pandas_gbq/constants.py
  • Defined various constants used in the new package.
• packages/pandas-gbq/pandas_gbq/contexts.py
  • Implemented context management for credentials and project in the new package.
• packages/pandas-gbq/pandas_gbq/core/init.py
  • Initialized the core subpackage within the new package.
• packages/pandas-gbq/pandas_gbq/core/biglake.py
  • Implemented utilities for BigLake tables in the new package.
• packages/pandas-gbq/pandas_gbq/core/pandas.py
  • Provided pandas-related core utilities for the new package.
• packages/pandas-gbq/pandas_gbq/core/read.py
  • Implemented core functionality for reading data in the new package.
• packages/pandas-gbq/pandas_gbq/core/resource_references.py
  • Implemented parsing for BigQuery and BigLake table IDs in the new package.
• packages/pandas-gbq/pandas_gbq/core/sample.py
  • Implemented table sampling functionality in the new package.
• packages/pandas-gbq/pandas_gbq/dry_runs.py
  • Provided utilities for dry run queries in the new package.
• packages/pandas-gbq/pandas_gbq/environment.py
  • Implemented environment detection for VS Code and Jupyter in the new package.
• packages/pandas-gbq/pandas_gbq/exceptions.py
  • Defined custom exception classes for the new package.
• packages/pandas-gbq/pandas_gbq/features.py
  • Implemented feature detection based on dependency versions in the new package.
• packages/pandas-gbq/pandas_gbq/gbq.py
  • Contained the main read_gbq and to_gbq functions for the new package.
• packages/pandas-gbq/pandas_gbq/gbq_connector.py
  • Implemented the BigQuery connector logic for the new package.
• packages/pandas-gbq/pandas_gbq/load/init.py
  • Initialized the load subpackage within the new package.
• packages/pandas-gbq/pandas_gbq/load/core.py
  • Implemented core data loading functionality for the new package.
• packages/pandas-gbq/pandas_gbq/query.py
  • Provided utilities for BigQuery queries in the new package.
• packages/pandas-gbq/pandas_gbq/schema/init.py
  • Initialized the schema subpackage within the new package.
• packages/pandas-gbq/pandas_gbq/schema/bigquery.py
  • Implemented BigQuery schema utilities for the new package.
• packages/pandas-gbq/pandas_gbq/schema/pandas_to_bigquery.py
  • Implemented conversion from pandas DataFrame to BigQuery schema for the new package.
• packages/pandas-gbq/pandas_gbq/schema/pyarrow_to_bigquery.py
  • Implemented conversion from PyArrow types to BigQuery schema for the new package.
• packages/pandas-gbq/pandas_gbq/timestamp.py
  • Provided utilities for handling BigQuery TIMESTAMP types in the new package.
• packages/pandas-gbq/pandas_gbq/version.py
  • Defined the package version for the new package.
• packages/pandas-gbq/release-procedure.md
  • Documented the release process for the new package.
• packages/pandas-gbq/renovate.json
  • Configured Renovate bot for dependency updates in the new package.
• packages/pandas-gbq/requirements-dev.txt
  • Listed development dependencies for the new package.
• packages/pandas-gbq/requirements.txt
  • Listed production dependencies for the new package.
• packages/pandas-gbq/samples/snippets/init.py
  • Initialized the samples snippets package within the new package.
• packages/pandas-gbq/samples/snippets/conftest.py
  • Configured pytest fixtures for samples in the new package.
• packages/pandas-gbq/samples/snippets/noxfile.py
  • Configured Nox sessions for samples in the new package.
• packages/pandas-gbq/samples/snippets/noxfile_config.py
  • Provided Nox configuration overrides for samples in the new package.
• packages/pandas-gbq/samples/snippets/read_gbq_legacy.py
  • Provided a sample for reading BigQuery data using legacy SQL in the new package.
• packages/pandas-gbq/samples/snippets/read_gbq_simple.py
  • Provided a simple sample for reading BigQuery data in the new package.
• packages/pandas-gbq/samples/snippets/read_gbq_test.py
  • Implemented system tests for read_gbq samples in the new package.
• packages/pandas-gbq/samples/snippets/requirements-test.txt
  • Listed test requirements for samples in the new package.
• packages/pandas-gbq/samples/snippets/requirements.txt
  • Listed dependencies for samples in the new package.
• packages/pandas-gbq/samples/snippets/to_gbq_simple.py
  • Provided a simple sample for writing data to BigQuery in the new package.
• packages/pandas-gbq/samples/snippets/to_gbq_test.py
  • Implemented system tests for to_gbq samples in the new package.
• packages/pandas-gbq/scripts/decrypt-secrets.sh
  • Added a script to decrypt secrets for testing in the new package.
• packages/pandas-gbq/scripts/readme-gen/readme_gen.py
  • Implemented a script for generating READMEs from templates in the new package.
• packages/pandas-gbq/scripts/readme-gen/templates/README.tmpl.rst
  • Provided a Jinja2 template for README generation in the new package.
• packages/pandas-gbq/scripts/readme-gen/templates/auth.tmpl.rst
  • Provided an authentication template for READMEs in the new package.
• packages/pandas-gbq/scripts/readme-gen/templates/auth_api_key.tmpl.rst
  • Provided an API key authentication template for READMEs in the new package.
• packages/pandas-gbq/scripts/readme-gen/templates/install_deps.tmpl.rst
  • Provided an install dependencies template for READMEs in the new package.
• packages/pandas-gbq/scripts/readme-gen/templates/install_portaudio.tmpl.rst
  • Provided a PortAudio installation template for READMEs in the new package.
• packages/pandas-gbq/setup.py
  • Configured the package for distribution in its new location.
• packages/pandas-gbq/testing/.gitignore
  • Specified Git ignore rules for testing files in the new package.
• packages/pandas-gbq/testing/constraints-3.9.txt
  • Defined dependency constraints for Python 3.9 testing in the new package.
• packages/pandas-gbq/tests/data/dummy_key.json
  • Provided a dummy JSON key for testing in the new package.
• packages/pandas-gbq/tests/system/init.py
  • Initialized the system tests package within the new package.
• packages/pandas-gbq/tests/system/conftest.py
  • Configured pytest fixtures for system tests in the new package.
• packages/pandas-gbq/tests/system/test_auth.py
  • Implemented system tests for authentication in the new package.
• packages/pandas-gbq/tests/system/test_gbq.py
  • Implemented system tests for the gbq module in the new package.
• packages/pandas-gbq/tests/system/test_read_gbq.py
  • Implemented system tests for read_gbq functionality in the new package.
• packages/pandas-gbq/tests/system/test_read_gbq_with_bqstorage.py
  • Implemented system tests for read_gbq using BigQuery Storage API in the new package.
• packages/pandas-gbq/tests/system/test_sample.py
  • Implemented system tests for table sampling in the new package.
• packages/pandas-gbq/tests/system/test_to_gbq.py
  • Implemented system tests for to_gbq functionality in the new package.
• packages/pandas-gbq/tests/unit/init.py
  • Initialized the unit tests package within the new package.
• packages/pandas-gbq/tests/unit/conftest.py
  • Configured pytest fixtures for unit tests in the new package.
• packages/pandas-gbq/tests/unit/core/init.py
  • Initialized the core unit tests package within the new package.
• packages/pandas-gbq/tests/unit/core/test_biglake.py
  • Implemented unit tests for BigLake utilities in the new package.
• packages/pandas-gbq/tests/unit/core/test_core_resource_references.py
  • Implemented unit tests for resource reference parsing in the new package.
• packages/pandas-gbq/tests/unit/core/test_core_sample.py
  • Implemented unit tests for core sampling logic in the new package.
• packages/pandas-gbq/tests/unit/schema/init.py
  • Initialized the schema unit tests package within the new package.
• packages/pandas-gbq/tests/unit/schema/test_pandas_to_bigquery.py
  • Implemented unit tests for pandas to BigQuery schema conversion in the new package.
• packages/pandas-gbq/tests/unit/schema/test_pyarrow_to_bigquery.py
  • Implemented unit tests for PyArrow to BigQuery schema conversion in the new package.
• packages/pandas-gbq/tests/unit/test_auth.py
  • Implemented unit tests for authentication in the new package.
• packages/pandas-gbq/tests/unit/test_context.py
  • Implemented unit tests for context management in the new package.
• packages/pandas-gbq/tests/unit/test_dry_runs.py
  • Implemented unit tests for dry run utilities in the new package.
• packages/pandas-gbq/tests/unit/test_features.py
  • Implemented unit tests for feature detection in the new package.
• packages/pandas-gbq/tests/unit/test_gbq.py
  • Implemented unit tests for the gbq module in the new package.
• packages/pandas-gbq/tests/unit/test_load.py
  • Implemented unit tests for data loading functionality in the new package.
• packages/pandas-gbq/tests/unit/test_query.py
  • Implemented unit tests for query utilities in the new package.
• packages/pandas-gbq/tests/unit/test_timestamp.py
  • Implemented unit tests for timestamp utilities in the new package.
• packages/pandas-gbq/tests/unit/test_to_gbq.py
  • Implemented unit tests for to_gbq functionality in the new package.
• scripts/split_repo_migration/single-library.git-migrate-history.sh
  • Updated the migration script to ensure proper handling of submodules during cloning and filtering operations by adding an extra --recurse-submodules flag.

Activity

• The pull request description explicitly states to "See Adopt split repo: _python-bigquery-pandas_ #10988", indicating this PR is part of a larger, tracked migration effort.
• The author has provided specific merge instructions: "This PR should be merged with a merge-commit, not a squash-commit, in order to preserve the git history." This highlights the importance of maintaining a clear and complete commit history during this significant migration.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request migrates the pandas-gbq package into this monorepo. The changes consist of adding the package source code, tests, documentation, and various configuration files for CI/CD and development. My review focuses on the correctness and consistency of these newly added files.

I've found a few minor issues:

• An invalid .readthedocs.yml file due to a duplicate key.
• Some outdated comments in CI scripts and documentation referring to unsupported Python versions.

Overall, this is a large but well-structured migration. Addressing these small issues will improve the quality and maintainability of the new package within the repository.

_{Note: Security Review did not run due to the size of the PR.}

I am having trouble creating individual review comments. Click here to see my feedback.

packages/pandas-gbq/.readthedocs.yml (22)

This file contains a duplicate version: 2 key at the top level (also on line 9), which makes it invalid YAML. Please remove this duplicate key to ensure the file is parsed correctly by ReadTheDocs.

packages/pandas-gbq/.github/workflows/unittest.yml (9)

This comment mentions that ubuntu-22.04 is used because of Python 3.7, but Python 3.7 is not in the test matrix anymore (the minimum is 3.9). This comment seems outdated and could be removed to avoid confusion.

packages/pandas-gbq/.kokoro/test-samples-impl.sh (36)

This comment mentions Python 3.7 compatibility for pinning virtualenv. Since Python 3.7 is no longer supported (the minimum is 3.9), this comment is outdated and could be removed or updated.

packages/pandas-gbq/CONTRIBUTING.rst (198)

The example command nox -s py-3.8 uses Python 3.8, which is no longer supported according to this document (supported versions start from 3.9). Please update the example to use a supported version, for example py-3.9.

   $ nox -s py-3.9