Enhance BigQuery and MySQL documentation for clarity and accuracy#148
Enhance BigQuery and MySQL documentation for clarity and accuracy#148ayush-shah merged 2 commits intomainfrom
Conversation
ayush-shah
commented
Mar 25, 2026
ayush-shah
commented
- Updated BigQuery authentication section to clarify the purpose of Billing Project ID and Project ID, including examples for same-project and cross-project setups.
- Improved MySQL lineage and usage instructions, specifying the need for SELECT access on both general and slow logs, with corresponding SQL commands.
- Expanded MySQL authentication options to include Azure auth, detailing required parameters and their usage.
- Revised YAML configuration documentation for MySQL to reflect new authentication options and improved structure.
- Updated Python SDK lineage documentation to streamline entity creation and lineage addition processes, ensuring clarity on using the new SDK methods.
- Updated BigQuery authentication section to clarify the purpose of Billing Project ID and Project ID, including examples for same-project and cross-project setups. - Improved MySQL lineage and usage instructions, specifying the need for SELECT access on both general and slow logs, with corresponding SQL commands. - Expanded MySQL authentication options to include Azure auth, detailing required parameters and their usage. - Revised YAML configuration documentation for MySQL to reflect new authentication options and improved structure. - Updated Python SDK lineage documentation to streamline entity creation and lineage addition processes, ensuring clarity on using the new SDK methods.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
There was a problem hiding this comment.
Pull request overview
This PR updates the documentation across v1.11.x, v1.12.x, and v1.13.x-SNAPSHOT to clarify authentication and lineage/usage requirements for BigQuery and MySQL, and to refresh Python SDK lineage examples to the newer metadata.sdk surface.
Changes:
- BigQuery: clarify the difference between
Project ID(scan targets) vsBilling Project ID(who pays), including same-project and cross-project examples. - MySQL: clarify lineage/usage prerequisites (general vs slow logs) and document Azure (Entra) auth options in both UI docs and YAML examples.
- Python SDK: update lineage docs to use
metadata.sdkhelpers and new lineage APIs (with a note about SQL-based lineage being legacy/CLI-driven).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| v1.13.x-SNAPSHOT/connectors/database/mysql/yaml.mdx | Updates YAML docs for MySQL lineage log permissions and adds Azure auth section/example. |
| v1.13.x-SNAPSHOT/connectors/database/mysql.mdx | Updates MySQL connector docs for general/slow log requirements and Azure auth fields. |
| v1.13.x-SNAPSHOT/connectors/database/bigquery.mdx | Clarifies Billing Project vs Project ID semantics and adds concrete setup examples. |
| v1.13.x-SNAPSHOT/api-reference/sdk/python/ingestion/lineage.mdx | Refreshes Python lineage examples to metadata.sdk APIs (but currently has inconsistencies—see comments). |
| v1.12.x/connectors/database/mysql/yaml.mdx | Same MySQL YAML doc updates as v1.13.x-SNAPSHOT. |
| v1.12.x/connectors/database/mysql.mdx | Same MySQL connector doc updates as v1.13.x-SNAPSHOT. |
| v1.12.x/connectors/database/bigquery.mdx | Same BigQuery clarification updates as v1.13.x-SNAPSHOT. |
| v1.12.x/api-reference/sdk/python/ingestion/lineage.mdx | Same Python lineage doc refresh as v1.13.x-SNAPSHOT (same inconsistencies). |
| v1.11.x/connectors/database/mysql/yaml.mdx | Backports MySQL lineage/usage logging clarification and Azure auth mentions to v1.11.x. |
| v1.11.x/connectors/database/mysql.mdx | Backports MySQL lineage/usage logging clarification and Azure auth mentions to v1.11.x. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| The simple `Lineage.add_lineage(...)` helper covers direct edges. For advanced payloads, construct an `AddLineageRequest` | ||
| and submit it with `Lineage.add_lineage_request(...)`. | ||
|
|
||
| Let's see how to do that and play with the possible combinations. | ||
|
|
||
| First, import the required classes and create a new table: |
There was a problem hiding this comment.
The example code block immediately below this text has a couple of issues that make it fail as written: it references create_schema_entity.ifullyQualifiedName (typo) and uses metadata.create_or_update(...) even though this page no longer defines a metadata client. Please update that snippet to use create_schema_entity.fullyQualifiedName and the current metadata.sdk entity helpers (e.g., Tables.create(...)).
| securityConfig=OpenMetadataJWTClientConfig( | ||
| jwtToken="<token>" | ||
| ), | ||
| configure( |
There was a problem hiding this comment.
The client setup snippet now only calls configure(...) and no longer creates a metadata/client instance, but later examples in this doc still call metadata.create_or_update(...). This makes the code samples inconsistent and not runnable as written; either reintroduce a client object in the setup step or update the later snippets to use the metadata.sdk entity helpers consistently.
| configure( | |
| metadata = configure( |
| securityConfig=OpenMetadataJWTClientConfig( | ||
| jwtToken="<token>" | ||
| ), | ||
| configure( |
There was a problem hiding this comment.
The client setup snippet now only calls configure(...) and no longer creates a metadata/client instance, but later examples in this doc still call metadata.create_or_update(...). This makes the code samples inconsistent and not runnable as written; either reintroduce a client object in the setup step or update the later snippets to use the metadata.sdk entity helpers consistently.
| configure( | |
| metadata = configure( |
| The simple `Lineage.add_lineage(...)` helper covers direct edges. For advanced payloads, construct an `AddLineageRequest` | ||
| and submit it with `Lineage.add_lineage_request(...)`. | ||
|
|
||
| Let's see how to do that and play with the possible combinations. | ||
|
|
||
| First, import the required classes and create a new table: |
There was a problem hiding this comment.
The example code block immediately below this text has a couple of issues that make it fail as written: it references create_schema_entity.ifullyQualifiedName (typo) and uses metadata.create_or_update(...) even though this page no longer defines a metadata client. Please update that snippet to use create_schema_entity.fullyQualifiedName and the current metadata.sdk entity helpers (e.g., Tables.create(...)).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ```python | ||
| from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import ( | ||
| OpenMetadataConnection, | ||
| ) | ||
| from metadata.generated.schema.security.client.openMetadataJWTClientConfig import ( | ||
| OpenMetadataJWTClientConfig, | ||
| ) | ||
| from metadata.ingestion.ometa.ometa_api import OpenMetadata | ||
| from metadata.sdk import configure | ||
|
|
||
| server_config = OpenMetadataConnection( | ||
| hostPort="http://localhost:8585/api", | ||
| authProvider="openmetadata", | ||
| securityConfig=OpenMetadataJWTClientConfig( | ||
| jwtToken="<token>" | ||
| ), | ||
| configure( | ||
| host="http://localhost:8585/api", | ||
| jwt_token="<token>", | ||
| ) | ||
| metadata = OpenMetadata(server_config) | ||
|
|
||
| assert metadata.health_check() # Will fail if we cannot reach the server | ||
| ``` |
There was a problem hiding this comment.
After switching the client setup to configure(...) (and removing the metadata = OpenMetadata(...) instance), later examples in this page still reference metadata.* methods (e.g., metadata.create_or_update(...)). Either reintroduce the metadata client initialization here or update the rest of the page to use the new metadata.sdk entity helpers consistently so the snippets are copy/paste runnable.
| ```python | ||
| from metadata.sdk import Tables | ||
| from metadata.generated.schema.api.lineage.addLineage import AddLineageRequest | ||
| from metadata.generated.schema.type.entityReference import EntityReference | ||
| from metadata.generated.schema.type.entityLineage import ( | ||
| ColumnLineage, | ||
| EntitiesEdge, |
There was a problem hiding this comment.
In the advanced lineage example that follows this import block, the snippet uses create_schema_entity.ifullyQualifiedName (typo) and metadata.create_or_update(...) when creating table_c. Update it to create_schema_entity.fullyQualifiedName and use the new SDK helper (e.g., Tables.create(...)) to keep the example consistent with the earlier sections and runnable.
| ```python | ||
| from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import ( | ||
| OpenMetadataConnection, | ||
| ) | ||
| from metadata.generated.schema.security.client.openMetadataJWTClientConfig import ( | ||
| OpenMetadataJWTClientConfig, | ||
| ) | ||
| from metadata.ingestion.ometa.ometa_api import OpenMetadata | ||
| from metadata.sdk import configure | ||
|
|
||
| server_config = OpenMetadataConnection( | ||
| hostPort="http://localhost:8585/api", | ||
| authProvider="openmetadata", | ||
| securityConfig=OpenMetadataJWTClientConfig( | ||
| jwtToken="<token>" | ||
| ), | ||
| configure( | ||
| host="http://localhost:8585/api", | ||
| jwt_token="<token>", | ||
| ) | ||
| metadata = OpenMetadata(server_config) | ||
|
|
||
| assert metadata.health_check() # Will fail if we cannot reach the server | ||
| ``` |
There was a problem hiding this comment.
After switching the client setup to configure(...) (and removing the metadata = OpenMetadata(...) instance), later examples in this page still reference metadata.* methods (e.g., metadata.create_or_update(...)). Either reintroduce the metadata client initialization here or update the rest of the page to use the new metadata.sdk entity helpers consistently so the snippets are copy/paste runnable.
| ```python | ||
| from metadata.sdk import Tables | ||
| from metadata.generated.schema.api.lineage.addLineage import AddLineageRequest | ||
| from metadata.generated.schema.type.entityReference import EntityReference | ||
| from metadata.generated.schema.type.entityLineage import ( | ||
| ColumnLineage, | ||
| EntitiesEdge, |
There was a problem hiding this comment.
In the advanced lineage example that follows this import block, the snippet uses create_schema_entity.ifullyQualifiedName (typo) and metadata.create_or_update(...) when creating table_c. Update it to create_schema_entity.fullyQualifiedName and use the new SDK helper (e.g., Tables.create(...)) to keep the example consistent with the earlier sections and runnable.
