Alexeypilyugin/kafka consumer gcp iam auth#23906
Conversation
Add support for Google Cloud Managed Kafka using SASL_SSL with
OAUTHBEARER mechanism, where tokens are GCP IAM access tokens.
This follows the same pattern as the existing AWS MSK IAM support.
Users can configure it with:
sasl_mechanism: OAUTHBEARER
sasl_oauth_token_provider:
method: gcp_cloud_managed_kafka
gcp_credentials_file: /path/to/sa.json # optional, uses ADC if omitted
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ity check, add changelog - Fix AWS branding: "AWS MSK IAM" -> "Amazon MSK IAM" in spec.yaml and conf.yaml.example - Fix GCP description: "Google Cloud Managed Kafka" -> "Google Cloud managed Kafka", "GCP IAM access tokens" -> "Google Cloud IAM access tokens" - Update gcp_credentials_file description to match reviewer suggestions; remove misleading default path example (no default -- application default credentials are used) - Restore SECURE_FIELD_NAMES and check_field_trusted_provider security check in instance.py that was accidentally removed in the initial commit - Add changelog entry for the GCP IAM authentication feature Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Managed Kafka's OAUTHBEARER validator expects a three-part dot-separated token (header.claims.raw_access_token), not the raw google-auth access token. Returning the raw token caused every authentication attempt to fail with "invalid credentials". Reference: https://github.com/googleapis/managedkafka/blob/main/kafka-auth-local-server/kafka_gcp_credentials_server.py Verified end-to-end against a real Managed Kafka cluster in datadog-sandbox (us-central1) with the compute VM service account holding roles/managedkafka.client. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aba6fcfd8a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "exp": expiry.timestamp(), | ||
| "iss": "Google", | ||
| "iat": now.timestamp(), | ||
| "sub": getattr(credentials, "service_account_email", "user"), |
There was a problem hiding this comment.
When ADC resolves to user or workload-identity credentials that do not expose service_account_email, this silently emits sub: "user" even though Managed Kafka validates the token against the actual IAM principal email. The new example explicitly advertises gcloud auth application-default login, but that path will build a syntactically valid token with the wrong subject and the broker rejects authentication; this should either read an explicit principal (for example the Google helper uses GOOGLE_MANAGED_KAFKA_AUTH_PRINCIPAL) or fail fast instead of defaulting to "user".
Useful? React with 👍 / 👎.
Validation Report
Run Passed validations (20)
|
2 participants
What does this PR do?
Motivation
Review checklist (to be filled by reviewers)
qa/requiredif this PR needs QA validation, orqa/skip-qaif it does not. Exactly one of the two is required.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged