[Daryna Ishchenko (darynaishchenko)]

Lets RateLimitedMultipleTokenAuthenticator rotate between credentials that log in and refresh their own tokens, not only between static tokens, with the same quota tracking, rotation and sharing across streams that static tokens get today.

Intent

Some APIs give each credential its own rate-limit budget, but the token used on the wire is short-lived: a login exchanges a long-lived secret for a session token that expires and has to be fetched again. Connectors for those APIs want what RateLimitedMultipleTokenAuthenticator already offers static tokens:

• per-credential quota counters
• reconciliation against response headers
• rotation when the server rejects a credential
• bounded exhaustion waits
• one instance shared by every stream

It can't do that today, for two reasons:

• Tokens are fixed strings. tokens is interpolated once at build time, so there is nowhere to plug in a login, and a token that expires mid-sync can't be refreshed.
• State is keyed on the token string. A refreshed token would look like a brand-new credential with an unseeded budget, while the real budget belongs to the credential and carries over the refresh.

So a connector in this position has to write a custom authenticator. A custom authenticator doesn't get the factory's instance cache, so each stream ends up with its own copy and its own counters. It also has to re-implement the quota logic.

This PR adds that capability to the existing component, so such connectors can stay declarative.

Changes

• New credentials: CredentialSet on RateLimitedMultipleTokenAuthenticator, as an alternative to tokens. Exactly one of the two must be set.
  • items evaluates to a list of objects, one per credential, usually an interpolated config field.
  • authenticator (a SessionTokenAuthenticator) is built once per item, with that item visible in interpolation as config['credential']. Existing components interpolate it without any new Jinja variable.
  • key is a stable, non-secret identifier per credential, evaluated as a string, so ids such as 1_000 and 1000 stay distinct. It defaults to the item's position.
• Quota state is kept per credential key. A refreshed token keeps its credential's counters.
  • Requests are signed with the credential's current header value; the login refreshes lazily.
  • Responses are attributed back to the credential through the header values it has issued. The last 3 per credential are kept, so a response to a request sent just before a refresh still lands on the right credential.
  • Each credential's quota-status request is signed by its own authenticator.
• No network and no failure at construction. An empty, missing or malformed list, a duplicate key, or a missing key is reported as a config_error on first use. SelectiveAuthenticator builds every branch up front, so a config that selects another branch is unaffected.
  • The request header is read from the credentials on first use too, so an API-key header session (whose header name comes from its token's request options) doesn't log in while streams are being built.
  • A failed login names the credential key in its error, whether the login was rejected, its response had no token, or it failed during header resolution. So does a failed quota-status request.
• Manifest checks at build time, with no network: credentials is rejected together with header, auth_method or token_delimiter, without an expiration_duration on its authenticator, or with an API key injected anywhere but a header. Without expiration_duration the session would be fetched for every request, and responses could no longer be traced back to their credential.
• Factory cache. Identical definitions share one instance across streams.
  • The cache key identifies credentials by key plus a SHA-256 of each item, so secrets never end up in the key.
  • It is built from the fields the models declare, without $parameters. Stream parameters reach the login only as its $parameters (see below), and the key leaves those out, so ordinary streams share one instance. Every declared field stays in the key, even one whose value happens to equal a parameter.
  • A custom component inside the login is keyed with everything it receives: its declared fields, its extras and its $parameters. Logins that differ only in its settings therefore never share. Such a login gets one instance per stream, because the component receives each stream's parameters.
  • A hash of the config is part of the key, so a login that reads config['api_url'] isn't shared across configs. All streams of a source get the same config.
  • A credential authenticator that interpolates parameters is keyed with them, so streams with different parameters get their own instance.
  • A tokens definition keys exactly as before.
• Stream $parameters don't fill the login's fields. Propagation normally fills any field a nested component leaves empty, so a stream's url would land in the login's empty url and give each stream its own instance and quota counters. Under CredentialSet.authenticator the stream parameters are still passed down as $parameters, so the login can read them, but they fill no fields; parameters declared inside the login still do. This is one new entry, STREAM_INDEPENDENT_FIELDS, in manifest_component_transformer.py; propagation everywhere else is unchanged.
  • The parameters a login component declares itself are recorded under $own_parameters. Dynamic streams are propagated a second time once resolved, and without that record the stream parameters merged in by the first pass would look like the login's own and fill it after all.
  • $parameters set on the CredentialSet itself don't fill the login either; declare them on the authenticator.
  • A warning is logged once when the same credentials (same keys, items, config and quota pools) still end up in more than one instance: a login that interpolates parameters or contains a custom component, or a quota-status request filled from a stream's $parameters. Substream factories share the registry. A credential list that can't be used, such as an unselected SelectiveAuthenticator branch, isn't reported.
  • An items or key path whose parent is missing from the config, such as config['credentials']['accounts'] with no credentials object, is reported on first use too, instead of failing while streams are built. The internal message names the expression and the Jinja error, since it may be a manifest mistake rather than a config one.
• One runtime path for static tokens and credentials. Each static token is a credential keyed by the token itself, signing with a fixed header value, so signing, quota-status requests and response attribution no longer fork. The manifest contract, factory and cache key are unchanged.
  • State stays keyed by the token string, and the token list keeps its configured order and duplicates.
  • A static token's quota-status failure is still re-raised unchanged, so a token never appears in an error.
  • Fix: with auth_method: "", the quota-status request was signed " <token>", which requests rejects as an invalid header. It is now the bare token.
• Static tokens behave exactly as before apart from that fix. All existing tests are untouched and pass, and so do source-github's 22 authenticator unit tests run against this branch.

Example

authenticator:
  type: RateLimitedMultipleTokenAuthenticator
  credentials:
    type: CredentialSet
    items: "{{ config['credentials']['accounts'] }}"
    key: "{{ config['credential']['account_id'] }}"
    authenticator:
      type: SessionTokenAuthenticator
      login_requester:
        type: HttpRequester
        url_base: "{{ config['api_url'] }}"
        path: "/accounts/{{ config['credential']['account_id'] }}/sessions"
        http_method: POST
        authenticator:
          type: JwtAuthenticator
          secret_key: "{{ config['credential']['private_key'] }}"
          algorithm: RS256
          token_duration: 540
          jwt_payload:
            iss: "{{ config['credential']['client_id'] }}"
      session_token_path: ["token"]
      expiration_duration: PT55M
      request_authentication:
        type: Bearer
  quota_status_source:
    type: QuotaStatusSource
    url: "{{ config['api_url'] }}/rate_limit"
  quotas:
    - type: TokenQuota
      name: default
      remaining_path: ["remaining"]
      reset_path: ["reset"]
      remaining_header: X-RateLimit-Remaining
      reset_header: X-RateLimit-Reset

Files

• auth/rate_limited_multiple_token.py:
  • new RotatingCredential, and a private _StaticTokenAuthenticator that runs a static token as a credential
  • credentials / credentials_error / credentials_error_detail constructor arguments
  • credential-aware header values, quota-status clients and response attribution
  • deferred validation
• parsers/model_to_component_factory.py: _resolve_credential_set (evaluates items and keys without building anything), per-credential construction after a cache miss, and the cache-key extension.
• declarative_component_schema.yaml and models/declarative_component_schema.py: the CredentialSet definition, tokens made optional, and a credentials field. The model was edited by hand. Checked against datamodel-codegen==0.26.3 with the flags in bin/generate_component_manifest_files.py: the two changed models have identical pydantic schemas, and only class order and forward-ref style differ. "Exactly one of tokens/credentials" is enforced in the factory rather than as a schema oneOf, because the generator turns a oneOf into two model classes.
• parsers/manifest_component_transformer.py: STREAM_INDEPENDENT_FIELDS, and a parent_field_parameters argument that tracks which parameters may fill fields. It defaults to None, which means all of them, so other callers are unaffected. Inside the boundary, $own_parameters records each component's own parameters for a second propagation.
• unit_tests/.../auth/test_rate_limited_multiple_token_credentials.py: 80 tests.
• unit_tests/.../parsers/test_manifest_component_transformer.py: 6 new tests for the STREAM_INDEPENDENT_FIELDS boundary.

Testing

• Runtime (11 tests):
  • live header signing, with per-credential seeding using each credential's own token
  • a refresh keeps quota state
  • a response from before a refresh is attributed correctly
  • old header values are evicted
  • a 429 rotates onto the next credential, with has_alternative_token returning True
  • a foreign header value is ignored
  • deferred errors: empty list, a factory error, duplicate keys
  • tokens together with credentials is rejected, as are mixed injection headers
• Factory (13 tests):
  • one credential per item, with no network call until first use and each login getting its own item's values
  • numeric-looking keys stay strings; keys default to positions
  • shared instance across streams with different propagated $parameters; distinct instances for different credentials
  • the cache key contains no credential values; a tokens cache key has exactly the same fields as before
  • unusable items (absent, empty, not a list, missing key) build fine and fail on use
  • definitions with both or neither of tokens and credentials are rejected
• Review follow-ups (23 tests). Of these, 9 fail on the previous head f5c2c7a5:
  • an API-key header credential makes no request at build time and signs X-Api-Key on first use
  • mixed injection headers are rejected on first use, not at construction
  • a login that interpolates parameters gets one instance per parameter set, each signed with its own session
  • a field equal to a parameter value stays in the cache key
  • definitions without expiration_duration, with an API key outside a header, or with header/auth_method/token_delimiter are rejected
  • a failed login names the credential
  • the other 14 pin behaviour that already held:
    • instances aren't shared across different secrets or login paths
    • unusable lists fail the same way through __call__, token and get_auth_header()
    • an unchanged token stays attributed past the 3-value window
    • 8 threads × 20 requests are all attributed
    • a real session refresh past expiration_duration (freezegun) keeps the credential's counters
    • the tokens cache key equals the base commit's, value for value
• Second review round (6 new tests, 2 extended). All 8 fail on 40f9acf7:
  • streams whose $parameters carry primary_key and data_field share one instance, for Bearer and API-key sessions, at factory level and end to end
  • credential ids that look like numbers stay distinct
  • login bodies that differ in a field named parameters don't share
  • an API-key 401, and a login with no token in its response, both name the credential
  • the parameters-reference test now fails when the guard is removed
• Third review round (3 tests). All 3 fail on a920c8c2:
  • a custom component's settings in the login are in the cache key
  • a custom component is keyed with the stream parameters it receives
  • logins that read different config values don't share an instance
• Fourth review round (11 tests, 1 extended). 5 fail on ab695b1e:
  • streams whose $parameters carry url and use_cache share one instance, and the login's fields stay unset
  • a missing parent object in items (factory level, and end to end behind SelectiveAuthenticator), and a missing level in key, fail on first use as config errors
  • a second instance for the same credentials logs one warning
  • the other 6 pin behaviour that already held, as mutation tests found unguarded:
    • parameters declared inside the login still fill its fields; components outside the login are still filled
    • streams sharing a login log no warning
    • a static-token quota-status failure is raised unchanged, with no token in it
    • literal items differing only in secrets don't share
    • an empty header is rejected; failed logins keep their failure type
• Fifth review round (7 tests in the credentials file, 6 in the transformer file). 8 fail on 56d0c996:
  • static and dynamic streams whose $parameters carry url and request_parameters share one instance, the login sends no stream request params, and no warning is logged
  • the transformer leaves every field inside the login unfilled, and still fills the login's own $parameters, after a second propagation like a dynamic stream's
  • a split through a parent stream, and one by the quota-status request, are each warned about once
  • an unselected credentials branch is not warned about
  • an items or key expression that can't be evaluated is named in the internal message, with the user message unchanged
  • the other 5 pin behaviour that already held: lists, untyped objects and the cycle exclusion inside the login, components around the login, and the same credentials under different quota pools. Mutating any of the transformer's list, untyped-object or cycle-exclusion branches, or dropping the $own_parameters record, now fails a test.
• Static tokens as credentials (4 tests). 2 fail on 3d927ce8; the other 2 pin behaviour that already held:
  • auth_method: "" signs both the data and the quota-status requests with the bare token (fails on 3d927ce8 with InvalidHeader)
  • static tokens stay keyed by the token, keep order and duplicates, and use header
  • a response to a request another component signed with a static token is still attributed
  • an empty header never makes a lookup error name the token
  • Mutating any merge point (the up-front header registration, the fixed-header lookup, the unchanged re-raise, the trim, remembering issued values) fails a test.
• End to end (2 tests), with a two-stream manifest behind a SelectiveAuthenticator:
  • in credentials mode both streams, whose $parameters include primary_key and data_field, share one authenticator instance, and each credential logs in exactly once for the whole read
  • a config selecting the static-token branch builds and isn't affected by the credentials branch
• Existing test_rate_limited_multiple_token.py (65 tests) and test_selective_authenticator.py pass unchanged.
• Full unit_tests/sources/declarative, unit_tests/sources/streams/http and unit_tests/legacy: 3366 passed, 4 skipped. The 2 local failures (test_read_concurrent_declarative_source with a partition router, test_per_partition_cursor_within_limit) fail the same way on 56d0c996, which passed CI, and the failing case varies between runs.
• The transformer gives identical output before and after on all 540 monorepo manifests, including a second pass over their 14 dynamic stream templates.
• ruff and mypy are clean on the changed files.

Open questions for review

• Naming: credentials / CredentialSet / config['credential']. The config overlay avoids touching every component's interpolation context; a dedicated Jinja variable would read more clearly but has a larger blast radius.
• Scope of authenticator: only SessionTokenAuthenticator in this PR. OAuth (one refresh token per credential) could follow if there's demand.
• Not in scope: routing a request only to credentials that can serve it, for credentials scoped to different resources. Every credential is still assumed interchangeable, as static tokens are today.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@daryna/rate-limited-credential-set#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch daryna/rate-limited-credential-set

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

6 572 tests  +86   6 536 ✅ +86   6m 58s ⏱️ -1s
    1 suites ± 0      36 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit d3a815a. ± Comparison against base commit 39244c7.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

6 575 tests  +86   6 539 ✅ +86   11m 13s ⏱️ + 1m 59s
    1 suites ± 0      36 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit d3a815a. ± Comparison against base commit 39244c7.

♻️ This comment has been updated with latest results.