AWS: Cache LakeFormation vended credentials to prevent rate exceeded errors - #17789
Open
jarosmpost wants to merge 1 commit into
Open
AWS: Cache LakeFormation vended credentials to prevent rate exceeded errors#17789jarosmpost wants to merge 1 commit into
jarosmpost wants to merge 1 commit into
Conversation
…errors LakeFormationCredentialsProvider called GetTemporaryGlueTableCredentials on every resolveCredentials() invocation, which happens once per S3 file open on Spark executors. At high DPU counts (e.g. 50x G.2X workers) this exhausts the LakeFormation API rate limit causing job failures with LakeFormationException: Rate exceeded. Wrap the credential fetch in a CachedSupplier<AwsCredentials> so each executor reuses a credential for its full TTL window. The stale time is set to expiration minus a configurable lead time (default 60 s), matching the pattern already used by VendedCredentialsProvider. Two new AwsProperties constants control the behaviour: - lakeformation.cache.enabled (default true) - disable for debugging - lakeformation.cache.refresh-lead-time-ms (default 60000) - how many ms before expiry the cache is proactively refreshed Validated internally: reduced GetTemporaryGlueTableCredentials call count from 12,281 (failing) to 598 (succeeding) on a 50x G.2X Spark job. Generated-by: claude-sonnet-4.6 (OpenCode)
singhpk234
reviewed
Aug 24, 2026
| .credentialsProvider( | ||
| new LakeFormationCredentialsProvider(lakeFormation(), buildTableArn())) | ||
| new LakeFormationCredentialsProvider( | ||
| lakeFormation(), buildTableArn(), cacheEnabled, cacheRefreshLeadTimeMs)) |
Contributor
There was a problem hiding this comment.
[orthogonal] wow, we take props rather than dynamically inferring like VendedCredProvider ? so if there is more than > 1 what would one do ?
| private final CachedSupplier<AwsCredentials> cache; | ||
|
|
||
| LakeFormationCredentialsProvider(LakeFormationClient lakeFormationClient, String tableArn) { | ||
| LakeFormationCredentialsProvider( |
Contributor
There was a problem hiding this comment.
can we structure this more like VendedCredProvider in terms of cache construction and modularization
https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Iceberg reads LakeFormation-governed tables at high executor counts (e.g. 50× G.2X Spark workers),
LakeFormationCredentialsProvidercallsGetTemporaryGlueTableCredentialsonce perresolveCredentials()invocation. Because this is called on every S3 file open, hundreds of concurrent tasks simultaneously hit the LakeFormation API, causing:This was observed in production: a job that succeeded at 25× G.1X (12,281 API calls) started failing after scaling to 50× G.2X. Reducing executors back below the rate limit was the only workaround.
Solution
Wrap the
GetTemporaryGlueTableCredentialscall in aCachedSupplier<AwsCredentials>(the same AWS SDK mechanism already used byVendedCredentialsProvider) so each executor reuses a credential for its full TTL window instead of fetching on every file open.The
staleTimeis set toexpiration - refreshLeadTimeMs(default 60 s), ensuring credentials are refreshed before they expire.Two new
AwsPropertiesconstants control the behaviour:lakeformation.cache.enabledtruefalseto disable caching (e.g. for debugging)lakeformation.cache.refresh-lead-time-ms60000Caching is on by default — no config change required for existing users. The fix is contained entirely within the existing
LakeFormationCredentialsProviderinner class andLakeFormationAwsClientFactory; no new classes are introduced.Validation
Validated on a 50× G.2X Spark job against a LakeFormation-governed cross-account Iceberg table:
LakeFormationException: Rate exceededChanges
AwsProperties: 2 new constants (LAKE_FORMATION_CACHE_ENABLED,LAKE_FORMATION_CACHE_REFRESH_LEAD_TIME_MS) with defaultsLakeFormationAwsClientFactory: read new properties ininitialize(); pass toLakeFormationCredentialsProviderconstructor; addCachedSuppliercaching path in the inner classTestLakeFormationCredentialsProvider: new test class (5 tests) covering cache hit, cache miss after stale time, and cache-disabled pathAI Disclosure