Repository navigation
Conversation
When upstream answered a revalidation with 304, the proxy read the cached document back from storage and then stored it again: a full upload and a SHA-256 over identical bytes, for every popular package once per TTL. A 304 now only moves the cache row's fetched_at on, keeping its digest. The update is conditional on the row still recording the digest of the document that was read, so a concurrent fetch that replaced it is not overwritten; in that case, and for rows written before digests were recorded, the document is stored as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When upstream answers a metadata revalidation with 304,
fetchUpstreamMetadatareads the cached document back from storage, andfetchMetadataFromUpstreamthen hands it tocacheMetadataBlob, which uploads it again and hashes it with SHA-256, for bytes that are already stored under the same digest. For large npm packuments that is several MB of storage reads and writes per package everymetadata_ttl, and on S3/GCS an extra PUT each time.Now a 304 only moves the cache row's
fetched_aton (RefreshMetadataCache), keeping the row's digest, sostoredRewriteand the rewrite cache keep working off it. The update only applies while the row still records the digest of the document that was read. If a concurrent fetch (say, one with another Accept or Accept-Encoding, which isn't coalesced with this one) has replaced the document in the meantime, it isn't overwritten. In that case, and for rows written before digests were recorded, the document is stored as before, and the row gets its digest.The document is still read on a 304, since callers need the body. Avoiding that read too would mean going back through
storedRewriteafter revalidation, which I've left for a separate change.Tests
TestMetadataNotModifiedKeepsStoredDocument: two revalidations answered with 304 serve the document with its original digest, movefetched_aton, and store it once in total (three times before).TestMetadataNotModifiedStoresRowWithoutDigest: a 304 for a row without a digest stores the document and records its digest.TestRefreshMetadataCache: the refresh changes onlyfetched_at, and doesn't apply when the digest differs or the row is missing.The first fails on main, and the database test fails if the digest condition is dropped.
go test -race ./...andgolangci-lintpass.