Skip to content

Resolve Composer downloads from the cached metadata - #494

Open
simonchrz wants to merge 1 commit into
git-pkgs:mainfrom
simonchrz:perf/composer-cached-download-lookup
Open

simonchrz wants to merge 1 commit into
git-pkgs:mainfrom
simonchrz:perf/composer-cached-download-lookup

Conversation

@simonchrz

Copy link
Copy Markdown
Contributor

Every Composer archive download fetched the package's p2 metadata straight from upstream to find the archive URL (findDownloadURLFromMetadata, a plain HTTPClient.Do), before the archive cache was checked. So a cached archive still cost one or two uncached Packagist requests per package on every composer install, and failed with a 502 while Packagist was down, even though the archive itself was on disk.

With cache_metadata on, handleDownload now first looks the dist up in the metadata the proxy already caches for Composer, through fetchMetadataDocument and under the same cache keys handlePackageMetadata stores the p2 and ~dev documents under (routed packages included). That means:

  • Within metadata_ttl a download needs no upstream request, since Composer has usually just fetched that document through the proxy. When the TTL has passed it revalidates with the ETag and shares the fetch with concurrent callers like any other metadata request.
  • When upstream fails, the stale cached document still resolves the archive, so cached archives stay installable.
  • A version or file the cached documents don't list, e.g. one published after they were cached and named by a lock file made elsewhere, falls back to the existing upstream lookup, with the same error responses as before.

Without cache_metadata nothing changes; the cached step is skipped so a download still makes exactly one upstream metadata request.

The cooldown check and the dev-branch ?sha= / commit handling run on the resolved dist as before. A short note is added to the Metadata Caching section of docs/configuration.md.

Tests

composer_download_lookup_test.go:

  • TestComposerDownloadResolvesFromCachedMetadata: metadata request plus two downloads cost one upstream metadata request (3 before).
  • TestComposerCachedArchiveServedWhileUpstreamDown: a cached archive is served while upstream returns 503 (404 before, as the failed lookup read as "version not found").
  • TestComposerDownloadOfVersionNewerThanCachedMetadata: a version missing from the cached metadata is still resolved upstream.
  • TestComposerDownloadWithoutMetadataCache: one metadata request per download without cache_metadata.

The first two fail on main; the third fails if the upstream fallback is removed. go test -race ./... and golangci-lint pass.

Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.

With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant