Skip to content

feat(nvd): retry 429ed records after conversion is completed#5389

Open
jess-lowe wants to merge 4 commits into
google:masterfrom
jess-lowe:feat/retry-429
Open

feat(nvd): retry 429ed records after conversion is completed#5389
jess-lowe wants to merge 4 commits into
google:masterfrom
jess-lowe:feat/retry-429

Conversation

@jess-lowe
Copy link
Copy Markdown
Contributor

@jess-lowe jess-lowe commented May 18, 2026

Call me hopeful, but we should attempt to convert the 429ed records again.

this maintains a threadsafe list of records that 429ed and then retries them at the end of conversion.

  • Updated RepoTags so that temporary 429 (rate limit / too many requests) errors do not mark a repository as permanently invalid in the repository tags cache (SetInvalid).
  • Updated the worker pool and worker function to collect any CVE records that fail with models.Error (rate limits) into a thread-safe retryList. After the initial processing pass completes, a secondary worker pool is launched to attempt converting those rate-limited records again.

Copy link
Copy Markdown
Contributor

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just do the first point. And rather than not setting it as invalid in the cache, set it as invalid, just with a shorter timeout than the standard 1 week. (Maybe just 1 hour is good enough, just so we are not spamming requests when multiple records relies on 1 429 host)

It'll run again in 12 hours and pick it back up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants