Skip to content

Default URL cache builder to use max size - #2291

Merged
jnioche merged 1 commit into
mainfrom
2289
Oct 31, 2026
Merged

jnioche merged 1 commit into
mainfrom
2289

Conversation

@jnioche

@jnioche jnioche commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #2289, partial revert of #1854

Default now set to 1M

The size should be set to the worker’s heap at about 200 bytes per entry, assuming an average URL size of 59-characters.
1M URLs in the cache would take ~200MB per instance of StatusUpdaterBolt.

Cheaper entries: the cache could store a 64-bit hash of the URL instead of the String, which takes about a half the memory per entry.

Signed-off-by: Julien Nioche <julien@digitalpebble.com>
@jnioche jnioche changed the title change back to setting max size Default URL cache builder to use max size Oct 31, 2026
@jnioche jnioche added the bug label Oct 31, 2026
@jnioche jnioche added this to the 4.0.0 milestone Oct 31, 2026
@jnioche
jnioche merged commit 5452c1c into main Oct 31, 2026
2 checks passed
@jnioche
jnioche deleted the 2289 branch October 31, 2026 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default cache spec of AbstractStatusUpdaterBolt is unbounded, softValues has no effect

2 participants