Skip to content

Increase Alloy remote_write max_shards from 3 to 10#274

Merged
timtalbot merged 1 commit into
mainfrom
alloy-max-shards
Apr 24, 2026
Merged

Increase Alloy remote_write max_shards from 3 to 10#274
timtalbot merged 1 commit into
mainfrom
alloy-max-shards

Conversation

@timtalbot
Copy link
Copy Markdown
Contributor

Summary

Increase max_shards from 3 to 10 on both control_room and workload
prometheus.remote_write queue_config blocks.

Context

max_shards=3 (set in #270) was too aggressive — workloads with high series
cardinality can't drain their queue fast enough with only 3 concurrent senders
per Alloy pod, causing repeated "Remote storage resharding" log spam and
delayed metric delivery.

max_shards=10 provides enough throughput for high-cardinality workloads while
still limiting the fan-in blast radius during recovery (10 shards × 8 pods ×
9 clusters = 720 max concurrent connections, down from 3,600 at default 50).

Test plan

  • Verify no "Remote storage resharding" log spam on previously affected workload
  • Confirm push latencies remain sub-second under normal load
  • Verify control room Traefik stays stable under steady-state push traffic

@claude
Copy link
Copy Markdown

claude Bot commented Apr 24, 2026

Claude finished @timtalbot's task —— View job


  • Gather context and review guidelines
  • Review the diff and changed file
  • Submit PR review

Reviewed and submitted. Both remote_write blocks are updated consistently, the concurrency math checks out, and there are no concerns.

@timtalbot timtalbot marked this pull request as ready for review April 24, 2026 18:45
@timtalbot timtalbot requested a review from a team as a code owner April 24, 2026 18:45
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Both remote_write blocks (control_room and workload) are updated consistently. The math in the PR description checks out (10 × 8 × 9 = 720 max connections), which is a reasonable middle ground between the previous 3 and the default 50. No concerns.

@timtalbot timtalbot added this pull request to the merge queue Apr 24, 2026
Merged via the queue into main with commit 0017482 Apr 24, 2026
7 checks passed
@timtalbot timtalbot deleted the alloy-max-shards branch April 24, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants