fix: add accepted outcomes consumer#4419
Conversation
Keep only the snuba-outcomes-accepted-consumer addition.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
4a5b931 to
5914cca
Compare
5914cca to
7b19e8d
Compare
| <<: *file_healthcheck_defaults | ||
| snuba-outcomes-accepted-consumer: | ||
| <<: *snuba_defaults | ||
| command: accepted-outcomes-consumer --storage eap_items --consumer-group accepted_outcomes --auto-offset-reset=earliest --max-batch-time-ms 750 --no-strict-offset-reset --accepted-outcomes-topic outcomes-billing --max-poll-interval-ms ${SENTRY_KAFKA_MAX_POLL_INTERVAL_MS:-300000} |
There was a problem hiding this comment.
Is 750 in --max-batch-time-ms 750 correct?
There was a problem hiding this comment.
I do not know the reasoning behind the proposed arguments in the PR I mentioned, which is why I decided to use the default of other consumers in self-hosted. This might need changing, but someone from Sentry would have to pitch in.
There was a problem hiding this comment.
Yeah, some are 1000, I don't know which one is correct here.
There was a problem hiding this comment.
I do not think 750 vs 1000 makes a big difference here, as the PR mentioned 120,000. Which is a lot, in my opinion, those are the types of nitpicks that you adjust when you start digging deeper into the data flow and the load on ClickHouse.
There was a problem hiding this comment.
We always aim to insert at about once per second in all our production clusters but we have usually only one set of consumers inserting into a given cluster.
Now, in self-hosted, you usually insert to the same ClickHouse cluster so maybe setting it to even higher is useful.
Higher means we insert less often into ClickHouse, meaning fewer parts created meaning less work (fewer merges to do) for ClickHouse, at the expense of having to wait a bit longer to see the data.
Back to the self-hosted use case, we have many consumers inserting into the same cluster, usually at low throughput per consumer, so I would probably set them to 30s at least to buffer as much data as possible and insert as little as possible (so less parts created meaning less work for ClickHouse). It means consumers will use more memory (since they buffer more) and data will come every 30s as opposed to every 1s so you'll see it a little later, but the system should be working less.
Make sure the max batch size is tuned accordingly so it hits the max batch time every time.
There was a problem hiding this comment.
Tried this on my instance, I set it to 2000 lol. With 750ms, the number of processed events are 30-90, but when I set it to 2000, it does not really affect memory much on high throughput.
4 participants
Related to: getsentry/snuba#7781
Without this, on v26.7.0 (maybe earlier, as I updated from 26.5.0), the stats & usage page does not show data for logs, application metrics, and dynamic sampling results.
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.