Fix NOT_ENOUGH_SPACE error from CH during monthly clean sites job by apata · Pull Request #6551 · plausible/analytics

apata · 2026-07-27T14:32:06Z

Changes

Changes the job to clean data of sites that have been deleted
- to run sequentially one table partition at a time. See module comment as to why.
- to scan other tables besides events_v2. This is to prevent orphaned data for sites that don't have native events and to ensure consistency between runs
Fixes test to assert count properly

Tests

Automated tests have been added

Changelog

Entry has been added to changelog

Documentation

This change does not need a documentation update

Dark mode

This PR does not change the UI

aerosol · 2026-07-27T17:02:57Z

+        [database, table]
+      )
+
+    Enum.map(rows, fn [partition_id] -> partition_id end)


Suggested change

Enum.map(rows, fn [partition_id] -> partition_id end)

List.flatten(rows)

aerosol · 2026-07-27T17:04:42Z

+  @partition_delete_timeout :timer.minutes(15)

  def perform(_job) do
    deleted_sites = get_deleted_sites_with_clickhouse_data()


Not sure if you like to do that, but we could store the deleted IDs upon deletion and pick them up from there instead of dancing around with mapset and somewhat greedy queries. We could run this exact cleanup for the last time manually and keep reading exactly what we need from a trusted source.

Good idea. These queries are scanning a lot of rows. We will need to discuss how to implement the removal of a site_id from that table: it must not happen before all mutations cleaning the data for the site for all tables for all partitions have run, but it's not obvious to me where we can get that signal.

aerosol · 2026-07-27T17:04:51Z

  ]

-  @settings if Mix.env() in [:test, :ce_test, :e2e_test], do: [mutations_sync: 2], else: []
+  @settings [mutations_sync: 2]


Pretty sure this should be async on prod

It is a bit of a step away from tradition and you're right to point this out. If mutations_sync: 2, we wait for all nodes in the cluster to finish the mutation, which is a bit flimsy if one of the nodes is in trouble at the time.

If mutations_sync: 0, we dump the mutations into a list on CH cluster Keeper, hang up and the job completes with :ok. CH then churns through these as and when able (https://clickhouse.com/docs/reference/statements/alter/index#mutations). The mutations may succeed or they may fail. How do we know that we actually cleaned up that which we intended? There's also the matter of mutation concurrency. With this method, we lose control over it (beyond settings levers), which can lead to mutations using up all the available disk space.

I'm ok with mutations_sync: 0, but in this case we should run another job soon after to check up on these mutations.

If mutations are absolutely necessary, monitor them carefully using the system.mutations table and use KILL MUTATION if a process is stuck or misbehaving. Misusing mutations can lead to degraded performance, excessive storage churn, and potential service instability—so apply them with caution and sparingly.

(from https://clickhouse.com/docs/concepts/best-practices/avoid-mutations)

I think running a single oban job potentially for hours/days is unacceptable design-wise. If you think you can recover from a synchronous timeout or other failure by retrying the job, please make it so each partition prune is a job on its own and oban takes care of scheduling.

The mutations may succeed or they may fail. How do we know that we actually cleaned up that which we intended?

Why do you need to know right away? What is the consequence of one partition not getting pruned because site id 2387 was deleted? On a scale of no one is affected to catastrophic, what is the severity?

Monitoring ClickHouse for its inability to execute mutations consistently is another matter, completely out of scope, so we're talking a one-off event where a partition isn't fully rewritten. What is the big deal?

Monitoring ClickHouse for its inability to execute mutations consistently is another matter, completely out of
scope, so we're talking a one-off event where a partition isn't fully rewritten. What is the big deal?

With the currently deployed method that's using mutations_sync: 0, if the mutation cleaning up events_v2 table for a batch of sites succeeded, but cleaning up other tables failed, when will the data for those sites be cleaned up from the other tables?

Data deletion is a critical part of security and privacy guarantees to users and I look at it with utmost severity, so I don't think monitoring mutations we rely on to ensure this is out of scope. CH docs explicitly advise monitoring mutations. Still, it can be done over our monitoring platform and with alerts, not in app code as I suggested.

I think running a single oban job potentially for hours/days is unacceptable design-wise.

Agreed, converting to draft and will discuss alternatives, separate job per partition per table or similar.

Data deletion is a critical part of security and privacy guarantees to users and I look at it with utmost severity

Care to elaborate how the data we store at rest is not private and how this is important for security? The site ID doesn't exist anymore, there's a bunch of anonymous visits with meaningless identifiers worst case, on a rare event (we have no record of) of a partial mutation failing.

so I don't think monitoring mutations we rely on to ensure this is out of scope

Out of scope of ensuring ingestion is uninterrupted due to deletions happening

The data about website visits is valuable, as evidenced by analytics platforms that offer their service for free in exchange for some access to the data.

This value and meaning persists even if the site_id that it is for doesn't exist any more. In events_v2 we have hostname, page, meta, ...

Failure to clean up this properly means that in the event of a data leak, data of website visitors of sites that have stopped using Plausible would be exposed as well.

aerosol · 2026-07-27T17:06:31Z

-  @settings if Mix.env() in [:test, :ce_test, :e2e_test], do: [mutations_sync: 2], else: []
+  @settings [mutations_sync: 2]
+
+  @partition_delete_timeout :timer.minutes(15)


So are we ok to crash if this takes more than 15 mins? What's the reasoning here?

Important question, we must take fails / timeouts into account when working on another solution.

aerosol · 2026-07-27T17:07:23Z

+      [table, partition_id, deleted_sites_ids],
+      settings: @settings,
+      timeout: @partition_delete_timeout,
+      checkout_retries: 0


Why checkout_retries: 0? Are we effectively rescheduling this partition for next month if the pool is busy?

Valid question! Need to discuss when working on another solution

Right, but why did you write it even? 😅

I appreciate your input and wanted to clarify that it will not be ignored, just that I didn't want to go in depth for a solution that is inadequate for other reasons.

Since timeouts and failures are such an integral part of any new solution, I'd prefer to discuss it with the proposal of the new system at hand.

These comments are a public record of how things are done at Plausible, so I thought it's appropriate to take an extra minute to ack.

Fix NOT_ENOUGH_SPACE error from CH during monthly clean sites job

84ca882

aerosol reviewed Jul 27, 2026

View reviewed changes

apata marked this pull request as draft July 29, 2026 06:52

	Enum.map(rows, fn [partition_id] -> partition_id end)
	List.flatten(rows)

Uh oh!

Uh oh!

Conversation

apata commented Jul 27, 2026

Changes

Tests

Changelog

Documentation

Dark mode

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants