-
Notifications
You must be signed in to change notification settings - Fork 850
Description
Is your feature request related to a problem? Please describe.
When running multiple compactors with partitioning enabled, the partition group info file version (and so its creation time) may change between runs of different compactors. The blocks cleaner currently cleans up partition plan visit markers if the last visit time is before the plan creation time but this can be inaccurate if the plan is recreated while the visit marker is still heartbeating.
Describe the solution you'd like
In order to validate whether a visit marker is actually pointing to the correct version of the partition group info file
the partition group info file creation time could be added to the visit marker when determining if the visit marker has expired.
Describe alternatives you've considered
Another alternative is to add a creation time to the visit marker and compare that creation time to the partition group creation time (but it would make more sense to compare fields on the partition group than to compare the marker and the group)
Additional context
After adding this creation time to the visit marker, this code in partitioned_group_info.go could be updated like so to allow the blocks cleaner to delete an invalid version of the visit marker.
Current
} else if visitMarker.VisitTime < p.CreationTime {
status.DeleteVisitMarker = true
allPartitionCompleted = false
}
Proposed
} else if (visitMarker.VisitTime < p.CreationTime || visitMarker.PartitionedGroupCreationTime < p.CreationTime) {
status.DeleteVisitMarker = true
allPartitionCompleted = false
}