Skip to content

Add PartitionGroupCreationTime to Compactor VisitMarkerΒ #7205

@anna-tran

Description

@anna-tran

Is your feature request related to a problem? Please describe.
When running multiple compactors with partitioning enabled, the partition group info file version (and so its creation time) may change between runs of different compactors. The blocks cleaner currently cleans up partition plan visit markers if the last visit time is before the plan creation time but this can be inaccurate if the plan is recreated while the visit marker is still heartbeating.

Describe the solution you'd like
In order to validate whether a visit marker is actually pointing to the correct version of the partition group info file
the partition group info file creation time could be added to the visit marker when determining if the visit marker has expired.

Describe alternatives you've considered
Another alternative is to add a creation time to the visit marker and compare that creation time to the partition group creation time (but it would make more sense to compare fields on the partition group than to compare the marker and the group)

Additional context
After adding this creation time to the visit marker, this code in partitioned_group_info.go could be updated like so to allow the blocks cleaner to delete an invalid version of the visit marker.

Current

		} else if visitMarker.VisitTime < p.CreationTime {
			status.DeleteVisitMarker = true
			allPartitionCompleted = false
		} 

Proposed

		} else if (visitMarker.VisitTime < p.CreationTime || visitMarker.PartitionedGroupCreationTime < p.CreationTime) {
			status.DeleteVisitMarker = true
			allPartitionCompleted = false
		} 

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions