Draft
Conversation
a6b8510 to
594b759
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to: #900
This branch contains prep work for
kamu applycommand that would diff the current and desired state of datasets and apply necessary changes.It's an important step to make large pipelines easier to maintain and a part of IaC efforts.
I approached this problem from envisioning the
--dry-runflag as the crux of the problem - a flag that allows you to preview changes.I decided to implement this flag via complete separation of planning and execution stages in use cases.
To prototype this, I started with adding
--dry-runflag forkamu addcommand:AddCommandintoCreateDatasetFromSnapshotUseCase- thus moving the complex dependency-based sorting into the use caseprepare()andapply()methods to the use caseprepare()returns a complete plan of what will be done - you can see generated IDs, keys, content and hashes of every metadata block that will be added to new datasets etckamu add --dry-runsimply dumps this plan as YAML into outputI like this approach, but still have some doubts:
While I really like having a detailed plan outputted for
--dry-run- it shows what "will be done", not "what is different", so we may need a separatekamu difforkamu apply --diffcommand to show the differences between current and desired states (e.g. diff between readmes, or SQL queries, or schemas)Kubernetes API essentially operates on state and diffs - you apply the manifest as the target state of the resource. So their
--dry-runwill show only whether some resource is created or updated.The way I implemented
--dry-runhere essentially shows a state transition plan that will be done within one transaction ... which is much more powerful. But because of Kubernetes async operators model - k8s never knows upfront how operators will act on diffs and can't plan them ahead - it may take multiple operators many steps and a long time to reconcile the current and desired states.So I wonder if we will run into issues with
--dry-runfor more complex state transitions.An alternative approach could be:
--dry-runsimply as a rollback of transaction - i.e. execution prints what it usually prints, but progress is un-done at the very end.