Cachfiles to support .json.gz in addition to .json - #405
Open
benvanwerkhoven wants to merge 1 commit into
Open
benvanwerkhoven wants to merge 1 commit into
benvanwerkhoven wants to merge 1 commit into
Conversation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks to a tip from @stijnh I learned that Python has a built-in gzip module that makes it very easy to support read and writing gzipped files. With some trivial changes, I was able to integrate full support for using gzipped cache files. This saves quite some space. Of course, we could make cache files storage much, much more space efficient using a binary format like HDF5, but that would be a much more involved change and more likely more difficult to make crash resistant. This change barely touches the code and saves disk space substantially.
I've tested it on my system, also with deliberately crashing Kernel Tuner and resuming the tuning session, both with .json and .json.gz. To my surprise this works perfectly also with the gzipped file. Need to test on more systems, but if this performs well on all systems we could merge it.