Query parquet/csv pins in place via DuckDB views - #232
Draft
cpsievert wants to merge 3 commits into
Draft
Conversation
First use of a pinned table used to pin_read() the data into R and dbWriteTable() a copy into DuckDB. For the formats DuckDB reads natively with built-in table functions (parquet, csv), first use now registers a view over the pin's versioned file instead: no copy, no R round-trip, and queries get column and predicate pushdown. The view references the resolved version's path, so it is as stable as the version itself. The locked-down connection gains an allowed_directories exception scoped to exactly the board's pin root (its cache for remote boards, its directory for folder boards), set before external access is disabled -- pin files become the only files the agent's queries can read. Extension readers (e.g. read_json_auto) stay off the view path because autoload is disabled; json/rds/qs2/arrow pins keep the eager path. A view dangles if its file disappears after registration (a rewrite on a non-versioned board deletes the old version's directory, or the cache is pruned). source_query()'s error-driven retry loop and source_describe()'s sample now re-resolve the latest version and re-register -- view again, or an eager load if the format changed -- before surfacing an error.
- The cache_hit span attribute in prewarm_context() probes the store path with the side-effect-free context_cache_dir() resolver (context_store_path() gains an injectable cache_dir), so recording telemetry cannot create directories or trigger the fallback warning. - Add local_context_cache_state() to snapshot/restore the package-level cache housekeeping state in tests that poke it. - Fix view tests to reach connection/pending state via data_source_state() (sources are R6 since #228). - Document the deliberate cache-before-path field order in board_pin_root().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Board-backed data sources previously loaded every pin eagerly into the in-process DuckDB: first use of a table meant
pin_read()into R followed bydbWriteTable()— a full copy of the data through R's memory for formats DuckDB could have read itself. This PR registers parquet and CSV pins as zero-copy views over the downloaded file in the pins cache instead. Formats DuckDB doesn't read with built-in table functions (RDS, qs2, arrow, uploads) keep the eager path unchanged.What this means in practice
CREATE VIEW(milliseconds) instead of a full read +dbWriteTable()copy.prewarm_sources()), first use is just view registration — there's nothing left to load in-process.The key ideas
Views are as stable as the pin version they point at. A table reflects the pin version resolved at first use, same as the eager path. When the underlying file disappears (a rewrite on a non-versioned board, cache pruning), the next failed query re-resolves the pin's latest version and retries once, transparently; a persistent failure surfaces the original error.
The lockdown relaxes to exactly one directory. The source's DuckDB connection stays locked down (no extensions, no filesystem access), but now allowlists the board's pin root via
allowed_directoriesso views can read pin files — and nothing else. Boards with an unknown layout skip the allowlist and load eagerly as before.The pins cache becomes single-writer. pins has no cache locking, so a background prewarm download racing a first-use
pin_read()could leave a truncated cache entry. Both sides now take an exclusivefilelockper board cache + pin (the new filelock dependency); lock files are left behind deliberately, since unlinking one a waiter holds would break the exclusion.Also in this PR
source_describe()gets the same dangling-view retry assource_query().cache_hitspan attribute now probes the store path without filesystem side effects, cache-state-mutating tests snapshot/restore that state, and the view tests go throughdata_source_state()(sources are R6 since Makedata_source(),semantic_layer(), andcontext_layer()output R6 #228).