Repository navigation
refactor(flow-php/etl): splittable sources and the etl layer model - #2710
Merged
Merged
Conversation
- splittable file sources, one schema inference contract, a launcher seam on Config - every namespace sits in one layer root (Foundation, Values, Storage, Kernels, Functions, IO, Operators, Planning, Runtime, API), guarded per layer, 0 upward dependencies - adapters follow the same layers (IO\Extractor, IO\Loader, IO\Inference, Values\Schema, Foundation\Exception) - one mechanism per operation: branch vocabulary, FileWriter, one FileCommit protocol, NestedLoader, Launch, BucketingAlgorithmBuilder - correctness: one count semantic, stateful steps never split, limit stop without an exception, readers skip hidden paths - tests mirror the layers, shared contracts run per package, new Architecture documentation page
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2710 +/- ##
============================================
+ Coverage 90.11% 90.40% +0.29%
- Complexity 0 30819 +30819
============================================
Files 2220 2174 -46
Lines 85881 89093 +3212
============================================
+ Hits 77389 80545 +3156
- Misses 8492 8548 +56
🚀 New features to boost your workflow:
|
- mod() refuses a negative modulus on every brick/math version - flow-php-ext formatted with cargo fmt - removed DSL example attributes pointing at the deleted data_frame example, docs and completers regenerated - completer commands take --output-dir, so their tests no longer write tracked assets
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #xxx
Change Log
Added
flow-php/etl- splittable file sources with byte-window splitsflow-php/etl- one schema inference contract with per-file sniffersflow-php/etl- launcher seam on Configflow-php/etl- Architecture documentation pageflow-php/etl-adapter-http- http_stop_when_any_of() and http_stop_when_all_of()Fixed
flow-php/etl- count(ref) skips nulls in group-by, pivot and window functions alikeflow-php/etl- overwrite removes only files Flow wrote and never loses data on a failed commitflow-php/etl- 0-row loads write no file for every formatflow-php/etl- a frame with constraints runs more than onceflow-php/etl-adapter-excel- Ignore no longer overwrites an existing workbookflow-php/filesystem- MemoryFilesystem::mv() moves files and directoriesChanged
flow-php/etl- every namespace sits in one layer root, guarded per layerflow-php/etl- Sink renamed to Branch, file sinks to FileWriter, one FileCommit protocolflow-php/etl- readers skip hidden paths and temporary outputflow-php/etl- the planner holds its settings, Rule::apply() takes a Backendflow-php/etl- GroupBy and Repartition builders merged into BucketingAlgorithmBuilderflow-php/etl-adapter-csv- classes follow the etl layers (IO\Extractor, IO\Loader, IO\Inference)flow-php/etl-adapter-json- classes follow the etl layersflow-php/etl-adapter-parquet- classes follow the etl layers, one writer per laneflow-php/etl-adapter-text- classes follow the etl layersflow-php/etl-adapter-xml- classes follow the etl layersflow-php/etl-adapter-excel- classes follow the etl layers, Excel writes describe themselvesflow-php/etl-adapter-doctrine- classes follow the etl layersflow-php/etl-adapter-postgresql- classes follow the etl layersflow-php/etl-adapter-http- classes follow the etl layers, StopWhen is an interfaceflow-php/etl-adapter-google-sheet- classes follow the etl layersflow-php/etl-adapter-avro- classes follow the etl layersflow-php/etl-adapter-chartjs- classes follow the etl layersflow-php/etl-adapter-logger- classes follow the etl layersflow-php/etl-adapter-seal- classes follow the etl layersflow-php/flow-php-ext- native classes follow the etl layer namespacesflow-php/cli- follows the etl namespace movesRemoved
flow-php/etl- LimitReachedException, replaced by the Limiting contractflow-php/etl- FilteredBranch::withTransformation()flow-php/etl- window frame accumulators, windows reuse aggregate functionsDeprecated
Security