Repository navigation
Rework error handler mechanism #1549
Description
Currently ErrorHandler's are a bit problematic since they operate on Rows.
It was done this way due to historical reason, Transformers are also operating on Rows.
But with all recently added features we should move towards operating on a single Row.
Once we move following building blocks to operate on Row instead of Rows we can tackle ErrorHandlers.
- Pipelines
- Transformers
The extractors should still yield Rows due to performance reasons.
Moving transformers and Pipelines to operate on Row instead of Rows will also reduce the number of loops, currently each transformer needs to loop over all rows, but once we are dont with this only Pipeline will iterate through rows (as pipeline is the first touch point) and then transformer's are going to just execute their logic without iterating trough rows.
That might bring some performance boost, especially on massive datasets.
Activity
norberttech commentedon Nov 8, 2025
After doing some research about this issue, I still think it's a good idea to make transformers operates on single Row however pipelines should be able to accumulate Rows.
One might ask what's the difference then between Transformer and Scalar Function?
State.
Scalar Functions - they should not have any state. Scalar function should get operate on a single row and never keep any state.
Transformers - role of transformers is to maintain some state when needed. Like for example in LimitTransformer that with each row that passes through it increases it's internal state and compares it with the limit.
Currently the biggest blocker from moving towards row by row processing is WindowFunctionTransfomer which due to his complexity should become a pipeline just like HashJoinPipeline, SortingPipeline or GroupByPipeline
norberttech commentedon Nov 8, 2025
#1970 was pretty much the only blocker to move towards row by row processing transformers, there should be no more roadblocks.
norberttech commentedon Oct 31, 2026
Closed by #2712