Add NDJSON support in polars#1431
Merged
skrawcz merged 1 commit intoapache:mainfrom Jan 12, 2026
Merged
Conversation
Contributor
|
@AvinashYerra would you mind rebasing against the latest hamilton main? I should have merged a fix for a bunch of unit tests. |
5dc8d8a to
a6ea097
Compare
Contributor
Author
|
Done @skrawcz, rebased to latest main. Thanks! |
Contributor
|
ugh sorry -- fix here #1437 -- if you wanted to apply it to this PR that would also work... |
Contributor
|
One more rebase please :) |
Contributor
Author
|
Hi @skrawcz , just for confirmation. you want me to rebase to the latest main right? |
Contributor
Author
|
Contributor
|
Strange. You can see that I merged #1437 ? So it's on main. Do you have that commit? |
a6ea097 to
4342d84
Compare
Contributor
Author
|
@skrawcz, It is done now. |
Contributor
Author
|
Hi @skrawcz , hope you had a great weekend. Can you check now whether the PR is mergeable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

This PR implements NDJSON (newline-delimited JSON) reader and writer support for Polars, following the same pattern as the existing JSON reader/writer implementation.
coming from #1197
Changes
Added PolarsNDJSONReader class in hamilton/plugins/polars_post_1_0_0_extensions.py
Added PolarsNDJSONWriter class in hamilton/plugins/polars_post_1_0_0_extensions.py
Supports writing NDJSON files using data.write_ndjson()
Registered both classes in
register_data_loaders()function to make them available through the materialization systemAdded the following tests:
Added example in examples/polars/materialization/my_script.py
How I tested this
Ran the specific NDJSON tests:
Both tests passed
Notes
Checklist