Build the sparse block of from_pandas directly from pandas sparse columns - #542
abelianbee wants to merge 5 commits into
Conversation
| else: | ||
| # SparseMatrix only represents zero fill. A sparse array with | ||
| # any other fill value is logically dense, so store it that way. | ||
| warnings.warn( |
There was a problem hiding this comment.
It might be slightly more elegant to collect all non-sparsified columns first and emit a single warning that lists them instead of possibly dozens of separate warnings.
|
Thanks for the review, all three are in. Batched the warning: the non-zero-fill columns are collected during the column loop and reported in one warning after it, the same way Took the other two suggestions as-is. |
Martin Stancsics (stanmart)
left a comment
Martin Stancsics (stanmart)
left a comment
There was a problem hiding this comment.
Thank you for the follow-up, lgtm! Let's wait for CI but we can merge as soon as it's green.
|
Pushed a fix for the nightly failure. It was the test, not the change: it built its input by multiplying two sparse arrays to get explicitly stored zeros, and newer pandas prunes those, so the setup assertion failed. It now builds the column with The new run needs workflow approval before it can go green. |
|
There is a trivial conflict in the changelog due to merging the other PR, but looks good! |
…umns sps.coo_matrix(df[:, sparse_dfidx]) materialized the whole block densely before converting, so constructing a SparseMatrix from pandas sparse columns cost O(n * k) time and memory on data that is O(nnz). Assemble the CSC from each column's sp_index and sp_values instead, and eliminate the explicit zeros that pandas sparse arrays can store. SparseMatrix only represents a fill value of 0. Sparse columns with any other fill value are logically dense, so they are now stored as dense with a warning instead of being pushed into sparse storage with every row explicitly stored.
Co-authored-by: Martin Stancsics <martin.stancsics@gmail.com>
Co-authored-by: Martin Stancsics <martin.stancsics@gmail.com>
Collect the columns whose sparse dtype has a non-zero fill value during the column loop and warn once after it, matching how ignored_cols is already handled, instead of emitting one warning per column.
The test relied on pandas keeping explicit zeros when multiplying two sparse arrays. Newer pandas prunes them, so the setup assertion failed on the nightly job. Use SparseArray.from_spmatrix, which stores them deterministically, and also assert that the genuine non-zeros survive.
70cad7e to
7ed2c56
Compare
|
Rebased on main, both changelog entries kept. |
2 participants
Follows up on points 2 and 3 of the discussion in #537. xref #378.
from_pandasbuilt its sparse block withsps.coo_matrix(df[:, sparse_dfidx]), which materializes the whole block densely before converting. For pandas sparse columns that is O(n * k) time and memory on data that is O(nnz). This assembles the CSC from each column'ssp_indexandsp_valuesinstead, and callseliminate_zeros()since pandas sparse arrays can hold explicit zeros.from_pandasend to end, median of 5 after a warmup, M1 Max:Both shapes have the same nnz and now take the same time.
On the fill value:
SparseMatrixonly represents a fill of 0, and nothing checked that, so a sparse column with another fill went into sparse storage with every row explicitly stored. Those columns are logically dense and now go to dense storage with a warning. I went with a warning rather than an error because pandas' default fill for a float sparse array is NaN, so.astype("Sparse")on a float column would otherwise start raising. Happy to make it an error if you would rather.Checklist
CHANGELOG.rstentry