Skip to content

Conversation

@rburghol
Copy link
Contributor

@rburghol rburghol commented Jan 27, 2026

Hold on to merge while I undo some morning mistakes @PaulDudaRESPEC -- All fixes with respect to the pandas 3.0 transition are now passing with full coverage from python 3.9-12. There was one lingering issue with the transform() function that just needed me to understand it a little more fully to insure that the calculations would be correct. Thus, SOLRAD was the only timeseries that was not identical (in test10), but that resulted in many changes to the PWATER simulation, which then cascaded into differences in HYDR, and other NPS constituents. This is also preliminarily valid for pandas 3.0.0, that is, code the will execute without error under pandas 3.0.0 and and this same code produces identical results to hspf in pandas < 3.0.0. So this is a good step forward IMO.

Under pandas 3.0.0 the test coverage fails -- an error related to pandas, I suspect that there may be deprecated pandas code in RegressTest, and if so, likely it can be resolved, and then we can see if the test coverage is actually valid. That will be tracked in the formal pandas 3.0.0 issue #209

  • Adds a close() method to the custom local HDF5 class
  • Invokes this method a the end of the run() function to insure that the hdf5 is handled cleanly -- will at last eliminate a warning.
  • Closing the hdf5 at the end of the run eliminates the pytest error: ValueError: The file '/opt/model/HSPsquared/tests/test10/HSPFresults/test10.h5' is already opened, but not in read-only mode (as requested). in pandas >= 3.0.0
  • Integrates the changes to utilities.py and main.py from a brief comment to trigger git #207
  • Enhances the changes from a brief comment to trigger git #207 to do all timeseries transformations correctly.
  • Fixes the value comparison problems in python < 3.11
  • This also appears to fix the pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match)
  • Adds a script cmd_regression.py that creates a command prompt friendly tester, as well as leveraging the intermediate products of the RegressTest class for simulation debugging.
  • Updates the class RegressTest in the file tests/convert/regression_base.py to have a silent mode to be a bit more screen friendly (@austinorr this is False by default so no change in previous behavior is made).
  • The model has run successfully under pandas 3.0.0, and I have manually verified the model flow quartiles are identical with hspf. However, this should just be considered a first step in pandas 3.0.0 update.

THIS IS NOT YET Pandas 3.0 safe. The change to pandas3 are substantial. Most specifically, there have been some things removed from classes, such as the to_timedelta() method of the .delta property of a pandas.timerseries.freq, but most importantly, pandas no longer allows Timedelta to have things like Y, or M me valid intervals, since they are non-ambiguous. Thus, I am not super optimistic about the IF-THEN constructs in the utilities.py transform() function continuing to work since it looks at things like the string value of a frequency to see if they have the characters Y and M. The results look OK with test10, but I think we should consider that with more complicated datasets we could definitely encounter timeseries resampling issues. Or maybe not. We can track that in #209

@rburghol rburghol changed the title create new HDF5 close() method and call at the end of run(). Pandas 3.0, create new HDF5 close() method Jan 29, 2026
rburghol and others added 7 commits January 29, 2026 16:38
…ile analysis reveals identical results vs hspf but RegressTest fails in pandas 3.0.0 with pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).
@rburghol rburghol mentioned this pull request Jan 29, 2026
rburghol and others added 9 commits January 30, 2026 07:44
This reverts commit a3a68d4, reversing
changes made to 3b3e56d.
…t. QUantile analysis reveals identical results vs hspf but RegressTest fails in pandas 3.0.0 with pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match)."

This reverts commit 3b3e56d.
…lass' into develop-hdf5"""""

This reverts commit 5d71f32.
@rburghol
Copy link
Contributor Author

Closed due to erroneous merge. Revived as new PR #216

@rburghol rburghol closed this Jan 30, 2026
@rburghol rburghol deleted the develop-hdf5 branch January 30, 2026 13:41
@rburghol rburghol changed the title Pandas 3.0, create new HDF5 close() method Draft Pandas 3.0, create new HDF5 close() method Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant