BTHofmann2023 additions

[steinnymir]

Bug fixes and utilities added during the beamtime at FLASH in nov2023.

Additions:

• A pretty print of the processor, including some statistical information about the data loaded. Still requires testing for non-flash loaders.
• filter the dataframe before binning to only look at the required columns. This avoids errors when there are issues with values in other columns (i.e. if a column is entirely nans)
• added the option of sequentially loading data in the flash loader. This helps to track errors otherwise lost in the parallel processes.

Bugs:

• added an error catching when trying to bin a column which has only nans. The error handled by numba was otherwise unclear.

[rettigl]

I saw this before in your commits. I don't think this is helpful, as it introduces another graph layer. Did you try whether this really improve computation time? In my tests, it slowed things down. Or why did you introduce this in the first place?

[rettigl]

• filter the dataframe before binning to only look at the required columns. This avoids errors when there are issues with values in other columns (i.e. if a column is entirely nans)

I don't quite understand this issue, and as I have commented, it makes computation of the dataframe, and thus binning, slower. Can you elaborate on this problem? I tried to reproduce by adding a NaN column to the df, which worked without problems...

[steinnymir]

it makes computation of the dataframe, and thus binning, slower.

did you test this? does it really make it slower?

I cannot recall exactly what the problem was, but will get back to it during analysis certainly. The problem was with a column which had dtypes not accepted by numba. I don't know why numba needed to test all dtypes of all columns, but this was a quick fix which solved the problem.

[rettigl]

it makes computation of the dataframe, and thus binning, slower.

did you test this? does it really make it slower?

I cannot recall exactly what the problem was, but will get back to it during analysis certainly. The problem was with a column which had dtypes not accepted by numba. I don't know why numba needed to test all dtypes of all columns, but this was a quick fix which solved the problem.

For mpes, I get something like this pretty consistently:

And also binning is notably slower. For Flash dataframes, the difference is smaller, but also there.

Note the added graph layer:

[rettigl]

Tests for the repr methods are failing. This is due to various reasons, the first being that you don't check if the dataframe is loaded, another that you don't check if the columns you request exist (e.g. for non-Flash-loaders).