Test more packages' latest builds in upstream ci

[Sevans711]

Closes #1785

Overview

Adds geoviews (and more other packages, too) to the upstream ci job (via install-upstream.sh). Using the latest geoviews should be sufficient to fix #1785; the upstream ci is failing because it is testing the latest geoviews release (which is not compatible with the latest cartopy; see #1780) when it probably should be testing the latest version of geoviews instead (where a fix has already been merged; see holoviz/geoviews#884).

Closing the original issue only requires adding geoviews to the upstream ci job, but this PR adds more packages there, too. There have recently been other bugs related to the latest versions of packages including breaking changes (see, e.g., #1542, where some cartopy<0.26 plots don't work properly with matplotlib>=3.11). Including more packages in upstream ci may help spot this before the relevant releases actually occur.

Minor sidenote: moved the pip uninstall packages at start of install-upstream.sh into the conda remove command, because they were both being installed by conda. This comes from upstream-dev-ci.yml which uses ci/environment.yml which does not have a pip: ... block; everything there is installed via conda; confirmed by looking at the conda list output from an actual CI Upstream run (e.g.: https://github.com/UXARRAY/uxarray/actions/runs/35949787277/job/107475635076).

(Tiny sidenote: woops, misspelled the branch name! Fixing it means deleting the PR, though, so that typo is here to stay....)

PR Checklist

General

• An issue is created and linked
• Added appropriate labels (if your uxarray repo permissions allow it)
• Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

• There is adequate test coverage of changes from this PR (add new tests if needed)
  • (Upstream CI workflow shows tests pass when run on this branch, see: https://github.com/UXARRAY/uxarray/actions/runs/36005027889/job/107650878616)
• [N/A] If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation and Examples

• [N/A] Docstrings updated with any function changes, and included in all new functions
• [N/A] User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)
• [N/A] If touched any notebook files, cleared the output of all cells before committing
• [N/A] If added new notebook files, put into appropriate directories and referenced in appropriate files

AI Disclosure

AI Usage: GitHub Copilot's inline code suggestions, plus asked claude about how to decide whether packages belong in the conda remove or pip uninstall block.

• I have tested and take responsibility for all AI-generated content in my PR.

[dylannelson]

Hey I tried to follow the order of these commands and may see a potential problem. Breaking some of this file into pieces and testing each line individually can show an error where matplotlib is installed

PackagesNotFoundError: The following packages are missing from the target environment:
  - matplotlib

but I think the reason this doesn't come up during the CI run is because this isn't an error, and just a warning. I think this causes some/all of the rest of the code to not run at all, and fail silently. I think if it's changed to matplotlib-base it works as intended though. Or if you want to test this, I believe if you add set -e to the top of the script, it will result in the error above, and can warn you of future ones.

The reason it seems to work with matplotlib-base is because /ci/environment.yml requires matplotlib-base<3.11 and calling conda remove matplotlib doesn't find matplotlib-base

I think you can reproduce with the set -e or by making the env yourself and trying to run the commands:

conda env create -n upstream-check -f ci/environment.yml
conda activate upstream-check
conda remove -y --force matplotlib

or even the full

conda remove -y --force antimeridian cartopy dask datashader distributed matplotlib holoviews hvplot geoviews pandas pyarrow requests scikit-learn scipy shapely spatialpandas xarray

assuming your terminal is at the location of this branch's environment.yml

I'm also not seeing an issue with uninstalling pooch as part of the conda list. If I create an env and run
conda remove -y --force pooch it seems to work. Though I'm not as sure what you saw while running that warranted the change in the first place, so I'm not as certain about this one being a concern. For me though it does appear on conda list:

pooch                               1.9.0                  pyhd8ed1ab_0             conda-forge

and it works fine uninstalling for me

conda remove -y --force pooch
3 channel Terms of Service accepted

## Package Plan ##

  environment location: ...

  removed specs:
    - pooch


The following packages will be REMOVED:

  pooch-1.9.0-pyhd8ed1ab_0



Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Hoping these aren't just a local quirk/windows machine bug too, if so, then carry on, haha.

[Sevans711]

Good finds, I fixed those two issues (I think) by:

1. swapping the spelling to matplotlib-base and matplotlib-inline instead of just matplotlib, during the conda remove command
2. adding pooch into the conda remove command. I originally put pooch separately because I was mistakenly looking at the second conda list during the job, instead of the first one (which happens right before the install-upstream.sh script runs). It does indeed appear in the first one.

Now trying to debug what appears to be a completely separate issue, occurring while collecting the tests:

__________________ ERROR collecting test/io/test_geopandas.py __________________
ImportError while importing test module '/home/runner/work/uxarray/uxarray/test/io/test_geopandas.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../micromamba/envs/uxarray_build/lib/python3.13/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
test/io/test_geopandas.py:1: in <module>
    import geopandas as gpd
../../../micromamba/envs/uxarray_build/lib/python3.13/site-packages/geopandas/__init__.py:3: in <module>
    from geopandas.geoseries import GeoSeries
../../../micromamba/envs/uxarray_build/lib/python3.13/site-packages/geopandas/geoseries.py:12: in <module>
    import shapely
../../../micromamba/envs/uxarray_build/lib/python3.13/site-packages/shapely/__init__.py:20: in <module>
    from shapely.lib import GEOSException
E   ImportError: libgeos_c.so.1: cannot open shared object file: No such file or directory

This is also occurring on main, so at least it wasn't introduced here. But, it wasn't occurring on main when #1785 (and this PR) were first created. Summarizing latest upstream CI runs here:

• Night of Sept 22 (and previous nights before that): passed.
• Night of Sept 23: failed due to "lon_0" KeyError from geoviews
• Night of Sept 24: failed due to "lon_0" KeyError from geoviews
• Night of Sept 25: failed due to "lon_0" KeyError from geoviews
• Night of Sept 26: failed while collecting tests; ImportError: libgeos_c.so.1: cannot open shared object file
• Night of Sept 27: failed while collecting tests; ImportError: libgeos_c.so.1: cannot open shared object file
• Night of Sept 28 (i.e. last night): failed while collecting tests; ImportError: libgeos_c.so.1: cannot open shared object file

Hoping to get a chance to look into this further later today; will follow up here with any progress.

[Sevans711]

It looks like the libgeos_c.so.1 ImportError has stopped occurring. The last run which had that error was on the night of Sept 30. Since then, upstream CI runs on main have once again been failing due to "lon_0" KeyError from geoviews (e.g. see runs from night of Oct 1 and night of Oct 4), which this PR would fix.

Since things seem to be working now, I ran the upstream CI action on this branch again, and confirmed it seems to be working here now, so I think this PR is once again ready for review!

[rajeeja]

Checked the run you linked (37353199049) — all 20 packages in the conda remove list are found and removed, and the pip step then pulls cartopy 0.26.1.dev, geoviews 1.16.0b0.dev and matplotlib 3.12.0.dev, so it is genuinely testing upstream rather than the releases. That is the fix for #1785, and distributed resolves fine even though it is not in environment.yml directly, since dask brings it.

One thing I would still take from @dylannelson's review: the set -e. The matplotlib vs matplotlib-base bug is fixed, but the reason it was invisible is not. I checked, and conda remove is atomic — one missing name aborts the whole removal and exits 1:

$ conda remove -y --force pooch nonexistentpkg requests
PackagesNotFoundError: The following packages are missing from the target environment:
  - nonexistentpkg
EXIT CODE: 1
$ conda list | grep -E "pooch|requests"
pooch     1.9.0    # still there
requests  2.34.2   # still there

So it is not a warning that skips one package, it skips all of them. Without set -e the script carries on, the pip step reinstalls over whatever is left, and the job goes green while silently testing the conda releases. That is the failure mode this PR exists to fix, and it would come back the next time a package gets renamed or dropped from environment.yml — which is exactly the sort of thing an upstream job runs into.

set -e at the top is a one-liner and makes the whole script fail loudly instead. Approving since this is a clear improvement as it stands, but I would add that before merging.

[Sevans711]

Thank you for taking a look @rajeeja!

Understood. Added set -e at the top of the install-upstream.sh file, and reran the upstream ci job just to make sure it still works (it does): https://github.com/UXARRAY/uxarray/actions/runs/37482442348/job/112333797598

[dylannelson]

What does the addition of matplotlib-inline do in this case? from what I see the matplotlib-base makes sense, but without a seperate install of matplotlib-inline (which I don't see anywhere), it doesn't ever get installed or uninstalled, making this line not do anything. Unless it's getting installed some other way I don't see and what I'm reading about matplotlib-inline online may be wrong? Either way I think this all works as intended, but that line may not be needed, and I may just be confused as to what you're able to see from side of the the env. Approving though as it all looks like it's working as intended now.

[Sevans711]

Thank you for reviewing, I'll merge this into main!

To address your final question:

What does the addition of matplotlib-inline do in this case? from what I see the matplotlib-base makes sense, but without a seperate install of matplotlib-inline (which I don't see anywhere), it doesn't ever get installed or uninstalled, making this line not do anything. Unless it's getting installed some other way I don't see and what I'm reading about matplotlib-inline online may be wrong? Either way I think this all works as intended, but that line may not be needed, and I may just be confused as to what you're able to see from side of the the env. Approving though as it all looks like it's working as intended now.

The matplotlib-inline comes from ci/environment.yml, which provides the packages that get installed during upstream-dev-ci.yml, before running install-upstream.sh. In install-upstream.sh I decided to conda remove both the matplotlib-base and the matplotlib-inline because the later commands pip install matplotlib (from nightly wheels) which corresponds to both the base and inline conda recipes, if I understand correctly.