[Proposal] Refactor project structure

[cvanelteren]

After the CI build is merged (#614 #613), I believe the package structure needs to be addressed. Currently, the package structure obfuscates the build structure which makes contributing as a new developer is challenging. I would therefore propose to refactor the build structure to address these concerns. This would address point 3 raised here. The following structure would work:

/basemap
├── basemap/
├── data/
│   ├── basemap_data/
│   ├── basemap_hires/
├── pyproject.toml
├── build scripts
└── README.md
└── examples

To achieve this, we need to

• update the CI builds with the new structure
• Unify the setup.py and setup.cfg in pyproject.toml for a single point of entry to understand how the package is built
• Move towards implicit namespaces

This restructuring would create a better user experience, allow for easier maintenance, consistent versioning, and simplify the build process.

Happy to hear your thoughts!

[DWesl]

I don't know that a pyproject.toml-only setup is possible quite yet, since setuptools is still working out how to specify C extension modules there, so at least that would still need setup.py. Most of the rest of setup.cfg supports pyproject.toml now.

I also like the src layout to make it a bit harder to make certain kinds of mistakes in development, but CI will catch some of them and the build-from-sdist CI will catch more

[cvanelteren]

I also like the src layout to make it a bit harder to make certain kinds of mistakes in development, but CI will catch some of them and the build-from-sdist CI will catch more

Could you elaborate on this?

[molinav]

The src layout is a safeguard to ensure that the test phase is performed on the installed basemap wheel and not on the basemap folder that you would have in the flat space. Otherwise, if your working directory is the package root, the basemap folder gets preference over the installed wheel when imported in the tests, and it could be that the basemap folder works but the wheel does not (i.e. because the wheels forgets to include something from the package tree), and you would not notice it.

The build-from-sdist is a safeguard to ensure that the wheels can be properly built from the source distribution that is uploaded to PyPI. Building the wheels directly from the package tree does not check that the sdist contains everything needed to build the wheel, and this may result in broken packages for the people that make pip install basemap without a precompiled wheel available (e.g. Cygwin). Indeed, this already occurred in the past (due to my lack of knowledge about it at the time), because the requirements file were not listed in the MANIFEST.in file and they were not included in the sdist.

[molinav]

Hi @cvanelteren. I am about to merge an issue branch that addresses your proposal partially.

I have converted again the root folder as the top folder for the basemap package. The basemap-data and basemap-data-hires packages have been moved to a data folder, and I have made the appropriate changes:

• Excluding data from the basemap package manifest.
• Updating the CI workflows with all the new paths.
• Updating the documentation files and the README to make use of the new paths.

On the other side, the three packages will keep the skeleton based on src folders, instead of the skeleton based on plain layout, because it helps with the maintenance of the project and avoids some subtle errors during development.

For the data packages, I have removed the setup.cfg file, and setup.py is currently the single project configuration file. It can be considered at a later stage if we migrate everything to a pyproject.toml file.

For the basemap package, I have removed the setup.cfg file, but both the setup.py and pyproject.toml files still need to stay, because setup.py stores logic needed to build the package.

[molinav]

The changes are now merged into the develop branch and will go into the 2.0.0 release that I will cut in the next couple of hours.