diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 5e9e7354f..83a9d8329 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -7,7 +7,8 @@ exercises: 0 ::::::::::::::::::::::::::::::::::::::: objectives -- Set up a Python virtual environment for our software project using `venv` and `pip`. +- Set up a Python virtual environment for our software project using `pdm`. +- Declare, install and update our project's external dependencies using `pdm`. - Run our software from the command line. :::::::::::::::::::::::::::::::::::::::::::::::::: @@ -72,15 +73,9 @@ and manage our external dependencies. So what exactly are virtual environments, and why use them? -A Python virtual environment helps us create an **isolated working copy** of a software project -that uses a specific version of Python interpreter -together with specific versions of a number of external libraries -installed into that virtual environment. -Python virtual environments are implemented as -directories with a particular structure within software projects, -containing links to specified dependencies -allowing isolation from other software projects on your machine that may require -different versions of Python or external libraries. +A Python virtual environment helps us create an **isolated workspace** for our software project. +This workspace comes with a Python interpreter together with the versions of any external libraries that your project needs (e.g. NumPy or SciPy). +Python virtual environments are implemented as directories with a particular structure, containing links to specified dependencies allowing isolation from other software projects on your machine that may require different versions of Python or external libraries. As more external libraries are added to your Python project over time, you can add them to its specific virtual environment @@ -93,14 +88,14 @@ that they make sharing your code with others much easier Here are some typical scenarios where the use of virtual environments is highly recommended (almost unavoidable): -- You have an older project that only works under Python 2. - You do not have the time to migrate the project to Python 3 +- You have an older project that is pinned to an older version of Python. + You do not have the time to migrate the project to a newer version, or it may not even be possible as some of the third party dependencies - are not available under Python 3. - You have to start another project under Python 3. + have not been updated. + You have to start another project that requires a recent version of Python. The best way to do this on a single machine is to set up two separate Python virtual environments. -- One of your Python 3 projects is locked to use +- One of your projects is locked to use a particular older version of a third party dependency. You cannot use the latest version of the dependency as it breaks things in your project. In a separate branch of your project, @@ -127,89 +122,81 @@ from different virtual environments. ### Managing Python Virtual Environments -There are several commonly used command line tools for managing Python virtual environments: +There are many commonly used command line tools for managing Python virtual environments: - `venv`, available by default from the standard `Python` distribution from `Python 3.3+` -- `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions -- `pipenv`, created to fix certain shortcomings of `virtualenv` +- `virtualenv`, needs to be installed separately but offers more features and supports more Python versions - `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community) +- `pipenv`, created to fix certain shortcomings of `virtualenv` - `poetry`, a modern Python packaging tool which handles virtual environments automatically +- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by Astral +- `pdm`, a modern Python package and dependency manager supporting the latest PEP standards. +The first few of these tools manage *environments* only. +The later ones (`pipenv`, `poetry`, `uv`, `pdm`) are **project managers**: they look after the virtual environment, the packages installed into it, and the metadata describing your project, all through a single command line tool. While there are pros and cons for using each of the above, all will do the job of managing Python virtual environments for you and it may be a matter of personal preference which one you go for. -In this course, we will use `venv` to create and manage our virtual environment -(which is the preferred way for Python 3.3+). -The upside is that `venv` virtual environments created from the command line are -also recognised and picked up automatically by the IDEs we will use in this course, -as we will see in the next episode. +In this course, we will use `pdm` to create and manage our virtual environment because is that it removes a whole class of common mistakes: +you don't have to remember which environment is active, +which `pip` belongs to which Python installation, +or which packages you installed by hand three months ago. -### Managing External Packages +As this comic points out, managing Python and its environments used to be quite complex without these tools. -Part of managing your (virtual) working environment involves -installing, updating and removing external packages on your system. -The Python package manager tool `pip` is most commonly used for this - -it interacts and obtains the packages from the central repository called -[Python Package Index (PyPI)](https://pypi.org/). -`pip` can now be used with all Python distributions (including Anaconda). +![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} ::::::::::::::::::::::::::::::::::::::::: callout ## A Note on Anaconda and `conda` -Anaconda is an open source Python distribution commonly used for scientific programming - -it conveniently installs Python, package and environment management `conda`, -and a number of commonly used scientific computing packages -so you do not have to obtain them separately. +Anaconda is an open source Python distribution commonly used for scientific programming - it conveniently installs Python, package and environment management through `conda`, and a number of commonly used scientific computing packages so you do not have to obtain them separately. +However, recent [licence changes](https://www.datacamp.com/blog/navigating-anaconda-licensing) have made Anaconda less appealing. +There are truly open alternatives like [conda-forge](https://conda-forge.org/). + `conda` is an independent command line tool -(available separately from the Anaconda distribution too) with dual functionality: -(1) it is a package manager that helps you find Python packages -from remote package repositories and install them on your system, and -(2) it is also a virtual environment manager. -So, you can use `conda` for both tasks instead of using `venv` and `pip`. +(available separately from the Anaconda distribution) with dual functionality: +1. It is a package manager that helps you find Python and non-Python packages from remote package repositories and install them on your system, and +2. It is also a virtual environment manager. + So, you can use `conda` for both tasks instead of using `venv` and `pip`. :::::::::::::::::::::::::::::::::::::::::::::::::: -### Many Tools for the Job - -Installing and managing Python distributions, -external libraries and virtual environments is, well, complex. -There is an abundance of tools for each task, -each with its advantages and disadvantages, -and there are different ways to achieve the same effect -(and even different ways to install the same tool!). -Note that each Python distribution comes with its own version of `pip` - -and if you have several Python versions installed you have to be extra careful to -use the correct `pip` to manage external packages for that Python version. - -`venv` and `pip` are considered the *de facto* standards for virtual environment -and package management for Python 3. -However, the advantages of using Anaconda and `conda` are that -you get (most of the) packages needed for scientific code development included with the distribution. -If you are only collaborating with others who are also using Anaconda, -you may find that `conda` satisfies all your needs. -It is good, however, to be aware of all these tools, and use them accordingly. -As you become more familiar with them you will realise that -equivalent tools work in a similar way even though the command syntax may be different -(and that there are equivalent tools for other programming languages too -to which your knowledge can be ported). - -![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} +### Creating a Virtual Environment Using `pdm` -Let us have a look at how we can create and manage virtual environments from the command line -using `venv` and manage packages using `pip`. +Let us have a look at how we can create and manage a virtual environment and its packages from the command line using `pdm`. :::::::::::::::::::::::::::::::::::::::::: prereq -### Making Sure You Can Invoke Python +### Making Sure You Can Invoke PDM and Python + +Install PDM according to its [website instructions](https://pdm-project.org/en/latest/#recommended-installation-method), which at the time of writing is a `curl` command: + +```bash +curl -sSL https://pdm-project.org/install.sh | bash +``` + +You can inspect the Bash script at the URL `curl` is grabbing from if you want to make sure it isn't doing anything sketchy. -You can test your Python installation from the command line with: +Then, test that PDM is available on your `PATH` by executing: ```bash -$ python3 --version # on Mac/Linux -$ python --version # on Windows — Windows installation comes with a python.exe file rather than a python3.exe file +pdm --version +``` + +```output +PDM, version 2.28.2 +``` + +If this fails, revisit the [setup instructions](../learners/setup.md) for this course. + +PDM can manage Python interpreters for you, +but it is good to check that you also have a system Python available: + +```bash +python --version ``` If you are using Windows and invoking `python` command causes your Git Bash terminal to hang with no error message or output, you may @@ -218,410 +205,329 @@ need to create an alias for the python executable `python.exe`, as explained in :::::::::::::::::::::::::::::::::::::::::::::::::: -### Creating Virtual Environments Using `venv` +Our project already contains a `pyproject.toml` file, the standard file that describes a Python project, its metadata and its dependencies. +Because of this, we do not need to create the project from scratch; we can ask PDM to set everything up for us with a single command. -Creating a virtual environment with `venv` is done by executing the following command: +First, ensure you are within the project root directory, then: ```bash -$ python3 -m venv /path/to/new/virtual/environment +pdm install +``` + +```output +WARNING: Lockfile does not exist +Updating the lock file... +WARNING: Project requires a python version of >=3.10, The virtualenv is being created for you as it cannot be matched to the right version. +INFO: python.use_venv is on, creating a virtualenv for this project... +Virtualenv is created successfully at +/home/user/python-intermediate-inflammation/.venv +Changes are written to pdm.lock. + 0:00:00 🔒 Lock successful. +All packages are synced to date, nothing to do. + ✔ Install python-intermediate-inflammation 0.0.0 successful + + 0:00:00 🎉 All complete! 0/0 ``` -where `/path/to/new/virtual/environment` is a path to a directory where you want to place it - -conventionally within your software project so they are co-located. -This will create the target directory for the virtual environment -(and any parent directories that don't exist already). +Your output will look a little different depending on the Python version and paths on your machine. +That one command did several things for us: -::::::::::::::::::::::::::::::::::::::::: callout +1. it created a virtual environment for the project in the `.venv` directory, +2. it created a **lock file** called `pdm.lock`, which records the exact version of every package the environment should contain, and +3. it installed our own project into that environment (more on this below). + +Our project does not declare any external dependencies yet, so there was not much for PDM to install... yet. -## What is `-m` Flag in `python3` Command? +::::::::::::::::::::::::::::::::::::::::: callout -The Python `-m` flag means "module" and tells the Python interpreter to treat what follows `-m` -as the name of a module and not as a single, executable program with the same name. -Some modules (such as `venv` or `pip`) have main entry points -and the `-m` flag can be used to invoke them on the command line via the `python` command. -The main difference between running such modules as standalone programs -(e.g. executing "venv" by running the `venv` command directly) -versus using `python3 -m` command seems to be that -with latter you are in full control of which Python module will be invoked -(the one that came with your environment's Python interpreter vs. -some other version you may have on your system). -This makes it a more reliable way to set things up correctly -and avoid issues that could prove difficult to trace and debug. +## Choosing a Python Interpreter +By default PDM will pick a suitable Python interpreter from the ones installed on your machine and remember the choice in a file called `.pdm-python`. +You can inspect what it selected with `pdm info`, change it with `pdm use`, and even have PDM download and install a Python version for you with, for example, `pdm python install 3.14`. +This is handy when a project needs a Python version that is not available through your operating system. :::::::::::::::::::::::::::::::::::::::::::::::::: -For our project let us create a virtual environment called "venv". -First, ensure you are within the project root directory, then: +### What PDM Actually Created -```bash -$ python3 -m venv venv -``` - -If you list the contents of the newly created directory "venv", on a Mac or Linux system +The virtual environment PDM built for us is an ordinary Python virtual environment of the kind that `venv` produces; there is nothing magic about it. +If you list the contents of the `.venv` directory, on a Mac or Linux system (slightly different on Windows as explained below) you should see something like: ```bash -$ ls -l venv +$ ls -l .venv ``` ```output -total 8 -drwxr-xr-x 12 alex staff 384 5 Oct 11:47 bin -drwxr-xr-x 2 alex staff 64 5 Oct 11:47 include -drwxr-xr-x 3 alex staff 96 5 Oct 11:47 lib --rw-r--r-- 1 alex staff 90 5 Oct 11:47 pyvenv.cfg +total 20 +drwxrwxr-x 2 user user 4096 Aug 19 18:08 bin/ +-rw-rw-r-- 1 user user 194 Aug 19 18:08 CACHEDIR.TAG +drwxrwxr-x 2 user user 4096 Aug 19 18:08 include/ +drwxrwxr-x 3 user user 4096 Aug 19 18:08 lib/ +-rw-rw-r-- 1 user user 750 Aug 19 18:08 pyvenv.cfg ``` -So, running the `python3 -m venv venv` command created the target directory called "venv" -containing: +So, a virtual environment is a directory containing: -- `pyvenv.cfg` configuration file - with a home key pointing to the Python installation from which the command was run, -- `bin` subdirectory (called `Scripts` on Windows) - containing a symlink of the Python interpreter binary used to create the environment - and the standard Python library, -- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) - to contain its own independent set of installed Python packages isolated from other projects, and +- `pyvenv.cfg` configuration file with a home key pointing to the Python installation it was created from, +- `bin` subdirectory (called `Scripts` on Windows) containing a symlink of the Python interpreter binary used to create the environment and the standard Python library, +- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) to contain its own independent set of installed Python packages isolated from other projects, and - various other configuration and supporting files and subdirectories. +Had we not been using PDM, we could have created exactly this ourselves with `python -m venv .venv` and then installed packages into it by using `pip`. +It is worth knowing this is all a virtual environment is, so that the environment does not feel like a black box. +But from here on, we will let PDM do the work of managing this environment. + +Note that since our software project is being tracked by Git, the newly created `.venv` directory will show up in version control. +We will see how to tell Git to ignore it in one of the subsequent episodes. + ::::::::::::::::::::::::::::::::::::::::: callout -## Naming Virtual Environments - -What is a good name to use for a virtual environment? -Using "venv" or ".venv" as the name for an environment -and storing it within the project's directory seems to be the recommended way - -this way when you come across such a subdirectory within a software project, -by convention you know it contains its virtual environment details. -A slight downside is that all different virtual environments on your machine -then use the same name -and the current one is determined by the context of the path you are currently located in. -A (non-conventional) alternative is -to use your project name for the name of the virtual environment, -with the downside that there is nothing to indicate that such a directory contains a virtual environment. -In our case, we have settled to use the name "venv" instead of ".venv" -since it is not a hidden directory and we want it to be displayed by the command line -when listing directory contents -(the "." in its name that would, by convention, make it hidden). -In the future, you will decide what naming convention works best for you. -Here are some references for each of the naming conventions: +## Naming and Locating Virtual Environments + +Storing the environment inside the project directory and calling it "venv" or ".venv" is the usual convention. +This way when you come across such a subdirectory within a software project, you know it contains its virtual environment details. +PDM uses `.venv` in the project root by default. +You can ask PDM to create additional, named environments elsewhere (e.g. `pdm venv create --name py312 3.12`) and list them with `pdm venv list`, which is useful when you want to test your code against several Python versions. +Here are some references for the naming conventions: - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/dev/virtualenvs/) notes that "venv" is the general convention used globally - [The Python Documentation](https://docs.python.org/3/library/venv.html) indicates that ".venv" is common - ["venv" vs ".venv" discussion](https://discuss.python.org/t/trying-to-come-up-with-a-default-directory-name-for-virtual-environments/3750) - :::::::::::::::::::::::::::::::::::::::::::::::::: -Once you've created a virtual environment, you will need to activate it. +### Running Commands In Your Environment -On Mac or Linux, it is done as: +With PDM you do not normally activate the virtual environment at all. +Instead, you prefix commands with `pdm run` and PDM makes sure they are executed using the project's environment: ```bash -$ source venv/bin/activate -(venv) $ +pdm run python --version ``` -On Windows, recall that we have `Scripts` directory instead of `bin` -and activating a virtual environment is done as: - -```bash -$ source venv/Scripts/activate -(venv) $ +```output +Python 3.14.2 # your version will differ depending on your system ``` -Activating the virtual environment will change your command line's prompt -to show what virtual environment you are currently using -(indicated by its name in round brackets at the start of the prompt), -and modify the environment so that running Python will get you -the particular version of Python configured in your virtual environment. +This works from anywhere inside the project, and it works the same way for you and for your collaborators, regardless of what is or is not activated in their shell. -You can verify you are using your virtual environment's version of Python -by checking the path using the command `which`: +If you would still like an activated shell---for example because you are running many commands in a row---PDM will print the activation command for you to evaluate: ```bash -(venv) $ which python3 +echo $(pdm venv activate) ``` ```output -/home/alex/python-intermediate-inflammation/venv/bin/python3 +# The output will vary depending on your OS and current shell, which is precisely why this command is useful +source /home/user/python-intermediate-inflammation/.venv/bin/activate ``` -When you're done working on your project, you can exit the environment with: +Activating the virtual environment will change your command line's prompt to show what virtual environment you are currently using (indicated by its name in round brackets at the start of the prompt): ```bash -(venv) $ deactivate +eval $(pdm venv activate) ``` -If you have just done the `deactivate`, -ensure you reactivate the environment ready for the next part: - -```bash -$ source venv/bin/activate -(venv) $ +```output +# Your prompt should look something like this: +(python-intermediate-inflammation-3.14) $ ``` -::::::::::::::::::::::::::::::::::::::::: callout - -## Python Within A Virtual Environment - -Within an active virtual environment, -commands `python3` and `python` should both refer to the version of Python 3 -you created the environment with (note you may have multiple Python 3 versions installed). - -However, on some machines with Python 2 installed, -`python` command may still be hardwired to the copy of Python 2 -installed outside of the virtual environment - this can cause errors and confusion. +Materially, this modifies the environment so that running `python` will get you the particular version of Python configured in your virtual environment. +You can verify this by checking the path with the command `which`: -You can always check which version of Python you are using in your virtual environment -with the command `which python` to be absolutely sure. -We continue using `python3` in this material to avoid mistakes, -but the command `python` may work for you as expected. +```bash +which python +``` +```output +/home/user/python-intermediate-inflammation/.venv/bin/python +``` -:::::::::::::::::::::::::::::::::::::::::::::::::: +When you're done working on your project, you can exit the environment with: -Note that, since our software project is being tracked by Git, -the newly created virtual environment will show up in version control - -we will see how to handle it using Git in one of the subsequent episodes. +```bash +deactivate +``` -### Installing External Packages Using `pip` +For the rest of this course we will write commands using `pdm run`, so you do not need to keep an environment activated. -We noticed earlier that our code depends on two *external packages/libraries* - -`numpy` and `matplotlib`. -In order for the code to run on your machine, -you need to install these two dependencies into your virtual environment. +### Adding External Packages Using `pdm add` -To install the latest version of a package with `pip` -you use pip's `install` command and specify the package's name, e.g.: +We noticed earlier that our code depends on two *external packages*: `numpy` and `matplotlib`. +In order for the code to run on your machine, you need to install these two dependencies into your virtual environment. +If you tried to run our script right now, Python would stop with a `ModuleNotFoundError` complaining it cannot find `numpy`. +Go ahead and try it yourself! ```bash -(venv) $ python3 -m pip install numpy -(venv) $ python3 -m pip install matplotlib +pdm run inflammation-analysis.py data/inflammation-01.csv ``` -or like this to install multiple packages at once for short: +With PDM, installing a dependency and declaring that your project requires it are the same action. +You *add* it to the project: ```bash -(venv) $ python3 -m pip install numpy matplotlib +pdm add numpy +pdm add matplotlib ``` -::::::::::::::::::::::::::::::::::::::::: callout +or doing both at once: -## How About `pip3 install ` Command? +```bash +pdm add numpy matplotlib +``` -You may have seen or used the `pip3 install ` command in the past, which is shorter -and perhaps more intuitive than `python3 -m pip install`. However, the -[official Pip documentation](https://pip.pypa.io/en/stable/user_guide/#running-pip) recommends -`python3 -m pip install` and core Python developer Brett Cannon offers a -[more detailed explanation](https://snarky.ca/why-you-should-use-python-m-pip/) -of edge cases when the two commands may produce different results and why `python3 -m pip install` -is recommended. In this material, we will use `python3 -m` whenever we have to invoke a Python -module from command line. +These commands might take a little while to run if there are not binary builds available for your machine. +Behind that one short command, PDM does three things for each package: -:::::::::::::::::::::::::::::::::::::::::::::::::: +1. records the dependency in the `[project]` table of `pyproject.toml`, +2. re-resolves the specific concrete dependencies and updates `pdm.lock`, and +3. installs the package into the project's virtual environment. -If you run the `python3 -m pip install` command on a package that is already installed, -`pip` will notice this and do nothing. +The `pyproject.toml` file will now contain something like: -To install a specific version of a Python package -give the package name followed by `==` and the version number, -e.g. `python3 -m pip install numpy==1.21.1`. +```toml +[project] +dependencies = [ + "numpy>=2.2.3", + "matplotlib>=3.10.0", +] +``` -To specify a minimum version of a Python package, -you can do `python3 -m pip install numpy>=1.20`. +You can quickly see this by running `git diff pyproject.toml`. -To upgrade a package to the latest version, e.g. `python3 -m pip install --upgrade numpy`. +Note that PDM saved a *minimum* version specifier rather than pinning an exact version. +The exact versions actually being used live in `pdm.lock`. +This separation is deliberate: +`pyproject.toml` says what your project is *compatible with*, and `pdm.lock` says what you are *currently using*. +These reflect two different use cases. +If you are providing a *library* that other researchers or developers will use in their code, then the looser constraints in `pyproject.toml` are necessary because they will have other dependencies that also have their own requirements, and a mutually compatible set of dependencies needs to be found. +Stating that your package is only compatible with exact versions of its dependencies makes this dependency resolution very difficult and sometimes impossible. +On the other hand, if someone is directly running your code as an application, then having the strict `pdm.lock` will mean they have a better chance of getting a working version of the code on their system. +A win for reproducibility! -To display information about a particular installed package do: +To ask for a particular version or set your own constraint on a dependency version, give the constraint on the command line (quoting it so that your shell does not interpret the `>` character), e.g.: ```bash -(venv) $ python3 -m pip show numpy +pdm add "numpy==2.2.3" +pdm add "numpy>=2.2" ``` -```output -Name: numpy -Version: 1.26.2 -Summary: Fundamental package for array computing in Python -Home-page: https://numpy.org -Author: Travis E. Oliphant et al. -Author-email: -License: Copyright (c) 2005-2023, NumPy Developers. -All rights reserved. -... -Required-by: contourpy, matplotlib -``` +Respectively, these will: + +- put the dependency requirement `numpy==2.2.3` in `pyproject.toml`, meaning anyone who installs your package will also have this exact version of `numpy` installed +- put the dependency requirement `numpy>=2.2` in `pyproject.toml`, meaning anyone who installs your package will need to have a `numpy` version of *at least* `2.2.0` and above. -To list all packages installed with `pip` (in your current virtual environment): +To see what is installed in your current environment, use the command ```bash -(venv) $ python3 -m pip list +pdm list ``` ```output -Package Version ---------------- ------- -contourpy 1.2.0 -cycler 0.12.1 -fonttools 4.45.0 -kiwisolver 1.4.5 -matplotlib 3.8.2 -numpy 1.26.2 -packaging 23.2 -Pillow 10.1.0 -pip 23.0.1 -pyparsing 3.1.1 -python-dateutil 2.8.2 -setuptools 67.6.1 -six 1.16.0 +╭──────────────────────────────────┬─────────────┬──────────────────────────────────────────────────╮ +│ name │ version │ location │ +├──────────────────────────────────┼─────────────┼──────────────────────────────────────────────────┤ +│ contourpy │ 1.3.2 │ │ +│ cycler │ 0.12.1 │ │ +│ fonttools │ 4.63.0 │ │ +│ kiwisolver │ 1.5.0 │ │ +│ matplotlib │ 3.10.9 │ │ +│ numpy │ 2.2.6 │ │ +│ packaging │ 26.3 │ │ +│ pillow │ 12.3.0 │ │ +│ pyparsing │ 3.3.2 │ │ +│ python-dateutil │ 2.9.0.post0 │ │ +│ python-intermediate-inflammation │ 0.0.0 │ -e /home/user/python-intermediate-inflammation │ +│ six │ 1.17.0 │ │ +╰──────────────────────────────────┴─────────────┴──────────────────────────────────────────────────╯ ``` -To uninstall a package installed in the virtual environment do: `python3 -m pip uninstall `. -You can also supply a list of packages to uninstall at the same time. -### Installing Our Local Project as a Package Using `pip` +::: callout -Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or at the very least it might be useful to treat it like a package. -Said another way, it is usually the case we want a convenient way to call the Python code we are writing from another location, and making this code accessible as a package is the best way to do this. -We will save the details of Python packaging for [a future episode](43-software-release.md), and for the meantime we can use the minimal package setup that our project already comes with, which is contained in the `pyproject.toml` file. -Once again, we can use `pip` to install our local package: +## Where Do the Dependencies Come From? -```bash -python3 -m pip install --editable . -``` +By default, `pdm` obtains packages from the central repository called the [Python Package Index (PyPI)](https://pypi.org/). +This is where most third-party Python packages live, but it is possible to configure `pdm` to get your packages from other repositories. +PyPI does its best to try and ensure the security and trustworthiness of the packages it hosts, but there are always bad actors that slip through. +You should be aware of "name squatting" where a malicious package will use a very similar name or gain ownership of the actual name of a semi-popular package. +If you mistype or are unlucky enough to download before this is discovered, you will inadvertantly download malicious code onto your device. -If the above command fails for you - your `pip` installation is older than version 21.3. -Such older versions of `pip` do not support `pyproject.toml` as the package metadata. -Given these versions of `pip` are now over 4 years old, we strongly recommend that you update `pip` if you can with: +::: -```bash -python3 -m pip install --upgrade pip -``` +### Our Own Project is Installed Too -This is similar syntax to above, with two important differences: +Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or someone else would like a package. +Said another way, it is usually the case we want a convenient way to call the Python code we are writing from another location, and making this code accessible as a package is the best way to do this. +We will save the details of Python packaging for [a future episode](43-software-release.md), +and for the meantime the minimal package setup in our `pyproject.toml` is enough. -1. The `--editable` or `-e` flag indicates that the package we are specifying should be an "editable" install. - An "editable" install is one that allows the package in our environment to change dynamically based on source code locally. - This is very convenient when we are developing the package because we can instantly see changes when we call the code from within our virtual environment, rather than having to install the local package again to get the updates. -2. The argument `'.'` indicates that the package we want to install is located in the current directory. - The `pyproject.toml` file located in this directory then handles the rest. +This is the third thing `pdm install` did for us back near [the beginning of the episode](#creating-a-virtual-environment-using-pdm): it installed our own project into the environment in **editable** mode. +An **editable** install is one that allows the package in our environment to change dynamically based on source code locally. +This is very convenient when we are developing the package because we can instantly see changes when we call the code from within our virtual environment, rather than having to install the local package again to get the updates. +That is why `python-intermediate-inflammation` appears in the `pdm list` output above, and it is indicated as "editable" with the `-e` prefix for its location. +(If you ever want to skip this step and not have your project installed in your environment, `pdm install --no-self` will do so.) +### Sharing Your Environment With Collaborators -If we reissue the `pip list` command we should now see our local package with the name `python-intermediate-inflammation` in the output: +You are collaborating on a project with a team so, naturally, you will want to share your environment with your collaborators so they can easily "clone" your software project with all of its dependencies and everyone can replicate equivalent virtual environments on their machines. +With PDM there is nothing extra to do to achieve this. +The `pdm.lock` generated automatically contains the exact versions of every package (including sub-dependencies) that were resolved when creating the virtual environment for our project. +This file should be committed to version control, and we will get around to doing this using Git in one of the following episodes. +A collaborator then only has to run: -```output -Package Version Editable project location --------------------------------- ----------- ---------------------------------------------------------------------------------------------- -contourpy 1.3.1 -cycler 0.12.1 -exceptiongroup 1.2.2 -fonttools 4.56.0 -iniconfig 2.0.0 -kiwisolver 1.4.8 -matplotlib 3.10.0 -numpy 2.2.3 -packaging 24.2 -pillow 11.1.0 -pip 22.0.2 -pluggy 1.5.0 -pyparsing 3.2.1 -pytest 8.3.4 -python-dateutil 2.9.0.post0 -python-intermediate-inflammation 0.0.0 /path/to/your/project/directory/python-intermediate-inflammation -setuptools 59.6.0 -six 1.17.0 -tomli 2.2.1 +```bash +pdm install ``` -### Exporting/Importing Virtual Environments Using `pip` +and they will get an environment containing exactly the same package versions as yours. +This reproducibility is the main reason lock files exist. +If your collaborator already has a PDM-created virtual environment, then they would instead run `pdm sync` to get any of your updates contained in `pdm.lock`. -You are collaborating on a project with a team so, naturally, -you will want to share your environment with your collaborators -so they can easily 'clone' your software project with all of its dependencies -and everyone can replicate equivalent virtual environments on their machines. -`pip` has a handy way of exporting, saving and sharing virtual environments. +As your project grows you will need to update your environment for a variety of reasons. +For example, one of your project's dependencies has just released a new version, you need an additional package for data analysis, or you have found a better package and no longer need the older one. +Each of these is a single `pdm add`, `pdm remove` or `pdm update` command, and each of them updates `pyproject.toml` and `pdm.lock` for you. +You then just commit those changes and propagate them to your collaborators via your code sharing platform (e.g. GitHub). -To export your active environment use the `python3 -m pip freeze --exclude-editable` command to produce a list of packages installed in the virtual environment. -A common convention is to put this list in a `requirements.txt` file: - -```bash -(venv) $ python3 -m pip freeze --exclude-editable > requirements.txt -(venv) $ cat requirements.txt -``` - -```output -contourpy==1.2.0 -cycler==0.12.1 -fonttools==4.45.0 -kiwisolver==1.4.5 -matplotlib==3.8.2 -numpy==1.26.2 -packaging==23.2 -Pillow==10.1.0 -pyparsing==3.1.1 -python-dateutil==2.8.2 -six==1.16.0 -``` +::::::::::::::::::::::::::::::::::::::::: callout -The first of the above commands will create a `requirements.txt` file in your current directory. -Yours may look a little different, -depending on the version of the packages you have installed, -as well as any differences in the packages that they themselves use. -Also, we need to use the `--exclude-editable` command so that our local package is not included in the output, otherwise pip will try to pull from a specific commit at the time we made the editable install, which is not what we want. +## What About `requirements.txt`? -The `requirements.txt` file can then be committed to a version control system -(we will see how to do this using Git in one of the following episodes) -and get shipped as part of your software and shared with collaborators and/or users. -They can then replicate your environment -and install all the necessary packages from the project root as follows: +You will still meet projects (and tools, and deployment systems) that expect a `requirements.txt` file listing pinned versions. +PDM can generate one from the lock file when you need it: ```bash -(venv) $ python3 -m pip install -r requirements.txt --editable . +pdm export -f requirements --without-hashes -o requirements.txt ``` -As your project grows you may need to update your environment for a variety of reasons. -For example, one of your project's dependencies has just released a new version -(dependency version number update), -you need an additional package for data analysis (adding a new dependency) -or you have found a better package and no longer need the older package -(adding a new and removing an old dependency). -What you need to do in this case -(apart from installing the new and removing the packages that are no longer needed -from your virtual environment) -is update the contents of the `requirements.txt` file accordingly -by re-issuing `pip freeze` command -and propagate the updated `requirements.txt` file to your collaborators -via your code sharing platform (e.g. GitHub). +Treat any such file as a generated artefact; `pyproject.toml` and `pdm.lock` are a better source of truth for your project. + +:::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: testimonial ## Official Documentation -For a full list of options and commands, -consult the [official `venv` documentation](https://docs.python.org/3/library/venv.html) -and the [Installing Python Modules with `pip` guide](https://docs.python.org/3/installing/index.html#installing-index). -Also check out the guide -["Installing packages using `pip` and virtual environments"](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-packages-using-pip-and-virtual-environments). - +For a full list of options and commands, consult the [official PDM documentation](https://pdm-project.org/), in particular the pages on [managing dependencies](https://pdm-project.org/latest/usage/dependency/) and [working with virtual environments](https://pdm-project.org/latest/usage/venv/). +If you want to dig into what is happening underneath, the [`venv` documentation](https://docs.python.org/3/library/venv.html) is a good place to start. :::::::::::::::::::::::::::::::::::::::::::::::::: ## Running Python Scripts From Command Line Congratulations! -Your environment is now activated and set up -to run our `inflammation-analysis.py` script from the command line. +Your environment is now set up to run our `inflammation-analysis.py` script from the command line. -You should already be located in the root of the `python-intermediate-inflammation` directory -(if not, please navigate to it from the command line now). +You should already be located in the root of the `python-intermediate-inflammation` directory (if not, please navigate to it from the command line now). To run the script, type the following command: ```bash -(venv) $ python3 inflammation-analysis.py +pdm run python inflammation-analysis.py ``` ```output @@ -629,24 +535,28 @@ usage: inflammation-analysis.py [-h] infiles [infiles ...] inflammation-analysis.py: error: the following arguments are required: infiles ``` -In the above command, we tell the command line two things: - -1. to find a Python interpreter - (in this case, the one that was configured via the virtual environment), and -2. to use it to run our script `inflammation-analysis.py`, - which resides in the current directory. - As we can see, the Python interpreter ran our script, which threw an error - `inflammation-analysis.py: error: the following arguments are required: infiles`. -It looks like the script expects a list of input files to process, -so this is expected behaviour since we do not supply any. +It looks like the script expects a list of input files to process, so this is expected behaviour since we do not supply any. We should run our code as follows, passing one (or more) data file(s) as input: ```bash -(venv) $ python3 inflammation-analysis.py data/inflammation-01.csv +pdm run python inflammation-analysis.py data/inflammation-01.csv ``` +::: callout + +Some other `pdm` commands you will find useful: + +- `pdm list --tree` shows which package pulled in which, as a dependency tree. +- `pdm show numpy` displays information about a particular package. +- `pdm remove numpy` removes a dependency from `pyproject.toml`, the lock file and the environment - all in one step. +- `pdm outdated` lists packages for which a newer version is available. +- `pdm update` updates your dependencies (within the constraints in `pyproject.toml`) and writes the new versions to `pdm.lock`. + +::: + ## Optional Exercises Have a look at [some optional exercises](17-section1-optional-exercises.md). @@ -655,13 +565,11 @@ Have a look at [some optional exercises](17-section1-optional-exercises.md). :::::::::::::::::::::::::::::::::::::::: keypoints - Virtual environments keep Python versions and dependencies required by different projects separate. -- A virtual environment is itself a directory structure. -- Use `venv` to create and manage Python virtual environments. -- Use `pip` to install and manage Python external (third-party) libraries. -- `pip` allows you to declare all dependencies for a project in a separate file (by convention called `requirements.txt`) which can be shared with collaborators/users and used to replicate a virtual environment. -- Use `python3 -m pip freeze --exclude-editable > requirements.txt` to take snapshot of your project's dependencies. -- Use `python3 -m pip install -r requirements.txt` to replicate someone else's virtual environment on your machine from the `requirements.txt` file. +- A virtual environment is itself a directory structure, of the kind `venv` creates. +- `pdm` creates and manages that virtual environment for you, by convention in `.venv` in the project root. +- Use `pdm install` to set up (or reproduce) a project's environment from `pyproject.toml` and `pdm.lock`. +- Use `pdm add` and `pdm remove` to manage external (third-party) libraries; they are recorded in `pyproject.toml`. +- `pdm.lock` records the exact resolved versions of every dependency - commit it alongside `pyproject.toml` so collaborators get an identical environment. +- Use `pdm run` to run a command inside the project's environment without having to activate it. :::::::::::::::::::::::::::::::::::::::::::::::::: - - diff --git a/learners/setup.md b/learners/setup.md index b76d1a5bd..1346bf501 100644 --- a/learners/setup.md +++ b/learners/setup.md @@ -8,6 +8,7 @@ You will need the following software and accounts setup to be able to follow the - Command line tool (such as Bash, Zsh or Git Bash) - Git version control program +- The `curl` command line utility - GitHub account - Python 3 distribution - Integrated development environment (IDE) - PyCharm or Visual Studio Code (VS Code)