From a664d96e69081a73578d93a62b797cb177b61bf0 Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 23 Jul 2026 17:24:06 +0100 Subject: [PATCH 1/5] Initial changes for using pdm --- episodes/12-virtual-environments.md | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 5e9e7354f..21630ef76 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -72,15 +72,9 @@ and manage our external dependencies. So what exactly are virtual environments, and why use them? -A Python virtual environment helps us create an **isolated working copy** of a software project -that uses a specific version of Python interpreter -together with specific versions of a number of external libraries -installed into that virtual environment. -Python virtual environments are implemented as -directories with a particular structure within software projects, -containing links to specified dependencies -allowing isolation from other software projects on your machine that may require -different versions of Python or external libraries. +A Python virtual environment helps us create an **isolated workspace** for our software project. +This workspace comes with a Python interpreter together with the versions of any external libraries that your project needs (e.g. NumPy or SciPy). +Python virtual environments are implemented as directories with a particular structure, containing links to specified dependencies allowing isolation from other software projects on your machine that may require different versions of Python or external libraries. As more external libraries are added to your Python project over time, you can add them to its specific virtual environment @@ -127,7 +121,7 @@ from different virtual environments. ### Managing Python Virtual Environments -There are several commonly used command line tools for managing Python virtual environments: +There are many commonly used command line tools for managing Python virtual environments: - `venv`, available by default from the standard `Python` distribution from `Python 3.3+` - `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions @@ -135,15 +129,13 @@ There are several commonly used command line tools for managing Python virtual e - `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community) - `poetry`, a modern Python packaging tool which handles virtual environments automatically +- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by OpenAI +- `pdm`, a modern Python package and dependency manager supporting the latest PEP standards. While there are pros and cons for using each of the above, all will do the job of managing Python virtual environments for you and it may be a matter of personal preference which one you go for. -In this course, we will use `venv` to create and manage our virtual environment -(which is the preferred way for Python 3.3+). -The upside is that `venv` virtual environments created from the command line are -also recognised and picked up automatically by the IDEs we will use in this course, -as we will see in the next episode. +In this course, we will use `pdm` to create and manage our virtual environment. ### Managing External Packages From b7ca966a8611aa23cf98b3379d67b364a7ba310d Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Wed, 19 Aug 2026 18:20:45 +0100 Subject: [PATCH 2/5] Update a few more sections to use pdm in environment episode --- episodes/12-virtual-environments.md | 204 ++++++++++++++-------------- learners/setup.md | 1 + 2 files changed, 105 insertions(+), 100 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 21630ef76..e8f0140f3 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -7,7 +7,8 @@ exercises: 0 ::::::::::::::::::::::::::::::::::::::: objectives -- Set up a Python virtual environment for our software project using `venv` and `pip`. +- Set up a Python virtual environment for our software project using `pdm`. +- Declare, install and update our project's external dependencies using `pdm`. - Run our software from the command line. :::::::::::::::::::::::::::::::::::::::::::::::::: @@ -87,14 +88,14 @@ that they make sharing your code with others much easier Here are some typical scenarios where the use of virtual environments is highly recommended (almost unavoidable): -- You have an older project that only works under Python 2. - You do not have the time to migrate the project to Python 3 +- You have an older project that is pinned to an older version of Python. + You do not have the time to migrate the project to a newer version, or it may not even be possible as some of the third party dependencies - are not available under Python 3. - You have to start another project under Python 3. + have not been updated. + You have to start another project that requires a recent version of Python. The best way to do this on a single machine is to set up two separate Python virtual environments. -- One of your Python 3 projects is locked to use +- One of your projects is locked to use a particular older version of a third party dependency. You cannot use the latest version of the dependency as it breaks things in your project. In a separate branch of your project, @@ -124,84 +125,77 @@ from different virtual environments. There are many commonly used command line tools for managing Python virtual environments: - `venv`, available by default from the standard `Python` distribution from `Python 3.3+` -- `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions -- `pipenv`, created to fix certain shortcomings of `virtualenv` +- `virtualenv`, needs to be installed separately but offers more features and supports more Python versions - `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community) +- `pipenv`, created to fix certain shortcomings of `virtualenv` - `poetry`, a modern Python packaging tool which handles virtual environments automatically -- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by OpenAI +- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by Astral - `pdm`, a modern Python package and dependency manager supporting the latest PEP standards. +The first few of these tools manage *environments* only. +The later ones (`pipenv`, `poetry`, `uv`, `pdm`) are **project managers**: they look after the virtual environment, the packages installed into it, and the metadata describing your project, all through a single command line tool. While there are pros and cons for using each of the above, all will do the job of managing Python virtual environments for you and it may be a matter of personal preference which one you go for. -In this course, we will use `pdm` to create and manage our virtual environment. +In this course, we will use `pdm` to create and manage our virtual environment because is that it removes a whole class of common mistakes: +you don't have to remember which environment is active, +which `pip` belongs to which Python installation, +or which packages you installed by hand three months ago. -### Managing External Packages +As this comic points out, managing Python and its environments used to be quite complex without these tools. -Part of managing your (virtual) working environment involves -installing, updating and removing external packages on your system. -The Python package manager tool `pip` is most commonly used for this - -it interacts and obtains the packages from the central repository called -[Python Package Index (PyPI)](https://pypi.org/). -`pip` can now be used with all Python distributions (including Anaconda). +![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} ::::::::::::::::::::::::::::::::::::::::: callout ## A Note on Anaconda and `conda` -Anaconda is an open source Python distribution commonly used for scientific programming - -it conveniently installs Python, package and environment management `conda`, -and a number of commonly used scientific computing packages -so you do not have to obtain them separately. +Anaconda is an open source Python distribution commonly used for scientific programming - it conveniently installs Python, package and environment management through `conda`, and a number of commonly used scientific computing packages so you do not have to obtain them separately. +However, recent [licence changes](https://www.datacamp.com/blog/navigating-anaconda-licensing) have made Anaconda less appealing. +There are truly open alternatives like [conda-forge](https://conda-forge.org/). + `conda` is an independent command line tool -(available separately from the Anaconda distribution too) with dual functionality: -(1) it is a package manager that helps you find Python packages -from remote package repositories and install them on your system, and -(2) it is also a virtual environment manager. -So, you can use `conda` for both tasks instead of using `venv` and `pip`. +(available separately from the Anaconda distribution) with dual functionality: +1. It is a package manager that helps you find Python and non-Python packages from remote package repositories and install them on your system, and +2. It is also a virtual environment manager. + So, you can use `conda` for both tasks instead of using `venv` and `pip`. :::::::::::::::::::::::::::::::::::::::::::::::::: -### Many Tools for the Job - -Installing and managing Python distributions, -external libraries and virtual environments is, well, complex. -There is an abundance of tools for each task, -each with its advantages and disadvantages, -and there are different ways to achieve the same effect -(and even different ways to install the same tool!). -Note that each Python distribution comes with its own version of `pip` - -and if you have several Python versions installed you have to be extra careful to -use the correct `pip` to manage external packages for that Python version. - -`venv` and `pip` are considered the *de facto* standards for virtual environment -and package management for Python 3. -However, the advantages of using Anaconda and `conda` are that -you get (most of the) packages needed for scientific code development included with the distribution. -If you are only collaborating with others who are also using Anaconda, -you may find that `conda` satisfies all your needs. -It is good, however, to be aware of all these tools, and use them accordingly. -As you become more familiar with them you will realise that -equivalent tools work in a similar way even though the command syntax may be different -(and that there are equivalent tools for other programming languages too -to which your knowledge can be ported). +Let us have a look at how we can create and manage a virtual environment +and its packages from the command line using `pdm`. -![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} +:::::::::::::::::::::::::::::::::::::::::: prereq -Let us have a look at how we can create and manage virtual environments from the command line -using `venv` and manage packages using `pip`. +### Making Sure You Can Invoke PDM and Python -:::::::::::::::::::::::::::::::::::::::::: prereq +Install PDM according to its [website instructions](https://pdm-project.org/en/latest/#recommended-installation-method), which at the time of writing is a `curl` command: + +```bash +curl -sSL https://pdm-project.org/install.sh | bash +``` + +You can inspect the Bash script at the URL `curl` is grabbing from if you want to make sure it isn't doing anything sketchy. + +Then, test that PDM is available on your `PATH` by executing: + +```bash +pdm --version +``` + +```output +PDM, version 2.28.2 +``` -### Making Sure You Can Invoke Python +If this fails, revisit the [setup instructions](../learners/setup.md) for this course. -You can test your Python installation from the command line with: +PDM can manage Python interpreters for you, +but it is good to check that you also have a system Python available: ```bash -$ python3 --version # on Mac/Linux -$ python --version # on Windows — Windows installation comes with a python.exe file rather than a python3.exe file +python --version ``` If you are using Windows and invoking `python` command causes your Git Bash terminal to hang with no error message or output, you may @@ -210,73 +204,83 @@ need to create an alias for the python executable `python.exe`, as explained in :::::::::::::::::::::::::::::::::::::::::::::::::: -### Creating Virtual Environments Using `venv` +### Creating a Virtual Environment Using `pdm` -Creating a virtual environment with `venv` is done by executing the following command: +Our project already contains a `pyproject.toml` file, the standard file that describes a Python project, its metadata and its dependencies. +Because of this, we do not need to create the project from scratch; we can ask PDM to set everything up for us with a single command. + +First, ensure you are within the project root directory, then: ```bash -$ python3 -m venv /path/to/new/virtual/environment +pdm install ``` -where `/path/to/new/virtual/environment` is a path to a directory where you want to place it - -conventionally within your software project so they are co-located. -This will create the target directory for the virtual environment -(and any parent directories that don't exist already). +```output +WARNING: Lockfile does not exist +Updating the lock file... +WARNING: Project requires a python version of >=3.10, The virtualenv is being created for you as it cannot be matched to the right version. +INFO: python.use_venv is on, creating a virtualenv for this project... +Virtualenv is created successfully at +/home/user/python-intermediate-inflammation/.venv +Changes are written to pdm.lock. + 0:00:00 🔒 Lock successful. +All packages are synced to date, nothing to do. + ✔ Install python-intermediate-inflammation 0.0.0 successful + + 0:00:00 🎉 All complete! 0/0 +``` -::::::::::::::::::::::::::::::::::::::::: callout +Your output will look a little different depending on the Python version and paths on your machine. +That one command did several things for us: + +1. it created a virtual environment for the project in the `.venv` directory, +2. it created a **lock file** called `pdm.lock`, which records the exact version of every package the environment should contain, and +3. it installed our own project into that environment (more on this below). -## What is `-m` Flag in `python3` Command? +Our project does not declare any external dependencies yet, so there was not much for PDM to install... yet. -The Python `-m` flag means "module" and tells the Python interpreter to treat what follows `-m` -as the name of a module and not as a single, executable program with the same name. -Some modules (such as `venv` or `pip`) have main entry points -and the `-m` flag can be used to invoke them on the command line via the `python` command. -The main difference between running such modules as standalone programs -(e.g. executing "venv" by running the `venv` command directly) -versus using `python3 -m` command seems to be that -with latter you are in full control of which Python module will be invoked -(the one that came with your environment's Python interpreter vs. -some other version you may have on your system). -This makes it a more reliable way to set things up correctly -and avoid issues that could prove difficult to trace and debug. +::::::::::::::::::::::::::::::::::::::::: callout +## Choosing a Python Interpreter -:::::::::::::::::::::::::::::::::::::::::::::::::: +By default PDM will pick a suitable Python interpreter from the ones installed on your machine and remember the choice in a file called `.pdm-python`. +You can inspect what it selected with `pdm info`, change it with `pdm use`, and even have PDM download and install a Python version for you with, for example, `pdm python install 3.14`. +This is handy when a project needs a Python version that is not available through your operating system. -For our project let us create a virtual environment called "venv". -First, ensure you are within the project root directory, then: +:::::::::::::::::::::::::::::::::::::::::::::::::: -```bash -$ python3 -m venv venv -``` +### What PDM Actually Created -If you list the contents of the newly created directory "venv", on a Mac or Linux system +The virtual environment PDM built for us is an ordinary Python virtual environment of the kind that `venv` produces; there is nothing magic about it. +If you list the contents of the `.venv` directory, on a Mac or Linux system (slightly different on Windows as explained below) you should see something like: ```bash -$ ls -l venv +$ ls -l .venv ``` ```output -total 8 -drwxr-xr-x 12 alex staff 384 5 Oct 11:47 bin -drwxr-xr-x 2 alex staff 64 5 Oct 11:47 include -drwxr-xr-x 3 alex staff 96 5 Oct 11:47 lib --rw-r--r-- 1 alex staff 90 5 Oct 11:47 pyvenv.cfg +total 20 +drwxrwxr-x 2 user user 4096 Aug 19 18:08 bin/ +-rw-rw-r-- 1 user user 194 Aug 19 18:08 CACHEDIR.TAG +drwxrwxr-x 2 user user 4096 Aug 19 18:08 include/ +drwxrwxr-x 3 user user 4096 Aug 19 18:08 lib/ +-rw-rw-r-- 1 user user 750 Aug 19 18:08 pyvenv.cfg ``` -So, running the `python3 -m venv venv` command created the target directory called "venv" -containing: +So, a virtual environment is a directory containing: -- `pyvenv.cfg` configuration file - with a home key pointing to the Python installation from which the command was run, -- `bin` subdirectory (called `Scripts` on Windows) - containing a symlink of the Python interpreter binary used to create the environment - and the standard Python library, -- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) - to contain its own independent set of installed Python packages isolated from other projects, and +- `pyvenv.cfg` configuration file with a home key pointing to the Python installation it was created from, +- `bin` subdirectory (called `Scripts` on Windows) containing a symlink of the Python interpreter binary used to create the environment and the standard Python library, +- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) to contain its own independent set of installed Python packages isolated from other projects, and - various other configuration and supporting files and subdirectories. +Had we not been using PDM, we could have created exactly this ourselves with `python -m venv .venv` and then installed packages into it by using `pip`. +It is worth knowing this is all a virtual environment is, so that the environment does not feel like a black box. +But from here on, we will let PDM do the work of managing this environment. + +TODO current position + ::::::::::::::::::::::::::::::::::::::::: callout ## Naming Virtual Environments diff --git a/learners/setup.md b/learners/setup.md index b76d1a5bd..1346bf501 100644 --- a/learners/setup.md +++ b/learners/setup.md @@ -8,6 +8,7 @@ You will need the following software and accounts setup to be able to follow the - Command line tool (such as Bash, Zsh or Git Bash) - Git version control program +- The `curl` command line utility - GitHub account - Python 3 distribution - Integrated development environment (IDE) - PyCharm or Visual Studio Code (VS Code) From cc3ff6eb6c8904be7e84f3e9b14d5c2d9ede4b1b Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 20 Aug 2026 16:17:35 +0100 Subject: [PATCH 3/5] Amend up through new section "Running Command in Your Environment" --- episodes/12-virtual-environments.md | 104 ++++++++++------------------ 1 file changed, 38 insertions(+), 66 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index e8f0140f3..8b0ae71f8 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -279,110 +279,82 @@ Had we not been using PDM, we could have created exactly this ourselves with `py It is worth knowing this is all a virtual environment is, so that the environment does not feel like a black box. But from here on, we will let PDM do the work of managing this environment. -TODO current position +Note that since our software project is being tracked by Git, the newly created `.venv` directory will show up in version control. +We will see how to tell Git to ignore it in one of the subsequent episodes. ::::::::::::::::::::::::::::::::::::::::: callout -## Naming Virtual Environments - -What is a good name to use for a virtual environment? -Using "venv" or ".venv" as the name for an environment -and storing it within the project's directory seems to be the recommended way - -this way when you come across such a subdirectory within a software project, -by convention you know it contains its virtual environment details. -A slight downside is that all different virtual environments on your machine -then use the same name -and the current one is determined by the context of the path you are currently located in. -A (non-conventional) alternative is -to use your project name for the name of the virtual environment, -with the downside that there is nothing to indicate that such a directory contains a virtual environment. -In our case, we have settled to use the name "venv" instead of ".venv" -since it is not a hidden directory and we want it to be displayed by the command line -when listing directory contents -(the "." in its name that would, by convention, make it hidden). -In the future, you will decide what naming convention works best for you. -Here are some references for each of the naming conventions: +## Naming and Locating Virtual Environments + +Storing the environment inside the project directory and calling it "venv" or ".venv" is the usual convention. +This way when you come across such a subdirectory within a software project, you know it contains its virtual environment details. +PDM uses `.venv` in the project root by default. +You can ask PDM to create additional, named environments elsewhere (e.g. `pdm venv create --name py312 3.12`) and list them with `pdm venv list`, which is useful when you want to test your code against several Python versions. +Here are some references for the naming conventions: - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/dev/virtualenvs/) notes that "venv" is the general convention used globally - [The Python Documentation](https://docs.python.org/3/library/venv.html) indicates that ".venv" is common - ["venv" vs ".venv" discussion](https://discuss.python.org/t/trying-to-come-up-with-a-default-directory-name-for-virtual-environments/3750) - :::::::::::::::::::::::::::::::::::::::::::::::::: -Once you've created a virtual environment, you will need to activate it. +### Running Commands In Your Environment -On Mac or Linux, it is done as: +With PDM you do not normally activate the virtual environment at all. +Instead, you prefix commands with `pdm run` and PDM makes sure they are executed using the project's environment: ```bash -$ source venv/bin/activate -(venv) $ +pdm run python --version ``` -On Windows, recall that we have `Scripts` directory instead of `bin` -and activating a virtual environment is done as: - -```bash -$ source venv/Scripts/activate -(venv) $ +```output +Python 3.14.2 # your version will differ depending on your system ``` -Activating the virtual environment will change your command line's prompt -to show what virtual environment you are currently using -(indicated by its name in round brackets at the start of the prompt), -and modify the environment so that running Python will get you -the particular version of Python configured in your virtual environment. +This works from anywhere inside the project, and it works the same way for you and for your collaborators, regardless of what is or is not activated in their shell. -You can verify you are using your virtual environment's version of Python -by checking the path using the command `which`: +If you would still like an activated shell---for example because you are running many commands in a row---PDM will print the activation command for you to evaluate: ```bash -(venv) $ which python3 +echo $(pdm venv activate) ``` ```output -/home/alex/python-intermediate-inflammation/venv/bin/python3 +# The output will vary depending on your OS and current shell, which is precisely why this command is useful +source /home/user/python-intermediate-inflammation/.venv/bin/activate ``` -When you're done working on your project, you can exit the environment with: +Activating the virtual environment will change your command line's prompt to show what virtual environment you are currently using (indicated by its name in round brackets at the start of the prompt): ```bash -(venv) $ deactivate +eval $(pdm venv activate) ``` -If you have just done the `deactivate`, -ensure you reactivate the environment ready for the next part: - -```bash -$ source venv/bin/activate -(venv) $ +```output +# Your prompt should look something like this: +(python-intermediate-inflammation-3.14) $ ``` -::::::::::::::::::::::::::::::::::::::::: callout - -## Python Within A Virtual Environment +Materially, this modifies the environment so that running `python` will get you the particular version of Python configured in your virtual environment. +You can verify this by checking the path with the command `which`: -Within an active virtual environment, -commands `python3` and `python` should both refer to the version of Python 3 -you created the environment with (note you may have multiple Python 3 versions installed). - -However, on some machines with Python 2 installed, -`python` command may still be hardwired to the copy of Python 2 -installed outside of the virtual environment - this can cause errors and confusion. +```bash +which python +``` -You can always check which version of Python you are using in your virtual environment -with the command `which python` to be absolutely sure. -We continue using `python3` in this material to avoid mistakes, -but the command `python` may work for you as expected. +```output +/home/user/python-intermediate-inflammation/.venv/bin/python +``` +When you're done working on your project, you can exit the environment with: -:::::::::::::::::::::::::::::::::::::::::::::::::: +```bash +deactivate +``` -Note that, since our software project is being tracked by Git, -the newly created virtual environment will show up in version control - -we will see how to handle it using Git in one of the subsequent episodes. +For the rest of this course we will write commands using `pdm run`, so you do not need to keep an environment activated. ### Installing External Packages Using `pip` From fb4117d96aa3477f622f77cc0277e2d123d2cdbe Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:11:21 +0100 Subject: [PATCH 4/5] Add section "Add External Packages Using pdm add" --- episodes/12-virtual-environments.md | 154 ++++++++++++++++------------ 1 file changed, 86 insertions(+), 68 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 8b0ae71f8..602ca0a61 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -356,102 +356,120 @@ deactivate For the rest of this course we will write commands using `pdm run`, so you do not need to keep an environment activated. -### Installing External Packages Using `pip` +### Adding External Packages Using `pdm add` -We noticed earlier that our code depends on two *external packages/libraries* - -`numpy` and `matplotlib`. -In order for the code to run on your machine, -you need to install these two dependencies into your virtual environment. - -To install the latest version of a package with `pip` -you use pip's `install` command and specify the package's name, e.g.: +We noticed earlier that our code depends on two *external packages*: `numpy` and `matplotlib`. +In order for the code to run on your machine, you need to install these two dependencies into your virtual environment. +If you tried to run our script right now, Python would stop with a `ModuleNotFoundError` complaining it cannot find `numpy`. +Go ahead and try it yourself! ```bash -(venv) $ python3 -m pip install numpy -(venv) $ python3 -m pip install matplotlib +pdm run inflammation-analysis.py data/inflammation-01.csv ``` -or like this to install multiple packages at once for short: +With PDM, installing a dependency and declaring that your project requires it are the same action. +You *add* it to the project: ```bash -(venv) $ python3 -m pip install numpy matplotlib +$ pdm add numpy +$ pdm add matplotlib ``` -::::::::::::::::::::::::::::::::::::::::: callout +or doing both at once: -## How About `pip3 install ` Command? +```bash +$ pdm add numpy matplotlib +``` -You may have seen or used the `pip3 install ` command in the past, which is shorter -and perhaps more intuitive than `python3 -m pip install`. However, the -[official Pip documentation](https://pip.pypa.io/en/stable/user_guide/#running-pip) recommends -`python3 -m pip install` and core Python developer Brett Cannon offers a -[more detailed explanation](https://snarky.ca/why-you-should-use-python-m-pip/) -of edge cases when the two commands may produce different results and why `python3 -m pip install` -is recommended. In this material, we will use `python3 -m` whenever we have to invoke a Python -module from command line. +These commands might take a little while to run if there are not binary builds available for your machine. +Behind that one short command, PDM does three things for each package: -:::::::::::::::::::::::::::::::::::::::::::::::::: +1. records the dependency in the `[project]` table of `pyproject.toml`, +2. re-resolves the specific concrete dependencies and updates `pdm.lock`, and +3. installs the package into the project's virtual environment. -If you run the `python3 -m pip install` command on a package that is already installed, -`pip` will notice this and do nothing. +The `pyproject.toml` file will now contain something like: -To install a specific version of a Python package -give the package name followed by `==` and the version number, -e.g. `python3 -m pip install numpy==1.21.1`. +```toml +[project] +dependencies = [ + "numpy>=2.2.3", + "matplotlib>=3.10.0", +] +``` -To specify a minimum version of a Python package, -you can do `python3 -m pip install numpy>=1.20`. +You can quickly see this by running `git diff pyproject.toml`. -To upgrade a package to the latest version, e.g. `python3 -m pip install --upgrade numpy`. +Note that PDM saved a *minimum* version specifier rather than pinning an exact version. +The exact versions actually being used live in `pdm.lock`. +This separation is deliberate: +`pyproject.toml` says what your project is *compatible with*, and `pdm.lock` says what you are *currently using*. +These reflect two different use cases. +If you are providing a *library* that other researchers or developers will use in their code, then the looser constraints in `pyproject.toml` are necessary because they will have other dependencies that also have their own requirements, and a mutually compatible set of dependencies needs to be found. +Stating that your package is only compatible with exact versions of its dependencies makes this dependency resolution very difficult and sometimes impossible. +On the other hand, if someone is directly running your code as an application, then having the strict `pdm.lock` will mean they have a better chance of getting a working version of the code on their system. +A win for reproducibility! -To display information about a particular installed package do: +To ask for a particular version or set your own constraint on a dependency version, give the constraint on the command line (quoting it so that your shell does not interpret the `>` character), e.g.: ```bash -(venv) $ python3 -m pip show numpy +$ pdm add "numpy==2.2.3" +$ pdm add "numpy>=2.2" ``` -```output -Name: numpy -Version: 1.26.2 -Summary: Fundamental package for array computing in Python -Home-page: https://numpy.org -Author: Travis E. Oliphant et al. -Author-email: -License: Copyright (c) 2005-2023, NumPy Developers. -All rights reserved. -... -Required-by: contourpy, matplotlib -``` +Respectively, these will: + +- put the dependency requirement `numpy==2.2.3` in `pyproject.toml`, meaning anyone who installs your package will also have this exact version of `numpy` installed +- put the dependency requirement `numpy>=2.2` in `pyproject.toml`, meaning anyone who installs your package will need to have a `numpy` version of *at least* `2.2.0` and above. -To list all packages installed with `pip` (in your current virtual environment): +To see what is installed in your current environment, use the command ```bash -(venv) $ python3 -m pip list +$ pdm list ``` ```output -Package Version ---------------- ------- -contourpy 1.2.0 -cycler 0.12.1 -fonttools 4.45.0 -kiwisolver 1.4.5 -matplotlib 3.8.2 -numpy 1.26.2 -packaging 23.2 -Pillow 10.1.0 -pip 23.0.1 -pyparsing 3.1.1 -python-dateutil 2.8.2 -setuptools 67.6.1 -six 1.16.0 -``` - -To uninstall a package installed in the virtual environment do: `python3 -m pip uninstall `. -You can also supply a list of packages to uninstall at the same time. - -### Installing Our Local Project as a Package Using `pip` +╭──────────────────────────────────┬─────────────┬──────────────────────────────────────────────────╮ +│ name │ version │ location │ +├──────────────────────────────────┼─────────────┼──────────────────────────────────────────────────┤ +│ contourpy │ 1.3.2 │ │ +│ cycler │ 0.12.1 │ │ +│ fonttools │ 4.63.0 │ │ +│ kiwisolver │ 1.5.0 │ │ +│ matplotlib │ 3.10.9 │ │ +│ numpy │ 2.2.6 │ │ +│ packaging │ 26.3 │ │ +│ pillow │ 12.3.0 │ │ +│ pyparsing │ 3.3.2 │ │ +│ python-dateutil │ 2.9.0.post0 │ │ +│ python-intermediate-inflammation │ 0.0.0 │ -e /home/user/python-intermediate-inflammation │ +│ six │ 1.17.0 │ │ +╰──────────────────────────────────┴─────────────┴──────────────────────────────────────────────────╯ +``` + +Some other commands you will find useful: + +- `pdm list --tree` shows which package pulled in which, as a dependency tree. +- `pdm show numpy` displays information about a particular package. +- `pdm remove numpy` removes a dependency from `pyproject.toml`, + the lock file and the environment - all in one step. +- `pdm outdated` lists packages for which a newer version is available. +- `pdm update` updates your dependencies (within the constraints in `pyproject.toml`) + and writes the new versions to `pdm.lock`. + + +::: callout + +## Where Do the Dependencies Come From? + +By default, `pdm` obtains packages from the central repository called the [Python Package Index (PyPI)](https://pypi.org/). +This is where most third-party Python packages live, but it is possible to configure `pdm` to get your packages from other repositories. +PyPI does its best to try and ensure the security and trustworthiness of the packages it hosts, but there are always bad actors that slip through. +You should be aware of "name squatting" where a malicious package will use a very similar name or gain ownership of the actual name of a semi-popular package. +If you mistype or are unlucky enough to download before this is discovered, you will inadvertantly download malicious code onto your device. + +::: Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or at the very least it might be useful to treat it like a package. Said another way, it is usually the case we want a convenient way to call the Python code we are writing from another location, and making this code accessible as a package is the best way to do this. From f202c17896a3bc7e03df79db8c850398511e7ca0 Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 3 Sep 2026 17:11:20 +0100 Subject: [PATCH 5/5] Add complete first draft of 12-virtual-environments.md using pdm --- episodes/12-virtual-environments.md | 210 +++++++++------------------- 1 file changed, 66 insertions(+), 144 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 602ca0a61..83a9d8329 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -164,8 +164,9 @@ There are truly open alternatives like [conda-forge](https://conda-forge.org/). :::::::::::::::::::::::::::::::::::::::::::::::::: -Let us have a look at how we can create and manage a virtual environment -and its packages from the command line using `pdm`. +### Creating a Virtual Environment Using `pdm` + +Let us have a look at how we can create and manage a virtual environment and its packages from the command line using `pdm`. :::::::::::::::::::::::::::::::::::::::::: prereq @@ -204,8 +205,6 @@ need to create an alias for the python executable `python.exe`, as explained in :::::::::::::::::::::::::::::::::::::::::::::::::: -### Creating a Virtual Environment Using `pdm` - Our project already contains a `pyproject.toml` file, the standard file that describes a Python project, its metadata and its dependencies. Because of this, we do not need to create the project from scratch; we can ask PDM to set everything up for us with a single command. @@ -371,14 +370,14 @@ With PDM, installing a dependency and declaring that your project requires it ar You *add* it to the project: ```bash -$ pdm add numpy -$ pdm add matplotlib +pdm add numpy +pdm add matplotlib ``` or doing both at once: ```bash -$ pdm add numpy matplotlib +pdm add numpy matplotlib ``` These commands might take a little while to run if there are not binary builds available for your machine. @@ -414,8 +413,8 @@ A win for reproducibility! To ask for a particular version or set your own constraint on a dependency version, give the constraint on the command line (quoting it so that your shell does not interpret the `>` character), e.g.: ```bash -$ pdm add "numpy==2.2.3" -$ pdm add "numpy>=2.2" +pdm add "numpy==2.2.3" +pdm add "numpy>=2.2" ``` Respectively, these will: @@ -426,7 +425,7 @@ Respectively, these will: To see what is installed in your current environment, use the command ```bash -$ pdm list +pdm list ``` ```output @@ -448,16 +447,6 @@ $ pdm list ╰──────────────────────────────────┴─────────────┴──────────────────────────────────────────────────╯ ``` -Some other commands you will find useful: - -- `pdm list --tree` shows which package pulled in which, as a dependency tree. -- `pdm show numpy` displays information about a particular package. -- `pdm remove numpy` removes a dependency from `pyproject.toml`, - the lock file and the environment - all in one step. -- `pdm outdated` lists packages for which a newer version is available. -- `pdm update` updates your dependencies (within the constraints in `pyproject.toml`) - and writes the new versions to `pdm.lock`. - ::: callout @@ -471,143 +460,74 @@ If you mistype or are unlucky enough to download before this is discovered, you ::: -Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or at the very least it might be useful to treat it like a package. +### Our Own Project is Installed Too + +Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or someone else would like a package. Said another way, it is usually the case we want a convenient way to call the Python code we are writing from another location, and making this code accessible as a package is the best way to do this. -We will save the details of Python packaging for [a future episode](43-software-release.md), and for the meantime we can use the minimal package setup that our project already comes with, which is contained in the `pyproject.toml` file. -Once again, we can use `pip` to install our local package: +We will save the details of Python packaging for [a future episode](43-software-release.md), +and for the meantime the minimal package setup in our `pyproject.toml` is enough. -```bash -python3 -m pip install --editable . -``` +This is the third thing `pdm install` did for us back near [the beginning of the episode](#creating-a-virtual-environment-using-pdm): it installed our own project into the environment in **editable** mode. +An **editable** install is one that allows the package in our environment to change dynamically based on source code locally. +This is very convenient when we are developing the package because we can instantly see changes when we call the code from within our virtual environment, rather than having to install the local package again to get the updates. +That is why `python-intermediate-inflammation` appears in the `pdm list` output above, and it is indicated as "editable" with the `-e` prefix for its location. +(If you ever want to skip this step and not have your project installed in your environment, `pdm install --no-self` will do so.) + +### Sharing Your Environment With Collaborators -If the above command fails for you - your `pip` installation is older than version 21.3. -Such older versions of `pip` do not support `pyproject.toml` as the package metadata. -Given these versions of `pip` are now over 4 years old, we strongly recommend that you update `pip` if you can with: +You are collaborating on a project with a team so, naturally, you will want to share your environment with your collaborators so they can easily "clone" your software project with all of its dependencies and everyone can replicate equivalent virtual environments on their machines. +With PDM there is nothing extra to do to achieve this. +The `pdm.lock` generated automatically contains the exact versions of every package (including sub-dependencies) that were resolved when creating the virtual environment for our project. +This file should be committed to version control, and we will get around to doing this using Git in one of the following episodes. +A collaborator then only has to run: ```bash -python3 -m pip install --upgrade pip +pdm install ``` -This is similar syntax to above, with two important differences: - -1. The `--editable` or `-e` flag indicates that the package we are specifying should be an "editable" install. - An "editable" install is one that allows the package in our environment to change dynamically based on source code locally. - This is very convenient when we are developing the package because we can instantly see changes when we call the code from within our virtual environment, rather than having to install the local package again to get the updates. -2. The argument `'.'` indicates that the package we want to install is located in the current directory. - The `pyproject.toml` file located in this directory then handles the rest. - +and they will get an environment containing exactly the same package versions as yours. +This reproducibility is the main reason lock files exist. +If your collaborator already has a PDM-created virtual environment, then they would instead run `pdm sync` to get any of your updates contained in `pdm.lock`. -If we reissue the `pip list` command we should now see our local package with the name `python-intermediate-inflammation` in the output: +As your project grows you will need to update your environment for a variety of reasons. +For example, one of your project's dependencies has just released a new version, you need an additional package for data analysis, or you have found a better package and no longer need the older one. +Each of these is a single `pdm add`, `pdm remove` or `pdm update` command, and each of them updates `pyproject.toml` and `pdm.lock` for you. +You then just commit those changes and propagate them to your collaborators via your code sharing platform (e.g. GitHub). -```output -Package Version Editable project location --------------------------------- ----------- ---------------------------------------------------------------------------------------------- -contourpy 1.3.1 -cycler 0.12.1 -exceptiongroup 1.2.2 -fonttools 4.56.0 -iniconfig 2.0.0 -kiwisolver 1.4.8 -matplotlib 3.10.0 -numpy 2.2.3 -packaging 24.2 -pillow 11.1.0 -pip 22.0.2 -pluggy 1.5.0 -pyparsing 3.2.1 -pytest 8.3.4 -python-dateutil 2.9.0.post0 -python-intermediate-inflammation 0.0.0 /path/to/your/project/directory/python-intermediate-inflammation -setuptools 59.6.0 -six 1.17.0 -tomli 2.2.1 -``` - -### Exporting/Importing Virtual Environments Using `pip` +::::::::::::::::::::::::::::::::::::::::: callout -You are collaborating on a project with a team so, naturally, -you will want to share your environment with your collaborators -so they can easily 'clone' your software project with all of its dependencies -and everyone can replicate equivalent virtual environments on their machines. -`pip` has a handy way of exporting, saving and sharing virtual environments. +## What About `requirements.txt`? -To export your active environment use the `python3 -m pip freeze --exclude-editable` command to produce a list of packages installed in the virtual environment. -A common convention is to put this list in a `requirements.txt` file: +You will still meet projects (and tools, and deployment systems) that expect a `requirements.txt` file listing pinned versions. +PDM can generate one from the lock file when you need it: ```bash -(venv) $ python3 -m pip freeze --exclude-editable > requirements.txt -(venv) $ cat requirements.txt -``` - -```output -contourpy==1.2.0 -cycler==0.12.1 -fonttools==4.45.0 -kiwisolver==1.4.5 -matplotlib==3.8.2 -numpy==1.26.2 -packaging==23.2 -Pillow==10.1.0 -pyparsing==3.1.1 -python-dateutil==2.8.2 -six==1.16.0 +pdm export -f requirements --without-hashes -o requirements.txt ``` -The first of the above commands will create a `requirements.txt` file in your current directory. -Yours may look a little different, -depending on the version of the packages you have installed, -as well as any differences in the packages that they themselves use. -Also, we need to use the `--exclude-editable` command so that our local package is not included in the output, otherwise pip will try to pull from a specific commit at the time we made the editable install, which is not what we want. - -The `requirements.txt` file can then be committed to a version control system -(we will see how to do this using Git in one of the following episodes) -and get shipped as part of your software and shared with collaborators and/or users. -They can then replicate your environment -and install all the necessary packages from the project root as follows: - -```bash -(venv) $ python3 -m pip install -r requirements.txt --editable . -``` +Treat any such file as a generated artefact; `pyproject.toml` and `pdm.lock` are a better source of truth for your project. -As your project grows you may need to update your environment for a variety of reasons. -For example, one of your project's dependencies has just released a new version -(dependency version number update), -you need an additional package for data analysis (adding a new dependency) -or you have found a better package and no longer need the older package -(adding a new and removing an old dependency). -What you need to do in this case -(apart from installing the new and removing the packages that are no longer needed -from your virtual environment) -is update the contents of the `requirements.txt` file accordingly -by re-issuing `pip freeze` command -and propagate the updated `requirements.txt` file to your collaborators -via your code sharing platform (e.g. GitHub). +:::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: testimonial ## Official Documentation -For a full list of options and commands, -consult the [official `venv` documentation](https://docs.python.org/3/library/venv.html) -and the [Installing Python Modules with `pip` guide](https://docs.python.org/3/installing/index.html#installing-index). -Also check out the guide -["Installing packages using `pip` and virtual environments"](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-packages-using-pip-and-virtual-environments). - +For a full list of options and commands, consult the [official PDM documentation](https://pdm-project.org/), in particular the pages on [managing dependencies](https://pdm-project.org/latest/usage/dependency/) and [working with virtual environments](https://pdm-project.org/latest/usage/venv/). +If you want to dig into what is happening underneath, the [`venv` documentation](https://docs.python.org/3/library/venv.html) is a good place to start. :::::::::::::::::::::::::::::::::::::::::::::::::: ## Running Python Scripts From Command Line Congratulations! -Your environment is now activated and set up -to run our `inflammation-analysis.py` script from the command line. +Your environment is now set up to run our `inflammation-analysis.py` script from the command line. -You should already be located in the root of the `python-intermediate-inflammation` directory -(if not, please navigate to it from the command line now). +You should already be located in the root of the `python-intermediate-inflammation` directory (if not, please navigate to it from the command line now). To run the script, type the following command: ```bash -(venv) $ python3 inflammation-analysis.py +pdm run python inflammation-analysis.py ``` ```output @@ -615,24 +535,28 @@ usage: inflammation-analysis.py [-h] infiles [infiles ...] inflammation-analysis.py: error: the following arguments are required: infiles ``` -In the above command, we tell the command line two things: - -1. to find a Python interpreter - (in this case, the one that was configured via the virtual environment), and -2. to use it to run our script `inflammation-analysis.py`, - which resides in the current directory. - As we can see, the Python interpreter ran our script, which threw an error - `inflammation-analysis.py: error: the following arguments are required: infiles`. -It looks like the script expects a list of input files to process, -so this is expected behaviour since we do not supply any. +It looks like the script expects a list of input files to process, so this is expected behaviour since we do not supply any. We should run our code as follows, passing one (or more) data file(s) as input: ```bash -(venv) $ python3 inflammation-analysis.py data/inflammation-01.csv +pdm run python inflammation-analysis.py data/inflammation-01.csv ``` +::: callout + +Some other `pdm` commands you will find useful: + +- `pdm list --tree` shows which package pulled in which, as a dependency tree. +- `pdm show numpy` displays information about a particular package. +- `pdm remove numpy` removes a dependency from `pyproject.toml`, the lock file and the environment - all in one step. +- `pdm outdated` lists packages for which a newer version is available. +- `pdm update` updates your dependencies (within the constraints in `pyproject.toml`) and writes the new versions to `pdm.lock`. + +::: + ## Optional Exercises Have a look at [some optional exercises](17-section1-optional-exercises.md). @@ -641,13 +565,11 @@ Have a look at [some optional exercises](17-section1-optional-exercises.md). :::::::::::::::::::::::::::::::::::::::: keypoints - Virtual environments keep Python versions and dependencies required by different projects separate. -- A virtual environment is itself a directory structure. -- Use `venv` to create and manage Python virtual environments. -- Use `pip` to install and manage Python external (third-party) libraries. -- `pip` allows you to declare all dependencies for a project in a separate file (by convention called `requirements.txt`) which can be shared with collaborators/users and used to replicate a virtual environment. -- Use `python3 -m pip freeze --exclude-editable > requirements.txt` to take snapshot of your project's dependencies. -- Use `python3 -m pip install -r requirements.txt` to replicate someone else's virtual environment on your machine from the `requirements.txt` file. +- A virtual environment is itself a directory structure, of the kind `venv` creates. +- `pdm` creates and manages that virtual environment for you, by convention in `.venv` in the project root. +- Use `pdm install` to set up (or reproduce) a project's environment from `pyproject.toml` and `pdm.lock`. +- Use `pdm add` and `pdm remove` to manage external (third-party) libraries; they are recorded in `pyproject.toml`. +- `pdm.lock` records the exact resolved versions of every dependency - commit it alongside `pyproject.toml` so collaborators get an identical environment. +- Use `pdm run` to run a command inside the project's environment without having to activate it. :::::::::::::::::::::::::::::::::::::::::::::::::: - -