Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 5 additions & 16 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,12 @@ jobs:
with:
persist-credentials: false
-
name: 'Validate that template/* are used to generate Dockerfiles'
name: 'Validate that the generated files are up to date'
run: |-
docker run --rm -v $PWD:/app -w /app webdevops/dockerfile-build-env make provision
git diff --exit-code --color=always
-
name: 'Validate .github/workflows/build.yaml is up to date'
run: |-
docker run --rm -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine composer install --no-dev
docker run --rm -v $PWD:/app -w /app webdevops/php:8.4-alpine ci/console github:generate-ci
git diff --exit-code --color=always
-
name: 'Validate the README banner is up to date'
run: |-
docker run --rm -v $PWD:/app -w /app webdevops/php:8.4-alpine ci/console readme-banner:generate
make generate
git status --porcelain
git diff --exit-code --color=always
test -z "$(git status --porcelain)"
check-ci-code:
name: 'Check ci/ (GrumPHP and PHPUnit)'
runs-on: ubuntu-latest
Expand All @@ -45,9 +36,7 @@ jobs:
persist-credentials: false
-
name: 'Run GrumPHP'
run: |-
docker run --rm -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine composer install
docker run --rm -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine sh -c 'apk add --no-cache file && git config --global --add safe.directory /app && vendor/bin/grumphp run --no-interaction'
run: 'make check'
apache_alpine:
strategy:
fail-fast: false
Expand Down
5 changes: 2 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
/.idea
.DS_Store
/.cache
/.vscode
/tests/serverspec/vendor
/tests/serverspec/.bundle/
/tests/serverspec/Dockerfile*
/BLACKLIST
/ci/BLACKLIST
*.log
*.pyc
.doit.db
31 changes: 15 additions & 16 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,31 @@

| Path | Content |
| --- | --- |
| `template/` | Shared Jinja2 templates for the Dockerfiles (source) |
| `docker/<image>/<tag>/Dockerfile.jinja2` | Per-image template (source) |
| `template/` | Shared Twig templates for the Dockerfiles: the layout, which writes the header, and the macros (source) |
| `docker/<image>/<tag>/Dockerfile.twig` | Per-image Twig template; the layout includes it, and it imports the macros it uses from `template/Dockerfile/` (source) |
| `provisioning/` | Scripts and configuration files copied into the images (source); `conf/provision.yml` says which go where |
| `docker/<image>/<tag>/Dockerfile`, `docker/<image>/<tag>/conf/` | Generated by `make provision`, never edited by hand |
| `bin/` | Old Python 2 tools behind `make provision` |
| `ci/` | PHP generators for `.github/workflows/build.yaml` (`ci/console github:generate-ci`) and the README banner (`ci/console readme-banner:generate`) |
| `docker/<image>/<tag>/Dockerfile`, `docker/<image>/<tag>/conf/` | Generated by `ci/console generate`, never edited by hand |
| `ci/` | PHP generators behind `ci/console generate`: the Dockerfiles (`generate:dockerfile`), their `conf/` (`generate:provision`), `.github/workflows/build.yaml` (`generate:github-ci`) and the README banner (`generate:readme-banner`) |
| `ci/tests/` | PHPUnit tests of `ci/src`; `ci/tests/Fixtures/repository` is a small repository they run against |
| `ci/grumphp.yml`, `ci/phpstan.neon`, `ci/rector.php`, `ci/fractor.php` | Checks of the PHP code in `ci/`, from pluswerk/grumphp-config |
| `conf/readme-banner.yml`, `ci/resources/readme-banner/` | Texts and font outlines of the README banner (source) |
| `docs/assets/readme-banner-*.svg` | README banner, generated by `ci/console readme-banner:generate`, never edited by hand |
| `docs/assets/readme-banner-*.svg` | README banner, generated by `ci/console generate:readme-banner`, never edited by hand |
| `tests/` | serverspec (Ruby) and container-structure-test definitions run by the CI |
| `docs/` | User documentation |

`docker/php/<version>` are symlinks to `docker/php-official/<version>`; both images are built from the same Dockerfiles. The tags that are built are the `-t "webdevops/..."` lines in `.github/workflows/build.yaml`.
`docker/php/<version>` are symlinks to `docker/php-official/<version>`, kept by `ci/console generate:dockerfile`; both images are built from the same Dockerfiles. The tags that are built are the `-t "webdevops/..."` lines in `.github/workflows/build.yaml`.

## Build and test

Only these commands are known to work:

- `docker run --rm -v $PWD:/app -w /app webdevops/dockerfile-build-env make provision` regenerates `docker/`. Run it after every change to `template/`, `provisioning/` or a `Dockerfile.jinja2`, and commit the result.
- `make generate` installs the PHP tools of `ci/` and runs `ci/console generate` in `webdevops/php:8.4-alpine`, which regenerates `docker/`, `.github/workflows/build.yaml` and the README banner. Run it after every change to `template/`, `provisioning/`, `conf/` or a `Dockerfile.twig`, and commit the result; the CI fails if a generated file is out of date or missing.
- `docker build -t webdevops/<image>:<tag> docker/<image>/<tag>` builds one image; its parent and `webdevops/toolbox` come from local images or Docker Hub.
- `./build-local.sh [version]` builds all PHP images of one version.
- `docker run --rm -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine composer install` installs the PHP tools of `ci/`.
- `docker run --rm -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine sh -c 'apk add --no-cache file && git config --global --add safe.directory /app && vendor/bin/grumphp run'` checks the PHP code in `ci/`: coding standard, PHPStan, Rector, `composer audit` and the PHPUnit tests. Run it after every change to `ci/`. GrumPHP and PHPStan only check files that Git knows (`git ls-files`), so `git add` new files first; PHPUnit runs every test anyway. `vendor/bin/phpunit` instead of `vendor/bin/grumphp run` runs only the tests. Changes to `ci/src` come with tests.
- `make check` installs the PHP tools of `ci/` and checks the PHP code in `ci/` with GrumPHP: coding standard, PHPStan, Rector, `composer audit` and the PHPUnit tests. Run it after every change to `ci/`. GrumPHP and PHPStan only check files that Git knows (`git ls-files`), so `git add` new files first; PHPUnit runs every test anyway. `make test` runs only the tests. Changes to `ci/src` come with tests.
- `make generate`, `make check` and `make test` run as your user, so the files they write belong to you. Do not write into the checkout as root.

Besides `provision`, the Makefile has `structure-test` and `setup`, which the `webdevops/dockerfile-build-env` image runs while it is built. `make structure-test` runs container-structure-test against your local images of `bootstrap`, `base`, `base-app`, `php`, `php-dev` and `php-nginx`, the images that have a suite in `tests/structure-test`. It tests only the tags in `.github/workflows/build.yaml` and fails if a test fails, if `docker` fails or if no such image exists locally; run it inside `webdevops/dockerfile-build-env` with the Docker socket mounted. The CI runs serverspec for every image except `toolbox` and `dockerfile-build-env`, and container-structure-test for the six images above; to check a change by hand, start the image and try it as described below.
Besides `generate`, `check` and `test`, the Makefile has `structure-test`. `make structure-test` runs container-structure-test against your local images of `bootstrap`, `base`, `base-app`, `php`, `php-dev` and `php-nginx`, the images that have a suite in `tests/structure-test`. It tests only the tags in `.github/workflows/build.yaml` and fails if a test fails, if `docker` fails or if no such image exists locally; run it inside `webdevops/dockerfile-build-env` with the Docker socket mounted. The CI runs serverspec for every image except `toolbox` and `dockerfile-build-env`, and container-structure-test for the six images above; to check a change by hand, start the image and try it as described below.

### Packages come with tests

Expand All @@ -53,7 +52,7 @@ Where the documentation lives:
| `docs/environment-variables.md` | Every environment variable with its default and effect; uploads and long requests |
| `docs/customization.md` | Commands, scripts at start and build time, tools for your Dockerfile, services, cron jobs and their time zone, healthcheck, configuration files and placeholders, supervisord, locales, application user and changing its UID |
| `docs/images/*.md` | One page per image family: tags, base image, services, settings, development variant. `php.md` also has Debian or Alpine, production, installing extensions, mail and Xdebug with an IDE |
| `docs/known-issues.md` | Options and tools that do not work, with affected images and workaround; migration from older images; build tools that no longer work |
| `docs/known-issues.md` | Options and tools that do not work, with affected images and workaround; migration from older images |
| `docs/discontinued-images.md` | Replacements for images that are no longer built (`storage`, `vsftp`, `ssh`) |

### When to update it
Expand All @@ -73,12 +72,12 @@ Update the documentation whenever a change touches any of these:

| Question | Where to look |
| --- | --- |
| Published images and tags | `-t "webdevops/..."` tags in `.github/workflows/build.yaml` (generated by `ci/console github:generate-ci` from `conf/console.yml` and `ci/`) |
| Published images and tags | `-t "webdevops/..."` tags in `.github/workflows/build.yaml` (generated by `ci/console generate:github-ci` from `conf/console.yml` and `ci/`) |
| Base image | `FROM` line in `docker/<image>/<tag>/Dockerfile` |
| Variables and defaults | `ENV` in `template/Dockerfile/**` and `docker/**/Dockerfile.jinja2`, scripts in `provisioning/**/bin` and `provisioning/**/provision` |
| Variables and defaults | `ENV` in `template/Dockerfile/**` and `docker/**/Dockerfile.twig`, scripts in `provisioning/**/bin` and `provisioning/**/provision` |
| Which files end up in which image | `conf/provision.yml` |

2. **Change sources, not generated files.** `Dockerfile` and `conf/` under `docker/` are generated. Edit `Dockerfile.jinja2`, `template/` or `provisioning/` and run `make provision`.
2. **Change sources, not generated files.** `Dockerfile` and `conf/` under `docker/` are generated. Edit `Dockerfile.twig`, `template/` or `provisioning/` and run `make generate`.

3. **Verify every statement in a running container.** Build the image (or use the image the CI built) and try the option. Do not rely on reading the scripts alone. Examples:
- PHP settings: `php -r 'echo ini_get("memory_limit");'`, or `php-fpm -i` for values the CLI overrides
Expand All @@ -91,7 +90,7 @@ Update the documentation whenever a change touches any of these:

4. **Document only what works.** Never describe an option that has no effect as working. Fix it, or leave it out and list it in `docs/known-issues.md`. If it works only in some images, say exactly which. Anything `docs/known-issues.md` lists as not working must never appear as working anywhere else. An `ENV` line or a script that reads a variable is no proof that it works: `SERVICE_POSTFIX_OPTS` is passed to `postfix start`, which ignores it. When you fix a known issue, remove its entry in the same pull request.

5. **Keep lists in sync.** The tag lists in `README.md` and in `docs/images/*.md` must match the tags pushed by `.github/workflows/build.yaml`. The README banner follows the built tags by itself when you run `ci/console readme-banner:generate`; change its texts in `conf/readme-banner.yml`. `webdevops/php-official` is not listed in the README; it is built from the same Dockerfiles as `webdevops/php`. The defaults in `docs/environment-variables.md` must match what a container reports.
5. **Keep lists in sync.** The tag lists in `README.md` and in `docs/images/*.md` must match the tags pushed by `.github/workflows/build.yaml`. The README banner follows the built tags by itself when you run `ci/console generate:readme-banner`; change its texts in `conf/readme-banner.yml`. `webdevops/php-official` is not listed in the README; it is built from the same Dockerfiles as `webdevops/php`. The defaults in `docs/environment-variables.md` must match what a container reports.

6. **Follow the style.**
- English, GitHub Markdown, relative links, diagrams as Mermaid. No documentation tooling.
Expand Down
35 changes: 20 additions & 15 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,22 +1,27 @@
MAKEFLAGS += --silent
.PHONY: list setup requirements structure-test provision
.PHONY: generate check test structure-test

list:
sh -c "echo; $(MAKE) -p no_targets__ 2>/dev/null |\
awk -F':' '/^[a-zA-Z0-9][^\$$#\\t=]*:([^=]|$$)/ {split(\$$1,A,/ /);\
for(i in A)print A[i]}' | \
grep -v '__\$$\|Makefile\|%' | \
sort"
# The containers run as the user who calls make, so that the files they write belong to them. In rootless Docker,
# root in the container already is that user. sh replaces the entrypoint of webdevops/php, which would switch to root.
ROOTLESS = $(findstring rootless,$(shell docker info --format '{{.SecurityOptions}}' 2>/dev/null))
USER_ID = $(shell id -u):$(shell id -g)
DOCKER_USER = $(if $(ROOTLESS),,--user $(USER_ID))
GOSU = $(if $(ROOTLESS),,gosu $(USER_ID))
# git trusts the checkout without a write to a .gitconfig
GIT_TRUST = -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0=/app

setup: requirements
# Installs the PHP tools of ci/ and writes every generated file, in webdevops/php:8.4-alpine
generate:
docker run --rm $(DOCKER_USER) -e COMPOSER_HOME=/tmp/composer -v "$(CURDIR)":/app -w /app --entrypoint sh webdevops/php:8.4-alpine -c 'composer install --working-dir=ci && ci/console generate'

requirements:
pip install --upgrade -I -r ./bin/requirements.txt
cd tests/serverspec && bundle install --path=vendor
# Installs the PHP tools of ci/ and checks the PHP code in ci/ with GrumPHP. Only apk runs as root, for the "file"
# command of GrumPHP's BOM check.
check:
docker run --rm $(GIT_TRUST) -e COMPOSER_HOME=/tmp/composer -v "$(CURDIR)":/app -w /app/ci --entrypoint sh webdevops/php:8.4-alpine -c 'apk add --no-cache file >/dev/null && exec $(GOSU) sh -c "composer install && vendor/bin/grumphp run --no-interaction"'

# Runs only the PHPUnit tests of ci/
test:
docker run --rm $(DOCKER_USER) -e COMPOSER_HOME=/tmp/composer -v "$(CURDIR)":/app -w /app/ci --entrypoint sh webdevops/php:8.4-alpine -c 'composer install && vendor/bin/phpunit'

structure-test:
cd tests/structure-test && ./run.sh

provision:
python2 bin/console generate:dockerfile
python2 bin/console generate:provision
61 changes: 36 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,43 +45,46 @@ Or join our discord https://discord.gg/BpBZtXAgJz

### File generation

The general build process is currently a mixture of python jinja2 legacy and PHP.

First we build the files inside the `docker` directory using this command:
The files inside the `docker` directory, the GitHub Actions workflow (`.github/workflows/build.yaml`) and the banner at the top of this README are generated with PHP:

```
docker run --rm -ti -v $PWD:/app -w /app webdevops/dockerfile-build-env make provision
```

It will use the Jinja2 templates inside of the `template` directory and the
config files from the `provisioning` directory.

**The generated files in the `docker` directory (`Dockerfile` and `conf/`) are never modified manually.**
Edit the `Dockerfile.jinja2` templates next to them, the shared templates in `template` or the files in `provisioning` instead.
`make provision` copies the `provisioning` files into the `conf/` directories as configured in `conf/provision.yml`.
make generate
```

*This will be replaced with PHP twig templates in the future to streamline everything.*
It installs the PHP tools of `ci/` with Composer and runs `ci/console generate` in `webdevops/php:8.4-alpine`, so you only need `make` and Docker.
The container runs as your user, so the files it writes belong to you.
`ci/console generate` runs these commands in this order; each of them also runs on its own:

| Command | Writes |
| --- | --- |
| `ci/console generate:dockerfile` | `docker/<image>/<tag>/Dockerfile` from the `Dockerfile.twig` next to it and the Twig templates in the `template` directory, and the links `docker/php/<version>` to `docker/php-official/<version>` |
| `ci/console generate:provision` | `docker/<image>/<tag>/conf/` from the `provisioning` directory, as configured in `conf/provision.yml` |
| `ci/console generate:github-ci` | `.github/workflows/build.yaml`, a build and a publish job for every Dockerfile |
| `ci/console generate:readme-banner` | `docs/assets/readme-banner-*.svg` |

**The generated files are never modified manually.**
Edit the `Dockerfile.twig` templates, the shared templates in `template`, the files in `provisioning` or the settings in `conf` instead.
`template/Dockerfile/layout.twig` writes the header of every Dockerfile and includes its `Dockerfile.twig`.
A `Dockerfile.twig` imports the macros it uses from `template/Dockerfile`, e.g. `{% import 'docker.twig' as docker %}`; a macro imports the macros it calls itself.
`generate:provision` empties the `conf/` directories of an image first, so a file removed from `provisioning` disappears there too.
It checks all of `conf/provision.yml` before it changes a file and stops if a source directory or a tag does not exist, or if an image has another setting than `configuration`.
`generate:dockerfile` stops if a directory `docker/<image>/<tag>` has files but no `Dockerfile.twig`; it first removes the `.DS_Store` files of the macOS Finder under `docker`.
To stop building a tag, remove its directory and those of the tags that build on it; `generate:github-ci` names any it finds. `generate:dockerfile` adds and removes the links in `docker/php` along with the `php-official` tags.

### Building in CI

We generate the GitHub Actions workflow (`.github/workflows/build.yaml`) and the banner at the top of this README using PHP:

```
docker run --rm -ti -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine composer install
docker run --rm -ti -v $PWD:/app -w /app webdevops/php:8.4-alpine ci/console github:generate-ci
docker run --rm -ti -v $PWD:/app -w /app webdevops/php:8.4-alpine ci/console readme-banner:generate
```

The banner (`docs/assets/readme-banner-*.svg`) takes the PHP versions, web servers, base OS suffixes, Debian releases and platforms from the images that are built.
Its texts are in `conf/readme-banner.yml`; the SVG files are never modified manually. The CI fails if the workflow or the banner is out of date.
Its texts are in `conf/readme-banner.yml`. The CI fails if a generated file is out of date or missing.

The PHP code in `ci/` follows the rules of [pluswerk/grumphp-config](https://github.com/andersundsehr/grumphp-config) (PSR-12, PHPStan level 8, Rector, `composer audit`) and has PHPUnit tests in `ci/tests`.
GrumPHP runs all of them; after a change to `ci/`, run it with:

```
docker run --rm -ti -v $PWD:/app -w /app/ci webdevops/php:8.4-alpine sh -c 'apk add --no-cache file && git config --global --add safe.directory /app && vendor/bin/grumphp run'
make check
```

It installs the PHP tools of `ci/` and runs GrumPHP as your user. `make test` runs only the PHPUnit tests.

The CI job "Check ci/" runs the same checks. No image build waits for it.

GitHub Actions builds every image independently and runs the serverspec and structure tests
Expand All @@ -102,8 +105,16 @@ Build all PHP images of one version (default `8.5`), in the order they depend on
./build-local.sh 8.4
```

Then try an image with the quick start above. `make provision` calls `bin/console`, which needs Python 2;
run it inside `webdevops/dockerfile-build-env`. See [Known issues](docs/known-issues.md#build-tools).
Then try an image with the quick start above.

The CI runs the serverspec tests for every image except `toolbox` and `dockerfile-build-env`, and the structure tests for `bootstrap`, `base`, `base-app`, `php`, `php-dev` and `php-nginx`.
To run the structure tests against the images you built locally:

```
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/app -w /app webdevops/dockerfile-build-env make structure-test
```

It tests the local tags that the CI builds and fails if no such image exists.

### PHP extension builds

Expand Down
Empty file removed bin/__init__.py
Empty file.
Empty file removed bin/command/__init__.py
Empty file.
Loading
Loading