What happened?
Thanks to my recent task to check that my MariaDB backups are working fine (had issues with Ofelia not firing, hence I switched to Supercronic), I noticed issues on my server to day. I saw an hour old error.flag file, which normally creates an email, which is sent by a job, that's triggered by Supercronic on FrankenPHP container, if that container is healthy. But it was not triggered, because the container was not healthy. Although it seemed to be working fine.
Current healthcheck calls curl -f http://localhost:2019/metrics, and I did that. I got a flood of entries like
caddy_rate_limit_process_time_seconds_sum{key="130.180.236.170/32",zone="reads"} 5.0729e-05
caddy_rate_limit_process_time_seconds_count{key="130.180.236.170/32",zone="reads"} 1
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.001"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.005"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.01"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.025"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.05"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.1"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.25"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.5"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="1"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="+Inf"} 2
caddy_rate_limit_process_time_seconds_sum{key="130.180.236.176/32",zone="reads"} 7.129200000000001e-05
It continued for like 20 seconds, when I broke it. I did a docker inspect --format='{{json .State.Health}}' frankenphp and it last healthcheck exceeded 30s timeout. But that's not all: looks like this also resulted in OOM that killed mariadb container. I do not have hard proof of that, but OOM did kill mariadb, and the timing is mighty suspicious. I see there was also a #439, which was caused by a memory leak, but it does suggest that healthcheck can cause a spike in memory usage, and if it will trigger OOM-killer, mariadb will be selected since it uses the most of RAM.
The entries are likely caused by https://github.com/mholt/caddy-ratelimit so I disabled metrics there for now (at least), but... Someone may need these metrics. Maybe not from this plugin, but from something else, that can also result in growth of the output from /metrics endpoint. Maybe not to the same extent, as rate-limiter, but depending on what else is happening on the server even a small increase in size can trigger an OOM or slowness of output of the metrics and you get issues.
As such, I am raising this ticket so that an alternative healthcheck is considered in the FrankenPHP Docker image. And since this is built into FrankenPHP image, I am raising it for FrankenPHP. It also may be worthwhile to have something that not only checks that Caddy is up, but that PHP interpreter is working, too. For Caddy side it looks like /config endpoint is the most neutral, but my config is 122KBs, which is relatively large, too, so may not be ideal.
Build Type
Docker (Debian Trixie)
Worker Mode
No
Operating System
GNU/Linux
CPU Architecture
x86_64
PHP configuration
phpinfo() output
Relevant log output
Relevant log output
What happened?
Thanks to my recent task to check that my MariaDB backups are working fine (had issues with Ofelia not firing, hence I switched to Supercronic), I noticed issues on my server to day. I saw an hour old
error.flagfile, which normally creates an email, which is sent by a job, that's triggered by Supercronic on FrankenPHP container, if that container is healthy. But it was not triggered, because the container was not healthy. Although it seemed to be working fine.Current healthcheck calls
curl -f http://localhost:2019/metrics, and I did that. I got a flood of entries likeIt continued for like 20 seconds, when I broke it. I did a
docker inspect --format='{{json .State.Health}}' frankenphpand it last healthcheck exceeded 30s timeout. But that's not all: looks like this also resulted in OOM that killed mariadb container. I do not have hard proof of that, but OOM did kill mariadb, and the timing is mighty suspicious. I see there was also a #439, which was caused by a memory leak, but it does suggest that healthcheck can cause a spike in memory usage, and if it will trigger OOM-killer, mariadb will be selected since it uses the most of RAM.The entries are likely caused by https://github.com/mholt/caddy-ratelimit so I disabled metrics there for now (at least), but... Someone may need these metrics. Maybe not from this plugin, but from something else, that can also result in growth of the output from
/metricsendpoint. Maybe not to the same extent, as rate-limiter, but depending on what else is happening on the server even a small increase in size can trigger an OOM or slowness of output of the metrics and you get issues.As such, I am raising this ticket so that an alternative healthcheck is considered in the FrankenPHP Docker image. And since this is built into FrankenPHP image, I am raising it for FrankenPHP. It also may be worthwhile to have something that not only checks that Caddy is up, but that PHP interpreter is working, too. For Caddy side it looks like
/configendpoint is the most neutral, but my config is 122KBs, which is relatively large, too, so may not be ideal.Build Type
Docker (Debian Trixie)
Worker Mode
No
Operating System
GNU/Linux
CPU Architecture
x86_64
PHP configuration
phpinfo() output
Relevant log output
Relevant log output