CI hardening, accurate worker docs, announcement banner, 404 page

mariaob1201 · claude · mariaob1201 · commit c72f9c062dcc · 2026-06-06T03:22:17.000-05:00
- deploy.yml: bump actions to Node 24 versions (checkout v6,
  setup-python v6, configure-pages v6, upload-pages-artifact v5,
  deploy-pages v5); add PR build job (build + linkcheck, no deploy);
  fail the build on Sphinx warnings (--warningiserror)
- worker.md: rewrite to match the real OpenLambda API (/run/&lt;name&gt;,
  ol worker init/up/down/status, SOCK sandbox, HTTP/Kafka/cron
  triggers) — the old page documented a nonexistent API
- Add announcement banner and a branded 404 page (sphinx-notfound-page,
  absolute asset paths so it styles at any depth)
- Pin sphinx-notfound-page

Co-Authored-By: Claude Opus 4.8 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -7,6 +7,11 @@ on:
       - 'README.md'
       - 'CONTRIBUTING.md'
       - 'LICENSE'
+  pull_request:
+    paths-ignore:
+      - 'README.md'
+      - 'CONTRIBUTING.md'
+      - 'LICENSE'
 
 permissions:
   contents: read
@@ -18,34 +23,45 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
+  build:
     runs-on: ubuntu-latest
-
     steps:
-      - uses: actions/checkout@v4
-
-      - name: Setup Pages
-        uses: actions/configure-pages@v5
+      - uses: actions/checkout@v6
 
       - name: Setup Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: '3.11'
 
       - name: Install dependencies
         run: pip install -r requirements.txt
 
       - name: Build Jupyter Book
-        run: jupyter-book build .
+        run: jupyter-book build . --warningiserror
+
+      - name: Check links
+        # Informational: external links can be flaky, so this never blocks the build.
+        continue-on-error: true
+        run: jupyter-book build . --builder linkcheck
+
+      - name: Setup Pages
+        if: github.event_name == 'push'
+        uses: actions/configure-pages@v6
 
       - name: Upload artifact
-        uses: actions/upload-pages-artifact@v3
+        if: github.event_name == 'push'
+        uses: actions/upload-pages-artifact@v5
         with:
           path: _build/html
 
+  deploy:
+    needs: build
+    if: github.event_name == 'push' && github.ref == 'refs/heads/master'
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v4
+        uses: actions/deploy-pages@v5
diff --git a/404.md b/404.md
@@ -0,0 +1,21 @@
+---
+orphan: true
+myst:
+  html_meta:
+    description: "Page not found."
+    robots: "noindex"
+---
+
+# 404 — Page Not Found
+
+Sorry, the page you were looking for doesn't exist or may have moved.
+
+Try one of these instead:
+
+- [Home](index.md)
+- [Worker](worker.md)
+- [Applications](applications/index.md)
+- [Blog](blog/index.md)
+
+If you followed a link here, please
+[open an issue](https://github.com/open-lambda/open-lambda.github.io/issues) so we can fix it.
diff --git a/_config.yml b/_config.yml
@@ -49,6 +49,7 @@ sphinx:
   extra_extensions:
     - ablog
     - sphinxext.opengraph
+    - notfound.extension
   config:
     html_theme: "sphinx_book_theme"
     html_show_sourcelink: false
@@ -59,10 +60,15 @@ sphinx:
     ogp_site_name: "OpenLambda"
     ogp_image: "https://open-lambda.github.io/_static/open-lambda.png"
     ogp_use_first_image: true
+
+    # ---- Custom 404 page ----
+    # This is a root (user/org) Pages site, so served from "/".
+    notfound_urls_prefix: "/"
     html_theme_options:
       use_download_button: false
       navigation_depth: 2
       show_navbar_depth: 1
+      announcement: "🚧 OpenLambda is a research project under active development — <a href='https://github.com/open-lambda/open-lambda'>follow along on GitHub</a>."
       logo:
         text: "OpenLambda"
 
diff --git a/requirements.txt b/requirements.txt
@@ -7,3 +7,4 @@ myst-parser==3.0.1
 sphinx-copybutton==0.5.2
 ablog==0.11.13
 sphinxext-opengraph==0.9.1
+sphinx-notfound-page==1.0.4
diff --git a/worker.md b/worker.md
@@ -1,69 +1,113 @@
 ---
 myst:
   html_meta:
-    description: "The OpenLambda worker — the core node component that handles HTTP requests, manages containers, and scales horizontally."
-    keywords: "OpenLambda, worker, serverless, Linux containers, HTTP, configuration"
+    description: "The OpenLambda worker — the core node component that runs lambdas in SOCK sandboxes and serves HTTP, Kafka, and cron triggers."
+    keywords: "OpenLambda, worker, serverless, SOCK, sandbox, HTTP, triggers, ol"
 ---
 
 # Worker
 
 The OpenLambda **worker** is the core server-side component of a node. It listens for
-incoming HTTP requests, manages container lifecycle, and returns responses to callers.
+incoming requests, runs lambdas inside isolated sandboxes, and returns responses to callers.
+The worker is part of the `ol` binary.
 
-## Overview
+```{note}
+This page is a high-level summary. The authoritative, version-tracked documentation lives in
+the main repository under
+[`docs/worker`](https://github.com/open-lambda/open-lambda/tree/main/docs/worker).
+```
 
-Each worker is a standalone Go binary that exposes a single HTTP endpoint:
+## Architecture
 
-```
-POST /runLambda/<lambda-name>
-```
+A worker is built from three layers, from the bottom up:
+
+1. **Sandbox** — isolates lambdas from one another. This layer is pluggable; the main
+   implementation today is **SOCK** (serverless-optimized containers), described in
+   [Oakes et al., ATC '18](https://www.usenix.org/conference/atc18/presentation/oakes).
+   Early versions used Docker.
+2. **Lambda** — a *lambda instance* is a robust virtual container backed by zero or one
+   sandboxes; a *lambda function* routes incoming requests to instances and decides how many
+   instances to provision based on load.
+3. **Event** — the trigger sources that invoke lambdas.
 
-When a request arrives the worker:
+## Triggers
 
-1. Checks whether the lambda's container image is already present on the node; if not, pulls it from the registry.
-2. Starts a Linux container from the image.
-3. Passes the request payload to the lambda function running inside the container.
-4. Waits for the function to return a result, then forwards that result back to the caller.
-5. Optionally keeps the container warm for a short period to reduce cold-start latency on subsequent calls.
+The worker supports three kinds of triggers:
 
-## Configuration
+- **HTTP** — a request to `http(s)://<worker-addr>:<port>/run/<lambda-name>` invokes a lambda
+  directly.
+- **Kafka** — a lambda can be configured to consume messages from a Kafka topic.
+- **Cron** — a lambda can be invoked on a schedule.
 
-The worker is configured via a JSON file (default `config.json`) in the working directory.
-Key fields:
+## Building
+
+OpenLambda is actively tested on **Ubuntu 24.04 LTS**, requires **cgroups v2**, and relies on
+operations that need root privilege. After installing the
+[dependencies](https://github.com/open-lambda/open-lambda/blob/main/docs/worker/getting-started.md#dependencies),
+build the Python-only ("min") deployment:
+
+```bash
+make ol imgs/ol-min
+```
 
-| Field | Description | Default |
-|---|---|---|
-| `worker_port` | Port the HTTP server listens on | `8080` |
-| `registry` | URL of the lambda registry | `""` |
-| `sandbox` | Container backend (`docker` or `sock`) | `docker` |
-| `log_output` | Where to write logs (`stdout` or a file path) | `stdout` |
+## Running a worker
 
-## Starting the Worker
+The `ol worker` subcommands manage a worker's lifecycle (run `./ol worker --help` for the
+full list):
 
 ```bash
-# From the repo root after building:
-./bin/worker --config config.json
+# Create a worker directory with a config.json and base image
+./ol worker init -i ol-min
+
+# Start the worker (use -d to run in the background)
+./ol worker up -d
+
+# Check status
+./ol worker status
+
+# Stop the worker
+./ol worker down
 ```
 
-The worker prints its listening address on startup. You can verify it is running with:
+`init` creates a worker directory (named `default-ol` by default) containing the worker
+configuration (`config.json`), the read-only base image shared by all lambda instances, and
+other resources. Per-worker settings such as memory limits are edited in `config.json`:
+
+```json
+"limits": {
+    "mem_mb": 512
+}
+```
+
+## Deploying a lambda
+
+Functions can be installed directly from a Git repository, with dependencies pinned via a
+`requirements.txt` and behavior configured via an `ol.yaml` file:
 
 ```bash
-curl -w "\n" localhost:8080/status
+./ol admin install -c ol.yaml -r requirements.txt https://github.com/<org>/<repo>.git
+
+# Invoke it
+curl http://localhost:5000/run/<lambda-name>/
 ```
 
-## Deploying Multiple Workers
+See the [Applications](applications/index.md) section and the
+[Ag Forecasting case study](blog/post/2026-05-18-ag-forecasting-case-study.md) for a complete
+worked example, including `ol.yaml` configuration and ASGI entry points.
+
+## Scaling out
 
-Workers are stateless with respect to routing — each one operates independently. To scale
-horizontally, start one worker process per node and place a standard HTTP load balancer
-(Nginx, HAProxy, or similar) in front of them. No coordination between workers is required.
+Workers are independent and require no coordination, so you can scale horizontally by running
+one worker per node behind a standard HTTP load balancer (Nginx, HAProxy, or similar).
 
 ```{note}
-A centralized **boss** component for cluster-wide management is currently under development.
-Until then, manual deployment behind a load balancer is the recommended approach for
-multi-node setups.
+A centralized **boss** component for cluster-wide management is under development. Until then,
+manual deployment behind a load balancer is the recommended approach for multi-node setups.
 ```
 
-## Further Reading
+## Further reading
 
-- [Quickstart guide](https://github.com/open-lambda/open-lambda/blob/main/docs/quickstart.md) — get a single worker running locally in minutes
-- [SOCK: Rapid Task Provisioning with Serverless-Optimized Containers](https://www.usenix.org/conference/atc18/presentation/oakes) — the research paper describing the container backend.
+- [Getting started](https://github.com/open-lambda/open-lambda/blob/main/docs/worker/getting-started.md) — build, deploy, and run your first lambda
+- [Lambda configuration](https://github.com/open-lambda/open-lambda/blob/main/docs/worker/lambda-config.md) — the `ol.yaml` format
+- [Deploying applications](https://github.com/open-lambda/open-lambda/blob/main/docs/worker/apps.md) — example app walkthroughs
+- [SOCK: Rapid Task Provisioning with Serverless-Optimized Containers](https://www.usenix.org/conference/atc18/presentation/oakes) — the sandbox research paper
