Mqtt endpoints

docs/mqtt/README.md

@@ -0,0 +1,125 @@
+---
+protocol: mqtt
+title: MQTT Test Broker
+desc: A one-command local Mosquitto broker plus a deterministic test publisher for exercising API Dash's MQTT client (QoS, retained, wildcards, LWT, MQTT v5).
+path: mqtt
+---
+
+This is a **local** MQTT test rig for API Dash's MQTT client. MQTT is not an
+HTTP route, so unlike the WebSocket endpoints it cannot be hosted inside the
+FastAPI app. Instead we ship a Dockerized [Eclipse Mosquitto](https://mosquitto.org/)
+2.x broker plus a small `paho-mqtt` publisher that generates deterministic,
+subscribable traffic -- the MQTT analogue of the WebSocket echo / ticker /
+heartbeat routes.
+
+Running a real, spec-compliant broker is the only way to genuinely test QoS 1/2
+handshakes, retained messages, persistent sessions, Last Will and MQTT v5
+properties.
+
+## Why this is different from the other test endpoints (and why Docker)
+
+**Before — HTTP / WebSocket / SSE:** every test endpoint is a FastAPI route
+inside this app, served over its single HTTP(S) port. That works because HTTP,
+WebSocket (an HTTP upgrade) and SSE all ride on HTTP — and Azure App Service
+exposes exactly one HTTP/HTTPS port, which is all those need. No extra process,
+no extra infra.
+
+**MQTT is fundamentally different.** It's a stateful, connection-oriented
+**pub/sub** protocol over its **own TCP ports** (1883 plaintext, 8883 TLS,
+8083/8084 for MQTT-over-WebSocket), and it needs a **real broker** to hold
+sessions, subscriptions, retained messages and QoS state. It is **not** HTTP, so:
+
+- It **can't be a FastAPI route** like `/ws/echo` — there is no HTTP request to
+  answer; the client keeps a long-lived MQTT connection open to a broker.
+- Azure App Service **can't host a broker** either — it forwards only one
+  HTTP/HTTPS port and cannot open raw TCP 1883/8883. (The WS endpoints work there
+  *only* because WS is an HTTP upgrade over that same port.)
+- Faking broker semantics (QoS 1/2 handshakes, retained, sessions, LWT, v5
+  properties) as a FastAPI endpoint would mean re-implementing a broker, with
+  poor fidelity — not worth it.
+
+**So to test locally we run a real broker (Eclipse Mosquitto) + a publisher via
+Docker** — one command, full fidelity, no cloud needed. That is the "extra
+hassle": it buys you a spec-compliant broker to point the client at, instead of a
+fake HTTP shim.
+
+**What about production?** A shared, always-on hosted MQTT test endpoint (the
+equivalent of `api.apidash.dev/ws/echo`) is a **separate, maintainer-owned
+decision** — it needs a broker somewhere *outside* App Service (a managed broker
+like HiveMQ/EMQX Cloud, or a self-hosted VM). **This change intentionally covers
+local testing only;** the Azure/production infra is wired up separately.
+
+## What this adds
+
+- `mqtt/docker-compose.yml`, `mqtt/Dockerfile`, `mqtt/mosquitto/mosquitto.conf` —
+  the Dockerized broker + publisher service.
+- `mqtt/publisher.py` — the deterministic test-topic publisher (ticker, retained,
+  echo request/response incl. v5 `response_topic`, Last Will).
+- `docs/mqtt/` — this README plus per-scenario pages.
+- `tests/mqtt/test_mqtt.py` — broker round-trip tests that skip when no broker is
+  running (CI-safe).
+- `paho-mqtt` added to `requirements-dev.txt` only — the production app
+  (`requirements.txt`) is unchanged and pulls in no MQTT dependency.
+
+## Run it
+
+From the repository root:
+
+```
+docker compose -f mqtt/docker-compose.yml up
+```
+
+This starts two services:
+
+| Service | What it is |
+| ----------- | ----------- |
+| `broker` | Eclipse Mosquitto 2.x (MQTT v3.1 / v3.1.1 / v5) |
+| `publisher` | A `paho-mqtt` client that emits the test scenarios below |
+
+Stop it with `Ctrl-C`, or run detached with `-d` and stop via
+`docker compose -f mqtt/docker-compose.yml down`.
+
+## Endpoints
+
+| Transport | URL |
+| ----------- | ----------- |
+| MQTT over TCP | `mqtt://localhost:1883` |
+| MQTT over WebSocket | `ws://localhost:9001` |
+
+Anonymous access is allowed by default (no username/password needed). See
+[Auth](auth.md) to test username/password.
+
+## Point API Dash at it
+
+In API Dash's MQTT client, create a connection to:
+
+- **Host:** `localhost`
+- **Port:** `1883` (TCP) or `9001` (WebSocket)
+- **Protocol:** MQTT v3.1.1 or v5 (both work)
+- **Auth:** none (anonymous), unless you enabled it
+
+Then subscribe to `apidash/test/#` to see every test topic at once.
+
+## Test topics
+
+| Topic | Scenario | Doc |
+| ----------- | ----------- | ----------- |
+| `apidash/test/ticker` | JSON message every 2s (QoS 0) | [ticker](ticker.md) |
+| `apidash/test/retained` | Retained message, delivered on subscribe | [retained](retained.md) |
+| `apidash/test/echo/request` -> `apidash/test/echo/response` | Request/response echo | [echo](echo.md) |
+| `apidash/test/status` | `online` / `offline` via Last Will (retained) | [lwt](lwt.md) |
+
+Further reference pages: [wildcards](wildcards.md), [qos](qos.md),
+[auth](auth.md), [MQTT v5 properties](v5_properties.md),
+[WebSocket transport](websocket.md).
+
+## Tests
+
+`tests/mqtt/test_mqtt.py` exercises the broker with `paho-mqtt`. The tests skip
+gracefully if no broker is reachable, so CI without a broker still passes:
+
+```
+docker compose -f mqtt/docker-compose.yml up -d
+pip install -r requirements-dev.txt
+pytest tests/mqtt/test_mqtt.py
+```

docs/mqtt/auth.md

@@ -0,0 +1,54 @@
+---
+protocol: mqtt
+title: MQTT Username/Password Auth
+desc: Anonymous by default for frictionless testing; optionally enable a password file to test username/password authentication.
+path: mqtt/auth
+---
+
+By default the broker allows **anonymous** connections so local testing is
+frictionless (`allow_anonymous true` in `mqtt/mosquitto/mosquitto.conf`). You
+can switch on username/password auth to test API Dash's credential handling.
+
+## Enable auth
+
+1. Create a password file inside the broker container:
+
+   ```
+   docker compose -f mqtt/docker-compose.yml exec broker \
+     mosquitto_passwd -c -b /mosquitto/config/passwd testuser testpass
+   ```
+
+2. In `mqtt/mosquitto/mosquitto.conf`, uncomment the `password_file` line and
+   set `allow_anonymous false` (a commented-out template block is already
+   there).
+
+3. Recreate the broker:
+
+   ```
+   docker compose -f mqtt/docker-compose.yml up -d --force-recreate broker
+   ```
+
+The publisher itself reads optional `MQTT_USER` / `MQTT_PASS` env vars, so set
+those in `docker-compose.yml` if you enable auth and want the publisher to keep
+working.
+
+## Behavior
+
+| Credentials | Result |
+| ----------- | ----------- |
+| `allow_anonymous true` (default) | Any client may connect without credentials |
+| `allow_anonymous false` + valid `testuser` / `testpass` | Connection accepted |
+| `allow_anonymous false` + missing/wrong credentials | Broker refuses the connection (CONNACK "not authorized") |
+
+## Sample Usage
+
+### Python (`paho-mqtt`)
+
+```python
+import paho.mqtt.client as mqtt
+
+c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv5)
+c.username_pw_set("testuser", "testpass")
+c.connect("localhost", 1883)
+c.loop_forever()
+```

docs/mqtt/echo.md

@@ -0,0 +1,47 @@
+---
+protocol: mqtt
+title: MQTT Echo
+desc: Publish to apidash/test/echo/request and the publisher mirrors the payload back on apidash/test/echo/response (honouring the MQTT v5 response topic if present).
+path: mqtt/echo
+---
+
+The publisher subscribes to `apidash/test/echo/request`. Every message it
+receives is republished unchanged to `apidash/test/echo/response`. This is the
+MQTT analogue of the WebSocket echo route and is useful for testing a full
+publish -> subscribe round-trip.
+
+If the incoming PUBLISH carries an **MQTT v5 `response_topic` property**, the
+reply is sent there instead, and any `correlation_data` is echoed back. See
+[MQTT v5 properties](v5_properties.md).
+
+## Topics
+
+| Topic | Direction |
+| ----------- | ----------- |
+| `apidash/test/echo/request` | You publish here |
+| `apidash/test/echo/response` | Publisher replies here (default) |
+
+## Behavior
+
+| Event | Result |
+| ----------- | ----------- |
+| Publish to `.../echo/request` | Payload is republished to `.../echo/response`, unchanged |
+| Request carries v5 `response_topic` | Reply is sent to that topic instead |
+| Request carries v5 `correlation_data` | The same `correlation_data` is set on the reply |
+
+## Sample Usage
+
+### Python (`paho-mqtt`)
+
+```python
+import paho.mqtt.client as mqtt
+
+c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv5)
+c.on_message = lambda cl, ud, msg: print("reply:", msg.payload.decode())
+c.connect("localhost", 1883)
+c.subscribe("apidash/test/echo/response", qos=1)
+c.loop_start()
+
+c.publish("apidash/test/echo/request", payload="ping", qos=1)
+# reply: ping
+```

docs/mqtt/lwt.md

@@ -0,0 +1,57 @@
+---
+protocol: mqtt
+title: MQTT Last Will (LWT)
+desc: The publisher registers a Last Will on apidash/test/status; killing it makes the broker publish "offline" automatically.
+path: mqtt/lwt
+---
+
+A **Last Will and Testament (LWT)** is a message the client registers at connect
+time; the broker publishes it automatically if the client disconnects
+**ungracefully** (crash, network drop, kill). It is how MQTT signals presence.
+
+The publisher registers a Will on `apidash/test/status` with payload `offline`
+(retained), and publishes `online` (retained) once connected.
+
+## Topic
+
+```
+apidash/test/status
+```
+
+## Behavior
+
+| Event | Value on `apidash/test/status` |
+| ----------- | ----------- |
+| Publisher connected | `online` (retained) |
+| Publisher killed / crashes / loses connection | `offline` (retained, published by the broker as the Will) |
+
+Because the value is retained, a client that subscribes at any time
+immediately sees the current status.
+
+## Try it
+
+1. Subscribe to `apidash/test/status` -- you see `online`.
+2. Ungracefully stop the publisher so it cannot send a clean disconnect:
+
+   ```
+   docker compose -f mqtt/docker-compose.yml kill publisher
+   ```
+
+3. The broker publishes the Will -- you now see `offline`.
+
+## Sample Usage
+
+### Python (`paho-mqtt`)
+
+```python
+import paho.mqtt.client as mqtt
+
+c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv5)
+c.on_message = lambda cl, ud, msg: print("status:", msg.payload.decode())
+c.connect("localhost", 1883)
+c.subscribe("apidash/test/status", qos=1)
+c.loop_forever()
+# status: online
+# (after `docker compose -f mqtt/docker-compose.yml kill publisher`)
+# status: offline
+```

docs/mqtt/qos.md

@@ -0,0 +1,47 @@
+---
+protocol: mqtt
+title: MQTT Quality of Service (QoS)
+desc: Test QoS 0, 1 and 2 delivery guarantees against the broker; the effective QoS is the minimum of the publish and subscribe QoS.
+path: mqtt/qos
+---
+
+MQTT defines three Quality of Service (QoS) levels for message delivery. A real
+broker is required to exercise the QoS 1 and QoS 2 handshakes, which is one of
+the main reasons this rig ships a Dockerized Mosquitto.
+
+## Levels
+
+| QoS | Guarantee | Handshake |
+| ----------- | ----------- | ----------- |
+| 0 | At most once ("fire and forget") | none |
+| 1 | At least once (may duplicate) | PUBLISH / PUBACK |
+| 2 | Exactly once | PUBLISH / PUBREC / PUBREL / PUBCOMP |
+
+The QoS a subscriber actually receives is the **minimum** of the publisher's
+QoS and the subscription's QoS. For example, a QoS 2 publish delivered to a QoS
+1 subscription is received at QoS 1.
+
+## Try it with the test topics
+
+- The [ticker](ticker.md) publishes at **QoS 0** -- subscribe at any QoS.
+- The [echo](echo.md) and [retained](retained.md) topics use **QoS 1**.
+- To exercise **QoS 2**, publish to your own topic (e.g.
+  `apidash/test/scratch`) at QoS 2 and subscribe at QoS 2.
+
+## Sample Usage
+
+### Python (`paho-mqtt`)
+
+```python
+import paho.mqtt.client as mqtt
+
+c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv5)
+c.on_message = lambda cl, ud, msg: print("qos", msg.qos, msg.payload.decode())
+c.connect("localhost", 1883)
+c.subscribe("apidash/test/scratch", qos=2)
+c.loop_start()
+
+info = c.publish("apidash/test/scratch", payload="exactly once", qos=2)
+info.wait_for_publish(5)  # completes only after the full QoS 2 handshake
+# qos 2 exactly once
+```

docs/mqtt/retained.md

@@ -0,0 +1,46 @@
+---
+protocol: mqtt
+title: MQTT Retained Message
+desc: The publisher stores a retained message on apidash/test/retained so any client that subscribes later receives it immediately.
+path: mqtt/retained
+---
+
+At startup (and on every reconnect) the publisher stores a **retained** message
+on `apidash/test/retained`. The broker keeps the last retained message for a
+topic and delivers it immediately to any client that subscribes **later** -- a
+core MQTT feature that has no WebSocket equivalent.
+
+## Topic
+
+```
+apidash/test/retained
+```
+
+## Behavior
+
+| Event | Result |
+| ----------- | ----------- |
+| Subscribe (any time) | You immediately receive the retained JSON payload, with the retain flag set |
+| Publisher restarts | The retained message is refreshed with a new timestamp |
+
+The payload is a JSON object with a `note` and a `ts` (ISO-8601) field.
+
+To clear a retained message on any topic, publish an **empty** payload to it
+with the retain flag set.
+
+## Sample Usage
+
+### Python (`paho-mqtt`)
+
+```python
+import paho.mqtt.client as mqtt
+
+c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv5)
+c.on_message = lambda cl, ud, msg: print("retain=%s" % msg.retain, msg.payload.decode())
+c.connect("localhost", 1883)
+# Even though this runs long after the publisher started, the message arrives
+# immediately on subscribe.
+c.subscribe("apidash/test/retained", qos=1)
+c.loop_forever()
+# retain=True {"note": "This is a retained message. ...", "ts": "..."}
+```