Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,13 @@ Shared utilities live in `utils/`:

| Module | Purpose |
|---|---|
| `utils/logging.py` | Structured logging via `get_logger(name)` |
| `utils/logger.py` | Structured logging via `get_logger(name)` |
| `utils/telegram.py` | Telegram alert delivery |
| `utils/cache.py` | File-based key:value persistence |
| `utils/cache.py` | Key:value state persistence between runs (SQLite-backed) |
| `utils/web3_wrapper.py` | Web3 connection management (`ChainManager`) |
| `utils/config.py` | Environment config (`Config`) |
| `utils/formatting.py` | Number formatting helpers (`format_usd`, `format_token_amount`) |
| `utils/http.py` | HTTP request helper (`fetch_json`) |
| `utils/http_client.py` | HTTP helpers with retries (`fetch_json`, `request_with_retry`) |
| `utils/chains.py` | Chain enum and explorer URLs |
| `utils/abi.py` | ABI loader |
| `utils/gauntlet.py` | Gauntlet risk parameter helpers |
Expand Down Expand Up @@ -141,7 +141,7 @@ with client.batch_requests() as batch:

### Caching

Use `utils/cache.py` for persisting state between runs (e.g. last processed timestamp or proposal ID):
Use `utils/cache.py` for persisting state between runs (e.g. last processed timestamp or proposal ID). Values live in SQLite at `$CACHE_DIR/monitoring.db`; the filename argument is only used as a namespace:

```python
from utils.cache import cache_filename, get_last_value_for_key_from_file, write_last_value_to_file
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ Monitoring scripts for DeFi protocols to track key metrics and send alerts. Join

## Supported Protocols

Live status for every monitor is on [curation.yearn.fi/monitoring](https://curation.yearn.fi/monitoring/), generated from [`monitoring.yaml`](./monitoring.yaml).

- [3Jane](./protocols/3jane/README.md)
- [Aave V3](./protocols/aave/README.md)
- [APYUSD](./protocols/apyusd/README.md)
- [Bedrock uniBTC](./protocols/unibtc/README.md)
Expand All @@ -32,6 +35,7 @@ Monitoring scripts for DeFi protocols to track key metrics and send alerts. Join

- [Timelock Alerts](./protocols/timelock/README.md) — monitors OpenZeppelin `TimelockController` contracts for `CallScheduled` events across multiple protocols and sends Telegram alerts to protocol-specific channels.
- [Safe Multisigs](./protocols/safe/main.py) — monitors Safe multisig wallets for queued transactions across multiple protocols.
- [Stablecoins](./protocols/stables/) — depeg alerts via DeFiLlama and Chainlink oracle health checks (staleness, round health, peg and market divergence).

## Telegram Alerts

Expand Down
104 changes: 4 additions & 100 deletions api/README.md
Original file line number Diff line number Diff line change
@@ -1,109 +1,13 @@
# Monitoring Alerts API

Read-only HTTP API for persisted monitoring alerts and the list of monitored
protocols.
Read-only HTTP API for persisted monitoring alerts, monitored protocols, and
monitoring card metadata.

Run locally:

```sh
CACHE_DIR=/tmp/monitoring-cache uv run python -m api
```

Production runs as `monitoring-api.service` and binds to `127.0.0.1:8923`.
Public auth and rate limiting should be handled by the reverse proxy.

## Health

```sh
curl http://127.0.0.1:8923/healthz
```

```json
{"status":"ok"}
```

## Protocols

Returns enabled protocol objects from `automation/jobs.yaml`, grouped with the
tasks that monitor each protocol.

```sh
curl http://127.0.0.1:8923/v1/protocols
```

```json
{
"data": [
{
"name": "aave",
"tasks": [
{
"name": "aave",
"script": "protocols/aave/main.py",
"args": {},
"profile": "hourly",
"cron": "5 * * * *"
}
]
}
],
"count": 1
}
```

## Alerts

```sh
curl 'http://127.0.0.1:8923/v1/alerts?limit=10'
curl 'http://127.0.0.1:8923/v1/alerts?source=protocol&protocol=aave'
curl 'http://127.0.0.1:8923/v1/alerts?from=2026-06-11T00:00:00Z&to=2026-06-12T00:00:00Z'
```

Query parameters:

- `limit`: default `100`, max `500`.
- `cursor`: previous response `next_cursor`, for older rows.
- `from`: inclusive timestamp with timezone.
- `to`: exclusive timestamp with timezone.
- `since`: alias for `from`.
- `protocol`: exact protocol filter.
- `severity`: `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL`.
- `source`: `protocol`, `ops_error`, `crash`, or `automation_digest`.

Response:

```json
{
"data": [
{
"id": 5021,
"created_at": "2026-06-11T10:23:45.123456Z",
"source": "protocol",
"protocol": "aave",
"channel": "aave",
"severity": "LOW",
"message": "message text",
"plain_text": false,
"silent": true,
"delivery_status": "delivered",
"delivered_at": "2026-06-11T10:23:45.456789Z",
"delivery_error": null,
"metadata": {}
}
],
"next_cursor": "5021",
"limit": 100
}
```

Fetch the next page:

```sh
curl 'http://127.0.0.1:8923/v1/alerts?cursor=5021&limit=100'
```

Fetch one alert:

```sh
curl http://127.0.0.1:8923/v1/alerts/5021
```
See [`deploy/alerts-api.md`](../deploy/alerts-api.md) for endpoints, response
shapes, pagination, and production setup.
12 changes: 2 additions & 10 deletions deploy/emergency-dispatch-demo.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,7 @@ monitoring-scripts-py liquidity-monitoring

## Protocols with dispatch enabled

| Protocol | Telegram channel | Example alerts |
|---|---|---|
| infinifi | infinifi | Backing < 0.999, reserves < $15M |
| cap | cap | Withdrawable liquidity < $50M |
| ethena | ethena | USDe not fully backed |
| ethplus | rtoken | Coverage below threshold, StRSR rate drop |
| origin | pegs | Wrapped OETH redeem value drop, backing ratio drop |
| usdai | usdai | _(hook registered)_ |
| 3jane | 3jane | USD3/sUSD3 PPS decrease, junior buffer low, vault shutdown, protocol pause |
The authoritative list is `DISPATCHABLE_PROTOCOLS` in [`utils/dispatch.py`](../utils/dispatch.py). Only HIGH and CRITICAL alerts whose `protocol` is in that set trigger a dispatch.

## Safety mechanisms

Expand Down Expand Up @@ -99,4 +91,4 @@ curl -sS -X POST http://127.0.0.1:8080/webhook/emergency \
--data-binary "$body"
```

Replace `usdai` with any protocol from the list above.
Replace `usdai` with any protocol from `DISPATCHABLE_PROTOCOLS`.
54 changes: 6 additions & 48 deletions protocols/timelock/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ Monitors all timelock contract types (TimelockController, Aave, Compound, Lido,
1. Queries the Envio GraphQL indexer (`ENVIO_GRAPHQL_URL`) for new `TimelockEvent` events across all monitored timelocks (all types).
2. Groups events by `operationId` so batch operations (`scheduleBatch`) are sent as a single alert.
3. Routes each alert to the correct Telegram channel based on the protocol mapping.
4. Stores the latest processed `blockTimestamp` in `cache-id.txt` (key: `TIMELOCK_LAST_TS`) to avoid duplicate alerts between runs.
4. Stores the latest processed `blockTimestamp` in the SQLite state store (namespace `cache-id.txt`, key `TIMELOCK_LAST_TS`) to avoid duplicate alerts between runs.

The script runs hourly via the [monitoring runner](../automation/jobs.yaml).
The script runs hourly via the [monitoring runner](../../automation/jobs.yaml).

## Stale Ready Operations

Expand Down Expand Up @@ -85,51 +85,13 @@ The `TimelockEvent` type includes fields that vary by timelock type:
- **Lido**: `creator`, `metadata`, `operationId` (voteId)
- **Maple**: `delay` (absolute timestamp/delayedUntil), `operationId` (proposalId)

For complete field mapping details, see [`detils.md`](./detils.md).

## Monitored Timelocks

| Address | Chain | Protocol | Label |
|---------|-------|----------|-------|
| [0xd8236031d8279d82e615af2bfab5fc0127a329ab](https://etherscan.io/address/0xd8236031d8279d82e615af2bfab5fc0127a329ab) | Mainnet | CAP | CAP TimelockController |
| [0x5d8a7dc9405f08f14541ba918c1bf7eb2dace556](https://etherscan.io/address/0x5d8a7dc9405f08f14541ba918c1bf7eb2dace556) | Mainnet | RTOKEN | ETH+ Timelock |
| [0x055e84e7fe8955e2781010b866f10ef6e1e77e59](https://etherscan.io/address/0x055e84e7fe8955e2781010b866f10ef6e1e77e59) | Mainnet | LRT | Lombard TimeLock |
| [0x9f26d4c958fd811a1f59b01b86be7dffc9d20761](https://etherscan.io/address/0x9f26d4c958fd811a1f59b01b86be7dffc9d20761) | Mainnet | LRT | EtherFi Timelock |
| [0x49bd9989e31ad35b0a62c20be86335196a3135b1](https://etherscan.io/address/0x49bd9989e31ad35b0a62c20be86335196a3135b1) | Mainnet | LRT | KelpDAO(rsETH) Timelock |
| [0x3d18480cc32b6ab3b833dcabd80e76cfd41c48a9](https://etherscan.io/address/0x3d18480cc32b6ab3b833dcabd80e76cfd41c48a9) | Mainnet | INFINIFI | Infinifi Longtimelock |
| [0x4b174afbed7b98ba01f50e36109eee5e6d327c32](https://etherscan.io/address/0x4b174afbed7b98ba01f50e36109eee5e6d327c32) | Mainnet | INFINIFI | Infinifi Shorttimelock |
| [0x9aee0b04504cef83a65ac3f0e838d0593bcb2bc7](https://etherscan.io/address/0x9aee0b04504cef83a65ac3f0e838d0593bcb2bc7) | Mainnet | AAVE | Aave Governance V3 |
| [0x6d903f6003cca6255d85cca4d3b5e5146dc33925](https://etherscan.io/address/0x6d903f6003cca6255d85cca4d3b5e5146dc33925) | Mainnet | COMP | Compound Timelock |
| [0x2386dc45added673317ef068992f19421b481f4c](https://etherscan.io/address/0x2386dc45added673317ef068992f19421b481f4c) | Mainnet | FLUID | Fluid Timelock |
| [0x2e59a20f205bb85a89c53f1936454680651e618e](https://etherscan.io/address/0x2e59a20f205bb85a89c53f1936454680651e618e) | Mainnet | LIDO | Lido Timelock |
| [0x2efff88747eb5a3ff00d4d8d0f0800e306c0426b](https://etherscan.io/address/0x2efff88747eb5a3ff00d4d8d0f0800e306c0426b) | Mainnet | MAPLE | Maple GovernorTimelock |
| [0x1dccd4628d48a50c1a7adea3848bcc869f08f8c2](https://etherscan.io/address/0x1dccd4628d48a50c1a7adea3848bcc869f08f8c2) | Mainnet | 3JANE | 3Jane 24h TimelockController |
| [0x3d3c41419ab401cd25055e8f9421d7d96d887885](https://etherscan.io/address/0x3d3c41419ab401cd25055e8f9421d7d96d887885) | Mainnet | 3JANE | 3Jane 7d TimelockController |
| [0xf817cb3092179083c48c014688d98b72fb61464f](https://basescan.org/address/0xf817cb3092179083c48c014688d98b72fb61464f) | Base | LRT | superOETH Timelock |
| [0x88ba032be87d5ef1fbe87336b7090767f367bf73](https://etherscan.io/address/0x88ba032be87d5ef1fbe87336b7090767f367bf73) | Mainnet | YEARN | Yearn TimelockController |
| [0x88ba032be87d5ef1fbe87336b7090767f367bf73](https://basescan.org/address/0x88ba032be87d5ef1fbe87336b7090767f367bf73) | Base | YEARN | Yearn TimelockController |
| [0x88ba032be87d5ef1fbe87336b7090767f367bf73](https://arbiscan.io/address/0x88ba032be87d5ef1fbe87336b7090767f367bf73) | Arbitrum | YEARN | Yearn TimelockController |
| [0x88ba032be87d5ef1fbe87336b7090767f367bf73](https://polygonscan.com/address/0x88ba032be87d5ef1fbe87336b7090767f367bf73) | Polygon | YEARN | Yearn TimelockController |
| [0x88ba032be87d5ef1fbe87336b7090767f367bf73](https://katanascan.com/address/0x88ba032be87d5ef1fbe87336b7090767f367bf73) | Katana | YEARN | Yearn TimelockController |
The list of monitored timelocks (address, chain, protocol, label) is `TIMELOCK_LIST` in [`timelock_alerts.py`](./timelock_alerts.py).

## How to Add a New Timelock

1. **Add the address to the Envio indexer config.** The address must be indexed before this script can query events for it. Open the [Envio config.yaml](https://github.com/yearn/yearn-envio/blob/main/config.yaml), add the address under the correct chain's `TimelockController` contract list, and deploy the updated indexer.

2. **Add a `TimelockConfig` entry** in [`timelock_alerts.py`](./timelock_alerts.py) in the `TIMELOCK_LIST` list:

```python
(TimelockConfig("0xabcdef...lowercase_address", 1, "PROTOCOL_NAME", "Human Readable Label"),)
```

Parameters:
- **address**: Timelock contract address, **must be lowercase**.
- **chain_id**: Chain ID (`1` for Mainnet, `8453` for Base, etc.). Must match the network in the Envio config.
- **protocol**: Protocol identifier used for Telegram routing. Maps to `TELEGRAM_CHAT_ID_{PROTOCOL}` and `TELEGRAM_BOT_TOKEN_{PROTOCOL}` env variables. Falls back to `TELEGRAM_BOT_TOKEN_DEFAULT` if no protocol-specific bot token exists.
- **label**: Human-readable name shown in the alert message.

3. If the chain is new, make sure it exists in [`utils/chains.py`](../utils/chains.py) (`Chain` enum and `EXPLORER_URLS` dict).
4. If the protocol needs a dedicated Telegram channel, add `TELEGRAM_CHAT_ID_{PROTOCOL}` and optionally `TELEGRAM_BOT_TOKEN_{PROTOCOL}` to the environment and GitHub Actions secrets.
Follow [`SKILL.md`](./SKILL.md). It covers the Envio indexer change, the `TimelockConfig` entry, and the deployment order.

## Alert Format

Expand Down Expand Up @@ -185,7 +147,7 @@ For batch operations (`scheduleBatch`), all calls are included in a single messa
## Usage

```bash
uv run timelock/timelock_alerts.py
uv run protocols/timelock/timelock_alerts.py
```

Optional flags:
Expand All @@ -198,8 +160,4 @@ Optional flags:

## Caching

The script stores the latest processed `blockTimestamp` in `cache-id.txt` under key `TIMELOCK_LAST_TS`. This value is universal across chains (unlike block numbers) so a single cache entry covers all monitored timelocks. On the first run (or with `--no-cache`), it falls back to querying events from the last 12 hours.

## Schema Details

For comprehensive information about the unified `TimelockEvent` schema, including field mappings for all supported timelock types (TimelockController, Aave, Compound, Lido, Maple), see [`detils.md`](./detils.md).
The script stores the latest processed `blockTimestamp` via `utils/cache.py` (namespace `cache-id.txt`, key `TIMELOCK_LAST_TS`). This value is universal across chains (unlike block numbers) so a single cache entry covers all monitored timelocks. On the first run (or with `--no-cache`), it falls back to querying events from the last 12 hours.
11 changes: 6 additions & 5 deletions protocols/timelock/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,14 +117,16 @@ The indexer must be deployed and indexing events before the monitoring script ca

**Repository**: [yearn/monitoring-scripts-py](https://github.com/yearn/monitoring-scripts-py)

### 3.1 — Add `TimelockConfig` entry in `timelock/timelock_alerts.py`
### 3.1 — Add `TimelockConfig` entry in `protocols/timelock/timelock_alerts.py`

```python
(TimelockConfig("0xlowercase_address", chain_id, "PROTOCOL", "Human Label"),)
```

- Address **must be lowercase**
- Protocol name is used for Telegram routing (`TELEGRAM_CHAT_ID_{PROTOCOL}`)
- `chain_id` must match the network in the Envio config; a new chain must also exist in `utils/chains.py` (`Chain` enum and `EXPLORER_URLS`)
- Protocol name is used for Telegram routing (`TELEGRAM_CHAT_ID_{PROTOCOL}`, optional `TELEGRAM_BOT_TOKEN_{PROTOCOL}`, falling back to `TELEGRAM_BOT_TOKEN_DEFAULT`)
- If the routing key is not a website page key, map it in `ALERT_HISTORY_PROTOCOLS`; `tests/test_alert_protocol_keys.py` fails otherwise

### 3.2 — Handle delay format (only for new contract types)

Expand All @@ -151,8 +153,7 @@ elif timelock_type in ("TimelockController", "Compound", "Puffer", "NewProtocol"

### 3.4 — Update documentation

- Add the new timelock to the table in `timelock/README.md`
- Update the `timelockType` list in the schema fields section
- For a new contract type, update the `timelockType` list in the schema fields section of `protocols/timelock/README.md`
- If using a new protocol, add `TELEGRAM_CHAT_ID_{PROTOCOL}` to the monitoring env file (`/etc/monitoring/.env` on the VPS) and document it in `.env.example`

## Deployment Order
Expand All @@ -166,7 +167,7 @@ elif timelock_type in ("TimelockController", "Compound", "Puffer", "NewProtocol"
After both are deployed, run manually to verify:

```bash
uv run timelock/timelock_alerts.py --no-cache --since-seconds 604800 --log-level DEBUG
uv run protocols/timelock/timelock_alerts.py --no-cache --since-seconds 604800 --log-level DEBUG
```

This looks back 7 days to catch any recent events from the new timelock.
Loading
Loading