Skip to content

fix(telemetry): cap auto-created variables per project#2

Open
Salah-XD wants to merge 5 commits into
decoded-cipher:masterfrom
Salah-XD:fix/variable-cardinality-cap
Open

fix(telemetry): cap auto-created variables per project#2
Salah-XD wants to merge 5 commits into
decoded-cipher:masterfrom
Salah-XD:fix/variable-cardinality-cap

Conversation

@Salah-XD
Copy link
Copy Markdown

@Salah-XD Salah-XD commented Jun 1, 2026

Heads up: I know the repo is pre-alpha and contributions may not be open yet — totally fine to close this if it's premature; happy to move it to an issue instead.

What

The telemetry ingress auto-creates a project_variables row on first sight of each metric key. The per-request caps (MAX_POINTS, MAX_KEY_LEN, …) bound a single POST, but nothing bounded the total number of distinct variables a project accumulates across requests. A buggy or compromised device with a valid project token can therefore grow the table without bound — which also contradicts the existing comment that says these caps "stop … exploding variable cardinality" (they only do so per request).

Change

  • Add MAX_VARIABLES_PER_PROJECT = 1000.
  • In upsertVariables, split due keys into existing vs. new:
    • Existing keys always refresh last_seen (unchanged behavior).
    • New keys are created only while the project is under the cap.
  • Best-effort guardrail: stays in waitUntil (never blocks the device's 204); a small overshoot under concurrent upserts is acceptable since it bounds growth, not an exact invariant.
  • Reworded the comment so it accurately distinguishes the per-request caps from the aggregate cap.

No behavior change for normal projects (well under 1000 variables).

Testing

Verified against a local wrangler dev instance:

  • Normal first-sight auto-create still works; existing-key last_seen refresh still works, including when the project is at the cap.
  • Sending 1201 distinct keys yields exactly 1000 project_variables rows — the cap engages.
  • typecheck passes for @nodrix/worker and @nodrix/web.

Separate note (not in this PR)

While testing I hit what looks like an unrelated bug: POST /v1/telemetry accepts up to MAX_POINTS = 200 metrics, but ProjectDO.ingest builds single multi-row SQL statements (the IN (…) SELECT + multi-row UPSERT/INSERT) that exceed the Durable Object SQLite bound-variable limit well below 200 — a valid ~200-key batch returns 500. Happy to open a separate issue/PR to chunk those writes if useful.

decoded-cipher and others added 5 commits June 1, 2026 20:18
…or automation graph editing

- Added BlockNode.vue to represent individual nodes in the automation graph with visual handles for connections.
- Implemented NodeInspector.vue for detailed configuration of selected nodes, supporting various input types including dropdowns and text areas.
- Updated automation-catalog.ts to reflect changes in trigger types, renaming 'scene' to 'manual'.
- Created graph-edit.ts for managing conversions between automation graphs and Vue Flow structures.
- Enhanced project store to support graph data in automation creation and updates.
- Modified types.ts to include AutomationGraph type for better type safety.
- Developed graph-build.ts for server-side graph handling, ensuring backward compatibility with legacy trigger configurations.
- Updated routes and service layers to accommodate new graph structure in automation creation and updates.
- Adjusted database migrations to include graph column in automations table.
- Refactored engine logic to prioritize graph column over legacy trigger/action columns for automation execution.
- Updated tests to validate new manual trigger behavior and ensure proper graph handling.
The per-request ingest caps bound a single POST, but the auto-create path
inserted a project_variables row for every new key with no aggregate limit.
A buggy or compromised device with a valid project token could therefore
grow the table without bound — contradicting the comment that says these
caps "stop ... exploding variable cardinality."

Add MAX_VARIABLES_PER_PROJECT (1000). Existing keys still refresh last_seen
unconditionally; new keys are created only while the project is under the
cap. Best-effort guardrail (a small overshoot under concurrent waitUntil
upserts is acceptable), kept off the device's response path.
@decoded-cipher
Copy link
Copy Markdown
Owner

Thanks for this, and for the clear writeup — your reasoning is right: the per-request caps never bounded the total variable count, so a valid token could grow project_variables without bound.

Heads up that the project is still pre-stable and under active development — even the schema changes between commits right now, so branches go stale fast and this one had too many conflicts to merge cleanly. Instead I've applied your change directly on master, with two tweaks:

  • Set the cap to 250 instead of 1000 — real projects run in the tens of keys, so this catches a runaway device much sooner while staying well clear of legit use.
  • Also fixed the ProjectDO.ingest issue from your note: confirmed DO SQLite/D1 cap 100 bound params per query, so I chunked the multi-row inserts and IN(...) lookups to stay under it. Good catch.

Going to leave this PR open for reference, but the substance is shipping and credited to you here.

Because things move this fast, it'd be a big help to open anything else you spot as issues rather than PRs — much easier to track and discuss until things settle. We'll be launching the initial stable version very soon. Thanks again!

@decoded-cipher decoded-cipher added the bug Something isn't working label Jun 1, 2026
@Salah-XD
Copy link
Copy Markdown
Author

Salah-XD commented Jun 1, 2026

Awesome, really glad it was useful — and thanks for the credit! 🙏

250 makes total sense; real projects sit in the tens of keys, so that catches a runaway device way sooner. And nice that the ingest chunking landed too.

Totally get the issues-over-PRs thing while the schema's still moving — way easier to track, I'll go that route from here. I'll keep poking around and file anything I spot as issues. Pumped for the stable release! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants