fix(telemetry): cap auto-created variables per project#2
Conversation
…ctor scheduling logic
…or automation graph editing - Added BlockNode.vue to represent individual nodes in the automation graph with visual handles for connections. - Implemented NodeInspector.vue for detailed configuration of selected nodes, supporting various input types including dropdowns and text areas. - Updated automation-catalog.ts to reflect changes in trigger types, renaming 'scene' to 'manual'. - Created graph-edit.ts for managing conversions between automation graphs and Vue Flow structures. - Enhanced project store to support graph data in automation creation and updates. - Modified types.ts to include AutomationGraph type for better type safety. - Developed graph-build.ts for server-side graph handling, ensuring backward compatibility with legacy trigger configurations. - Updated routes and service layers to accommodate new graph structure in automation creation and updates. - Adjusted database migrations to include graph column in automations table. - Refactored engine logic to prioritize graph column over legacy trigger/action columns for automation execution. - Updated tests to validate new manual trigger behavior and ensure proper graph handling.
The per-request ingest caps bound a single POST, but the auto-create path inserted a project_variables row for every new key with no aggregate limit. A buggy or compromised device with a valid project token could therefore grow the table without bound — contradicting the comment that says these caps "stop ... exploding variable cardinality." Add MAX_VARIABLES_PER_PROJECT (1000). Existing keys still refresh last_seen unconditionally; new keys are created only while the project is under the cap. Best-effort guardrail (a small overshoot under concurrent waitUntil upserts is acceptable), kept off the device's response path.
|
Thanks for this, and for the clear writeup — your reasoning is right: the per-request caps never bounded the total variable count, so a valid token could grow Heads up that the project is still pre-stable and under active development — even the schema changes between commits right now, so branches go stale fast and this one had too many conflicts to merge cleanly. Instead I've applied your change directly on
Going to leave this PR open for reference, but the substance is shipping and credited to you here. Because things move this fast, it'd be a big help to open anything else you spot as issues rather than PRs — much easier to track and discuss until things settle. We'll be launching the initial stable version very soon. Thanks again! |
|
Awesome, really glad it was useful — and thanks for the credit! 🙏 250 makes total sense; real projects sit in the tens of keys, so that catches a runaway device way sooner. And nice that the ingest chunking landed too. Totally get the issues-over-PRs thing while the schema's still moving — way easier to track, I'll go that route from here. I'll keep poking around and file anything I spot as issues. Pumped for the stable release! 🎉 |
Heads up: I know the repo is pre-alpha and contributions may not be open yet — totally fine to close this if it's premature; happy to move it to an issue instead.
What
The telemetry ingress auto-creates a
project_variablesrow on first sight of each metric key. The per-request caps (MAX_POINTS,MAX_KEY_LEN, …) bound a single POST, but nothing bounded the total number of distinct variables a project accumulates across requests. A buggy or compromised device with a valid project token can therefore grow the table without bound — which also contradicts the existing comment that says these caps "stop … exploding variable cardinality" (they only do so per request).Change
MAX_VARIABLES_PER_PROJECT = 1000.upsertVariables, split due keys into existing vs. new:last_seen(unchanged behavior).waitUntil(never blocks the device's 204); a small overshoot under concurrent upserts is acceptable since it bounds growth, not an exact invariant.No behavior change for normal projects (well under 1000 variables).
Testing
Verified against a local
wrangler devinstance:last_seenrefresh still works, including when the project is at the cap.project_variablesrows — the cap engages.typecheckpasses for@nodrix/workerand@nodrix/web.Separate note (not in this PR)
While testing I hit what looks like an unrelated bug:
POST /v1/telemetryaccepts up toMAX_POINTS = 200metrics, butProjectDO.ingestbuilds single multi-row SQL statements (theIN (…)SELECT + multi-row UPSERT/INSERT) that exceed the Durable Object SQLite bound-variable limit well below 200 — a valid ~200-key batch returns 500. Happy to open a separate issue/PR to chunk those writes if useful.