Skip to content

feat(billing): daily profile usage from service_usage - #2681

Merged
Makisuo merged 2 commits into
mainfrom
profiles-daily-usage
Nov 16, 2026
Merged

Makisuo merged 2 commits into
mainfrom
profiles-daily-usage

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #2678: fills the billing page's per-day profiles band, which reads $0 until this lands.

Warehouse

  • Migration 0068_service_usage_profiles: ProfileRowCount and ProfileSizeBytes (UInt64, default 0) on service_usage, and a service_usage_profiles_mv view from profiles, shaped like the logs view. Not required for ingest, no backfill.
  • Local Maple schema v59 to v60 with the same columns and view, plus the migration step that adds them first.
  • A profile row's bytes are estimated as length(Frames) * 16 + 200. The chart rescales each day to the metered Autumn total, so only the daily shape matters.

Read path: dailySignalVolumeQuery selects profile bytes, and DailySpendService sets profilesGB. A cluster without the new columns yet falls back to a query that reports 0 for profiles, so the chart still loads.

Tests: domain migration tests, the compiled query and its SQL baseline, DailySpendService (including the fallback), and a new chDB test (apps/cli/test/service-usage-profiles.test.ts, added to CI's list) showing profile rows land in service_usage and that the real v59 to v60 local migration keeps old rows.

Validated against prod Tinybird with the read-only bun run tinybird:deploy check: "Deployment is valid" in us and eu. Changes: service_usage_profiles_mv new; service_usage and its six signal views modified (each now emits 0 AS ProfileRowCount, 0 AS ProfileSizeBytes, which Tinybird requires of every view writing the table). ProfileRowCount counts stack rows, since a summing table cannot count distinct profiles.

Migration 0068 adds the columns, then drops and recreates all seven service_usage views from the generated schema; local v60 does the same.

After deploy: service_usage carries a forward query naming every column, with the two new ones defaulted. Without it, Tinybird rebuilds the table from source tables that keep 14 to 90 days, so its 365 days of history would be lost. Once this deployment is live it must go back to SELECT * (a follow-up PR), or the next rebuild would reset profile counts.

Gap: on self-hosted ClickHouse and local stores, rows ingested in the brief window while the views are recreated are not counted in service_usage (the same trade-off migration 0065 made).

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

Summary by CodeRabbit

  • New Features
    • Daily usage reporting now includes estimated profile volume alongside logs, traces, and metrics.
    • Profile usage is aggregated by organization, service, and hour. Existing usage history is not backfilled, so profile usage appears for data collected after the update.
  • Compatibility
    • Environments without profile usage data continue to report zero profile volume.

@maple-review-bot

This comment has been minimized.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@Makisuo
Makisuo added this pull request to stack #2682 November 16, 2026 00:41
Base automatically changed from profiles-billing to main November 16, 2026 00:50
Profiles land in service_usage through a new view (ProfileCount, ProfileSizeBytes,
16 bytes per frame plus 200 per stack row), ClickHouse migration 0068 and local
schema v60. The billing chart's daily volume reads them as profilesGB, and a cluster
whose service_usage predates the columns reads no profile usage instead of a 502.
@Makisuo
Makisuo force-pushed the profiles-daily-usage branch from 3d242b7 to 12c492b Compare November 16, 2026 00:50
@maple-review-bot

This comment has been minimized.

…enamed ProfileRowCount

Tinybird refuses a view whose output lacks target columns, so the six signal views
write 0 to ProfileRowCount and ProfileSizeBytes, and migration 0068 and the local
v60 step recreate them. The row count is named for stack rows, which is what it
counts. service_usage keeps an explicit forward query: without one Tinybird would
rebuild it from sources that retain 14 to 90 days.
@maple-review-bot

maple-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. No defect found; the new view, the v59 to v60 step and the fallback are each pinned by a test. The one-shot forwardQuery is the only live risk.
no findings · tests covered · risk high · 1/1 new units observable

Adds estimated continuous-profiling volume to the billing page's daily chart: a service_usage_profiles_mv over profiles, migration 0068 recreating all seven service_usage views, and the read path with a fallback for clusters that lack the columns. Safe to merge once the Tinybird preview passes.

  • service_usage_profiles_mv counts profile stack rows and estimates bytes per org, service and hour
  • dailySignalVolumeQuery sums ProfileSizeBytes; DailySpendService maps it to profilesGB
  • Migration 0068 adds both columns and recreates every service_usage view
  • A cluster without the columns re-runs the query reading 0 profile bytes

Before merge

Fixed since the last review

  • ✅ F1 · count() AS ProfileCount counts stack rows, not profiles

210faa8 · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit 81559e2 into main Nov 16, 2026
48 of 49 checks passed
@Makisuo
Makisuo deleted the profiles-daily-usage branch November 16, 2026 01:09
@maple-review-bot

Copy link
Copy Markdown

Merged with 4 steps from "Before merge" still open. If they are done, tick them on the review comment; if not, now is the time.

  • Manual · After that deploy, revert service_usage's forwardQuery to SELECT * — a rebuild otherwise zeroes profile counts
  • Manual · Run a full preview deploy and confirm Tinybird accepts the six existing service_usage views omitting the new columns
  • Migration · Check it is safe on production data and with the code still deployed
  • Warehouse · Deploy the schema before the code that reads it

@maple-review-bot

Copy link
Copy Markdown

Merged with 1 step from "Before merge" still open. If it is done, tick it on the review comment; if not, now is the time.

  • Manual · Deploy with the full preview label, not preview:web: this changes Tinybird datasources and views

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant