diff --git a/.gitignore b/.gitignore index 36508301c3..a2bc888b85 100644 --- a/.gitignore +++ b/.gitignore @@ -87,3 +87,4 @@ docs/_cache/ .devcontainer/devcontainer-lock.json *.lscache +.gstack/ diff --git a/openspec/changes/add-usage-budget-controls/design.md b/openspec/changes/add-usage-budget-controls/design.md new file mode 100644 index 0000000000..c073d24c1c --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/design.md @@ -0,0 +1,893 @@ +# Design: Add Usage Budget Controls + +## Overview + +Add three complementary usage budget controls: + +1. **Organization budget alert emails** — warnings sent when organization accepted event usage crosses configured percentage thresholds. +2. **Automatic smart project throttling** — automatic project-isolated throttling/sampling when a project spikes enough to threaten organization usage. +3. **Optional project event budgets** — user-configured project-level accepted event caps. + +These are event-usage controls, not generic API request throttling controls. + +The implementation should extend existing usage and notification paths: + +- `UsageService` remains responsible for usage totals, budget threshold detection, smart throughput calculations, and limit calculations. +- Existing organization overage notification behavior remains unchanged. +- `OverageMiddleware` remains the event-post gate for coarse request-level enforcement. +- `EventPostsJob` / event post processing is where project-level sampled acceptance should happen after events are parsed. +- Existing organization usage enforcement remains the outer hard limit. +- Existing `ThrottlingMiddleware` remains unchanged for this feature. +- Budget alert and smart throttling emails are asynchronous side effects and must not block event ingestion. + +## Existing code paths + +### Event post enforcement + +`OverageMiddleware` currently: + +1. Skips non-event posts. +2. Gets the default organization id from the authenticated request. +3. Rejects when event submission is globally disabled. +4. Rejects invalid/oversized posts. +5. Calls `UsageService.GetEventsLeftAsync(organizationId)`. +6. Rejects organization overage. +7. Allows the request to continue. + +This change keeps that coarse gate but does not rely on it as the only project-level enforcement layer. + +### Usage tracking + +`UsageService.IncrementTotalAsync(organizationId, projectId, eventCount)` already increments both: + +- organization bucket total +- project bucket total + +Existing cache key helpers accept an optional `projectId`. + +Use these existing counters for organization budget alerts, smart project throttling, and project event budgets. Do not introduce a parallel accepted-event counter. + +### Event post processing + +`EventPostsJob` already: + +- loads project and organization +- parses the post into events +- calculates how many events can be processed +- processes only allowed events +- increments blocked usage for events over plan limit +- increments accepted total usage for processed events +- increments discarded usage for discarded events + +Extend this path for sampled project-level acceptance. + +### Existing organization notice email path + +Existing flow for monthly/hourly overage notices: + +1. `UsageService` publishes `PlanOverage`. +2. `EnqueueOrganizationNotificationOnPlanOverage` subscribes to `PlanOverage`. +3. It enqueues `OrganizationNotificationWorkItem`. +4. `OrganizationNotificationWorkItemHandler` loads organization and users. +5. The handler sends organization emails only to users with verified email addresses and email notifications enabled. +6. `Mailer.SendOrganizationNoticeAsync` renders the organization notice email. + +Budget alerts and project smart throttling notifications should follow this pattern with new message/work item/mailer/template types. + +--- + +## Automatic Smart Project Throttling + +### Overview + +Automatic smart project throttling is the default guardrail for sudden event spikes. + +It addresses historical feedback that: + +- organization-level throttling can punish all projects when one project is noisy +- throttling should notify users when applied +- throttling should still allow a small percentage of events through for troubleshooting +- users should not need to configure many options before this protection works + +This is separate from optional project event budgets: + +- Smart throttling is automatic and adaptive. +- Project event budgets are explicit user-configured caps. + +### Design principle: minimal configuration + +Smart project throttling should not require users to configure project percentages, per-project caps, or per-stack settings. + +The UI may expose status and explanation, but not a large set of tuning options. + +### Throughput calculation + +Smart throttling should use remaining monthly allowance and remaining time in the monthly usage period. + +Preferred formula shape: + +```text +maxThroughput(window) = eventsLeftInMonth / windowsLeftInMonth * burstMultiplier +``` + +Where: + +* `eventsLeftInMonth` is the organization's current effective monthly allowance minus accepted usage. +* `windowsLeftInMonth` is the number of five-minute usage buckets remaining in the period. +* `burstMultiplier` is 10 to preserve existing Exceptionless burst tolerance. +* Accepted totals include the current and previous unsaved buckets. + +This is intentionally different from a static `monthlyPlanLimit / timeInMonth` calculation. Customers should not be throttled harshly late in the month when they still have substantial allowance remaining. + +### Project isolation + +When organization-level smart throttling would apply, the system should identify the project or projects contributing to the spike and throttle those projects first. + +Observable behavior: + +* One noisy project should not cause all other projects in the organization to stop ingesting events. +* Other projects should continue under organization-level usage enforcement unless they also exceed smart-throttling criteria. +* If the organization hard monthly limit is exhausted, all projects remain subject to existing organization overage behavior. + +### Sampling while throttled + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless should accept a small sample of events from that project. + +Recommended v1 behavior: + +```text +sampleRate = 5% +``` + +V1 uses stable hash selection across the whole post. Sampling is a long-run probability, so a single-event post is genuinely accepted about 5% of the time rather than being forced through. + +Sampling goals: + +* Preserve visibility into whether the problem is still occurring. +* Preserve visibility into whether a deployed fix reduced occurrences. +* Preserve a representative long-run sample without a per-post minimum that biases single-event traffic. +* Avoid allowing a noisy project to consume the entire organization allowance. + +### Event processing location + +Smart throttling and sampled acceptance should be implemented in the event processing path after event posts are parsed, not solely in `OverageMiddleware`. + +Reason: + +* `OverageMiddleware` can reject the entire request before the event count is known. +* It cannot preserve a 1–5% sample of events from a batch. +* `EventPostsJob` already parses the post, calculates `eventsToProcess`, processes only allowed events, and increments blocked usage for the rest. +* Extending this job/path allows project-level partial acceptance while preserving existing usage accounting. + +### Event selection + +When only part of a batch can be accepted, the implementation should avoid always taking only the first events if doing so would bias troubleshooting data. + +Acceptable v1 approaches: + +* deterministic sampling by event id/hash +* random sampling with stable seed per post +* first-N selection only if no safe sampling utility exists, but this should be documented as a known limitation + +### Smart throttling state + +The system may store project throttling state in cache to avoid recalculating expensive decisions on every event. + +Suggested key shape: + +```text +usage:{fiveMinuteBucket}:{organizationId}:{projectId}:smart-throttle +``` + +Suggested value: + +```json +{ + "is_throttled": true, + "sample_rate": 0.05, + "effective_until_utc": "..." +} +``` + +TTL: short window, aligned with usage bucket/window duration. + +If cached throttle state expires, the system can re-evaluate based on current usage counters. + +`EnableSmartProjectThrottling` defaults to enabled and acts only as an operational kill switch for automatic smart throttling. Project-list status enrichment bulk-reads these short-lived keys once. + +### Smart throttling notification + +When a project enters smart-throttled state, eligible organization users should receive an email notification. + +Notification behavior: + +* Send at most once per project per throttling period or cooldown window. +* Reuse the existing organization notification eligibility rules. +* Do not send repeated emails for every event post while the project remains throttled. +* Include project name, organization name, current project accepted usage, current fair-share limit, and a link to project usage. +* Re-check current plan, usage, kill-switch, and throttle state in the delayed handler before sending. +* Emit the dedicated throttled-event metric for sampled-out events and structured logs only when state transitions to active. + +Suggested message/work item: + +```csharp +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Suggested mailer method: + +```csharp +Task SendProjectThrottledNoticeAsync( + User user, + Organization organization, + Project project, + double sampleRate, + int currentEventCount, + int eventLimit); +``` + +### Relationship to organization overage + +Smart project throttling is not a substitute for organization hard overage enforcement. + +Behavior: + +* If organization monthly allowance remains, smart-throttled projects may still have sampled events accepted. +* If organization monthly allowance is exhausted, existing organization overage behavior applies. +* Project sampling must not allow accepted event totals to exceed the organization hard monthly allowance. + +### Relationship to optional project event budgets + +If a project has an explicit project event budget, that budget participates in the same allowed-event calculation. + +Recommended order: + +```text +organization hard allowance +↓ +explicit project budget, if configured +↓ +automatic smart throttling sample allowance, if active +``` + +The final number of events accepted from a batch should be the minimum allowed by all applicable controls. + +### Relationship to budget alert emails + +Smart throttling email notification is distinct from organization budget alert emails. + +* Budget alerts notify when organization accepted usage crosses configured percentages. +* Smart throttling notifications notify when the system starts sampling/throttling a noisy project. +* A project can be smart-throttled before any budget alert threshold is crossed. +* A budget alert can fire without any project being smart-throttled. + +--- + +## Organization Budget Alert Emails + +### Overview + +Add configurable organization-level event budget alert emails. These alerts warn users before monthly plan overage, based on accepted event usage crossing configured percentage thresholds. + +Budget alerts are informational only. They do not block ingestion and do not replace organization overage enforcement, smart throttling, or project event budgets. + +### Domain model + +Add budget alert settings to Organization. + +```csharp +public class Organization +{ + // existing properties... + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Add a new model: + +```csharp +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} +``` + +### Defaults + +Existing organizations: + +```text +budget_alert_settings = null +``` + +Null means disabled. + +New organizations: + +```text +budget_alert_settings = null +``` + +The UI may suggest default thresholds [50, 80], but alerts must not be enabled until a user explicitly saves settings. + +### Validation + +Validation rules: + +* `BudgetAlertSettings == null` is valid. +* `Enabled == false` with empty thresholds is valid. +* If enabled, thresholds must contain at least one value. +* Each threshold must be greater than 0. +* Each threshold must be less than 100. +* Threshold 100 is not allowed because existing monthly overage notification already handles reaching/exceeding the plan limit. +* Duplicate thresholds must be removed. +* Thresholds must be stored sorted ascending. +* Percentage budget alerts must be rejected or disabled for unlimited organizations. + +### API contract + +Add `BudgetAlertSettings` to organization update and view DTOs. + +Preferred cleanup: + +```csharp +public record UpdateOrganization +{ + public string Name { get; set; } = null!; + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Then update `OrganizationController` generic usage from: + +```csharp +RepositoryApiController +``` + +to: + +```csharp +RepositoryApiController +``` + +If minimizing backend churn is preferred, `BudgetAlertSettings` can be added to `NewOrganization` because the current controller uses `NewOrganization` for both create and update. However, adding `UpdateOrganization` is cleaner because `NewOrganization` currently only contains the required organization name. + +Add to `ViewOrganization`: + +```csharp +public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +``` + +Both `budget_alert_settings: null` and `ingest_limit: null` clear their settings. The canonical Delta OpenAPI transformer must generate the referenced complex component schemas and represent these PATCH properties as `oneOf: [null, $ref]`; snapshots and generated clients are regenerated from that output. + +Serialized JSON uses the existing snake_case policy. + +### Threshold calculation + +Use the same effective organization allowance as existing usage enforcement: + +```csharp +effectiveOrganizationAllowance = organization.GetMaxEventsPerMonthWithBonus(timeProvider) +``` + +If effective allowance is negative/unlimited: + +```text +budget alert thresholds are inactive +``` + +Threshold event count: + +```csharp +thresholdEventCount = ceil(effectiveOrganizationAllowance * thresholdPercent / 100) +``` + +### Triggering alerts + +Budget alert checks should run in the accepted-event usage increment path. + +Recommended location: + +```text +UsageService.IncrementTotalAsync +``` + +Observable behavior must be: + +* Alert fires when usage crosses threshold. +* Alert does not fire before threshold. +* Alert does not fire repeatedly after threshold. +* Multiple crossed thresholds can fire if a large batch jumps over more than one threshold. +* Crossing compares bucket-inclusive previous and current accepted totals. +* Enabling alerts after usage already exceeds a threshold evaluates the newly active thresholds once immediately after save. + +### Deduplication + +Each organization threshold must email at most once per monthly usage period. + +Recommended cache key: + +```text +usage-budget-alert:{yyyyMM}:{organizationId}:{threshold} +``` + +TTL: until end of current monthly usage period + safety buffer. + +The cache key is claimed atomically with `AddAsync` before publishing to prevent duplicate sends under concurrency. + +### Message and work item + +Add a new message: + +```csharp +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a new work item: + +```csharp +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a startup subscriber: + +```text +EnqueueOrganizationBudgetAlertOnUsageThreshold +``` + +### Work item handler + +Add: + +```text +OrganizationBudgetAlertWorkItemHandler +``` + +Behavior: + +* Load organization by id. +* Load users in organization. +* Send only to users with verified email addresses. +* Send only to users with `EmailNotificationsEnabled == true`. +* Use the same eligibility rules as existing organization notices. +* Do not send if organization no longer exists. +* Re-check current organization budget alert settings before sending. +* If budget alerts are disabled or the threshold was removed, skip sending. + +### Mailer + +Add to `IMailer`: + +```csharp +Task SendOrganizationBudgetAlertAsync( + User user, + Organization organization, + int threshold, + int thresholdEventCount, + int currentEventCount, + int eventLimit); +``` + +Add implementation in `Mailer`. + +Add template: + +```text +src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html +``` + +Template must include: + +* organization name +* threshold percentage +* threshold event count +* current accepted event count +* organization event limit +* remaining event count +* link to organization usage +* link to billing/plan management when applicable +* link to notification settings + +Do not include event payload data. + +--- + +## Optional Project Event Budgets + +### Overview + +Add optional project-level event budgets enforced during event processing. This is an event-usage control, not generic API request throttling. + +### Domain model + +Add a nullable event budget configuration to Project. + +```csharp +public class Project +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } +} +``` + +Add a new model in `src/Exceptionless.Core/Models/ProjectIngestLimit.cs`: + +```csharp +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} +``` + +### Naming + +Use `IngestLimit`, not `RateLimit`, because the feature limits accepted event volume for a project, not generic request rate. + +Use `ProjectIngestLimit`, not `ProjectQuota`, because this is a cap, not a reservation. + +### API contract + +Add `IngestLimit` to: + +```text +src/Exceptionless.Web/Models/Project/UpdateProject.cs +src/Exceptionless.Web/Models/Project/ViewProject.cs +``` + +Recommended DTO shape: + +```csharp +public record UpdateProject +{ + public string Name { get; set; } = null!; + public bool DeleteBotDataEnabled { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } +} + +public class ViewProject +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } + public int? EffectiveIngestLimit { get; set; } + public bool IsSmartThrottled { get; set; } + public double? SmartThrottleSampleRate { get; set; } +} +``` + +### Project update behavior + +Use existing `PATCH /api/v2/projects/{id}`. Do not add a new endpoint. + +Clear project budget: + +```json +{ + "ingest_limit": null +} +``` + +Fixed budget: + +```json +{ + "ingest_limit": { + "type": "fixed", + "fixed_limit": 20000 + } +} +``` + +Percentage budget: + +```json +{ + "ingest_limit": { + "type": "percent_of_organization_limit", + "percent_of_organization_limit": 20 + } +} +``` + +### Validation + +General validation: + +* `ingest_limit == null` is valid and means Off. +* Unknown limit type is invalid. +* Limit values must not be negative. +* Ingest limit validation must not change existing project name validation. + +For Fixed: + +* `fixed_limit` is required. +* `fixed_limit` must be greater than 0. +* `percent_of_organization_limit` should be ignored or cleared server-side. +* Backend should not reject a fixed limit greater than the current organization limit. The effective cap is clamped during evaluation. + +For PercentOfOrganizationLimit: + +* `percent_of_organization_limit` is required. +* Percentage must be greater than 0. +* Percentage must be less than or equal to 100. +* `fixed_limit` should be ignored or cleared server-side. +* If the organization currently has an unlimited event allowance, the API should reject setting a percentage cap because a percentage of unlimited is not meaningful. +* If an existing percentage cap later encounters an unlimited organization due to a plan change, enforcement should treat the percentage cap as inactive until the organization returns to a finite allowance. + +### Effective project limit calculation + +Use the same effective organization max that current usage enforcement uses. + +```csharp +private static int? GetEffectiveProjectIngestLimit(ProjectIngestLimit? limit, int organizationMaxEvents) +{ + if (limit is null) + return null; + + return limit.Type switch + { + ProjectIngestLimitType.Fixed when limit.FixedLimit is > 0 => + organizationMaxEvents < 0 + ? limit.FixedLimit.Value + : Math.Min(limit.FixedLimit.Value, organizationMaxEvents), + ProjectIngestLimitType.PercentOfOrganizationLimit + when organizationMaxEvents > 0 && limit.PercentOfOrganizationLimit is > 0 => + Math.Max(1, (int)Math.Ceiling(organizationMaxEvents * (limit.PercentOfOrganizationLimit.Value / 100m))), + _ => null + }; +} +``` + +### UsageService API + +Add project-aware/budget-aware event allowance calculation. + +The final event count accepted from a parsed batch should be the minimum allowed by: + +1. organization hard allowance +2. explicit project event budget, if configured +3. automatic smart throttling sampled allowance, if active + +Avoid a design that returns only a Boolean. The event processing path needs a count. + +Suggested result: + +```csharp +public record EventIngestAllowanceResult +{ + public int EventsAllowed { get; init; } + public int OrganizationEventsLeft { get; init; } + public int? ProjectEventsLeft { get; init; } + public int? EffectiveProjectLimit { get; init; } + public bool IsSmartThrottled { get; init; } + public double? SmartThrottleSampleRate { get; init; } + public string? Reason { get; init; } +} +``` + +Suggested method: + +```csharp +public Task GetEventIngestAllowanceAsync( + Organization organization, + Project project); +``` + +Behavior: + +* Compute organization and project totals from one bulk read of persisted-total/current-bucket/previous-bucket cache keys. +* If organization events left is <= 0, allow 0 events. +* Compute explicit project budget events left if configured. +* Compute smart throttling state/sample allowance if project is noisy. +* Return the minimum allowed count. +* Never return an allowed count greater than submitted event count. +* Never return an allowed count greater than organization events left. + +### Refactor current events-left logic + +The event job's project-aware path uses the organization and project it already loaded. It must not issue repository rereads in normal traffic. Existing middleware may retain the organization-id-only helper for its coarse gate. + +```csharp +public Task GetEventIngestAllowanceAsync( + Organization organization, + Project project) +``` + +Use existing total and bucket cache keys with optional project id. Query the repository's canonically invalidated organization project-count cache only when spike and remaining-allowance checks require fair-share evaluation. + +### OverageMiddleware behavior + +`OverageMiddleware` should continue handling request-level event-post gates: + +* non-event posts are skipped +* missing organization context is rejected +* globally disabled event submission is rejected +* missing content length is rejected +* oversized posts are rejected +* organization hard overage can still be rejected early when there are no events left + +Project-level smart throttling and sampled project-budget behavior should not be implemented solely as early middleware rejection because early rejection drops the entire post and cannot preserve a sample. + +If `OverageMiddleware` detects that the organization has no remaining monthly allowance, it may preserve the existing organization overage rejection behavior. + +If the organization has remaining allowance, project-level controls should be evaluated later in `EventPostsJob` / `UsageService` after events are parsed. + +### Status code behavior + +* Organization monthly/bucket hard overage remains `402 PaymentRequired` where existing behavior already returns that status. +* Oversized submissions continue to return `413 RequestEntityTooLarge`. +* Missing content length continues to return `411 LengthRequired`. +* Missing organization context continues to return `401 Unauthorized`. +* Disabled event submission continues to return `503 ServiceUnavailable`. +* Project smart throttling should not require changing the HTTP response for accepted queued posts; the job may accept a sample and block/discard the rest asynchronously. +* Project hard rejection may return `429 TooManyRequests` only in code paths where the server can make an immediate project-level decision without sacrificing sampled acceptance. For the normal queued event-post path, project throttling should be represented by partial processing and blocked/discarded usage accounting rather than a full-request 429. + +### Headers + +Do not repurpose existing generic API throttle headers for project monthly ingest caps or smart project throttling in this change. + +### API throttling middleware + +Do not change `ThrottlingMiddleware` for this feature. + +### Storage and indexing + +Persisted data: + +* Existing organizations have `BudgetAlertSettings = null`. +* Existing projects have `IngestLimit = null`. +* No migration/backfill is required. +* Null/default behavior must preserve current behavior. + +Elasticsearch: + +* Do not add Elasticsearch mappings for `Organization.BudgetAlertSettings` or `Project.IngestLimit` unless filtering/sorting/searching by those fields is required. +* Avoid reindexing. + +### Organization Usage UI + +Add budget alert settings to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte +``` + +Place above the usage chart. + +Suggested card: + +```text +Budget Alerts +Receive an email when your organization reaches selected percentages of its monthly event allowance. +[ ] Enable budget alerts +Thresholds +[50] [80] [90] +[+ Add threshold] +Current plan allowance: 100,000 events +50% = 50,000 events +80% = 80,000 events +Alerts are sent once per threshold per monthly usage period to organization users who have email notifications enabled. +``` + +### Project Usage UI + +Add project event budget and smart throttling status to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte +``` + +Copy: + +```text +Project Event Budget +Protect your organization's monthly event allowance by limiting how many accepted events this project can use. +This is a cap, not a reservation. Other projects are not guaranteed unused capacity. +``` + +Smart throttling status copy when active: + +```text +Smart throttling is currently active for this project. Exceptionless is accepting a small sample of events so you can continue troubleshooting while protecting your organization's monthly event allowance. +``` + +Do not expose many tuning knobs. + +### Project usage chart behavior + +If `effective_ingest_limit` is not null, display the dashed limit line as the project limit. + +If `effective_ingest_limit` is null, keep showing the organization limit. + +Show smart throttling status as text or badge if active. + +### Accessibility + +* Slider controls must have accessible labels. +* Numeric inputs must have associated labels. +* Computed threshold/effective limit text should update in screen-reader-friendly text. +* Do not rely on color only. +* Save success/error must use existing toast/error patterns. + +### Security + +* Only users authorized to update the organization may update budget alert settings. +* Only users authorized to update the project may update project event budgets. +* Budget alert and smart throttling emails are sent only to verified users with email notifications enabled. +* Project budgets do not grant additional access. +* Users cannot increase organization allowance through project budgets. +* Smart throttling must not allow accepted usage beyond organization hard allowance. + +### Privacy + +No new personal event data is collected. + +Budget alert and smart throttling emails include organization/project names and aggregate usage numbers only. They must not include event payload data, stack data, user-identifying event details, or project-specific event contents. + +### Failure modes + +**Budget alert email queue failure** + +If mail queue enqueue fails, do not block event ingestion. + +**Smart throttling email queue failure** + +If smart throttling notification enqueue fails, do not block event ingestion. + +**Dedupe cache failure** + +If dedupe cache fails, prefer avoiding ingestion failure. The implementation may skip alert sending rather than risk duplicate emails or ingestion failure. + +**Organization/project settings changed after message publication** + +Re-load settings in the work item handler. If the relevant notification is disabled or no longer applicable, skip sending. + +**Project lookup fails** + +If the project cannot be loaded during project-level evaluation, do not block solely due to project-level settings. Continue organization-level enforcement. + +**Invalid persisted settings** + +Treat invalid persisted settings as inactive for enforcement/notification. Do not crash event ingestion. diff --git a/openspec/changes/add-usage-budget-controls/proposal.md b/openspec/changes/add-usage-budget-controls/proposal.md new file mode 100644 index 0000000000..e4cee16752 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/proposal.md @@ -0,0 +1,224 @@ +# Proposal: Add Usage Budget Controls + +## Summary + +Add usage budget controls that help organizations both **warn before overage** and **prevent noisy projects from consuming the full organization event allowance**. + +This change includes three related capabilities: + +1. **Organization budget alert emails** — configurable percentage thresholds such as 50% and 80% of the organization's monthly event allowance. When accepted event usage crosses a configured threshold, Exceptionless sends a budget alert email to eligible organization users. +2. **Automatic smart project throttling** — when a project/environment starts sending enough events to threaten the organization's budget, Exceptionless should automatically isolate throttling to the noisy project where possible, instead of stopping ingestion for the entire organization. +3. **Optional project event budgets** — project-level budget caps that allow customers to explicitly limit how many accepted events a project can consume in the monthly usage period. + +Budget alerts are warning controls. Smart project throttling is the default automatic guardrail. Project event budgets are optional prevention controls for customers who want explicit per-project limits. + +This change addresses customer feedback asking for budget alerts before plan overage: + +> Hi support team +> +> Is there any budget alerts we can configure in terms of the number of events? For example I want to receive an email when i have reached 50% of my plan, and then when 80% of my plan. +> +> This month we have exceeded twice our plan, and we should have a mechanism to prevent this situation. + +It also addresses historical customer feedback from issue #112 requesting smarter throttling: + +- project-level throttling +- email notification when throttling is applied +- 1–5% of errors delivered even under throttling + +## User-visible behavior + +### Organization budget alerts + +Users managing an organization can configure event budget alert thresholds from Organization Settings → Usage. + +Supported behavior: + +- Alerts are disabled by default for existing organizations. +- Users can enable budget alerts and configure one or more percentage thresholds. +- Example thresholds: 50%, 80%, 90%. +- A threshold represents the organization's current monthly event allowance, including active bonus events if those are included by existing usage enforcement. +- When accepted usage crosses a configured threshold for the first time in a monthly usage period, eligible organization users receive an email. +- Each threshold is emailed at most once per monthly usage period. +- Alerts are not sent for unlimited organizations because a percentage of unlimited usage is undefined. +- Alerts do not block event ingestion. +- Existing monthly overage emails remain unchanged. + +### Automatic smart project throttling + +Exceptionless should automatically protect organization usage when one project starts sending a large event spike. + +Supported behavior: + +- Users should not be required to configure smart throttling. +- The system should avoid exposing a large number of throttling options. +- When throttling is needed, it should be scoped to the noisy project where possible rather than blocking all projects in the organization. +- While a project is throttled, Exceptionless should still accept a small sample of events from that project when the organization has remaining monthly allowance. +- The accepted sample should preserve troubleshooting visibility so users can tell whether the issue is still happening and whether a deployed fix helped. +- When smart throttling is applied, eligible organization users should receive an email notification. +- Smart throttling should use remaining monthly allowance and remaining time in the period, not only the static plan size, to avoid throttling customers who still have substantial monthly allowance left. + +### Optional project event budgets + +Users managing a project can configure a Project Event Budget from Project Settings → Usage. + +Supported modes: + +- **Off**: no project-specific cap; current behavior is preserved. +- **Fixed limit**: cap this project at a configured number of accepted events per monthly usage period. +- **Percentage of organization limit**: cap this project at a percentage of the organization's current monthly event allowance. + +Examples: + +- A project with no configured ingest limit can consume organization events as it does today. +- A project with a fixed limit of `20,000` can accept up to `20,000` events in the current monthly usage period, subject to the organization still having events available. +- A project with a `20%` limit in an organization with `100,000` monthly events has an effective project cap of `20,000`. +- If the organization plan changes, percentage-based project caps are recalculated from the organization's current event allowance. + +When a project limit is reached: + +- Event submissions for that project are limited or sampled. +- Event submissions for other projects in the organization are not limited solely because this project reached its cap. +- Blocked event usage is recorded for the project and organization. +- Existing organization overage behavior remains unchanged. + +## Classification + +- **Type:** Feature +- **Affected areas:** Backend/API, billing/usage enforcement, organization model, project model, Redis/cache usage counters, mail/work items, Svelte UI, generated API types, tests +- **OpenSpec justification:** This change affects event ingestion behavior, organization/project usage limits, smart throttling behavior, email notification behavior, API response contracts, persisted organization/project data, Redis/cache usage state, SDK/client expectations, jobs, and Svelte UI/API contracts. + +## Current implementation context + +Exceptionless currently enforces organization event overage in `OverageMiddleware` for event posts. The middleware checks the authenticated organization id, rejects disabled event submission, rejects oversized submissions, and calls `UsageService.GetEventsLeftAsync(organizationId)` before allowing the event post to continue. + +`UsageService` already tracks both organization and project usage counters. `IncrementTotalAsync` increments organization totals and project totals using existing cache keys that accept an optional `projectId`. + +`EventPostsJob` already parses queued event posts, calculates how many events may be processed, processes only allowed events, increments blocked usage for events over the limit, and increments total/discarded usage for processed events. This is the correct place to implement project-level sampled acceptance. + +Exceptionless already has an organization overage email path. Existing usage overage publishes `PlanOverage`, which enqueues `OrganizationNotificationWorkItem`, and the work item handler sends organization notice emails to verified users with email notifications enabled. Budget alerts and project throttling notifications should reuse that mail/work-item pattern with distinct alert types rather than overloading monthly/hourly overage booleans. + +The existing generic API throttling middleware remains separate. It limits API requests over a short period and is not the same as event-plan usage enforcement. + +## Affected areas + +### Backend/API + +- Add organization usage budget alert settings to organization model/API responses. +- Add project event budget fields to project model/API responses. +- Extend usage tracking to publish budget-alert notifications when configured thresholds are crossed. +- Extend usage/event post processing to compute smart project throttling and sampled acceptance. +- Extend usage/event post processing to compute optional project budget limits. +- Preserve existing organization-level overage behavior and status codes. +- Avoid implementing project sampled throttling solely in middleware. + +### Redis/cache + +- Reuse existing organization/project usage counter keys. +- Add budget alert dedupe state keyed by organization, threshold, and monthly usage period. +- Add smart project throttling state or notification dedupe state keyed by organization/project/window where useful. +- Preserve existing usage counter TTL behavior. +- Do not add new project usage counter families unless required for efficient limit evaluation. + +### Mail/work items + +- Reuse the existing organization notification pattern. +- Add budget-alert-specific message/work item/template. +- Add project-smart-throttling-specific message/work item/template. +- Send emails only to verified users with email notifications enabled, matching existing organization notice behavior. + +### Svelte UI + +- Add organization budget alert controls to the Svelte Organization Settings → Usage page. +- Add project event budget controls to the Svelte Project Settings → Usage page. +- Surface smart throttling status where feasible, especially on Project Usage. +- Show computed thresholds/caps and current usage. +- Update the project usage chart to display the project limit when configured. + +### Legacy Angular UI + +- Legacy Angular parity is out of scope for this change unless the release path requires these settings to be available in the legacy UI. +- This change must not break existing legacy Angular organization or project management behavior. + +### SDK/client compatibility + +- Existing event submission authentication mechanisms must continue to work. +- Existing project API keys and user tokens must continue to work. +- Organization overage behavior must remain compatible. +- Smart project throttling for queued event posts should prefer asynchronous sampled processing over changing the HTTP submission response. +- Budget alert emails must be asynchronous side effects and must not change normal event submission success responses. + +### Tests + +- Add backend unit/integration tests for budget threshold calculation, smart throughput calculation, sampled acceptance, and project budget computation. +- Add usage service tests for threshold crossing, dedupe, organization cap, project cap, and smart throttling behavior. +- Add event post job tests for sampled acceptance and blocked usage accounting. +- Add API tests for organization budget alert and project event budget update/validation. +- Add mail/work-item tests for budget alert and smart throttling email eligibility/dedupe behavior. +- Add Svelte UI tests or targeted manual QA for the Usage page controls. +- Update HTTP samples if request/response contracts are changed. + +## Compatibility risks + +| Risk | Mitigation | +|------|------------| +| Existing event submissions could be rejected unexpectedly | Project event budgets default to null/Off for all existing projects. Smart throttling should preserve sampled acceptance when organization allowance remains. | +| Existing organization overage behavior could change | Keep organization-level monthly/bucket enforcement intact. Organization hard overage remains the outer limit. | +| Middleware-level project rejection would drop 100% of events | Project smart throttling and sampled acceptance must be implemented in the event processing path after events are parsed, not only in `OverageMiddleware`. | +| Too much configuration conflicts with product direction | Smart project throttling must work automatically with minimal or no required user configuration. | +| Existing customers expect some data while throttled | When the organization still has allowance, throttled projects should retain a small accepted sample rather than being fully blocked. | +| Static monthly plan throttling can feel unfair late in the month | Throughput calculations should consider events remaining in the monthly period and time remaining in the monthly period. | +| Project cap could be confused with reserved quota | UI copy must state that this is a cap, not a reservation. Other projects are not guaranteed unused capacity. | +| Percentage caps could behave unexpectedly when organization plans change | Effective percentage caps are recalculated from the current organization event allowance each time limits are evaluated. | +| Unlimited organization plans make percentage caps ambiguous | Percentage mode must be disabled or invalid for unlimited organizations. Fixed limits remain supported for project event budgets. Budget alert percentages are unavailable for unlimited organizations. | +| Budget alerts could send unexpected emails | Default budget alert settings to disabled for existing organizations. Users must opt in. | +| Duplicate threshold emails could annoy users | Each configured threshold is sent at most once per organization per monthly usage period. | +| Existing overage emails could change | Keep existing `PlanOverage` monthly/hourly emails unchanged and add budget alerts and smart throttling emails as separate notification types. | +| Persisted project or organization model changes could require Elasticsearch mapping/reindex | Store new settings source-only unless filtering/searching is added. No Elasticsearch mapping or reindex is required for enforcement. | + +## Non-goals + +- Do not implement API-key/token-level ingest limits in this change. +- Do not implement rate-based billing. +- Do not implement reserved project quotas. +- Do not change plan billing, invoices, Stripe integration, or organization plan limits. +- Do not change generic API request throttling behavior in `ThrottlingMiddleware`. +- Do not add new public event-submission endpoints. +- Do not migrate historical usage data. +- Do not require Elasticsearch reindexing. +- Do not require existing organizations or projects to be backfilled. +- Do not implement per-recipient custom alert thresholds in this change. +- Do not implement Slack/webhook budget alerts in this change. +- Do not implement stack-level adaptive throttling in v1 unless it can be done with low risk; project-level isolation is the v1 scope. +- Do not expose a large set of smart throttling tuning options in the UI. + +## Future considerations + +This design should leave room for future layers: + +1. Organization plan/event allowance. +2. Organization budget alert thresholds. +3. Automatic smart project throttling. +4. Optional project event budget. +5. API key/token ingest cap. +6. Future rate-based or usage-based billing. + +Budget alerts should leave room for future channels and scopes: + +- fixed event-count alert thresholds +- project-specific budget alerts +- token/API-key-specific budget alerts +- Slack/webhook alert delivery +- role-based recipients +- rate-based billing alerts + +The original issue suggested stack-level adaptive throttling with approximate occurrence counts. This change should leave room for that future direction, but v1 should focus on project-level isolation because it is simpler, aligns with project usage counters already tracked by Exceptionless, and directly addresses the customer problem of one product/environment affecting all others. + +## Rollback plan + +- Because project event budgets are opt-in, disabling or removing the UI leaves existing unset project budgets unchanged. +- Because budget alerts are opt-in, disabling or removing the UI leaves existing unset organization alert settings unchanged. +- If smart project throttling causes issues, smart throttling can be disabled while leaving budget alerts and optional project budgets in place. +- If budget alert emails cause issues, budget alert publishing/subscribers can be disabled while leaving stored settings in place. +- Existing organizations and projects default to null/Off, so rolling back model/API exposure has no effect on existing ingestion behavior unless users have already configured settings. +- If stored settings must be disabled quickly, project budget and budget alert evaluation can be short-circuited to treat all settings as unset. diff --git a/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md new file mode 100644 index 0000000000..518366e482 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md @@ -0,0 +1,166 @@ +# Spec: API Compatibility + +## MODIFIED Requirements + +### Requirement: Event submission MUST preserve existing organization overage behavior + +Exceptionless event submission MUST preserve existing organization overage and API authentication behavior while adding automatic smart project throttling and optional project event budgets. + +#### Scenario: Organization overage status remains unchanged + +Given an organization has no remaining event allowance +When an event is submitted for a project in that organization +Then the response status code must remain the existing organization overage status code. + +#### Scenario: Project smart throttling preserves queued event post compatibility + +Given an organization has remaining event allowance and a project is under smart throttling +When an event post is submitted through the normal queued event submission path +Then the HTTP response should not be changed solely to report project throttling. + +#### Scenario: Generic API throttle behavior is unchanged + +Given API request throttling is enabled +When non-event API requests are made +Then existing generic API request throttle behavior must remain unchanged. + +#### Scenario: Existing client API keys continue to authenticate + +Given an existing project API key is used for event submission and the project has no configured event budget +When an event is submitted +Then authentication and authorization behavior must match existing behavior. + +### Requirement: Rate-limit headers MUST NOT be repurposed for project monthly controls + +Existing generic API throttle response headers MUST NOT be redefined to mean project monthly event budgets or smart project throttling. + +#### Scenario: Generic API throttle headers remain request-rate headers + +Given generic API throttling emits request-rate headers +When usage budget controls are added +Then those headers must continue to represent generic API request throttling behavior. + +## ADDED Requirements + +### Requirement: Organization API exposes budget alert settings + +The organization API may expose and update budget alert settings as an additive organization field. + +#### Scenario: Organization response includes budget alert settings + +Given an organization has budget alert settings +When the organization is returned from the API +Then the response must include budget_alert_settings. + +#### Scenario: Existing organization response remains compatible + +Given an existing organization has no budget alert settings +When the organization is returned from the API +Then budget_alert_settings may be null and existing fields must remain unchanged. + +#### Scenario: Authorized user updates budget alert settings + +Given a user is authorized to update an organization +When the user updates budget alert settings with valid thresholds +Then the API must persist the settings and include them in the response. + +#### Scenario: Invalid budget alert settings rejected + +Given a user is authorized to update an organization +When the user submits invalid budget alert settings +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Budget alert emails are asynchronous side effects + +Budget alert emails are side effects of accepted usage threshold crossing and must not change event submission success responses. + +#### Scenario: Event crosses budget alert threshold + +Given budget alerts are enabled and an accepted event causes usage to cross a threshold +When the event submission completes +Then the event submission response must remain the normal accepted response. + +#### Scenario: Budget alert failure does not reject event + +Given budget alerts are enabled and budget alert queuing fails +When an event is submitted +Then the event must not be rejected solely because budget alert queuing failed. + +## ADDED Requirements + +### Requirement: Project API accepts event budget configuration + +The project update API may accept an optional project event budget configuration as an additive project field. + +#### Scenario: Clear project event budget + +Given an authorized user can update a project +When the user patches the project with ingest_limit set to null +Then the project event budget must be cleared. + +#### Scenario: Set fixed project event budget + +Given an authorized user can update a project +When the user patches the project with a fixed positive ingest_limit +Then the project must persist the fixed event budget. + +#### Scenario: Set percentage project event budget + +Given an authorized user can update a project and the organization has finite allowance +When the user patches the project with a valid percentage event budget +Then the project must persist the percentage event budget. + +#### Scenario: Invalid fixed limit is rejected + +Given an authorized user can update a project +When the user patches the project with a fixed event budget less than or equal to 0 +Then the API must reject the request with a validation error. + +#### Scenario: Invalid percentage limit is rejected + +Given an authorized user can update a project +When the user patches the project with a percentage event budget less than or equal to 0 or greater than 100 +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Project response MUST include budget and throttling state + +Project responses used by the UI MUST expose the configured project event budget, currently effective cap, and smart throttling state where available. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget +When the project is returned from the API +Then ingest_limit must be null and effective_ingest_limit must be null. + +#### Scenario: Project has fixed event budget + +Given a project has a fixed event budget +When the project is returned from the API +Then effective_ingest_limit must contain the currently effective cap. + +#### Scenario: Project is smart-throttled + +Given a project is currently under smart throttling +When the project is returned from the API +Then the response should indicate smart throttling state where feasible. + +### Requirement: Nullable complex PATCH contracts MUST be generated canonically + +Organization and project settings MUST be clearable with JSON null, and their OpenAPI contracts MUST reference the canonical complex component schemas. + +#### Scenario: Clear organization budget settings + +Given an authorized user can update an organization +When the user patches budget_alert_settings with null +Then the settings must be cleared. + +#### Scenario: Delta OpenAPI schema + +Given a nullable complex property belongs to a Delta PATCH model +When OpenAPI is generated +Then the property must be represented as null or a component schema reference +And generated clients must come from that document rather than manual snapshot edits. diff --git a/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md new file mode 100644 index 0000000000..53833b64cc --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md @@ -0,0 +1,364 @@ +# Spec: Event Ingestion + +## ADDED Requirements + +### Requirement: Smart throttling works without required user configuration + +Exceptionless must provide automatic smart project throttling without requiring users to configure per-project or per-stack throttling settings. + +#### Scenario: No smart throttling configuration exists + +Given an organization has projects +And no smart throttling settings have been configured +When one project starts sending a large event spike +Then Exceptionless should still be able to apply smart throttling automatically. + +#### Scenario: UI does not require many throttling options + +Given smart throttling is available +When a user views organization or project usage settings +Then the UI must not require the user to configure many low-level throttling parameters before smart throttling can protect the organization. + +### Requirement: Smart throttling uses remaining allowance and remaining time + +Smart throttling must calculate allowed throughput using the organization's remaining monthly event allowance and the time remaining in the monthly usage period. + +#### Scenario: Organization has substantial allowance remaining late in the month + +Given an organization is late in the monthly usage period +And the organization still has substantial event allowance remaining +When smart throttling calculates allowed throughput +Then the calculation must account for remaining event allowance +And must not rely only on the static monthly plan size divided by the original month duration. + +#### Scenario: Organization has little allowance remaining + +Given an organization has little event allowance remaining +When smart throttling calculates allowed throughput +Then the allowed throughput should decrease to protect the remaining monthly allowance. + +### Requirement: Smart throttling isolates noisy projects where possible + +When one project is responsible for a usage spike, smart throttling must apply to that project where possible rather than fully throttling the entire organization. + +#### Scenario: One project spikes + +Given an organization has Project A and Project B +And Project A sends a large event spike +And Project B remains within normal usage +When smart throttling is applied +Then Project A may be throttled or sampled +And Project B must not be blocked solely because Project A spiked. + +#### Scenario: Multiple projects spike + +Given an organization has multiple projects +And more than one project exceeds smart throttling criteria +When smart throttling is applied +Then each noisy project may be evaluated and throttled independently. + +### Requirement: Smart throttling preserves a sample of events + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless must continue accepting a small sample of events for that project. + +#### Scenario: Project is throttled but organization has allowance + +Given an organization has remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then Exceptionless must accept a small sample of events +And must block or discard the remainder according to usage accounting rules. + +#### Scenario: Sample rate remains small + +Given a project is under smart throttling +When the accepted sample is calculated +Then the accepted sample should be within the intended 1 to 5 percent range +Unless a lower organization or project hard limit leaves fewer events available. + +#### Scenario: Organization allowance exhausted + +Given an organization has no remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then existing organization overage behavior must apply +And smart throttling must not allow sampled events beyond the organization hard allowance. + +### Requirement: Smart throttling operates after event posts are parsed + +Smart project throttling must be evaluated in a processing path that knows the number of events in the post so it can accept a sample and block the remainder. + +#### Scenario: Batch contains more events than allowed + +Given a queued event post contains many events +And smart throttling allows only a sample +When the event post is processed +Then the processor must process only the sampled/allowed events +And must record blocked usage for the rest. + +#### Scenario: Full request rejection would prevent sampling + +Given an event post would otherwise be rejected entirely by project-level throttling +When the organization still has remaining allowance +Then the system should prefer sampled processing over full request rejection. + +### Requirement: Smart throttling sends project throttling notification + +When smart throttling is applied to a project, Exceptionless must send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled +And smart throttling is applied to the project +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed while the project remains throttled +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization +And the user does not have a verified email address or has email notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. + +### Requirement: Smart throttling usage accounting remains accurate + +Smart throttling must record accepted, blocked, and discarded event counts consistently with existing usage accounting. + +#### Scenario: Sampled events accepted + +Given a project is smart-throttled +When a sample of events is accepted +Then accepted organization and project usage must increase by the number of processed events. + +#### Scenario: Non-sampled events blocked + +Given a project is smart-throttled +When some events in a post are not accepted due to throttling +Then blocked usage must increase by the number of non-accepted events. + +#### Scenario: Budget alerts use accepted events only + +Given budget alerts are enabled and a project is smart-throttled +When non-sampled events are blocked +Then blocked events must not count as accepted usage for organization budget alert thresholds. + +### Requirement: Smart throttling does not replace explicit project budgets + +Automatic smart throttling must coexist with optional project event budgets. + +#### Scenario: Project has no explicit budget + +Given a project has no configured project event budget +When the project spikes +Then automatic smart throttling may still apply. + +#### Scenario: Project has explicit budget + +Given a project has an explicit project event budget and automatic smart throttling also applies +When events are processed +Then the number of events accepted must not exceed the lowest applicable allowance from organization hard limit, project budget, and smart throttling sample. + +## ADDED Requirements + +### Requirement: Projects MUST be able to define optional event budgets + +Exceptionless projects MUST be able to define an optional project-level event budget that caps or limits the number of accepted events for that project within the current monthly usage period. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget and the organization has remaining event allowance +When an event is submitted for the project +Then the explicit project event budget must not block the event +And existing organization-level usage enforcement and automatic smart throttling must apply. + +#### Scenario: Existing projects remain uncapped + +Given a project existed before project event budgets were introduced +When the project is loaded +Then its project event budget must be treated as unset. + +### Requirement: Fixed project event budgets MUST cap accepted project events + +A fixed project event budget MUST cap accepted event volume for a project to a configured positive integer for the current monthly usage period. + +#### Scenario: Fixed project budget allows events below cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted fewer than 20000 events in the current monthly usage period +When an event is submitted for the project +Then the event must not be blocked by the explicit project event budget. + +#### Scenario: Fixed project budget limits events at cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted 20000 events in the current monthly usage period +When events are submitted for the project +Then accepted events for that project must be limited by the project event budget +And blocked usage must be recorded for non-accepted events. + +#### Scenario: Fixed project budget above organization allowance is clamped + +Given an organization has a finite monthly event allowance of 10000 events +And a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 10000 events. + +#### Scenario: Fixed project budget on unlimited organization + +Given an organization has unlimited event allowance and a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +### Requirement: Percentage project event budgets MUST derive from organization allowance + +A percentage project event budget MUST cap accepted event volume for a project to a percentage of the organization's current finite monthly event allowance. + +#### Scenario: Percentage project budget computes effective cap + +Given an organization has a finite monthly event allowance of 100000 events +And a project has a percentage event budget of 20 percent +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +#### Scenario: Percentage project budget on unlimited organization is inactive + +Given an organization has unlimited event allowance and a project has a percentage event budget +When the effective project event budget is evaluated +Then the percentage project event budget must not produce an effective project cap. + +### Requirement: Project event budgets MUST be caps, not reservations + +Project event budgets MUST prevent a project from exceeding its cap but must not reserve organization event allowance for other projects. + +#### Scenario: Unused project budget does not reserve events + +Given an organization has a monthly event allowance of 100000 events +And Project A has a percentage event budget of 20 percent and uses 0 events +When Project B submits events +Then Project B must not be blocked solely because Project A has unused project budget. + +#### Scenario: Project budget does not increase organization allowance + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project event budget exhaustion MUST limit only the capped project + +When one project reaches its project event budget, other projects in the same organization MUST continue to be evaluated independently. + +#### Scenario: One capped project does not block another project + +Given an organization has remaining event allowance +And Project A has reached its project event budget and Project B has no project event budget +When an event is submitted for Project B +Then the event must not be blocked because Project A reached its budget. + +#### Scenario: Multiple projects have independent budgets + +Given an organization has remaining event allowance +And Project A has a fixed event budget of 1000 events and Project B has a fixed event budget of 2000 events +When Project A reaches 1000 accepted events +Then Project A must be limited by its project event budget +And Project B must continue to be evaluated against its own 2000-event budget. + +### Requirement: Organization overage MUST remain the hard outer limit + +Organization-level event allowance MUST remain the hard billing/usage boundary. + +#### Scenario: Organization overage takes precedence + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project ingest controls MUST reuse existing usage accounting + +Project budget and smart throttling enforcement MUST use existing project usage counters. + +#### Scenario: Accepted event increments project usage + +Given an organization has remaining event allowance and a project has remaining project event budget +When an event is accepted for the project +Then existing project accepted event usage must be incremented. + +#### Scenario: Blocked event does not count as accepted project usage + +Given a project has reached its project event budget +When an event is submitted for the project +Then accepted project usage must not increase for non-accepted events +And blocked usage must be recorded. + +### Requirement: Project controls MUST support safe defaults on failure + +Project budget and smart throttling evaluation MUST avoid introducing new ingestion failures when project data is missing or invalid. + +#### Scenario: Missing project does not cause project-budget rejection + +Given an event submission has an organization id and the project cannot be loaded +When project controls are evaluated +Then the system must not reject the event solely due to project budget evaluation. + +#### Scenario: Invalid persisted project budget is treated as inactive + +Given a project has an invalid persisted event budget configuration +When event ingestion evaluates project controls +Then the invalid project budget must be treated as inactive. + +### Requirement: Project ingest controls MUST prefer sampled processing over full request rejection + +Project-level ingest controls MUST preserve sampled visibility where possible. + +#### Scenario: Queued event post contains many project events + +Given an event post has been queued and the project is over its smart throttling allowance +And the organization still has remaining allowance +When the event post is processed +Then the job should process an allowed sample and record blocked usage for non-sampled events. + +#### Scenario: Middleware cannot sample project events + +Given a project-level condition would require preserving only 1 to 5 percent of events +When the request is still in middleware before parsing +Then middleware must not be the only enforcement layer. + +### Requirement: Smart throttling MUST use stable five-percent sampling + +Automatic project throttling MUST select events with a stable five-percent hash sample across the complete parsed post. + +#### Scenario: Single-event post is genuinely sampled + +Given a project is smart-throttled and the organization has remaining allowance +When repeated single-event posts with stable distinct identities are processed +Then approximately five percent must be accepted over time +And no per-post minimum may force every single event through. + +#### Scenario: Nonaccepted events are counted once + +Given organization allowance, explicit project budget, and smart sampling are evaluated for one parsed post +When the final accepted set is selected +Then every event outside that set must increment blocked usage exactly once +And those events must not also increment discarded usage. + +### Requirement: Smart throttling MUST be cheap and operationally disableable + +Automatic project throttling MUST use bucket-inclusive totals, five-minute remaining windows, the existing ten-times burst tolerance, and current noisy-project fair share. + +#### Scenario: Normal unlimited traffic avoids extra work + +Given an organization has unlimited allowance and the project has no explicit cap +When ingestion evaluates the already-loaded organization and project +Then it must not reload either model or query the organization project count. + +#### Scenario: Kill switch is disabled + +Given EnableSmartProjectThrottling is false +When a noisy project submits events +Then automatic sampling must be bypassed +And explicit project and organization limits must remain enforced. diff --git a/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md new file mode 100644 index 0000000000..12eb011936 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md @@ -0,0 +1,218 @@ +# Spec: Jobs, Notifications & Queues + +## ADDED Requirements + +### Requirement: Organizations may configure budget alert thresholds + +Organizations may configure event budget alert thresholds as percentages of their effective monthly event allowance. Budget alerts are disabled by default. + +#### Scenario: Existing organization has no budget alerts + +Given an organization existed before budget alerts were introduced +When the organization is loaded +Then budget alerts must be treated as disabled. + +#### Scenario: Enable budget alerts with thresholds + +Given an authorized user can update an organization +When the user enables budget alerts with thresholds 50 and 80 +Then the organization must store budget alert settings as enabled with thresholds 50 and 80. + +#### Scenario: Disable budget alerts + +Given an organization has enabled budget alerts +When an authorized user disables budget alerts +Then budget alert emails must no longer be sent for that organization. + +### Requirement: Budget alert thresholds validate percentage values + +Budget alert threshold percentages must be valid pre-overage warning thresholds. + +#### Scenario: Reject zero threshold + +Given an authorized user can update budget alert settings +When the user configures threshold 0 +Then the API must reject the request with a validation error. + +#### Scenario: Reject threshold at or above one hundred percent + +Given an authorized user can update budget alert settings +When the user configures threshold 100 or above +Then the API must reject the request with a validation error. + +#### Scenario: Enabled alerts require at least one threshold + +Given an authorized user can update budget alert settings +When the user enables budget alerts with no thresholds +Then the API must reject the request with a validation error. + +### Requirement: Budget alerts use effective organization allowance + +Budget alert thresholds must be evaluated against the same effective organization event allowance used by existing usage enforcement. + +#### Scenario: Threshold uses plan allowance + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 +When the threshold event count is calculated +Then the threshold event count must be 50000. + +#### Scenario: Unlimited organization has no percentage budget alert + +Given an organization has unlimited event allowance +When budget alert thresholds are evaluated +Then percentage budget alerts must be inactive and no budget alert email must be sent. + +### Requirement: Budget alerts send when accepted usage crosses thresholds + +Budget alert emails must be triggered when accepted organization event usage crosses a configured threshold for the first time in a monthly usage period. + +#### Scenario: Usage below threshold does not send alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49998 +When one event is accepted +Then no budget alert email must be sent. + +#### Scenario: Usage crossing threshold sends alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49999 +When one event is accepted +Then a 50 percent budget alert email must be queued. + +#### Scenario: Usage already above threshold does not resend alert + +Given the 50 percent alert has already been sent in the current monthly usage period +When another event is accepted +Then another 50 percent budget alert email must not be queued. + +#### Scenario: Large batch crosses multiple thresholds + +Given budget alerts are enabled for thresholds 50 and 80 and accepted event usage is 45000 +When a batch of 40000 events is accepted +Then both the 50 percent and 80 percent budget alert emails must be queued. + +### Requirement: Budget alert emails are sent once per threshold per monthly usage period + +Each configured threshold may generate at most one budget alert email per organization per monthly usage period. + +#### Scenario: Threshold already sent in period + +Given the 80 percent alert was sent earlier in the current monthly usage period +When usage remains above 80 percent +Then no additional 80 percent budget alert email must be sent. + +#### Scenario: New monthly period resets sent threshold state + +Given an organization received an 80 percent alert in the previous monthly usage period +When usage crosses 80 percent in the new period +Then a new 80 percent budget alert email may be sent. + +### Requirement: Budget alert emails use organization email notification eligibility + +Budget alert emails must be sent only to organization users who are eligible for existing organization email notices. + +#### Scenario: Verified user with email notifications enabled receives alert + +Given a user belongs to an organization with a verified email and email notifications enabled +When a budget alert is sent for the organization +Then the user must receive the budget alert email. + +#### Scenario: Unverified user does not receive alert + +Given a user belongs to an organization and does not have a verified email address +When a budget alert is sent for the organization +Then the user must not receive the budget alert email. + +### Requirement: Budget alerts do not block ingestion + +Budget alert processing must not block accepted event ingestion. + +#### Scenario: Alert email enqueue fails + +Given budget alerts are enabled and usage crosses a configured threshold +When the budget alert email cannot be queued +Then the accepted event must not be rejected solely because the alert email failed. + +### Requirement: Budget alerts preserve existing overage notifications + +Budget alerts must not replace or suppress existing monthly/hourly overage notifications. + +#### Scenario: Existing monthly overage still sends + +Given an organization reaches or exceeds its monthly event allowance +When existing monthly overage notification behavior is triggered +Then the existing monthly overage email must still be sent. + +### Requirement: Budget alert emails are separate from smart throttling emails + +Organization budget alerts and project smart throttling notifications must be distinct notifications. + +#### Scenario: Project throttling below budget threshold + +Given organization budget alerts are enabled and accepted usage has not crossed a threshold +When a project enters smart-throttled state +Then no organization budget threshold email must be sent solely because smart throttling was applied. + +## ADDED Requirements + +### Requirement: Smart throttling MUST send project throttling notification email + +When smart throttling is applied to a project, Exceptionless MUST send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled and smart throttling is applied +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization and does not have a verified email or has notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. + +### Requirement: Budget alert crossings MUST be current and atomically deduplicated + +Budget alert publication MUST compare bucket-inclusive previous/current accepted totals and atomically claim each organization-threshold-period key. + +#### Scenario: Concurrent crossing + +Given concurrent accepted batches cross the same configured threshold +When each publisher attempts AddAsync for the dedupe key +Then only the successful claimant may publish the alert. + +#### Scenario: Alerts enabled above threshold + +Given accepted usage already exceeds a threshold while alerts are disabled +When an authorized user enables that threshold +Then it must be evaluated once immediately after settings are saved. + +#### Scenario: Invalid disabled threshold + +Given budget alert settings exist but are disabled +When a threshold is outside 1 through 99 +Then validation must reject the settings. + +### Requirement: Delayed usage notifications MUST suppress stale mail + +Notification handlers MUST re-check current persisted settings, plan, bucket-inclusive usage, and throttle state before sending. + +#### Scenario: Budget alert becomes stale + +Given a budget alert work item is queued +When the threshold is removed, alerts are disabled, usage is below the current threshold count, or the plan becomes unlimited before handling +Then the handler must not send the email. + +#### Scenario: Smart throttle becomes stale + +Given a smart-throttle work item is queued +When the operational kill switch is disabled or the project's current throttle state expires before handling +Then the handler must not send the email. diff --git a/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md new file mode 100644 index 0000000000..18c006aa47 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md @@ -0,0 +1,115 @@ +# Spec: Organizations, Projects, Users & Auth + +## ADDED Requirements + +### Requirement: Budget alert settings belong to an organization + +Budget alert settings belong to an organization and may be changed only by users authorized to update that organization. + +#### Scenario: Authorized organization user updates budget alerts + +Given a user is authorized to update an organization +When the user updates budget alert settings +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update budget alerts + +Given a user is not authorized to update an organization +When the user attempts to update budget alert settings +Then the API must reject the operation according to existing organization authorization behavior. + +## ADDED Requirements + +### Requirement: Budget alert email recipients respect user preferences + +Budget alert email recipients must respect existing user email verification and email notification preferences. + +#### Scenario: User email notifications disabled + +Given a user belongs to an organization and has email notifications disabled +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: User email unverified + +Given a user belongs to an organization and the user's email address is not verified +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: Verified user with email notifications enabled + +Given a user belongs to an organization with a verified email and notifications enabled +When a budget alert is sent +Then the user may receive the budget alert email. + +## ADDED Requirements + +### Requirement: Project event budget authorization follows project authorization + +A project event budget belongs to a project and is scoped by the project's owning organization. Only users authorized to update the project may configure its event budget. + +#### Scenario: Authorized project user updates project event budget + +Given a user is authorized to update a project +When the user updates the project's event budget +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update project event budget + +Given a user is not authorized to update a project +When the user attempts to update the project's event budget +Then the API must reject the operation according to existing project authorization behavior. + +#### Scenario: Project event budget cannot grant organization capacity + +Given a user configures a project event budget +When events are submitted for the project +Then the project event budget must not allow accepted event usage beyond the organization's effective allowance. + +## ADDED Requirements + +### Requirement: Budget controls preserve existing token and auth behavior + +Budget alert settings, smart project throttling, and project event budget configuration must not change API key authentication, token scopes, or user authorization roles. + +#### Scenario: Existing project token behavior is preserved + +Given an existing project API key has client scope and the project has no configured event budget +When the key is used for event submission +Then token authentication and scope behavior must match existing behavior. + +#### Scenario: Project smart throttling does not disable API key + +Given a project is under smart throttling +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project is smart-throttled. + +#### Scenario: Project budget does not disable API key + +Given a project reaches its project event budget +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project reached its event budget. + +#### Scenario: Disabled or suspended token remains rejected + +Given a project has remaining project event budget and the API key is disabled or suspended +When an event is submitted with that API key +Then the event must still be rejected due to token authentication state. + +## ADDED Requirements + +### Requirement: Project event budget MUST NOT preclude future token-level caps + +Project event budget and smart throttling behavior MUST NOT preclude future API-key/token-level ingest caps. + +#### Scenario: Future token cap is absent + +Given token-level ingest caps are not implemented +When a project event budget or smart throttling is evaluated +Then enforcement must consider organization and project scopes only. + +#### Scenario: Project budget remains independent of token count + +Given a project has multiple API keys and a configured project event budget +When events are submitted using different API keys for that project +Then all accepted events for the project must count against the same project event budget. diff --git a/openspec/changes/add-usage-budget-controls/tasks.md b/openspec/changes/add-usage-budget-controls/tasks.md new file mode 100644 index 0000000000..d63e599416 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/tasks.md @@ -0,0 +1,368 @@ +# Tasks: Add Usage Budget Controls + +## OpenSpec + +- [x] 1. Create OpenSpec change + * Create proposal, design, tasks, and spec deltas for: + * organization budget alerts + * automatic smart project throttling + * project event budgets + * API compatibility + * organization/project auth behavior + * Verification: + * openspec validate add-usage-budget-controls --strict --no-interactive + +## Backend model and API contract + +- [x] 2. Add organization budget alert settings model + * Add OrganizationBudgetAlertSettings. + * Add nullable BudgetAlertSettings to Organization. + * Validate every present threshold as 1-99 even while disabled; enabled settings require at least one threshold. + * Verification: + * dotnet build + * Add model/validation tests if existing model validation tests are available. +- [x] 3. Add budget alert settings to organization API contract + * Add BudgetAlertSettings to ViewOrganization. + * Prefer adding UpdateOrganization with Name and BudgetAlertSettings. + * Update OrganizationController generic update DTO if using UpdateOrganization. + * Regenerate generated API and Svelte schemas. + * Verification: + * dotnet build + * dotnet test -- --filter-class OrganizationControllerTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check +- [x] 4. Add project event budget domain model + * Add ProjectIngestLimit and ProjectIngestLimitType under src/Exceptionless.Core/Models/. + * Add nullable Project.IngestLimit. + * Add validation for fixed and percentage modes. + * Verification: + * dotnet build + * Add/adjust serializer tests if project model serialization coverage exists. +- [x] 5. Add project event budget to project DTOs + * Add ProjectIngestLimit? IngestLimit to UpdateProject. + * Add ProjectIngestLimit? IngestLimit to ViewProject. + * Add int? EffectiveIngestLimit to ViewProject. + * Add smart throttling state fields to ViewProject if feasible: + * bool IsSmartThrottled + * double? SmartThrottleSampleRate + * Ensure Mapperly maps the new fields or add explicit mapping if required. + * Verification: + * dotnet build + * dotnet test -- --filter-class ProjectControllerTests +- [x] 6. Regenerate OpenAPI and Svelte generated types + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts. + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts. + * Confirm UpdateOrganization, ViewOrganization, UpdateProject, ViewProject, and validation schemas include new fields. + * Generate nullable complex Delta properties as `null | $ref` from the canonical transformer and add regression coverage. + * Verification: + * dotnet build + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + +## Organization budget alert emails + +- [x] 7. Add budget alert threshold calculation + * Use the same effective organization allowance as existing usage enforcement. + * Support configured percentage thresholds from 1 through 99. + * Treat thresholds as inactive for unlimited organizations. + * Verification: + * Add tests to UsageServiceTests. + * dotnet test -- --filter-class UsageServiceTests +- [x] 8. Publish budget alert messages when thresholds are crossed + * Add OrganizationBudgetAlert message. + * In UsageService.IncrementTotalAsync, detect threshold crossings after accepted event usage increments. + * Compare bucket-inclusive previous/current totals and immediately evaluate newly enabled thresholds after settings save. + * Publish one message per crossed threshold. + * Do not publish repeatedly after threshold was already sent in the current monthly usage period. + * Verification: + * Tests: + * below threshold does not publish + * crossing threshold publishes + * already-sent threshold does not publish + * batch crossing multiple thresholds publishes each crossed threshold + * dotnet test -- --filter-class UsageServiceTests +- [x] 9. Add budget alert dedupe + * Add dedupe key per organization, threshold, and monthly usage period. + * Atomically claim each key with `AddAsync` and set TTL through the end of the monthly usage period plus a safety buffer. + * Prefer non-blocking behavior if dedupe cache fails. + * Verification: + * Add tests for one-email-per-threshold-per-period behavior. + * dotnet test -- --filter-class UsageServiceTests +- [x] 10. Add budget alert work item and subscriber + * Add OrganizationBudgetAlertWorkItem. + * Add startup subscriber that turns OrganizationBudgetAlert messages into work items. + * Add work item handler that loads organization/users and sends emails. + * Re-check current organization budget alert settings before sending. + * Re-check current usage and effective plan allowance before sending stale queued work. + * Verification: + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * If no dedicated handler test class exists, add one. +- [x] 11. Add budget alert mailer method and template + * Add IMailer.SendOrganizationBudgetAlertAsync. + * Implement the method in Mailer. + * Add organization-budget-alert.html template. + * Include current usage, threshold percentage, threshold event count, event limit, remaining events, and links to organization usage/billing. + * Verification: + * Mailer/template rendering test if existing template tests exist. + * dotnet build + +## Automatic Smart Project Throttling + +- [x] 12. Add smart throughput calculation + * Calculate allowed throughput from events left in the monthly period and time/windows left in the period. + * Use five-minute windows and preserve the existing 10x burst tolerance. + * Avoid static plan-size-only calculations. + * Verification: + * Tests for early-month, late-month, high-remaining, and low-remaining allowance cases. + * dotnet test -- --filter-class UsageServiceTests +- [x] 13. Add project-level smart throttling decision + * Detect projects contributing to usage spikes. + * Apply smart throttling to noisy projects where possible instead of organization-wide full blocking. + * Use the canonically invalidated repository project-count cache only after spike criteria require fair-share evaluation. + * Add `EnableSmartProjectThrottling`, default enabled, as an operational kill switch. + * Verification: + * Tests where Project A spikes and Project B remains unaffected. + * Tests where multiple projects are evaluated independently. + * dotnet test -- --filter-class UsageServiceTests +- [x] 14. Add sampled acceptance for smart-throttled projects + * Implement fixed 5% stable hash sampling across the whole batch, including true sampling of single-event posts. + * Count every nonaccepted event exactly once as blocked, never again as discarded. + * Verification: + * Tests for sampled acceptance count. + * Tests that blocked usage increments for non-sampled events. + * Tests that accepted usage increments only for sampled events. + * dotnet test -- --filter-class EventPostsJobTests +- [x] 15. Move project-level sampled enforcement into event post processing + * Extend EventPostsJob / UsageService after event post parsing to calculate allowed events. + * Avoid using only OverageMiddleware for project-level sampled enforcement. + * Preserve existing organization hard overage middleware behavior. + * Verification: + * Tests for batch event posts where only a sample is processed. + * Tests for organization hard overage still blocking. + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class OverageMiddlewareTests +- [x] 16. Add smart throttling notification message/work item/email + * Add project throttling notification message. + * Add work item and handler. + * Add mailer method and template. + * Send to verified users with email notifications enabled. + * Deduplicate notifications during a throttling cooldown window. + * Re-check current plan, usage, and throttle state before sending; email fields describe project usage and fair-share limits. + * Add a dedicated smart-throttled event metric and transition-only structured logging. + * Verification: + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet build + +## Optional project event budgets + +- [x] 17. Refactor UsageService events-left calculation + * Make one allowance decision from the organization/project models already loaded by EventPostsJob. + * Bulk-read bucket-inclusive organization/project totals using existing project-aware cache key helpers. + * Preserve current organization behavior. + * Verification: + * Add tests for organization-only events-left behavior to ensure no regression. + * dotnet test -- --filter-class UsageServiceTests +- [x] 18. Add project effective budget calculation + * Implement effective fixed budget calculation. + * Implement effective percentage budget calculation. + * Clamp fixed budgets to finite organization allowance during enforcement. + * Treat percentage budgets as inactive when organization allowance is unlimited. + * Verification: + * dotnet test -- --filter-class UsageServiceTests +- [x] 19. Add parsed-event ingest allowance result + * Add EventIngestAllowanceResult. + * Add UsageService.GetEventIngestAllowanceAsync(Organization organization, Project project) or equivalent without repository reloads. + * Return event count allowed, organization remaining, project remaining, effective project budget, and smart throttling state. + * Verification: + * Tests: + * no project id or missing project falls back safely + * no project budget uses organization + smart throttling + * fixed budget limits allowed event count + * percentage budget computes effective cap from organization allowance + * organization overage takes precedence over project controls + * smart throttling sample and project budget combine by taking the minimum allowed count + * dotnet test -- --filter-class UsageServiceTests + +## Middleware and event processing + +- [x] 20. Keep OverageMiddleware as coarse gate + * Preserve existing status codes for organization overage, disabled submission, missing content length, oversized posts, and unauthorized organization context. + * Do not implement sampled project enforcement only in middleware. + * Verification: + * dotnet test -- --filter-class OverageMiddlewareTests +- [x] 21. Update EventPostsJob for sampled/project-budget allowance + * After parsing events, ask UsageService for allowed event count. + * Process only allowed events. + * Use deterministic sampling/selection when allowed count is lower than submitted count due to smart throttling. + * Increment blocked usage for non-accepted events. + * Increment blocked usage once after all applicable controls have selected the final accepted set. + * Increment accepted usage only for processed events. + * Verification: + * dotnet test -- --filter-class EventPostsJobTests + +## Backend API tests + +- [x] 22. Add OrganizationController tests for budget alert settings + * Test enabling budget alerts with thresholds. + * Test disabling budget alerts. + * Test threshold normalization/deduplication. + * Test invalid thresholds reject. + * Test unauthorized organization update remains rejected. + * Verification: + * dotnet test -- --filter-class OrganizationControllerTests +- [x] 23. Add ProjectController tests for event budget update behavior + * Test setting fixed event budget. + * Test setting percentage event budget. + * Test clearing event budget. + * Test invalid fixed budget rejects. + * Test invalid percentage rejects. + * Test percentage budget rejects for unlimited organization or is marked inactive according to final implementation decision. + * Verification: + * dotnet test -- --filter-class ProjectControllerTests +- [x] 24. Add event submission integration tests + * Verify event submission processing accepts samples under smart throttling. + * Verify organization overage behavior remains unchanged. + * Verify uncapped projects continue under the organization limit. + * Verify crossing budget alert thresholds does not change accepted event response. + * Verification: + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests +- [x] 25. Update HTTP samples if contracts are changed + * Update tests/http/*.http samples for organization update/get if organization request/response examples exist. + * Update tests/http/*.http samples for project update/get if project request/response examples exist. + * Verification: + * Manual review of tests/http/*.http. + * dotnet build + +## Svelte UI + +- [x] 26. Add budget alerts UI to Organization Usage page + * Update src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte. + * Add card for enabling/disabling budget alerts. + * Add threshold editor with default suggestions of 50 and 80. + * Use TanStack Form, Zod, existing shadcn components, accessible labels/live errors, and explicit loading/saving states. + * Show computed event counts for each threshold. + * Disable percentage alerts for unlimited organizations. + * Use existing organization API query/mutation patterns. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA on Organization Settings → Usage. +- [x] 27. Add project event budget UI component + * Create src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte. + * Support Off, Fixed, and Percentage modes. + * Show computed effective cap. + * Show current project usage against cap when active. + * Show fixed-cap warning when fixed cap exceeds current organization allowance. + * Disable percentage mode when organization allowance is unlimited. + * Reject truncated fixed-limit decimals and enforce backend-identical fixed/percentage ranges. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint +- [x] 28. Add smart throttling UI status + * Surface project smart-throttled status on Project Usage where feasible. + * Explain that a sample of events is still accepted. + * Link to organization/project usage. + * Avoid exposing many tuning options. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA with a smart-throttled project. +- [x] 29. Integrate project card into Project Usage page + * Add the card to src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte. + * Use existing getProjectQuery, getOrganizationQuery, and updateProject mutation. + * Invalidate/update project query data after saving. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * Manual localhost QA on /project/{projectId}/usage +- [x] 30. Update project usage chart limit/status display + * Use effective_ingest_limit as the chart limit when present. + * Otherwise preserve existing organization limit behavior. + * Label the limit line as Project Limit or Organization Limit. + * Show smart throttling status text/badge when active. + * Verification: + * Manual localhost QA with: + * no project budget + * fixed project budget + * percentage project budget + * smart-throttled project + * Confirm chart remains readable and accessible. +- [x] 31. Add frontend tests for budget control helpers + * Test organization budget alert threshold normalization. + * Test organization budget alert threshold validation. + * Test budget alert computed event counts. + * Test project budget mode-to-payload conversion. + * Test project percentage cap calculation display. + * Test fixed cap warning. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run test:unit + +## Compatibility and validation + +- [x] 32. Verify existing request throttling remains unchanged + * Confirm ThrottlingMiddleware behavior and tests are unchanged unless incidental generated code changes require updates. + * Verification: + * dotnet test -- --filter-class ThrottlingMiddlewareTests +- [x] 33. Verify existing overage notification remains unchanged + * Confirm existing PlanOverage monthly/hourly paths continue to enqueue and send existing organization notices. + * Confirm budget alerts and smart throttling use separate message/work item/template. + * Verification: + * Existing organization notification tests if present. + * New budget alert and smart throttling work item tests. +- [x] 34. Verify no Elasticsearch mapping/reindex is required + * Confirm budget alert settings and project event budget are not used for organization/project search/filter/sort. + * Confirm no organization/project index version bump is required. + * Verification: + * Code review checklist item. +- [x] 35. Local dogfood budget alerts + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Enable budget alerts for a low threshold. + * Submit events until threshold is crossed. + * Confirm budget alert work item/email is queued. + * Confirm alert does not send twice for same threshold in same monthly period. + * Confirm disabling alerts prevents future sends. + * Verification: + * Manual QA notes in PR. +- [x] 36. Local dogfood project budgets and smart throttling + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Configure a low fixed budget on a project. + * Submit events until budget is reached. + * Confirm: + * project is limited/sampled + * blocked usage increases + * another project can still ingest events + * clearing the budget restores behavior + * Create or simulate a noisy project. + * Confirm smart throttling applies without manual configuration. + * Confirm a sample is still accepted. + * Verification: + * Manual QA notes in PR. + +## Final validation + +- [x] 37. Run targeted validation + * Keep budget-specific tests out of UsageServiceTests.cs so the existing file does not grow past 1,000 lines. + * Commands: + * dotnet build + * dotnet test -- --filter-class UsageServiceTests + * dotnet test -- --filter-class OverageMiddlewareTests + * dotnet test -- --filter-class OrganizationControllerTests + * dotnet test -- --filter-class ProjectControllerTests + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class ThrottlingMiddlewareTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * cd src/Exceptionless.Web/ClientApp && npm run test:unit +- [x] 38. Run OpenSpec validation + * Command: + * openspec validate add-usage-budget-controls --strict --no-interactive +- [x] 39. Run full validation before merge + * Commands: + * dotnet test + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check && npm run lint && npm run test:unit && npm run build + * openspec validate --all --strict --no-interactive + * git diff --check + * Re-run the thermo-nuclear review against `origin/main...HEAD` and require no structural, hot-path, file-size, or generated-artifact drift blockers. diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index efcfae9efb..6da15491c6 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -70,6 +70,7 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO ResiliencePolicyProvider = s.GetRequiredService(), LoggerFactory = s.GetRequiredService() })); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(s => s.GetRequiredService().Client); @@ -90,6 +91,8 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); diff --git a/src/Exceptionless.Core/Configuration/AppOptions.cs b/src/Exceptionless.Core/Configuration/AppOptions.cs index ff6217829a..e6f47712c7 100644 --- a/src/Exceptionless.Core/Configuration/AppOptions.cs +++ b/src/Exceptionless.Core/Configuration/AppOptions.cs @@ -43,6 +43,11 @@ public class AppOptions public bool EventSubmissionDisabled { get; internal set; } + /// + /// Operational kill switch for automatic project-level sampling. Explicit project limits are unaffected. + /// + public bool EnableSmartProjectThrottling { get; internal set; } + internal List DisabledPipelineActions { get; set; } = null!; internal List DisabledPlugins { get; set; } = null!; @@ -99,6 +104,7 @@ public static AppOptions ReadFromConfiguration(IConfiguration config) options.EnableArchive = config.GetValue(nameof(options.EnableArchive), false); options.EnableSampleData = config.GetValue(nameof(options.EnableSampleData), options.AppMode == AppMode.Development); options.EventSubmissionDisabled = config.GetValue(nameof(options.EventSubmissionDisabled), false); + options.EnableSmartProjectThrottling = config.GetValue(nameof(options.EnableSmartProjectThrottling), true); options.DisabledPipelineActions = config.GetValueList(nameof(options.DisabledPipelineActions)); options.DisabledPlugins = config.GetValueList(nameof(options.DisabledPlugins)); options.MaximumEventPostSize = config.GetValue(nameof(options.MaximumEventPostSize), 200000).NormalizeValue(); diff --git a/src/Exceptionless.Core/Exceptionless.Core.csproj b/src/Exceptionless.Core/Exceptionless.Core.csproj index 9ff9a9608d..f6b70d923e 100644 --- a/src/Exceptionless.Core/Exceptionless.Core.csproj +++ b/src/Exceptionless.Core/Exceptionless.Core.csproj @@ -3,10 +3,12 @@ + + @@ -14,10 +16,12 @@ + + diff --git a/src/Exceptionless.Core/Jobs/EventPostsJob.cs b/src/Exceptionless.Core/Jobs/EventPostsJob.cs index 0b7aaffec3..f1b4cc3505 100644 --- a/src/Exceptionless.Core/Jobs/EventPostsJob.cs +++ b/src/Exceptionless.Core/Jobs/EventPostsJob.cs @@ -162,35 +162,55 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex return JobResult.Success; } - // Don't process all the events if it will put the account over its limits. - int eventsToProcess = await _usageService.GetEventsLeftAsync(organization.Id); - if (eventsToProcess < 1) + int submittedEventCount = events.Count; + bool isSingleEvent = submittedEventCount == 1; + var candidates = events + .Select((persistentEvent, index) => new EventCandidate(persistentEvent, index, GetStableEventHash(entry.Id, persistentEvent, index))) + .ToList(); + var reservation = await _usageService.ReserveEventIngestAsync( + organization, + project, + entry.Id, + candidates.Select(candidate => new EventIngestCandidate(candidate.Index, candidate.Hash)).ToArray(), + context.CancellationToken); + if (reservation.IsCompleted) { - if (!isInternalProject) - _logger.LogDebug("Unable to process EventPost {FilePath}: Over plan limits", payloadPath); - - await _usageService.IncrementBlockedAsync(organization.Id, project.Id, events.Count); - - await CompleteEntryAsync(entry, ep, _timeProvider.GetUtcNow().UtcDateTime); + await _usageService.CompleteEventIngestReservationAsync(reservation, organization, reservation.ProcessedCount); + await CompleteEntryAsync(entry, ep, createdUtc); return JobResult.Success; } - // Keep track of the original event payload size, we can save some processing for retries in the case it was a massive batch. - bool isSingleEvent = events.Count == 1; - - // Discard any events over the plan limit. - if (eventsToProcess < events.Count) - { - int discarded = events.Count - eventsToProcess; - events = events.Take(eventsToProcess).ToList(); - - await _usageService.IncrementBlockedAsync(organization.Id, project.Id, discarded); - } - int errorCount = 0; + bool reservationCompleted = false; + bool reservationCompletionStarted = false; var eventsToRetry = new List(); try { + var acceptedIndexes = reservation.AcceptedIndexes.ToHashSet(); + candidates = candidates.Where(candidate => acceptedIndexes.Contains(candidate.Index)).ToList(); + _usageService.RecordSmartThrottle(reservation.SmartThrottleBlockedCount); + + events = candidates + .OrderBy(candidate => candidate.Index) + .Select(candidate => candidate.Event) + .ToList(); + + int blockedEventCount = submittedEventCount - events.Count; + if (blockedEventCount > 0) + await _usageService.IncrementBlockedAsync(organization.Id, project.Id, blockedEventCount); + + if (events.Count == 0) + { + if (!isInternalProject) + _logger.LogDebug("Unable to process EventPost {FilePath}: accepted 0/{SubmittedEventCount} events after usage budget evaluation", payloadPath, submittedEventCount); + + reservationCompletionStarted = true; + await _usageService.CompleteEventIngestReservationAsync(reservation, organization, 0); + reservationCompleted = true; + await CompleteEntryAsync(entry, ep, _timeProvider.GetUtcNow().UtcDateTime); + return JobResult.Success; + } + var contexts = await _eventPipeline.RunAsync(events, organization, project, ep); if (!isInternalProject && isDebugLogLevelEnabled) { @@ -200,7 +220,9 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex // increment the plan usage counters (note: OverageHandler already incremented usage by 1) int processedEvents = contexts.Count(c => c.IsProcessed); - await _usageService.IncrementTotalAsync(organization.Id, project.Id, processedEvents); + reservationCompletionStarted = true; + await _usageService.CompleteEventIngestReservationAsync(reservation, organization, processedEvents); + reservationCompleted = true; int discardedEvents = contexts.Count(c => c.IsDiscarded); await _usageService.IncrementDiscardedAsync(organization.Id, project.Id, discardedEvents); @@ -228,6 +250,14 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex catch (Exception ex) { if (!isInternalProject) _logger.LogError(ex, "Error processing EventPost {QueueEntryId} {FilePath}: {Message}", entry.Id, payloadPath, ex.Message); + if (reservationCompletionStarted) + { + // Completion is an idempotent transition keyed by this queue entry. Never fork new + // queue IDs after it starts: retry the original so it can observe Active vs Completed. + await AbandonEntryAsync(entry); + return JobResult.FailedWithMessage($"Unable to finalize usage for EventPost '{entry.Id}': {ex.Message}"); + } + if (ex is ArgumentException || ex is DocumentNotFoundException) { await CompleteEntryAsync(entry, ep, createdUtc); @@ -238,6 +268,11 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex if (!isSingleEvent) eventsToRetry.AddRange(events); } + finally + { + if (!reservationCompleted && !reservationCompletionStarted) + await _usageService.ReleaseEventIngestReservationAsync(reservation); + } if (eventsToRetry.Count > 0) await AppDiagnostics.PostsRetryTime.TimeAsync(() => RetryEventsAsync(eventsToRetry, ep, entry, project, isInternalProject)); @@ -250,6 +285,34 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex return JobResult.Success; } + private static ulong GetStableEventHash(string queueEntryId, PersistentEvent persistentEvent, int index) + { + string value = $"{queueEntryId}:{index}:{persistentEvent.ReferenceId}"; + const ulong offsetBasis = 14695981039346656037; + const ulong prime = 1099511628211; + ulong hash = offsetBasis; + foreach (char character in value) + { + hash ^= character; + hash *= prime; + } + + return hash; + } + + internal static bool IsSelectedForSmartThrottle(string queueEntryId, PersistentEvent persistentEvent, int index, double sampleRate = SmartThrottleResult.DefaultSampleRate) + { + int sampleThreshold = (int)(sampleRate * 10_000); + return IsHashSelected(GetStableEventHash(queueEntryId, persistentEvent, index), sampleThreshold); + } + + private static bool IsHashSelected(ulong hash, int sampleThreshold) + { + return hash % 10_000 < (ulong)sampleThreshold; + } + + private sealed record EventCandidate(PersistentEvent Event, int Index, ulong Hash); + private List? ParseEventPost(EventPostInfo ep, DateTime createdUtc, byte[] uncompressedData, string queueEntryId, bool isInternalProject) { using (_logger.BeginScope(new ExceptionlessState().Tag("parsing"))) diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs new file mode 100644 index 0000000000..5ea20e7186 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs @@ -0,0 +1,150 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.DateTimeExtensions; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Lock; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueOrganizationBudgetAlertOnUsageThreshold : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + + public EnqueueOrganizationBudgetAlertOnUsageThreshold(IQueue workItemQueue, IMessageSubscriber subscriber, TimeProvider timeProvider, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _timeProvider = timeProvider; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async alert => + { + _logger.LogInformation("Enqueueing budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", alert.OrganizationId, alert.Threshold); + + await _workItemQueue.EnqueueAsync(new OrganizationBudgetAlertWorkItem + { + OrganizationId = alert.OrganizationId, + Threshold = alert.Threshold, + ThresholdEventCount = alert.ThresholdEventCount, + CurrentEventCount = alert.CurrentEventCount, + EventLimit = alert.EventLimit, + UsagePeriod = alert.UsagePeriod > 0 ? alert.UsagePeriod : _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }); + }, shutdownToken); + } +} + +public class OrganizationBudgetAlertWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + private readonly UsageService _usageService; + private readonly NotificationService _notificationService; + private readonly TimeProvider _timeProvider; + + public OrganizationBudgetAlertWorkItemHandler(IOrganizationRepository organizationRepository, IUserRepository userRepository, IMailer mailer, UsageService usageService, + NotificationService notificationService, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _userRepository = userRepository; + _mailer = mailer; + _usageService = usageService; + _notificationService = notificationService; + _timeProvider = timeProvider; + } + + public override Task GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default) + { + var wi = (OrganizationBudgetAlertWorkItem)workItem; + int currentUsagePeriod = _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch(); + return _notificationService.TryAcquireUsageNotificationLockAsync(wi.GetUniqueIdentifier(currentUsagePeriod)); + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", wi.OrganizationId, wi.Threshold); + + int usagePeriod = wi.UsagePeriod > 0 ? wi.UsagePeriod : _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch(); + string notificationIdentifier = wi.GetUniqueIdentifier(usagePeriod); + if (usagePeriod != _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch()) + { + Log.LogInformation("Budget alert period {UsagePeriod} is stale for organization {OrganizationId}, skipping", wi.UsagePeriod, wi.OrganizationId); + return; + } + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping budget alert", wi.OrganizationId); + return; + } + + // Re-check that budget alerts are still enabled and the threshold is still configured + if (organization.BudgetAlertSettings is not { Enabled: true, Thresholds: not null } || !organization.BudgetAlertSettings.Thresholds.Contains(wi.Threshold)) + { + Log.LogInformation("Budget alerts disabled or threshold {Threshold}% removed for organization: {OrganizationId}, skipping", wi.Threshold, wi.OrganizationId); + return; + } + + int eventLimit = await _usageService.GetMaxEventsPerMonthAsync(organization.Id); + if (eventLimit <= 0) + { + Log.LogInformation("Organization {OrganizationId} no longer has a finite event allowance, skipping budget alert", wi.OrganizationId); + return; + } + + int thresholdEventCount = UsageService.GetBudgetThresholdEventCount(eventLimit, wi.Threshold); + int currentEventCount = (await _usageService.GetUsageAsync(organization.Id)).CurrentUsage.Total; + if (currentEventCount < thresholdEventCount) + { + Log.LogInformation("Organization {OrganizationId} usage is now below budget threshold {Threshold}%, skipping", wi.OrganizationId, wi.Threshold); + return; + } + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + if (await _notificationService.IsUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id)) + continue; + + await using var recipientLock = await _notificationService.TryAcquireUsageNotificationRecipientLockAsync(notificationIdentifier, user.Id); + if (recipientLock is null || await _notificationService.IsUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id)) + continue; + + Log.LogTrace("Sending budget alert email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendOrganizationBudgetAlertAsync(user, organization, wi.Threshold, thresholdEventCount, currentEventCount, eventLimit); + await _notificationService.MarkUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id, usagePeriod); + } + + Log.LogTrace("Done sending budget alert emails"); + } +} diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs new file mode 100644 index 0000000000..16a3dc8462 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs @@ -0,0 +1,155 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.DateTimeExtensions; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Lock; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueProjectSmartThrottleOnThrottleApplied : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + + public EnqueueProjectSmartThrottleOnThrottleApplied(IQueue workItemQueue, IMessageSubscriber subscriber, TimeProvider timeProvider, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _timeProvider = timeProvider; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async throttle => + { + _logger.LogInformation("Enqueueing smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", throttle.ProjectId, throttle.OrganizationId); + + await _workItemQueue.EnqueueAsync(new ProjectSmartThrottleWorkItem + { + OrganizationId = throttle.OrganizationId, + ProjectId = throttle.ProjectId, + SampleRate = throttle.SampleRate, + CurrentEventCount = throttle.CurrentEventCount, + EventLimit = throttle.EventLimit, + UsagePeriod = throttle.UsagePeriod > 0 ? throttle.UsagePeriod : _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }); + }, shutdownToken); + } +} + +public class ProjectSmartThrottleWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IProjectRepository _projectRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + private readonly UsageService _usageService; + private readonly NotificationService _notificationService; + private readonly TimeProvider _timeProvider; + + public ProjectSmartThrottleWorkItemHandler(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, IUserRepository userRepository, IMailer mailer, UsageService usageService, + NotificationService notificationService, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _projectRepository = projectRepository; + _userRepository = userRepository; + _mailer = mailer; + _usageService = usageService; + _notificationService = notificationService; + _timeProvider = timeProvider; + } + + public override Task GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default) + { + var wi = (ProjectSmartThrottleWorkItem)workItem; + int currentUsagePeriod = _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch(); + return _notificationService.TryAcquireUsageNotificationLockAsync(wi.GetUniqueIdentifier(currentUsagePeriod)); + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", wi.ProjectId, wi.OrganizationId); + + int usagePeriod = wi.UsagePeriod > 0 ? wi.UsagePeriod : _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch(); + string notificationIdentifier = wi.GetUniqueIdentifier(usagePeriod); + if (usagePeriod != _timeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch()) + { + Log.LogInformation("Smart throttle period {UsagePeriod} is stale for project {ProjectId}, skipping", wi.UsagePeriod, wi.ProjectId); + return; + } + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping smart throttle notification", wi.OrganizationId); + return; + } + + var project = await _projectRepository.GetByIdAsync(wi.ProjectId, o => o.Cache()); + if (project is null) + { + Log.LogWarning("Project {ProjectId} not found, skipping smart throttle notification", wi.ProjectId); + return; + } + + if (!String.Equals(project.OrganizationId, organization.Id, StringComparison.Ordinal) || + !await _usageService.IsProjectSmartThrottledAsync(organization.Id, project.Id)) + { + Log.LogInformation("Project {ProjectId} is no longer smart throttled, skipping notification", wi.ProjectId); + return; + } + + int organizationLimit = await _usageService.GetMaxEventsPerMonthAsync(organization.Id); + if (organizationLimit <= 0) + { + Log.LogInformation("Organization {OrganizationId} no longer has a finite event allowance, skipping smart throttle notification", wi.OrganizationId); + return; + } + + int projectCount = (int)Math.Max(1, (await _projectRepository.GetCountByOrganizationIdAsync(organization.Id)).Total); + int fairShareLimit = organizationLimit / projectCount; + int currentProjectUsage = (await _usageService.GetUsageAsync(organization.Id, project.Id)).CurrentUsage.Total; + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + if (await _notificationService.IsUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id)) + continue; + + await using var recipientLock = await _notificationService.TryAcquireUsageNotificationRecipientLockAsync(notificationIdentifier, user.Id); + if (recipientLock is null || await _notificationService.IsUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id)) + continue; + + Log.LogTrace("Sending smart throttle email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendProjectThrottledNoticeAsync(user, organization, project, wi.SampleRate, currentProjectUsage, fairShareLimit); + await _notificationService.MarkUsageNotificationRecipientSentAsync(notificationIdentifier, user.Id, usagePeriod); + } + + Log.LogTrace("Done sending smart throttle emails"); + } +} diff --git a/src/Exceptionless.Core/Mail/IMailer.cs b/src/Exceptionless.Core/Mail/IMailer.cs index 6e8905cab5..7a121864ed 100644 --- a/src/Exceptionless.Core/Mail/IMailer.cs +++ b/src/Exceptionless.Core/Mail/IMailer.cs @@ -11,6 +11,8 @@ public interface IMailer Task SendOrganizationNoticeAsync(User user, Organization organization, bool isOverMonthlyLimit, bool isOverHourlyLimit); Task SendOrganizationPaymentFailedAsync(User owner, Organization organization); Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan); + Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit); + Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit); Task SendUserEmailVerifyAsync(User user); Task SendUserPasswordResetAsync(User user); } diff --git a/src/Exceptionless.Core/Mail/Mailer.cs b/src/Exceptionless.Core/Mail/Mailer.cs index 16394cca4d..c24e739983 100644 --- a/src/Exceptionless.Core/Mail/Mailer.cs +++ b/src/Exceptionless.Core/Mail/Mailer.cs @@ -231,6 +231,57 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza }, template); } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + const string template = "organization-budget-alert"; + string subject = $"[{organization.Name}] Budget Alert: {threshold}% of monthly event allowance used"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "Threshold", threshold }, + { "ThresholdEventCount", thresholdEventCount }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit }, + { "RemainingEventCount", Math.Max(0, eventLimit - currentEventCount) } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + const string template = "project-smart-throttle"; + string subject = $"[{organization.Name}] Smart Throttling Active: {project.Name}"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "ProjectId", project.Id }, + { "ProjectName", project.Name }, + { "SampleRate", sampleRate }, + { "SamplePercent", (int)(sampleRate * 100) }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { const string template = "project-daily-summary"; diff --git a/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html new file mode 100644 index 0000000000..c85c5cdc31 --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

{{OrganizationName}} has reached {{Threshold}}% of its monthly event allowance.

The organization has accepted {{CurrentEventCount}} events. The {{Threshold}}% alert is {{ThresholdEventCount}} events, the current allowance is {{EventLimit}}, and {{RemainingEventCount}} events remain.

View Usage
                                                           
\ No newline at end of file diff --git a/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html new file mode 100644 index 0000000000..a8891b00dd --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

Smart throttling is active for {{ProjectName}} in {{OrganizationName}}.

This project has accepted {{CurrentEventCount}} events this month, above its current fair-share limit of {{EventLimit}} events. To protect the organization's remaining allowance, Exceptionless is accepting a stable {{SamplePercent}}% sample from this project. Events outside that sample are counted as blocked.

Other projects are unaffected unless they independently meet the same smart-throttling criteria.

View Project Usage
                                                           
\ No newline at end of file diff --git a/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs new file mode 100644 index 0000000000..b7c5ae07f9 --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs @@ -0,0 +1,11 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + public int UsagePeriod { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs new file mode 100644 index 0000000000..b184414611 --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs @@ -0,0 +1,11 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + public int UsagePeriod { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Organization.cs b/src/Exceptionless.Core/Models/Organization.cs index 0a92a03660..0c13e6ccf7 100644 --- a/src/Exceptionless.Core/Models/Organization.cs +++ b/src/Exceptionless.Core/Models/Organization.cs @@ -174,6 +174,11 @@ public Organization() /// public DataDictionary? Data { get; set; } + /// + /// Budget alert settings for this organization. Null means disabled. + /// + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } + public DateTime CreatedUtc { get; set; } public DateTime UpdatedUtc { get; set; } public bool IsDeleted { get; set; } @@ -268,6 +273,19 @@ public IEnumerable Validate(ValidationContext validationContex [nameof(SuspendedByUserId)]); } } + + if (BudgetAlertSettings is not null) + { + if (BudgetAlertSettings.Enabled && BudgetAlertSettings.Thresholds is not { Count: > 0 }) + { + yield return new ValidationResult("At least one threshold is required when budget alerts are enabled.", + [nameof(BudgetAlertSettings)]); + } + + if (BudgetAlertSettings.Thresholds?.Any(t => t <= 0 || t >= 100) is true) + yield return new ValidationResult("Budget alert thresholds must be between 1 and 99.", + [nameof(BudgetAlertSettings)]); + } } } @@ -279,5 +297,3 @@ public enum BillingStatus Canceled = 3, Unpaid = 4 } - - diff --git a/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs new file mode 100644 index 0000000000..e8d55eccb2 --- /dev/null +++ b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs @@ -0,0 +1,12 @@ +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} diff --git a/src/Exceptionless.Core/Models/Project.cs b/src/Exceptionless.Core/Models/Project.cs index 4ac783e893..5b07d6089a 100644 --- a/src/Exceptionless.Core/Models/Project.cs +++ b/src/Exceptionless.Core/Models/Project.cs @@ -63,6 +63,11 @@ public Project() public bool DeleteBotDataEnabled { get; set; } + /// + /// Optional project-level event budget configuration. Null means no project-specific cap. + /// + public ProjectIngestLimit? IngestLimit { get; set; } + /// /// The tick count that represents the next time the daily summary job should run. This time is set to midnight of the /// projects local time. @@ -90,5 +95,24 @@ public IEnumerable Validate(ValidationContext validationContex yield return new ValidationResult("Please specify a valid next summary end of day ticks.", [nameof(NextSummaryEndOfDayTicks)]); } + + if (IngestLimit is null) + yield break; + + if (!Enum.IsDefined(IngestLimit.Type)) + { + yield return new ValidationResult("Please specify a valid project ingest limit type.", [nameof(IngestLimit)]); + yield break; + } + + if (IngestLimit.Type is ProjectIngestLimitType.Fixed && IngestLimit.FixedLimit is null or <= 0) + yield return new ValidationResult("The fixed project ingest limit must be greater than 0.", [nameof(IngestLimit)]); + + if (IngestLimit.Type is ProjectIngestLimitType.PercentOfOrganizationLimit && IngestLimit.PercentOfOrganizationLimit is null or <= 0 or > 100) + yield return new ValidationResult("The project ingest percentage must be greater than 0 and no more than 100.", [nameof(IngestLimit)]); + + if (IngestLimit.Type is ProjectIngestLimitType.PercentOfOrganizationLimit && + IngestLimit.PercentOfOrganizationLimit is { } percentage && percentage != Decimal.Round(percentage, 4)) + yield return new ValidationResult("The project ingest percentage can have no more than 4 decimal places.", [nameof(IngestLimit)]); } } diff --git a/src/Exceptionless.Core/Models/ProjectIngestLimit.cs b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs new file mode 100644 index 0000000000..67704e9de6 --- /dev/null +++ b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs @@ -0,0 +1,14 @@ +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} diff --git a/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs new file mode 100644 index 0000000000..db8394f7c2 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs @@ -0,0 +1,16 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + public int UsagePeriod { get; init; } + + public string UniqueIdentifier => $"BudgetAlert:{UsagePeriod}:{OrganizationId}:{Threshold}"; + public string GetUniqueIdentifier(int fallbackUsagePeriod) => $"BudgetAlert:{(UsagePeriod > 0 ? UsagePeriod : fallbackUsagePeriod)}:{OrganizationId}:{Threshold}"; +} diff --git a/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs new file mode 100644 index 0000000000..55e05c9ec1 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs @@ -0,0 +1,16 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record ProjectSmartThrottleWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + public int UsagePeriod { get; init; } + + public string UniqueIdentifier => $"SmartThrottle:{UsagePeriod}:{OrganizationId}:{ProjectId}"; + public string GetUniqueIdentifier(int fallbackUsagePeriod) => $"SmartThrottle:{(UsagePeriod > 0 ? UsagePeriod : fallbackUsagePeriod)}:{OrganizationId}:{ProjectId}"; +} diff --git a/src/Exceptionless.Core/Services/IAtomicCacheBatch.cs b/src/Exceptionless.Core/Services/IAtomicCacheBatch.cs new file mode 100644 index 0000000000..b7c222a036 --- /dev/null +++ b/src/Exceptionless.Core/Services/IAtomicCacheBatch.cs @@ -0,0 +1,37 @@ +using Foundatio.Caching; + +namespace Exceptionless.Core.Services; + +public interface IAtomicCacheBatch +{ + Task TrySetAllAsync(IReadOnlyDictionary expectedValues, IReadOnlyDictionary values, TimeSpan expiresIn, + IReadOnlyDictionary? listValues = null, TimeSpan? listExpiresIn = null); +} + +internal sealed class InMemoryAtomicCacheBatch(ICacheClient cacheClient) : IAtomicCacheBatch +{ + public async Task TrySetAllAsync(IReadOnlyDictionary expectedValues, IReadOnlyDictionary values, TimeSpan expiresIn, + IReadOnlyDictionary? listValues = null, TimeSpan? listExpiresIn = null) + { + // The in-memory provider has no remote partial-failure boundary. Callers hold the + // organization reservation lock while applying this batch. + var currentValues = await cacheClient.GetAllAsync(expectedValues.Keys); + foreach (var expected in expectedValues) + { + string? current = currentValues.TryGetValue(expected.Key, out var value) && value.HasValue ? value.Value : null; + if (!String.Equals(current, expected.Value, StringComparison.Ordinal)) + return false; + } + + if (await cacheClient.SetAllAsync(values.ToDictionary(entry => entry.Key, entry => entry.Value), expiresIn) != values.Count) + return false; + + if (listValues is not null) + { + foreach (var entry in listValues) + await cacheClient.ListAddAsync(entry.Key, entry.Value, listExpiresIn); + } + + return true; + } +} diff --git a/src/Exceptionless.Core/Services/NotificationService.cs b/src/Exceptionless.Core/Services/NotificationService.cs index 1c89766b9b..a90de5c425 100644 --- a/src/Exceptionless.Core/Services/NotificationService.cs +++ b/src/Exceptionless.Core/Services/NotificationService.cs @@ -78,6 +78,43 @@ public Task IsOrganizationNotificationLockedAsync(string organizationId, b return lockProvider.IsLockedAsync(GetOrganizationNotificationLockKey(organizationId, isOverMonthlyLimit)); } + public Task TryAcquireUsageNotificationLockAsync(string notificationIdentifier) + { + ArgumentException.ThrowIfNullOrEmpty(notificationIdentifier); + return lockProvider.TryAcquireAsync($"{notificationIdentifier}-lock", OrganizationNotificationLockTimeout, TimeSpan.Zero); + } + + public Task TryAcquireUsageNotificationRecipientLockAsync(string notificationIdentifier, string userId) + { + ArgumentException.ThrowIfNullOrEmpty(notificationIdentifier); + ArgumentException.ThrowIfNullOrEmpty(userId); + + return lockProvider.TryAcquireAsync($"{GetUsageNotificationRecipientKey(notificationIdentifier, userId)}-lock", OrganizationNotificationLockTimeout, TimeSpan.Zero); + } + + public Task IsUsageNotificationRecipientSentAsync(string notificationIdentifier, string userId) + { + ArgumentException.ThrowIfNullOrEmpty(notificationIdentifier); + ArgumentException.ThrowIfNullOrEmpty(userId); + return cacheClient.ExistsAsync(GetUsageNotificationRecipientKey(notificationIdentifier, userId)); + } + + public Task MarkUsageNotificationRecipientSentAsync(string notificationIdentifier, string userId, int usagePeriod) + { + ArgumentException.ThrowIfNullOrEmpty(notificationIdentifier); + ArgumentException.ThrowIfNullOrEmpty(userId); + + var utcNow = timeProvider.GetUtcNow().UtcDateTime; + if (usagePeriod <= 0) + usagePeriod = utcNow.StartOfMonth().ToEpoch(); + var periodStartUtc = DateTimeOffset.FromUnixTimeSeconds(usagePeriod).UtcDateTime; + var expiresIn = periodStartUtc.AddMonths(1).AddDays(1) - utcNow; + if (expiresIn < CacheClientExtensions.MinimumExpiration) + expiresIn = CacheClientExtensions.MinimumExpiration; + + return cacheClient.SetAsync(GetUsageNotificationRecipientKey(notificationIdentifier, userId), true, expiresIn); + } + private static string GetOrganizationNotificationLockKey(string organizationId, bool isOverMonthlyLimit) { return $"{OrganizationNotificationWorkItem.GetNotificationKey(organizationId, isOverMonthlyLimit)}-lock"; @@ -88,6 +125,11 @@ private static string GetOrganizationNotificationSentKey(string organizationId, return $"{OrganizationNotificationWorkItem.GetNotificationKey(organizationId, isOverMonthlyLimit)}-sent"; } + private static string GetUsageNotificationRecipientKey(string notificationIdentifier, string userId) + { + return $"{notificationIdentifier}:{userId}-sent"; + } + private DateTime GetOrganizationNotificationSentExpiresAtUtc() { var utcNow = timeProvider.GetUtcNow().UtcDateTime; diff --git a/src/Exceptionless.Core/Services/UsageService.Budgets.cs b/src/Exceptionless.Core/Services/UsageService.Budgets.cs new file mode 100644 index 0000000000..c5e4421932 --- /dev/null +++ b/src/Exceptionless.Core/Services/UsageService.Budgets.cs @@ -0,0 +1,406 @@ +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.DateTimeExtensions; +using Foundatio.Caching; +using Foundatio.Messaging; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services; + +public partial class UsageService +{ + private const int MinimumColdBucketSmartThrottleBatchSize = 100; + + /// + /// Calculates the complete organization, explicit project, and automatic smart-throttle allowance + /// from the models already loaded by the ingestion job. + /// + public async Task GetEventIngestAllowanceAsync(Organization organization, Project project) + { + ArgumentNullException.ThrowIfNull(organization); + ArgumentNullException.ThrowIfNull(project); + if (!String.Equals(project.OrganizationId, organization.Id, StringComparison.Ordinal)) + throw new ArgumentException("The project does not belong to the organization.", nameof(project)); + + int maxEventsPerMonth = organization.GetMaxEventsPerMonthWithBonus(_timeProvider); + int effectiveProjectLimit = GetEffectiveProjectLimit(project, maxEventsPerMonth); + + // This is the common self-hosted/unlimited path: no cache or repository reads are needed. + if (maxEventsPerMonth < 0 && effectiveProjectLimit < 0) + return EventIngestAllowance.Unlimited; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + var totals = await GetAcceptedUsageTotalsAsync(utcNow, organization, project); + + return await CalculateEventIngestAllowanceAsync(utcNow, organization, project, maxEventsPerMonth, effectiveProjectLimit, totals, 0, true); + } + + private async Task CalculateEventIngestAllowanceAsync(DateTime utcNow, Organization organization, Project project, + int maxEventsPerMonth, int effectiveProjectLimit, AcceptedUsageTotals totals, int submittedEventCount, bool activateSmartThrottle) + { + + int organizationEventsLeft = GetOrganizationEventsLeft(maxEventsPerMonth, totals.OrganizationTotal, totals.OrganizationCurrentBucket); + if (organizationEventsLeft <= 0) + { + return new EventIngestAllowance + { + EventsLeft = 0, + IsOverOrgLimit = true, + EffectiveProjectLimit = effectiveProjectLimit + }; + } + + int projectEventsLeft = effectiveProjectLimit < 0 + ? Int32.MaxValue + : Math.Max(0, effectiveProjectLimit - totals.ProjectTotal); + if (projectEventsLeft <= 0) + { + return new EventIngestAllowance + { + EventsLeft = 0, + IsOverProjectLimit = true, + EffectiveProjectLimit = effectiveProjectLimit + }; + } + + var smartThrottle = await GetSmartThrottleResultAsync(utcNow, organization, maxEventsPerMonth, totals, submittedEventCount); + if (activateSmartThrottle && smartThrottle.IsThrottled) + await ActivateSmartThrottleAsync(utcNow, organization, project, smartThrottle); + + return new EventIngestAllowance + { + EventsLeft = Math.Min(organizationEventsLeft, projectEventsLeft), + EffectiveProjectLimit = effectiveProjectLimit, + SmartThrottle = smartThrottle + }; + } + + public async Task GetEventIngestAllowanceAsync(string organizationId, string projectId) + { + var organizationTask = _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + var projectTask = _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + await Task.WhenAll(organizationTask, projectTask); + + var organization = await organizationTask ?? throw new UsageServiceException($"Organization '{organizationId}' not found."); + var project = await projectTask ?? throw new UsageServiceException($"Project '{projectId}' not found."); + return await GetEventIngestAllowanceAsync(organization, project); + } + + public async Task GetSmartThrottleRateAsync(string organizationId, string projectId) + { + var organizationTask = _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + var projectTask = _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + await Task.WhenAll(organizationTask, projectTask); + + var organization = await organizationTask; + var project = await projectTask; + if (organization is null || project is null) + return SmartThrottleResult.NoThrottle; + + return (await GetEventIngestAllowanceAsync(organization, project)).SmartThrottle; + } + + public int GetEffectiveProjectLimit(Project project, Organization organization) + { + ArgumentNullException.ThrowIfNull(project); + ArgumentNullException.ThrowIfNull(organization); + return GetEffectiveProjectLimit(project.IngestLimit, organization.GetMaxEventsPerMonthWithBonus(_timeProvider)); + } + + public int GetEffectiveProjectLimit(ProjectIngestLimit? ingestLimit, Organization organization) + { + ArgumentNullException.ThrowIfNull(organization); + return GetEffectiveProjectLimit(ingestLimit, organization.GetMaxEventsPerMonthWithBonus(_timeProvider)); + } + + private static int GetEffectiveProjectLimit(Project project, int maxEventsPerMonth) + { + return GetEffectiveProjectLimit(project.IngestLimit, maxEventsPerMonth); + } + + private static int GetEffectiveProjectLimit(ProjectIngestLimit? ingestLimit, int maxEventsPerMonth) + { + if (ingestLimit is null) + return -1; + + if (ingestLimit.Type is ProjectIngestLimitType.Fixed) + { + if (ingestLimit.FixedLimit is not > 0) + return -1; + + return maxEventsPerMonth < 0 + ? ingestLimit.FixedLimit.Value + : Math.Min(ingestLimit.FixedLimit.Value, maxEventsPerMonth); + } + + if (ingestLimit.Type is not ProjectIngestLimitType.PercentOfOrganizationLimit || + ingestLimit.PercentOfOrganizationLimit is not > 0 or > 100 || + maxEventsPerMonth < 0) + return -1; + + return Math.Min(maxEventsPerMonth, decimal.ToInt32(decimal.Ceiling(maxEventsPerMonth * ingestLimit.PercentOfOrganizationLimit.Value / 100m))); + } + + private async Task GetAcceptedUsageTotalsAsync(DateTime utcNow, Organization organization, Project project) + { + string organizationTotalKey = GetTotalCacheKey(utcNow, organization.Id); + string projectTotalKey = GetTotalCacheKey(utcNow, organization.Id, project.Id); + string organizationCurrentKey = GetBucketTotalCacheKey(utcNow, organization.Id); + string projectCurrentKey = GetBucketTotalCacheKey(utcNow, organization.Id, project.Id); + string organizationPreviousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id); + string projectPreviousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id, project.Id); + + var values = await _cache.GetAllAsync([ + organizationTotalKey, + projectTotalKey, + organizationCurrentKey, + projectCurrentKey, + organizationPreviousKey, + projectPreviousKey + ]); + + int organizationCurrent = GetCachedValue(values, organizationCurrentKey); + int projectCurrent = GetCachedValue(values, projectCurrentKey); + bool includePreviousBucket = GetTotalBucket(utcNow) == GetTotalBucket(utcNow.Subtract(_bucketSize)); + int organizationTotal = GetCachedValue(values, organizationTotalKey, organization.GetCurrentUsage(_timeProvider).Total) + + organizationCurrent + + (includePreviousBucket ? GetCachedValue(values, organizationPreviousKey) : 0); + int projectTotal = GetCachedValue(values, projectTotalKey, project.GetCurrentUsage(_timeProvider).Total) + + projectCurrent + + (includePreviousBucket ? GetCachedValue(values, projectPreviousKey) : 0); + + return new AcceptedUsageTotals(organizationTotal, projectTotal, organizationCurrent, projectCurrent); + } + + private static int GetCachedValue(IDictionary> values, string key, int defaultValue = 0) + { + return values.TryGetValue(key, out var value) && value.HasValue ? value.Value : defaultValue; + } + + private int GetOrganizationEventsLeft(int maxEventsPerMonth, int currentTotal, int currentBucketTotal) + { + if (maxEventsPerMonth < 0) + return Int32.MaxValue; + + int monthlyEventsLeft = Math.Max(0, maxEventsPerMonth - currentTotal); + int bucketEventsLeft = Math.Max(0, GetBucketEventLimit(maxEventsPerMonth) - currentBucketTotal); + return Math.Min(monthlyEventsLeft, bucketEventsLeft); + } + + private async Task GetSmartThrottleResultAsync(DateTime utcNow, Organization organization, int maxEventsPerMonth, AcceptedUsageTotals totals, int submittedEventCount) + { + if (!_appOptions.EnableSmartProjectThrottling || maxEventsPerMonth <= 0) + return SmartThrottleResult.NoThrottle; + + int remainingAllowance = Math.Max(0, maxEventsPerMonth - totals.OrganizationTotal); + if (remainingAllowance <= 0) + return SmartThrottleResult.NoThrottle; + + double windowsLeft = Math.Max(1, Math.Ceiling((utcNow.EndOfMonth() - utcNow).TotalMinutes / _bucketSize.TotalMinutes)); + double sustainableWindowAllowance = remainingAllowance / windowsLeft * 10; + bool isColdBucket = totals.OrganizationCurrentBucket == 0 && totals.ProjectCurrentBucket == 0; + // Do not turn a normal small first post into a 5% sample merely because it belongs to one + // project. The explicit organization/project allowance still caps it; smart throttling is + // reserved for meaningfully noisy cold-bucket batches. + if (isColdBucket && submittedEventCount < MinimumColdBucketSmartThrottleBatchSize) + return SmartThrottleResult.NoThrottle; + + int projectedOrganizationCurrentBucket = checked(totals.OrganizationCurrentBucket + submittedEventCount); + int projectedProjectCurrentBucket = checked(totals.ProjectCurrentBucket + submittedEventCount); + if (projectedOrganizationCurrentBucket < sustainableWindowAllowance * 0.8 || projectedProjectCurrentBucket <= 0) + return SmartThrottleResult.NoThrottle; + + int projectCount = (int)Math.Max(1, (await _projectRepository.GetCountByOrganizationIdAsync(organization.Id)).Total); + if (projectCount <= 1) + return SmartThrottleResult.NoThrottle; + + double fairShareWindowAllowance = sustainableWindowAllowance / projectCount; + double fairShareRatio = fairShareWindowAllowance > 0 ? projectedProjectCurrentBucket / fairShareWindowAllowance : Double.PositiveInfinity; + if (fairShareRatio <= 2) + return SmartThrottleResult.NoThrottle; + + var result = new SmartThrottleResult + { + IsThrottled = true, + SampleRate = SmartThrottleResult.DefaultSampleRate, + ProjectShare = (double)(totals.ProjectTotal + submittedEventCount) / Math.Max(1, totals.OrganizationTotal + submittedEventCount), + FairShareRatio = fairShareRatio, + CurrentProjectUsage = totals.ProjectTotal + submittedEventCount, + FairShareLimit = maxEventsPerMonth / projectCount + }; + return result; + } + + private async Task ActivateSmartThrottleAsync(DateTime utcNow, Organization organization, Project project, SmartThrottleResult result) + { + string stateKey = GetProjectSmartThrottleKey(utcNow, organization.Id, project.Id); + bool isNewTransition; + try + { + isNewTransition = await _cache.AddAsync(stateKey, result, _bucketSize + TimeSpan.FromMinutes(1)); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to record smart throttle state for project {ProjectId}", project.Id); + return; + } + + if (!isNewTransition) + return; + + _logger.LogInformation("Smart project throttling activated for {OrganizationId}/{ProjectId} at {SampleRate:P0}; project usage {ProjectUsage}, fair-share limit {FairShareLimit}", + organization.Id, project.Id, result.SampleRate, result.CurrentProjectUsage, result.FairShareLimit); + + string notificationKey = GetSmartThrottleNotificationKey(utcNow, organization.Id, project.Id); + bool notificationClaimed = false; + try + { + var notificationTtl = utcNow.EndOfMonth() - utcNow + TimeSpan.FromDays(1); + notificationClaimed = await _cache.AddAsync(notificationKey, true, notificationTtl); + if (!notificationClaimed) + return; + + await _messagePublisher.PublishAsync(new ProjectSmartThrottleApplied + { + OrganizationId = organization.Id, + ProjectId = project.Id, + SampleRate = result.SampleRate, + CurrentEventCount = result.CurrentProjectUsage, + EventLimit = result.FairShareLimit, + UsagePeriod = GetTotalBucket(utcNow) + }); + } + catch (Exception ex) + { + if (notificationClaimed) + { + try + { + await _cache.RemoveAsync(notificationKey); + } + catch (Exception cleanupException) + { + _logger.LogWarning(cleanupException, "Failed to release smart throttle notification claim for project {ProjectId}", project.Id); + } + } + + _logger.LogError(ex, "Failed to publish smart throttle notification for project {ProjectId}", project.Id); + } + } + + public async Task> GetProjectSmartThrottleStatesAsync(IReadOnlyCollection<(string OrganizationId, string ProjectId)> projects) + { + if (!_appOptions.EnableSmartProjectThrottling || projects.Count == 0) + return new Dictionary(); + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + var keysByProjectId = projects.ToDictionary( + project => project.ProjectId, + project => GetProjectSmartThrottleKey(utcNow, project.OrganizationId, project.ProjectId)); + var values = await _cache.GetAllAsync(keysByProjectId.Values); + + return keysByProjectId + .Where(pair => values.TryGetValue(pair.Value, out var value) && value.HasValue && value.Value.IsThrottled) + .ToDictionary(pair => pair.Key, pair => values[pair.Value].Value); + } + + public async Task IsProjectSmartThrottledAsync(string organizationId, string projectId) + { + if (!_appOptions.EnableSmartProjectThrottling) + return false; + + var state = await _cache.GetAsync(GetProjectSmartThrottleKey(_timeProvider.GetUtcNow().UtcDateTime, organizationId, projectId)); + return state is { HasValue: true, Value.IsThrottled: true }; + } + + public void RecordSmartThrottle(int blockedCount) + { + if (blockedCount > 0) + AppDiagnostics.EventsSmartThrottled.Add(blockedCount); + } + + public async Task EvaluateBudgetAlertsAfterSettingsChangeAsync(Organization organization, IReadOnlyCollection thresholds) + { + if (organization.BudgetAlertSettings is not { Enabled: true } || thresholds.Count == 0) + return; + + int maxEventsPerMonth = organization.GetMaxEventsPerMonthWithBonus(_timeProvider); + if (maxEventsPerMonth <= 0) + return; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + string totalKey = GetTotalCacheKey(utcNow, organization.Id); + string currentKey = GetBucketTotalCacheKey(utcNow, organization.Id); + string previousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id); + var values = await _cache.GetAllAsync([totalKey, currentKey, previousKey]); + bool includePreviousBucket = GetTotalBucket(utcNow) == GetTotalBucket(utcNow.Subtract(_bucketSize)); + int currentTotal = GetCachedValue(values, totalKey, organization.GetCurrentUsage(_timeProvider).Total) + + GetCachedValue(values, currentKey) + + (includePreviousBucket ? GetCachedValue(values, previousKey) : 0); + + await PublishCrossedBudgetAlertsAsync(organization, 0, currentTotal, maxEventsPerMonth, thresholds); + } + + private async Task PublishCrossedBudgetAlertsAsync(Organization organization, int previousTotal, int currentTotal, int maxEventsPerMonth, IReadOnlyCollection? thresholds = null) + { + if (organization.BudgetAlertSettings is not { Enabled: true, Thresholds: not null } || maxEventsPerMonth <= 0) + return; + + thresholds ??= organization.BudgetAlertSettings.Thresholds; + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + foreach (int threshold in thresholds.Order()) + { + if (threshold is <= 0 or >= 100) + continue; + + int thresholdEventCount = GetBudgetThresholdEventCount(maxEventsPerMonth, threshold); + if (previousTotal >= thresholdEventCount || currentTotal < thresholdEventCount) + continue; + + string alertSentKey = GetBudgetAlertSentKey(utcNow, organization.Id, threshold); + bool alertClaimed = false; + try + { + var alertTtl = utcNow.EndOfMonth() - utcNow + TimeSpan.FromDays(1); + alertClaimed = await _cache.AddAsync(alertSentKey, true, alertTtl); + if (!alertClaimed) + continue; + + await _messagePublisher.PublishAsync(new OrganizationBudgetAlert + { + OrganizationId = organization.Id, + Threshold = threshold, + ThresholdEventCount = thresholdEventCount, + CurrentEventCount = currentTotal, + EventLimit = maxEventsPerMonth, + UsagePeriod = GetTotalBucket(utcNow) + }); + } + catch (Exception ex) + { + if (alertClaimed) + { + try + { + await _cache.RemoveAsync(alertSentKey); + } + catch (Exception cleanupException) + { + _logger.LogWarning(cleanupException, "Failed to release budget alert {Threshold}% claim for organization {OrganizationId}", threshold, organization.Id); + } + } + + _logger.LogError(ex, "Failed to publish budget alert {Threshold}% for organization {OrganizationId}", threshold, organization.Id); + } + } + } + + internal static int GetBudgetThresholdEventCount(int eventLimit, int threshold) + { + return checked((int)((threshold * (long)eventLimit + 99) / 100)); + } + + private sealed record AcceptedUsageTotals(int OrganizationTotal, int ProjectTotal, int OrganizationCurrentBucket, int ProjectCurrentBucket); +} diff --git a/src/Exceptionless.Core/Services/UsageService.Reservations.cs b/src/Exceptionless.Core/Services/UsageService.Reservations.cs new file mode 100644 index 0000000000..611930cc27 --- /dev/null +++ b/src/Exceptionless.Core/Services/UsageService.Reservations.cs @@ -0,0 +1,330 @@ +using System.Globalization; +using System.Text.Json; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Models; +using Exceptionless.DateTimeExtensions; +using Foundatio.Lock; + +namespace Exceptionless.Core.Services; + +public partial class UsageService +{ + private static readonly TimeSpan IngestReservationLockLifetime = TimeSpan.FromSeconds(30); + private static readonly TimeSpan IngestReservationRetention = TimeSpan.FromDays(35); + + public async Task ReserveEventIngestAsync(Organization organization, Project project, string reservationId, + IReadOnlyCollection candidates, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(organization); + ArgumentNullException.ThrowIfNull(project); + ArgumentException.ThrowIfNullOrEmpty(reservationId); + if (!String.Equals(project.OrganizationId, organization.Id, StringComparison.Ordinal)) + throw new ArgumentException("The project does not belong to the organization.", nameof(project)); + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + int maxEventsPerMonth = organization.GetMaxEventsPerMonthWithBonus(_timeProvider); + int effectiveProjectLimit = GetEffectiveProjectLimit(project, maxEventsPerMonth); + + // This is the common self-hosted/unlimited path. It must not take the organization hot-path lock. + if (maxEventsPerMonth < 0 && effectiveProjectLimit < 0) + return EventIngestReservation.Unlimited(reservationId, organization.Id, project.Id, GetTotalBucket(utcNow), utcNow.Floor(_bucketSize), candidates); + + SmartThrottleResult? smartThrottleToActivate = null; + EventIngestReservation reservation; + await using (await _lockProvider.AcquireAsync(GetIngestReservationLockKey(organization.Id), IngestReservationLockLifetime, cancellationToken)) + { + string reservationKey = GetIngestReservationKey(organization.Id, reservationId); + var existingValue = await _cache.GetAsync(reservationKey); + if (existingValue.HasValue) + { + var existing = DeserializeReservation(existingValue.Value); + if (existing.State is EventIngestReservationState.Active or EventIngestReservationState.Completed) + return existing.ToReservation(); + } + + int[] recentUsagePeriods = GetRecentUsagePeriods(utcNow); + string[] organizationReservedKeys = recentUsagePeriods.Select(period => GetIngestReservedKey(period, organization.Id)).ToArray(); + string[] projectReservedKeys = recentUsagePeriods.Select(period => GetIngestReservedKey(period, organization.Id, project.Id)).ToArray(); + var reservedValues = await _cache.GetAllAsync(organizationReservedKeys.Concat(projectReservedKeys)); + int organizationReserved = organizationReservedKeys.Sum(key => GetCachedInt(reservedValues, key)); + int projectReserved = projectReservedKeys.Sum(key => GetCachedInt(reservedValues, key)); + string organizationReservedKey = organizationReservedKeys[0]; + string projectReservedKey = projectReservedKeys[0]; + int currentOrganizationReserved = GetCachedInt(reservedValues, organizationReservedKey); + int currentProjectReserved = GetCachedInt(reservedValues, projectReservedKey); + var totals = await GetAcceptedUsageTotalsAsync(utcNow, organization, project); + totals = totals with + { + OrganizationTotal = checked(totals.OrganizationTotal + organizationReserved), + ProjectTotal = checked(totals.ProjectTotal + projectReserved), + // Every outstanding reservation counts against the active bucket. This is deliberately + // conservative and closes both five-minute and monthly rollover races. + OrganizationCurrentBucket = checked(totals.OrganizationCurrentBucket + organizationReserved), + ProjectCurrentBucket = checked(totals.ProjectCurrentBucket + projectReserved) + }; + + var allowance = await CalculateEventIngestAllowanceAsync(utcNow, organization, project, maxEventsPerMonth, + effectiveProjectLimit, totals, candidates.Count, false); + var selectedCandidates = candidates; + int smartThrottleBlockedCount = 0; + if (allowance.SmartThrottle.IsThrottled) + { + int sampleThreshold = (int)(allowance.SmartThrottle.SampleRate * 10_000); + selectedCandidates = candidates.Where(candidate => candidate.Hash % 10_000 < (ulong)sampleThreshold).ToArray(); + smartThrottleBlockedCount = candidates.Count - selectedCandidates.Count; + } + + if (selectedCandidates.Count > allowance.EventsLeft) + selectedCandidates = selectedCandidates.OrderBy(candidate => candidate.Hash).Take(allowance.EventsLeft).ToArray(); + + int[] acceptedIndexes = selectedCandidates.OrderBy(candidate => candidate.Index).Select(candidate => candidate.Index).ToArray(); + var smartThrottle = allowance.SmartThrottle.IsThrottled + ? new SmartThrottleResult + { + IsThrottled = true, + SampleRate = allowance.SmartThrottle.SampleRate, + ProjectShare = (double)(totals.ProjectTotal + acceptedIndexes.Length) / Math.Max(1, totals.OrganizationTotal + acceptedIndexes.Length), + FairShareRatio = allowance.SmartThrottle.FairShareRatio, + CurrentProjectUsage = totals.ProjectTotal + acceptedIndexes.Length, + FairShareLimit = allowance.SmartThrottle.FairShareLimit + } + : SmartThrottleResult.NoThrottle; + + var record = new EventIngestReservationRecord( + reservationId, + organization.Id, + project.Id, + GetTotalBucket(utcNow), + utcNow.Floor(_bucketSize), + acceptedIndexes, + smartThrottleBlockedCount, + smartThrottle, + EventIngestReservationState.Active, + 0, + null, + 0); + + var updatedValues = new Dictionary + { + [reservationKey] = JsonSerializer.Serialize(record), + [organizationReservedKey] = FormatInt(checked(currentOrganizationReserved + record.ReservedCount)), + [projectReservedKey] = FormatInt(checked(currentProjectReserved + record.ReservedCount)) + }; + var expectedValues = new Dictionary + { + [reservationKey] = existingValue.HasValue ? existingValue.Value : null, + [organizationReservedKey] = GetCachedString(reservedValues, organizationReservedKey), + [projectReservedKey] = GetCachedString(reservedValues, projectReservedKey) + }; + if (!await _atomicCacheBatch.TrySetAllAsync(expectedValues, updatedValues, IngestReservationRetention)) + throw new UsageServiceException($"Ingest reservation '{reservationId}' changed while it was being reserved; retry the queue entry."); + + reservation = record.ToReservation(); + if (smartThrottle.IsThrottled) + smartThrottleToActivate = smartThrottle; + } + + if (smartThrottleToActivate is not null) + await ActivateSmartThrottleAsync(utcNow, organization, project, smartThrottleToActivate); + + return reservation; + } + + public async Task CompleteEventIngestReservationAsync(EventIngestReservation reservation, Organization organization, int processedCount) + { + ArgumentNullException.ThrowIfNull(reservation); + ArgumentNullException.ThrowIfNull(organization); + if (processedCount is < 0 || processedCount > reservation.ReservedCount) + throw new ArgumentOutOfRangeException(nameof(processedCount)); + if (!reservation.IsTracked) + { + await IncrementTotalAsync(organization, reservation.ProjectId, processedCount); + return; + } + + DateTime completionBucketUtc; + long organizationBucketTotal; + bool alreadyCompleted; + await using (await _lockProvider.AcquireAsync(GetIngestReservationLockKey(reservation.OrganizationId), IngestReservationLockLifetime, CancellationToken.None)) + { + string reservationKey = GetIngestReservationKey(reservation.OrganizationId, reservation.Id); + var recordValue = await _cache.GetAsync(reservationKey); + if (!recordValue.HasValue) + throw new UsageServiceException($"Ingest reservation '{reservation.Id}' is missing and cannot be completed safely."); + + var record = DeserializeReservation(recordValue.Value); + alreadyCompleted = record.State is EventIngestReservationState.Completed; + if (alreadyCompleted) + { + completionBucketUtc = record.CompletionBucketUtc ?? throw new UsageServiceException($"Completed ingest reservation '{reservation.Id}' is missing its completion bucket."); + organizationBucketTotal = record.CompletionOrganizationBucketTotal; + } + else + { + if (record.State is not EventIngestReservationState.Active) + throw new UsageServiceException($"Ingest reservation '{reservation.Id}' is not active and cannot be completed."); + + completionBucketUtc = _timeProvider.GetUtcNow().UtcDateTime.Floor(_bucketSize); + string organizationReservedKey = GetIngestReservedKey(reservation.UsagePeriod, reservation.OrganizationId); + string projectReservedKey = GetIngestReservedKey(reservation.UsagePeriod, reservation.OrganizationId, reservation.ProjectId); + string organizationBucketKey = GetBucketTotalCacheKey(completionBucketUtc, reservation.OrganizationId); + string projectBucketKey = GetBucketTotalCacheKey(completionBucketUtc, reservation.OrganizationId, reservation.ProjectId); + string[] keys = [reservationKey, organizationReservedKey, projectReservedKey, organizationBucketKey, projectBucketKey]; + var values = await _cache.GetAllAsync(keys); + organizationBucketTotal = checked(GetCachedInt(values, organizationBucketKey) + processedCount); + var updatedValues = new Dictionary + { + [reservationKey] = JsonSerializer.Serialize(record with + { + State = EventIngestReservationState.Completed, + ProcessedCount = processedCount, + CompletionBucketUtc = completionBucketUtc, + CompletionOrganizationBucketTotal = organizationBucketTotal + }), + [organizationReservedKey] = FormatInt(Math.Max(0, GetCachedInt(values, organizationReservedKey) - record.ReservedCount)), + [projectReservedKey] = FormatInt(Math.Max(0, GetCachedInt(values, projectReservedKey) - record.ReservedCount)), + [organizationBucketKey] = FormatInt(organizationBucketTotal), + [projectBucketKey] = FormatInt(checked(GetCachedInt(values, projectBucketKey) + processedCount)) + }; + var usageSetMembers = new Dictionary + { + [GetOrganizationSetKey(completionBucketUtc)] = reservation.OrganizationId, + [GetProjectSetKey(completionBucketUtc)] = reservation.ProjectId + }; + if (!await _atomicCacheBatch.TrySetAllAsync(GetExpectedValues(values, keys), updatedValues, IngestReservationRetention, usageSetMembers, TimeSpan.FromHours(8))) + throw new UsageServiceException($"Ingest reservation '{reservation.Id}' changed while it was being completed; retry the queue entry."); + } + } + + int committedCount = alreadyCompleted ? reservation.ProcessedCount : processedCount; + if (committedCount > 0) + await PublishUsageIncrementNotificationsAsync(organization, reservation.ProjectId, committedCount, completionBucketUtc, organizationBucketTotal); + } + + public async Task ReleaseEventIngestReservationAsync(EventIngestReservation reservation) + { + ArgumentNullException.ThrowIfNull(reservation); + if (!reservation.IsTracked || reservation.ReservedCount == 0 || reservation.IsCompleted) + return; + + await using (await _lockProvider.AcquireAsync(GetIngestReservationLockKey(reservation.OrganizationId), IngestReservationLockLifetime, CancellationToken.None)) + { + string reservationKey = GetIngestReservationKey(reservation.OrganizationId, reservation.Id); + string organizationReservedKey = GetIngestReservedKey(reservation.UsagePeriod, reservation.OrganizationId); + string projectReservedKey = GetIngestReservedKey(reservation.UsagePeriod, reservation.OrganizationId, reservation.ProjectId); + string[] keys = [reservationKey, organizationReservedKey, projectReservedKey]; + var values = await _cache.GetAllAsync(keys); + if (!values.TryGetValue(reservationKey, out var recordValue) || !recordValue.HasValue) + return; + + var record = DeserializeReservation(recordValue.Value); + if (record.State is EventIngestReservationState.Released or EventIngestReservationState.Completed) + return; + + var updatedValues = new Dictionary + { + [reservationKey] = JsonSerializer.Serialize(record with { State = EventIngestReservationState.Released }), + [organizationReservedKey] = FormatInt(Math.Max(0, GetCachedInt(values, organizationReservedKey) - record.ReservedCount)), + [projectReservedKey] = FormatInt(Math.Max(0, GetCachedInt(values, projectReservedKey) - record.ReservedCount)) + }; + if (!await _atomicCacheBatch.TrySetAllAsync(GetExpectedValues(values, keys), updatedValues, IngestReservationRetention)) + throw new UsageServiceException($"Ingest reservation '{reservation.Id}' changed while it was being released; retry the queue entry."); + } + } + + private static EventIngestReservationRecord DeserializeReservation(string value) => + JsonSerializer.Deserialize(value) ?? throw new UsageServiceException("Invalid ingest reservation state."); + + private static int GetCachedInt(IDictionary> values, string key) => + values.TryGetValue(key, out var value) && value.HasValue && Int32.TryParse(value.Value, NumberStyles.Integer, CultureInfo.InvariantCulture, out int result) ? result : 0; + + private static string? GetCachedString(IDictionary> values, string key) => + values.TryGetValue(key, out var value) && value.HasValue ? value.Value : null; + + private static Dictionary GetExpectedValues(IDictionary> values, IEnumerable keys) => + keys.ToDictionary(key => key, key => GetCachedString(values, key)); + + private static string FormatInt(long value) => value.ToString(CultureInfo.InvariantCulture); + private static int[] GetRecentUsagePeriods(DateTime utcNow) + { + var periodStart = utcNow.StartOfMonth(); + return [periodStart.ToEpoch(), periodStart.AddMonths(-1).ToEpoch(), periodStart.AddMonths(-2).ToEpoch()]; + } + + private static string GetIngestReservedKey(int usagePeriod, string organizationId, string? projectId = null) => + String.IsNullOrEmpty(projectId) ? $"usage:reserved:{{{organizationId}}}:{usagePeriod}:total" : $"usage:reserved:{{{organizationId}}}:{usagePeriod}:{projectId}:total"; + private static string GetIngestReservationKey(string organizationId, string reservationId) => $"usage:ingest-reservation:{{{organizationId}}}:{reservationId}"; + private static string GetIngestReservationLockKey(string organizationId) => $"usage:ingest-reservation-lock:{organizationId}"; + + private sealed record EventIngestReservationRecord( + string Id, + string OrganizationId, + string ProjectId, + int UsagePeriod, + DateTime BucketUtc, + int[] AcceptedIndexes, + int SmartThrottleBlockedCount, + SmartThrottleResult SmartThrottle, + EventIngestReservationState State, + int ProcessedCount, + DateTime? CompletionBucketUtc, + long CompletionOrganizationBucketTotal) + { + public int ReservedCount => AcceptedIndexes.Length; + + public EventIngestReservation ToReservation() => new( + Id, + OrganizationId, + ProjectId, + UsagePeriod, + BucketUtc, + AcceptedIndexes, + SmartThrottleBlockedCount, + SmartThrottle, + true, + State is EventIngestReservationState.Completed, + ProcessedCount, + CompletionBucketUtc); + } + + private enum EventIngestReservationState + { + Active, + Completed, + Released + } +} + +public sealed record EventIngestCandidate(int Index, ulong Hash); + +public sealed record EventIngestReservation( + string Id, + string OrganizationId, + string ProjectId, + int UsagePeriod, + DateTime BucketUtc, + int[] AcceptedIndexes, + int SmartThrottleBlockedCount, + SmartThrottleResult SmartThrottle, + bool IsTracked, + bool IsCompleted, + int ProcessedCount, + DateTime? CompletionBucketUtc) +{ + public int ReservedCount => AcceptedIndexes.Length; + + public static EventIngestReservation Unlimited(string id, string organizationId, string projectId, int usagePeriod, DateTime bucketUtc, + IReadOnlyCollection candidates) => new( + id, + organizationId, + projectId, + usagePeriod, + bucketUtc, + candidates.OrderBy(candidate => candidate.Index).Select(candidate => candidate.Index).ToArray(), + 0, + SmartThrottleResult.NoThrottle, + false, + false, + 0, + null); +} diff --git a/src/Exceptionless.Core/Services/UsageService.cs b/src/Exceptionless.Core/Services/UsageService.cs index 4a12491675..97a3b45e54 100644 --- a/src/Exceptionless.Core/Services/UsageService.cs +++ b/src/Exceptionless.Core/Services/UsageService.cs @@ -1,36 +1,45 @@ -using Exceptionless.Core.Extensions; +using Exceptionless.Core; +using Exceptionless.Core.Extensions; using Exceptionless.Core.Messaging.Models; using Exceptionless.Core.Models; using Exceptionless.Core.Repositories; using Exceptionless.DateTimeExtensions; using Foundatio.Caching; +using Foundatio.Lock; using Foundatio.Messaging; using Foundatio.Repositories; using Microsoft.Extensions.Logging; namespace Exceptionless.Core.Services; -public class UsageService +public partial class UsageService { private readonly IOrganizationRepository _organizationRepository; private readonly IProjectRepository _projectRepository; private readonly ICacheClient _cache; + private readonly ILockProvider _lockProvider; + private readonly IAtomicCacheBatch _atomicCacheBatch; private readonly IMessagePublisher _messagePublisher; private readonly NotificationService _notificationService; + private readonly AppOptions _appOptions; private readonly TimeProvider _timeProvider; private readonly ILogger _logger; private readonly TimeSpan _bucketSize = TimeSpan.FromMinutes(5); - public UsageService(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, ICacheClient cache, IMessagePublisher messagePublisher, + public UsageService(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, ICacheClient cache, ILockProvider lockProvider, IAtomicCacheBatch atomicCacheBatch, IMessagePublisher messagePublisher, NotificationService notificationService, + AppOptions appOptions, TimeProvider timeProvider, ILoggerFactory loggerFactory) { _organizationRepository = organizationRepository; _projectRepository = projectRepository; _cache = cache; + _lockProvider = lockProvider; + _atomicCacheBatch = atomicCacheBatch; _messagePublisher = messagePublisher; _notificationService = notificationService; + _appOptions = appOptions; _timeProvider = timeProvider; _logger = loggerFactory.CreateLogger(); } @@ -305,6 +314,9 @@ public async Task GetUsageAsync(string organizationId, string var bucketUtc = lastUsageSave; var currentBucketUtc = utcNow.Floor(_bucketSize); + var currentMonthUtc = utcNow.StartOfMonth(); + if (bucketUtc < currentMonthUtc) + bucketUtc = currentMonthUtc; var isThrottled = await _cache.GetAsync(GetThrottledKey(currentBucketUtc, organizationId)); UsageInfoResponse usage; @@ -407,8 +419,11 @@ public async Task GetEventsLeftAsync(string organizationId) currentTotal += bucketTotal.Value; // get previous bucket counter and add it to total since it might not be saved yet - var previousBucketTotal = await _cache.GetAsync(GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organizationId)); - if (previousBucketTotal.HasValue) + var previousBucketUtc = utcNow.Subtract(_bucketSize); + var previousBucketTotal = GetTotalBucket(previousBucketUtc) == GetTotalBucket(utcNow) + ? await _cache.GetAsync(GetBucketTotalCacheKey(previousBucketUtc, organizationId)) + : default; + if (previousBucketTotal is { HasValue: true }) currentTotal += previousBucketTotal.Value; // check to see if adding this bucket puts the org over the limit @@ -422,38 +437,72 @@ public async Task GetEventsLeftAsync(string organizationId) return Math.Max(eventsLeftInBucket, 0); } - public async Task IncrementTotalAsync(string organizationId, string projectId, int eventCount = 1) + public Task IncrementTotalAsync(string organizationId, string projectId, int eventCount = 1) + { + return IncrementTotalAsync(null, organizationId, projectId, eventCount); + } + + public Task IncrementTotalAsync(Organization organization, string projectId, int eventCount = 1) + { + ArgumentNullException.ThrowIfNull(organization); + return IncrementTotalAsync(organization, organization.Id, projectId, eventCount); + } + + private async Task IncrementTotalAsync(Organization? organization, string organizationId, string projectId, int eventCount) { if (eventCount <= 0) return; var utcNow = _timeProvider.GetUtcNow().UtcDateTime; - long bucketTotal = await _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId), eventCount, TimeSpan.FromHours(8)); - await _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId, projectId), eventCount, TimeSpan.FromHours(8)); - await _cache.ListAddAsync(GetOrganizationSetKey(utcNow), organizationId, TimeSpan.FromHours(8)); - await _cache.ListAddAsync(GetProjectSetKey(utcNow), projectId, TimeSpan.FromHours(8)); + await Task.WhenAll( + _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId, projectId), eventCount, TimeSpan.FromHours(8)), + _cache.ListAddAsync(GetOrganizationSetKey(utcNow), organizationId, TimeSpan.FromHours(8)), + _cache.ListAddAsync(GetProjectSetKey(utcNow), projectId, TimeSpan.FromHours(8)) + ); - int maxEventsPerMonth = await GetMaxEventsPerMonthAsync(organizationId); + await PublishUsageIncrementNotificationsAsync(organization, projectId, eventCount, utcNow, bucketTotal, organizationId); + } + + private async Task PublishUsageIncrementNotificationsAsync(Organization? organization, string projectId, int eventCount, DateTime utcNow, long bucketTotal, string? organizationId = null) + { + organizationId ??= organization?.Id ?? throw new ArgumentNullException(nameof(organization)); + + int maxEventsPerMonth = organization?.GetMaxEventsPerMonthWithBonus(_timeProvider) ?? await GetMaxEventsPerMonthAsync(organizationId); int bucketLimit = GetBucketEventLimit(maxEventsPerMonth); - var currentTotalCache = await _cache.GetAsync(GetTotalCacheKey(utcNow, organizationId)); - if (currentTotalCache.HasValue) + if (maxEventsPerMonth > 0) { - long monthTotal = currentTotalCache.Value + bucketTotal; - if (monthTotal >= maxEventsPerMonth && monthTotal - maxEventsPerMonth < eventCount) + organization ??= await _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + int persistedTotal = organization?.GetCurrentUsage(_timeProvider).Total ?? 0; + var totals = await _cache.GetAllAsync([ + GetTotalCacheKey(utcNow, organizationId), + GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organizationId) + ]); + + int baseTotal = GetCachedValue(totals, GetTotalCacheKey(utcNow, organizationId), persistedTotal); + int previousBucketTotal = GetTotalBucket(utcNow.Subtract(_bucketSize)) == GetTotalBucket(utcNow) + ? GetCachedValue(totals, GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organizationId)) + : 0; + int currentTotal = checked(baseTotal + previousBucketTotal + (int)bucketTotal); + int previousTotal = Math.Max(0, currentTotal - eventCount); + + if (previousTotal < maxEventsPerMonth && currentTotal >= maxEventsPerMonth) await _messagePublisher.PublishAsync(new PlanOverage { OrganizationId = organizationId }); + + if (organization is not null) + await PublishCrossedBudgetAlertsAsync(organization, previousTotal, currentTotal, maxEventsPerMonth); } - if (bucketTotal >= bucketLimit && bucketTotal - bucketLimit < eventCount) + if (bucketLimit >= 0 && bucketTotal >= bucketLimit && bucketTotal - bucketLimit < eventCount) { - // org will be throttled during the current bucket of time await _messagePublisher.PublishAsync(new PlanOverage { OrganizationId = organizationId, IsHourly = true }); - await _cache.SetAsync(GetThrottledKey(utcNow, organizationId), true, TimeSpan.FromMinutes(5)); + await _cache.SetAsync(GetThrottledKey(utcNow, organizationId), true, _bucketSize); } } + public async Task IncrementBlockedAsync(string organizationId, string? projectId, int eventCount = 1) { if (eventCount <= 0) @@ -620,6 +669,49 @@ private string GetProjectSetKey(DateTime utcTime) return $"usage:{bucket}:projects"; } + private string GetBudgetAlertSentKey(DateTime utcTime, string organizationId, int threshold) + { + int monthBucket = GetTotalBucket(utcTime); + return $"usage:budget-alert:{monthBucket}:{organizationId}:{threshold}"; + } + + private string GetProjectSmartThrottleKey(DateTime utcTime, string organizationId, string projectId) + { + int bucket = GetCurrentBucket(utcTime); + return $"usage:{bucket}:{organizationId}:{projectId}:smart-throttle"; + } + + private string GetSmartThrottleNotificationKey(DateTime utcTime, string organizationId, string projectId) + { + int monthBucket = GetTotalBucket(utcTime); + return $"usage:smart-throttle-notified:{monthBucket}:{organizationId}:{projectId}"; + } + private int GetCurrentBucket(DateTime utcTime) => utcTime.Floor(_bucketSize).ToEpoch(); private int GetTotalBucket(DateTime utcTime) => utcTime.StartOfMonth().ToEpoch(); } + +public class EventIngestAllowance +{ + public static readonly EventIngestAllowance Unlimited = new() { EventsLeft = Int32.MaxValue }; + + public int EventsLeft { get; init; } + public bool IsOverOrgLimit { get; init; } + public bool IsOverProjectLimit { get; init; } + public int EffectiveProjectLimit { get; init; } = -1; + public SmartThrottleResult SmartThrottle { get; init; } = SmartThrottleResult.NoThrottle; + public double SampleRate => SmartThrottle.SampleRate; +} + +public class SmartThrottleResult +{ + public const double DefaultSampleRate = 0.05; + public static readonly SmartThrottleResult NoThrottle = new() { SampleRate = 1.0 }; + + public double SampleRate { get; init; } = 1.0; + public bool IsThrottled { get; init; } + public double ProjectShare { get; init; } + public double FairShareRatio { get; init; } + public int CurrentProjectUsage { get; init; } + public int FairShareLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Utility/AppDiagnostics.cs b/src/Exceptionless.Core/Utility/AppDiagnostics.cs index c081925b6c..a88ec49519 100644 --- a/src/Exceptionless.Core/Utility/AppDiagnostics.cs +++ b/src/Exceptionless.Core/Utility/AppDiagnostics.cs @@ -96,6 +96,7 @@ public GaugeInfo(Meter meter, string name) internal static readonly Counter EventsProcessErrors = Meter.CreateCounter("ex.events.processing.errors", description: "Errors processing events"); internal static readonly Counter EventsDiscarded = Meter.CreateCounter("ex.events.discarded", description: "Events that were discarded"); internal static readonly Counter EventsBlocked = Meter.CreateCounter("ex.events.blocked", description: "Events that were blocked"); + internal static readonly Counter EventsSmartThrottled = Meter.CreateCounter("ex.events.smart_throttled", description: "Events blocked by automatic project throttling"); internal static readonly Counter EventsProcessCancelled = Meter.CreateCounter("ex.events.processing.cancelled", description: "Events that started processing and were cancelled"); internal static readonly Counter EventsDeleted = Meter.CreateCounter("ex.events.deleted", description: "Events that were deleted"); internal static readonly Counter EventsRetryCount = Meter.CreateCounter("ex.events.retry.count", description: "Events where processing was retried"); diff --git a/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html b/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html new file mode 100644 index 0000000000..c0f91e5f6f --- /dev/null +++ b/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html @@ -0,0 +1,20 @@ + + + + +

+ \{{OrganizationName}} has reached \{{Threshold}}% of its monthly event allowance. +

+

+ The organization has accepted \{{CurrentEventCount}} events. The \{{Threshold}}% alert is + \{{ThresholdEventCount}} events, the current allowance is \{{EventLimit}}, and \{{RemainingEventCount}} + events remain. +

+
+ +
+ + + diff --git a/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html b/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html new file mode 100644 index 0000000000..5269a9d8fd --- /dev/null +++ b/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html @@ -0,0 +1,23 @@ + + + + +

+ Smart throttling is active for \{{ProjectName}} in \{{OrganizationName}}. +

+

+ This project has accepted \{{CurrentEventCount}} events this month, above its current fair-share limit of + \{{EventLimit}} events. To protect the organization's remaining allowance, Exceptionless is accepting a + stable \{{SamplePercent}}% sample from this project. Events outside that sample are counted as blocked. +

+

+ Other projects are unaffected unless they independently meet the same smart-throttling criteria. +

+
+ +
+ + + diff --git a/src/Exceptionless.Insulation/Bootstrapper.cs b/src/Exceptionless.Insulation/Bootstrapper.cs index 5ce4f4f7b3..67f14254bc 100644 --- a/src/Exceptionless.Insulation/Bootstrapper.cs +++ b/src/Exceptionless.Insulation/Bootstrapper.cs @@ -9,6 +9,7 @@ using Exceptionless.Core.Jobs.Elastic; using Exceptionless.Core.Mail; using Exceptionless.Core.Queues.Models; +using Exceptionless.Core.Services; using Exceptionless.Core.Utility; using Exceptionless.Insulation.Geo; using Exceptionless.Insulation.HealthChecks; @@ -110,6 +111,7 @@ private static void RegisterCache(IServiceCollection container, CacheOptions opt container.ReplaceSingleton(s => new ScopedCacheClient(CreateRedisCacheClient(s), options.Scope)); else container.ReplaceSingleton(CreateRedisCacheClient); + container.ReplaceSingleton(s => new RedisAtomicCacheBatch(s.GetRequiredService(), options, s.GetRequiredService())); container.ReplaceSingleton(); } diff --git a/src/Exceptionless.Insulation/Redis/RedisAtomicCacheBatch.cs b/src/Exceptionless.Insulation/Redis/RedisAtomicCacheBatch.cs new file mode 100644 index 0000000000..83f2dd50c5 --- /dev/null +++ b/src/Exceptionless.Insulation/Redis/RedisAtomicCacheBatch.cs @@ -0,0 +1,65 @@ +using Exceptionless.Core.Configuration; +using Exceptionless.Core.Services; +using StackExchange.Redis; + +namespace Exceptionless.Insulation.Redis; + +public sealed class RedisAtomicCacheBatch(IConnectionMultiplexer connectionMultiplexer, CacheOptions cacheOptions, TimeProvider timeProvider) : IAtomicCacheBatch +{ + private const string CompareAndSetAllScript = """ + local ttl = ARGV[1] + local valueCount = tonumber(ARGV[2]) + local listExpiresAt = ARGV[3] + local missing = '__EX_MISSING__' + for index = 1, valueCount do + local current = redis.call('GET', KEYS[index]) + local expected = ARGV[index + 3] + if expected == missing then + if current then return 0 end + elseif current ~= expected then + return 0 + end + end + for index = 1, valueCount do + redis.call('SET', KEYS[index], ARGV[valueCount + index + 3], 'PX', ttl) + end + for index = valueCount + 1, #KEYS do + redis.call('ZADD', KEYS[index], listExpiresAt, ARGV[valueCount * 2 + index - valueCount + 3]) + end + return 1 + """; + + public async Task TrySetAllAsync(IReadOnlyDictionary expectedValues, IReadOnlyDictionary values, TimeSpan expiresIn, + IReadOnlyDictionary? listValues = null, TimeSpan? listExpiresIn = null) + { + if (values.Count == 0) + return true; + if (expectedValues.Count != values.Count || expectedValues.Keys.Any(key => !values.ContainsKey(key))) + throw new ArgumentException("Expected and updated cache batches must contain the same keys.", nameof(expectedValues)); + + string prefix = String.IsNullOrEmpty(cacheOptions.Scope) ? String.Empty : $"{cacheOptions.Scope}:"; + listValues ??= new Dictionary(); + var keys = values.Keys.Concat(listValues.Keys).Select(key => (RedisKey)$"{prefix}{key}").ToArray(); + var arguments = new RedisValue[(values.Count * 2) + listValues.Count + 3]; + arguments[0] = checked((long)expiresIn.TotalMilliseconds); + arguments[1] = values.Count; + arguments[2] = timeProvider.GetUtcNow().Add(listExpiresIn ?? expiresIn).ToUnixTimeMilliseconds(); + int index = 3; + foreach (string key in values.Keys) + arguments[index++] = expectedValues[key] is { } expected ? expected : "__EX_MISSING__"; + foreach (string value in values.Values) + arguments[index++] = value; + foreach (string value in listValues.Values) + arguments[index++] = value; + + try + { + var result = await connectionMultiplexer.GetDatabase().ScriptEvaluateAsync(CompareAndSetAllScript, keys, arguments); + return (int)result == 1; + } + catch (RedisServerException ex) when (ex.Message.Contains("CROSSSLOT", StringComparison.OrdinalIgnoreCase)) + { + throw new NotSupportedException("Atomic usage reservations require a single Redis endpoint; Redis Cluster hash-slot splitting is not supported.", ex); + } + } +} diff --git a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs index b90082f8d6..6f0fca470f 100644 --- a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs +++ b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs @@ -87,14 +87,14 @@ public static IEndpointRouteBuilder MapOrganizationEndpoints(this IEndpointRoute } }); - group.MapPatch("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => + group.MapPatch("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => { if (changes is null) return ApiValidation.MissingRequestBody(); return (await mediator.InvokeAsync>(new OrganizationMessages.UpdateOrganizationMessage(id, changes, httpContext))).ToHttpResult(resultMapper); }) - .Accepts>("application/json", "application/*+json") + .Accepts>("application/json", "application/*+json") .Produces() .ProducesProblem(StatusCodes.Status400BadRequest) .ProducesProblem(StatusCodes.Status404NotFound) @@ -112,14 +112,14 @@ public static IEndpointRouteBuilder MapOrganizationEndpoints(this IEndpointRoute } }); - group.MapPut("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => + group.MapPut("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => { if (changes is null) return ApiValidation.MissingRequestBody(); return (await mediator.InvokeAsync>(new OrganizationMessages.UpdateOrganizationMessage(id, changes, httpContext))).ToHttpResult(resultMapper); }) - .Accepts>("application/json", "application/*+json") + .Accepts>("application/json", "application/*+json") .Produces() .ProducesProblem(StatusCodes.Status400BadRequest) .ProducesProblem(StatusCodes.Status404NotFound) diff --git a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs index 9c910c721c..8f9c7c15df 100644 --- a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs @@ -9,6 +9,7 @@ using Exceptionless.Core.Repositories; using Exceptionless.Core.Repositories.Queries; using Exceptionless.Core.Services; +using Exceptionless.Core.Validation; using Exceptionless.Web.Api.Infrastructure; using Exceptionless.Web.Api.Messages; using Exceptionless.Web.Api.Results; @@ -135,9 +136,45 @@ public async Task> Handle(UpdateOrganizationMessage mes if (error is not null) return error; + var changed = message.Changes.GetEntity(); + bool budgetSettingsChanged = message.Changes.ContainsChangedProperty(p => p.BudgetAlertSettings!); + bool wereAlertsEnabled = original.BudgetAlertSettings is { Enabled: true }; + var previousThresholds = original.BudgetAlertSettings?.Thresholds?.ToHashSet() ?? []; + message.Changes.Patch(original); - await repository.SaveAsync(original, o => o.Cache()); - return await MapToViewAsync(original); + if (budgetSettingsChanged) + original.BudgetAlertSettings = changed.BudgetAlertSettings; + + Organization saved; + try + { + saved = await repository.SaveAsync(original, o => o.Cache()); + } + catch (MiniValidatorException ex) + { + return ValidationResult(ex); + } + + if (budgetSettingsChanged && saved.BudgetAlertSettings is { Enabled: true, Thresholds: not null }) + { + int[] thresholdsToEvaluate = wereAlertsEnabled + ? saved.BudgetAlertSettings.Thresholds.Except(previousThresholds).ToArray() + : saved.BudgetAlertSettings.Thresholds.ToArray(); + + if (thresholdsToEvaluate.Length > 0) + { + try + { + await usageService.EvaluateBudgetAlertsAfterSettingsChangeAsync(saved, thresholdsToEvaluate); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to evaluate newly enabled budget alerts for organization {OrganizationId}", saved.Id); + } + } + } + + return await MapToViewAsync(saved); } public async Task>> Handle(SetOrganizationIcon message) @@ -776,12 +813,17 @@ await messagePublisher.PublishAsync(new UserMembershipChanged return organization; } - private async Task?> CanUpdateAsync(Organization original, Delta changes, HttpContext httpContext) + private async Task?> CanUpdateAsync(Organization original, Delta changes, HttpContext httpContext) { var changed = changes.GetEntity(); - if (!await IsOrganizationNameAvailableInternalAsync(changed.Name, httpContext)) + if (changes.ContainsChangedProperty(p => p.Name) && !await IsOrganizationNameAvailableInternalAsync(changed.Name, httpContext, original.Id)) return Result.BadRequest("A organization with this name already exists."); + if (changes.ContainsChangedProperty(p => p.BudgetAlertSettings!) && + changed.BudgetAlertSettings is { Enabled: true } && + original.GetMaxEventsPerMonthWithBonus(timeProvider) < 0) + return Result.BadRequest("Budget alerts cannot be enabled for an organization with an unlimited event allowance."); + if (changes.GetChangedPropertyNames().Contains("OrganizationId")) return Result.BadRequest("OrganizationId cannot be modified."); @@ -915,14 +957,14 @@ private async Task> PopulateOrganizationStatsAsync(List IsOrganizationNameAvailableInternalAsync(string? name, HttpContext httpContext) + private async Task IsOrganizationNameAvailableInternalAsync(string? name, HttpContext httpContext, string? excludeOrganizationId = null) { if (String.IsNullOrWhiteSpace(name)) return false; string decodedName = Uri.UnescapeDataString(name).Trim().ToLowerInvariant(); var results = await repository.GetByIdsAsync(httpContext.Request.GetAssociatedOrganizationIds().ToArray(), o => o.Cache()); - return !results.Any(o => String.Equals(o.Name.Trim().ToLowerInvariant(), decodedName, StringComparison.OrdinalIgnoreCase)); + return !results.Any(o => o.Id != excludeOrganizationId && String.Equals(o.Name.Trim().ToLowerInvariant(), decodedName, StringComparison.OrdinalIgnoreCase)); } private static Result PermissionToResult(PermissionResult permission) @@ -939,6 +981,12 @@ private static Result PermissionToResult(PermissionResult permission) return Result.Forbidden(permission.Message ?? "Access denied."); } + private static Result ValidationResult(MiniValidatorException ex) + { + return Result.FromResult(Result.Invalid(ex.Errors.SelectMany(error => + error.Value.Select(message => ValidationError.Create(error.Key.ToLowerUnderscoredWords(), message))))); + } + private static User GetCurrentUser(HttpContext httpContext) => httpContext.Request.GetUser(); private static bool IsStatsMode(string? mode) => !String.IsNullOrEmpty(mode) && String.Equals(mode, "stats", StringComparison.OrdinalIgnoreCase); private static bool messageIsGlobalAdmin(HttpContext httpContext) => httpContext.Request.IsGlobalAdmin(); diff --git a/src/Exceptionless.Web/Api/Handlers/ProjectHandler.cs b/src/Exceptionless.Web/Api/Handlers/ProjectHandler.cs index d07f40001a..acedd25047 100644 --- a/src/Exceptionless.Web/Api/Handlers/ProjectHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/ProjectHandler.cs @@ -8,6 +8,7 @@ using Exceptionless.Core.Repositories.Queries; using Exceptionless.Core.Services; using Exceptionless.Core.Utility; +using Exceptionless.Core.Validation; using Exceptionless.Web.Api.Infrastructure; using Exceptionless.Web.Api.Messages; using Exceptionless.Web.Api.Results; @@ -134,11 +135,33 @@ public async Task> Handle(UpdateProjectMessage message) if (error is not null) return error; + var changed = message.Changes.GetEntity(); + bool ingestLimitChanged = message.Changes.ContainsChangedProperty(p => p.IngestLimit!); + message.Changes.Patch(original); + if (ingestLimitChanged) + original.IngestLimit = changed.IngestLimit; + if (message.Changes.ContainsChangedProperty(p => p.PromotedTabs!)) original.PromotedTabs = NormalizePromotedTabs(original.PromotedTabs); - await repository.SaveAsync(original, o => o.Cache()); + if (ingestLimitChanged && original.IngestLimit is not null) + { + if (original.IngestLimit.Type is ProjectIngestLimitType.Fixed) + original.IngestLimit.PercentOfOrganizationLimit = null; + else if (original.IngestLimit.Type is ProjectIngestLimitType.PercentOfOrganizationLimit) + original.IngestLimit.FixedLimit = null; + } + + try + { + await repository.SaveAsync(original, o => o.Cache()); + } + catch (MiniValidatorException ex) + { + return ValidationResult(ex); + } + return await MapToViewAsync(original); } @@ -496,7 +519,13 @@ private async Task AfterResultMapAsync(ICollection m if (viewProjects.Count == 0) return; - var organizations = await organizationRepository.GetByIdsAsync(viewProjects.Select(p => p.OrganizationId).Distinct().ToArray(), o => o.Cache()); + var organizationsTask = organizationRepository.GetByIdsAsync(viewProjects.Select(p => p.OrganizationId).Distinct().ToArray(), o => o.Cache()); + var throttleStatesTask = usageService.GetProjectSmartThrottleStatesAsync( + viewProjects.Select(project => (project.OrganizationId, project.Id)).ToArray()); + await Task.WhenAll(organizationsTask, throttleStatesTask); + + var organizations = await organizationsTask; + var throttleStates = await throttleStatesTask; foreach (var viewProject in viewProjects) { if (!viewProject.IsConfigured.HasValue) @@ -530,6 +559,18 @@ private async Task AfterResultMapAsync(ICollection m currentHourUsage.Discarded = realTimeUsage.CurrentHourUsage.Discarded; currentHourUsage.TooBig = realTimeUsage.CurrentHourUsage.TooBig; currentHourUsage.Deleted = realTimeUsage.CurrentHourUsage.Deleted; + + if (viewProject.IngestLimit is not null) + { + int effectiveLimit = usageService.GetEffectiveProjectLimit(viewProject.IngestLimit, organization); + viewProject.EffectiveIngestLimit = effectiveLimit > 0 ? effectiveLimit : null; + } + + if (throttleStates.TryGetValue(viewProject.Id, out var throttleState)) + { + viewProject.IsSmartThrottled = true; + viewProject.SmartThrottleSampleRate = throttleState.SampleRate; + } } } @@ -567,6 +608,13 @@ private Task AddModelAsync(Project value, HttpContext httpContext) if (changes.ContainsChangedProperty(p => p.Name) && !await IsProjectNameAvailableInternalAsync(original.OrganizationId, changed.Name, httpContext)) return Result.BadRequest("A project with this name already exists."); + if (changes.ContainsChangedProperty(p => p.IngestLimit!) && changed.IngestLimit?.Type is ProjectIngestLimitType.PercentOfOrganizationLimit) + { + var organization = await organizationRepository.GetByIdAsync(original.OrganizationId, o => o.Cache()); + if (organization?.GetMaxEventsPerMonthWithBonus(timeProvider) < 0) + return Result.BadRequest("A percentage project ingest limit cannot be used with an unlimited organization allowance."); + } + if (!httpContext.Request.CanAccessOrganization(original.OrganizationId)) return Result.BadRequest("Invalid organization id specified."); @@ -734,6 +782,12 @@ private static Result PermissionToResult(PermissionResult permission) return Result.Forbidden(permission.Message ?? "Access denied."); } + private static Result ValidationResult(MiniValidatorException ex) + { + return Result.FromResult(Result.Invalid(ex.Errors.SelectMany(error => + error.Value.Select(message => ValidationError.Create(error.Key.ToLowerUnderscoredWords(), message))))); + } + private static User GetCurrentUser(HttpContext httpContext) => httpContext.Request.GetUser(); private static string GetCurrentUserId(HttpContext httpContext) => GetCurrentUser(httpContext).Id; private static bool IsStatsMode(string? mode) => !String.IsNullOrEmpty(mode) && String.Equals(mode, "stats", StringComparison.OrdinalIgnoreCase); diff --git a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs index 672bd6eff9..e61b984869 100644 --- a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs +++ b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs @@ -11,7 +11,7 @@ public record GetAdminOrganizations(string? Criteria, bool? Paid, bool? Suspende public record GetOrganizationPlanStats(HttpContext Context); public record GetOrganizationById(string Id, string? Mode, HttpContext Context); public record CreateOrganization(NewOrganization Organization, HttpContext Context); -public record UpdateOrganizationMessage(string Id, Delta Changes, HttpContext Context); +public record UpdateOrganizationMessage(string Id, Delta Changes, HttpContext Context); public record SetOrganizationIcon(string Id, string FileName, HttpContext Context); public record DeleteOrganizationIcon(string Id, HttpContext Context); public record DeleteOrganizations(string[] Ids, HttpContext Context); diff --git a/src/Exceptionless.Web/Bootstrapper.cs b/src/Exceptionless.Web/Bootstrapper.cs index 721feeb4ea..b27bfef33f 100644 --- a/src/Exceptionless.Web/Bootstrapper.cs +++ b/src/Exceptionless.Web/Bootstrapper.cs @@ -41,5 +41,11 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); } } diff --git a/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs b/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs index 58935cf119..6180e71431 100644 --- a/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs +++ b/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs @@ -61,7 +61,7 @@ const typeToZod = (rawType, isRequired, nullable, field) => { // but outputs "X | null" for nullable arrays fixed via generate-api.mjs workaround. if (type.startsWith('null | ')) { let innerType = type.replace('null | ', ''); - if (innerType.startsWith('(') && innerType.endsWith(')') && !innerType.endsWith('[]')) { + if (innerType.startsWith('(') && innerType.endsWith(')')) { innerType = innerType.slice(1, -1); } return `${typeToZod(innerType, true, false, field)}.nullable()`; @@ -69,13 +69,13 @@ const typeToZod = (rawType, isRequired, nullable, field) => { if (type.endsWith(' | null')) { let innerType = type.replace(' | null', ''); - if (innerType.startsWith('(') && innerType.endsWith(')') && !innerType.endsWith('[]')) { + if (innerType.startsWith('(') && innerType.endsWith(')')) { innerType = innerType.slice(1, -1); } return `${typeToZod(innerType, true, false, field)}.nullable()`; } - if (type.startsWith('(') && type.endsWith(')') && !type.includes('[]')) { + if (type.startsWith('(') && type.endsWith(')')) { type = type.slice(1, -1); } @@ -191,7 +191,9 @@ const typeToZod = (rawType, isRequired, nullable, field) => { }; const buildZodField = (field) => { - const rawType = field.value || 'unknown'; + const hasNullableOneOf = Array.isArray(field.oneOf) && field.oneOf.some((alternative) => alternative.type === 'null'); + const nonNullAlternative = hasNullableOneOf ? field.oneOf.find((alternative) => alternative.type !== 'null') : undefined; + const rawType = nonNullAlternative?.$parsed?.content || field.value || 'unknown'; const type = rawType.replace(/\bany\b/g, 'unknown'); const cleanedType = cleanType(type); const readableName = toReadableName(field.name); @@ -210,13 +212,15 @@ const buildZodField = (field) => { const isUuidField = field.format === 'uuid' || fieldNameLower.endsWith('_uuid'); // Check if this is a nullable union type (e.g., "null | string" or "string | null") - const isNullableUnion = cleanedType.startsWith('null | ') || cleanedType.endsWith(' | null'); + const unwrappedType = cleanedType.startsWith('(') && cleanedType.endsWith(')') ? cleanedType.slice(1, -1) : cleanedType; + const isNullableUnion = unwrappedType.startsWith('null | ') || unwrappedType.endsWith(' | null'); + const isFieldNullable = field.nullable || field.isNullable || hasNullableOneOf || isNullableUnion; // Get the base type (without null union) - let baseType = cleanedType; + let baseType = unwrappedType; if (isNullableUnion) { baseType = cleanedType.replace('null | ', '').replace(' | null', ''); - if (baseType.startsWith('(') && baseType.endsWith(')') && !baseType.endsWith('[]')) { + if (baseType.startsWith('(') && baseType.endsWith(')')) { baseType = baseType.slice(1, -1); } } @@ -344,12 +348,12 @@ const buildZodField = (field) => { } // Handle nullable (adds .nullable()) - if (isNullableUnion && !zodType.includes('.nullable()')) { + if (isFieldNullable && !zodType.includes('.nullable()')) { zodType = zodType + '.nullable()'; } // Handle optional (non-required fields and nullable fields are optional) - if (field.nullable || !field.isRequired) { + if (isFieldNullable || !field.isRequired) { zodType = zodType + '.optional()'; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts index ecc2301730..e8764e67ff 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts @@ -8,7 +8,7 @@ import { queryKeys as userQueryKeys } from '$features/users/api.svelte'; import { type FetchClientResponse, type ProblemDetails, useFetchClient } from '@exceptionless/fetchclient'; import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query'; -import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, ViewOrganization } from './models'; +import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, UpdateOrganization, ViewOrganization } from './models'; export async function invalidateOrganizationQueries(queryClient: QueryClient, message: WebSocketMessageValue<'OrganizationChanged'>) { const { id } = message; @@ -413,9 +413,9 @@ export function getPlansQuery(request: GetPlansRequest) { export function patchOrganization(request: PatchOrganizationRequest) { const queryClient = useQueryClient(); - return createMutation(() => ({ + return createMutation(() => ({ enabled: () => !!accessToken.current && !!request.route.id, - mutationFn: async (data: NewOrganization) => { + mutationFn: async (data: UpdateOrganization) => { const client = useFetchClient(); const response = await client.patchJSON(`organizations/${request.route.id}`, data); return response.data!; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts new file mode 100644 index 0000000000..37b9f40a06 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from 'vitest'; + +import { getBudgetThresholdEventCount, parseBudgetThresholds } from './budget-utils'; +import { BudgetAlertCardSchema } from './schemas'; + +describe('organization budget controls', () => { + it('normalizes, sorts, and deduplicates threshold input', () => { + expect(parseBudgetThresholds('80, 50, 80, 90')).toEqual([50, 80, 90]); + expect(parseBudgetThresholds('')).toEqual([]); + expect(parseBudgetThresholds(' , ')).toEqual([]); + }); + + it('rejects decimals, out-of-range thresholds, and empty enabled settings', () => { + expect(BudgetAlertCardSchema.safeParse({ enabled: true, thresholds: '' }).success).toBe(false); + expect(BudgetAlertCardSchema.safeParse({ enabled: false, thresholds: '50.5' }).success).toBe(false); + expect(BudgetAlertCardSchema.safeParse({ enabled: false, thresholds: '100' }).success).toBe(false); + }); + + it('calculates threshold event counts and disables percentages for unlimited plans', () => { + expect(getBudgetThresholdEventCount(1001, 50)).toBe(501); + expect(getBudgetThresholdEventCount(-1, 50)).toBeNull(); + expect(getBudgetThresholdEventCount(1001, 50.5)).toBeNull(); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts new file mode 100644 index 0000000000..310def613c --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts @@ -0,0 +1,20 @@ +export function getBudgetThresholdEventCount(eventLimit: number, threshold: number): null | number { + if (!Number.isFinite(eventLimit) || !Number.isInteger(threshold) || eventLimit < 0 || threshold < 1 || threshold > 99) { + return null; + } + + return Math.ceil((eventLimit * threshold) / 100); +} + +export function parseBudgetThresholds(value: string): number[] { + return [ + ...new Set( + value + .split(',') + .map((threshold) => threshold.trim()) + .filter(Boolean) + .map(Number) + .filter(Number.isFinite) + ) + ].sort((left, right) => left - right); +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte index eae4237066..67831a2edf 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte @@ -25,6 +25,7 @@ has_slack_integration: false, id: '1', is_configured: false, + is_smart_throttled: false, name: 'Frontend App', organization_id: 'org1', organization_name: 'Test Org', diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte new file mode 100644 index 0000000000..57439ac359 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte @@ -0,0 +1,164 @@ + + + + + Budget alerts + Receive an email when accepted organization usage crosses selected percentages of the monthly allowance. + +
{ + event.preventDefault(); + event.stopPropagation(); + form.handleSubmit(); + }} + > + + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + + {#snippet children(field)} + + + Enable budget alerts + Alerts are sent once per threshold per monthly usage period. + + field.handleChange(checked)} + disabled={update.isPending || isUnlimited} + /> + + {/snippet} + + + + {#snippet children(field)} + + Threshold percentages + field.handleChange(event.currentTarget.value)} + aria-invalid={ariaInvalid(field)} + aria-describedby="budget-threshold-help" + disabled={update.isPending} + /> + Comma-separated whole numbers from 1 to 99. + + +

+ Current allowance: + {#if isUnlimited} + unlimited; percentage alerts are inactive. + {:else} + events. + {/if} +

+ {#if !isUnlimited} +
    + {#each parseBudgetThresholds(field.state.value).filter((threshold) => Number.isInteger(threshold) && threshold >= 1 && threshold <= 99) as threshold (threshold)} +
  • + {threshold}% = + events +
    • + {/each} +
+ {/if} + + {/snippet} + + + + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + + +
+ diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts index 85d1fe5de6..8296938f94 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts @@ -1,4 +1,6 @@ -export type { Invoice, InvoiceGridModel, NewOrganization, ViewOrganization } from '$generated/api'; +import type { UpdateOrganization as GeneratedUpdateOrganization } from '$generated/api'; + +export type { Invoice, InvoiceGridModel, NewOrganization, OrganizationBudgetAlertSettings, ViewOrganization } from '$generated/api'; // TODO: This should be generated from the backend enum - investigate why it wasn't included in the generated API export enum SuspensionCode { @@ -7,3 +9,5 @@ export enum SuspensionCode { Abuse = 2, Other = 100 } + +export type UpdateOrganization = Partial; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts index c380dca5e7..82e2451e77 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts @@ -1,9 +1,29 @@ -import { date, type infer as Infer, number, object, string, enum as zodEnum } from 'zod'; +import { boolean, date, type infer as Infer, number, object, string, enum as zodEnum } from 'zod'; import { SuspensionCode } from './models'; export { type NewOrganizationFormData, NewOrganizationSchema } from '$generated/schemas'; +export const BudgetAlertCardSchema = object({ + enabled: boolean(), + thresholds: string() +}).superRefine((settings, context) => { + const values = settings.thresholds + .split(',') + .map((value) => value.trim()) + .filter(Boolean); + + if (settings.enabled && values.length === 0) { + context.addIssue({ code: 'custom', message: 'Enter at least one threshold.', path: ['thresholds'] }); + return; + } + + if (values.some((value) => !/^\d+$/.test(value) || Number(value) < 1 || Number(value) > 99)) { + context.addIssue({ code: 'custom', message: 'Use whole-number percentages from 1 to 99.', path: ['thresholds'] }); + } +}); +export type BudgetAlertCardFormData = Infer; + export const SetBonusOrganizationSchema = object({ bonusEvents: number().int('Bonus events must be a whole number'), expires: date().optional() diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts new file mode 100644 index 0000000000..53a006d5ce --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from 'vitest'; + +import type { ViewOrganization } from './models'; + +import { getEffectiveEventLimit } from './utils'; + +function organization(overrides: Partial): ViewOrganization { + return { bonus_events_per_month: 0, max_events_per_month: 1000, usage: [], ...overrides } as ViewOrganization; +} + +describe('getEffectiveEventLimit', () => { + it('uses the current usage limit so active bonus events match backend enforcement', () => { + const now = new Date(); + const usageDate = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 1)); + const value = organization({ + bonus_events_per_month: 250, + max_events_per_month: 1000, + usage: [{ blocked: 0, date: usageDate.toISOString(), deleted: 0, discarded: 0, limit: 1250, too_big: 0, total: 0 }] + }); + + expect(getEffectiveEventLimit(value)).toBe(1250); + }); + + it('preserves unlimited organizations', () => { + expect(getEffectiveEventLimit(organization({ max_events_per_month: -1 }))).toBe(-1); + }); + + it('treats legacy zero-limit organizations as unlimited', () => { + expect(getEffectiveEventLimit(organization({ max_events_per_month: 0 }))).toBe(-1); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts index 11cdb6569a..43f00e3d00 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts @@ -2,6 +2,21 @@ import { isSameUtcMonth } from '$features/shared/dates'; import type { ViewOrganization } from './models'; +export function getEffectiveEventLimit(organization?: ViewOrganization): number { + if (organization?.max_events_per_month == null || organization.max_events_per_month <= 0) { + return -1; + } + + const now = new Date(); + const currentUsage = organization.usage?.[organization.usage.length - 1]; + if (currentUsage && isSameUtcMonth(new Date(currentUsage.date), now)) { + return currentUsage.limit; + } + + const bonusEvents = organization.bonus_expiration && new Date(organization.bonus_expiration) > now ? organization.bonus_events_per_month : 0; + return organization.max_events_per_month + bonusEvents; +} + export function getNextBillingDateUtc(organization?: ViewOrganization): Date { if (organization?.subscribe_date) { console.log('Organization subscribe date for next billing date:', organization.subscribe_date); @@ -12,21 +27,20 @@ export function getNextBillingDateUtc(organization?: ViewOrganization): Date { } export function getRemainingEventLimit(organization?: ViewOrganization): number { - if (!organization?.max_events_per_month) { + const eventLimit = getEffectiveEventLimit(organization); + if (eventLimit <= 0) { return 0; } const now = new Date(); - const bonusEvents = organization.bonus_expiration && new Date(organization.bonus_expiration) > now ? organization.bonus_events_per_month : 0; - - const usage = organization.usage && organization.usage[organization.usage.length - 1]; + const usage = organization?.usage?.[organization.usage.length - 1]; if (usage) { const usageDate = new Date(usage.date); if (isSameUtcMonth(usageDate, now)) { - const remaining = usage.limit - usage.total; + const remaining = eventLimit - usage.total; return remaining > 0 ? remaining : 0; } } - return organization.max_events_per_month + bonusEvents; + return eventLimit; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts new file mode 100644 index 0000000000..dea43b46c0 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, it } from 'vitest'; + +import { createProjectIngestLimit, getEffectiveProjectLimit } from './budget-utils'; +import { ProjectBudgetCardSchema } from './schemas'; + +describe('project budget controls', () => { + it('rejects truncated fixed decimals and out-of-range percentages', () => { + expect(ProjectBudgetCardSchema.safeParse({ type: 'fixed', value: '1.5' }).success).toBe(false); + expect(ProjectBudgetCardSchema.safeParse({ type: 'fixed', value: '10' }).success).toBe(true); + expect(ProjectBudgetCardSchema.safeParse({ type: 'percent', value: '100.1' }).success).toBe(false); + expect(ProjectBudgetCardSchema.safeParse({ type: 'percent', value: '0.0001' }).success).toBe(true); + expect(ProjectBudgetCardSchema.safeParse({ type: 'percent', value: '0.00001' }).success).toBe(false); + }); + + it('maps modes to additive API payloads', () => { + expect(createProjectIngestLimit('none', '')).toBeNull(); + expect(createProjectIngestLimit('fixed', 'not-a-number')).toBeNull(); + expect(createProjectIngestLimit('fixed', '20000')).toMatchObject({ fixed_limit: 20000, type: 0 }); + expect(createProjectIngestLimit('percent', '25.5')).toMatchObject({ percent_of_organization_limit: 25.5, type: 1 }); + }); + + it('calculates percentage caps and clamps fixed caps to finite organization allowance', () => { + expect(getEffectiveProjectLimit(1001, createProjectIngestLimit('percent', '50'))).toBe(501); + expect(getEffectiveProjectLimit(3000, createProjectIngestLimit('percent', '1.1'))).toBe(33); + expect(getEffectiveProjectLimit(1_000_000, createProjectIngestLimit('percent', '8.3'))).toBe(83_000); + expect(getEffectiveProjectLimit(1_000_000, createProjectIngestLimit('percent', '0.0001'))).toBe(1); + expect(getEffectiveProjectLimit(1000, createProjectIngestLimit('fixed', '2000'))).toBe(1000); + expect(getEffectiveProjectLimit(-1, createProjectIngestLimit('percent', '50'))).toBeNull(); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts new file mode 100644 index 0000000000..3ac872386d --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts @@ -0,0 +1,42 @@ +import { type ProjectIngestLimit, ProjectIngestLimitType } from './models'; + +export type ProjectBudgetType = 'fixed' | 'none' | 'percent'; + +export function createProjectIngestLimit(type: ProjectBudgetType, value: string): null | ProjectIngestLimit { + if (type === 'none') { + return null; + } + + const numericValue = Number(value.trim()); + if (!Number.isFinite(numericValue)) { + return null; + } + + return type === 'fixed' + ? { fixed_limit: numericValue, percent_of_organization_limit: null, type: ProjectIngestLimitType.Fixed } + : { fixed_limit: null, percent_of_organization_limit: numericValue, type: ProjectIngestLimitType.PercentOfOrganizationLimit }; +} + +export function getEffectiveProjectLimit(organizationLimit: number, ingestLimit: null | ProjectIngestLimit): null | number { + if (!ingestLimit) { + return null; + } + + if (ingestLimit.type === ProjectIngestLimitType.Fixed) { + if (!Number.isInteger(ingestLimit.fixed_limit) || (ingestLimit.fixed_limit ?? 0) <= 0) { + return null; + } + + return organizationLimit < 0 ? ingestLimit.fixed_limit! : Math.min(ingestLimit.fixed_limit!, organizationLimit); + } + + const percentage = ingestLimit.percent_of_organization_limit; + if (organizationLimit < 0 || percentage == null || !Number.isFinite(percentage) || percentage <= 0 || percentage > 100) { + return null; + } + + // The API accepts at most 4 decimal places. Scaling both operands to integers keeps this preview + // aligned with the server's exact decimal ceiling without binary floating-point boundary errors. + const scaledPercentage = Math.round(percentage * 10_000); + return Math.min(organizationLimit, Math.ceil((organizationLimit * scaledPercentage) / 1_000_000)); +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte new file mode 100644 index 0000000000..69919215e6 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte @@ -0,0 +1,193 @@ + + + + + Project event budget + Optionally cap accepted events for this project without changing the organization allowance. + +
{ + event.preventDefault(); + event.stopPropagation(); + form.handleSubmit(); + }} + > + + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + + {#snippet children(field)} + + Limit type + { + const nextType = value as ProjectBudgetType; + selectedType = nextType; + field.handleChange(nextType); + if (nextType === 'none') { + form.setFieldValue('value', ''); + } + }} + disabled={update.isPending} + > + + {selectedType === 'none' + ? 'No project limit' + : selectedType === 'fixed' + ? 'Fixed event count' + : 'Percentage of organization allowance'} + + + + No project limit + Fixed event count + Percentage of organization allowance + + + + + {/snippet} + + + {#if selectedType !== 'none'} + + {#snippet children(field)} + + {selectedType === 'fixed' ? 'Monthly event count' : 'Percentage'} + field.handleChange(event.currentTarget.value)} + aria-invalid={ariaInvalid(field)} + disabled={update.isPending} + /> + + {selectedType === 'fixed' + ? 'Use a whole number greater than 0. The effective cap cannot exceed a finite organization allowance.' + : 'Use a value greater than 0 and no more than 100.'} + + + + {/snippet} + + {/if} + + state.values}> + {#snippet children(values)} + {@const draftLimit = createProjectIngestLimit(values.type, values.value)} + {@const effectiveLimit = getEffectiveProjectLimit(organizationLimit, draftLimit)} +

+ Effective project limit: + {#if effectiveLimit == null} + organization allowance + {:else} + events per month + {/if} +

+ {#if values.type === 'fixed' && organizationLimit >= 0 && (draftLimit?.fixed_limit ?? 0) > organizationLimit} + + Fixed limit will be clamped + The effective cap cannot exceed the organization's finite event allowance. + + {/if} + {/snippet} + + + {#if project.is_smart_throttled} + + Smart throttling active 5% sample + This project is being sampled independently; other projects remain unaffected. + + {/if} + + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + + +
+ diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts index a055dfb212..f30fa01b48 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts @@ -1,6 +1,10 @@ -export type { ClientConfiguration, NewProject, NotificationSettings, UpdateProject, ViewProject } from '$generated/api'; +import type { UpdateProject as GeneratedUpdateProject } from '$generated/api'; +export { ProjectIngestLimitType } from '$generated/api'; +export type { ClientConfiguration, NewProject, NotificationSettings, ProjectIngestLimit, ViewProject } from '$generated/api'; export interface ClientConfigurationSetting { key: string; value: string; } + +export type UpdateProject = Partial; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts index e495e7dd4f..d548f4485c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts @@ -5,7 +5,7 @@ import { type NotificationSettingsFormData, NotificationSettingsSchema } from '$generated/schemas'; -import { type infer as Infer, object, string } from 'zod'; +import { type infer as Infer, object, string, enum as zodEnum } from 'zod'; export { type NewProjectFormData, NewProjectSchema, type NotificationSettingsFormData, NotificationSettingsSchema }; @@ -17,3 +17,28 @@ export type ClientConfigurationSettingFormData = Infer; + +export const ProjectBudgetCardSchema = object({ + type: zodEnum(['none', 'fixed', 'percent']), + value: string() +}).superRefine((budget, context) => { + if (budget.type === 'none') { + return; + } + + const value = budget.value.trim(); + if (budget.type === 'fixed') { + const numericValue = Number(value); + if (!/^\d+$/.test(value) || !Number.isSafeInteger(numericValue) || numericValue < 1 || numericValue > 2_147_483_647) { + context.addIssue({ code: 'custom', message: 'Enter a whole number from 1 to 2,147,483,647.', path: ['value'] }); + } + + return; + } + + const numericValue = Number(value); + if (!/^(?:\d+(?:\.\d{0,4})?|\.\d{1,4})$/.test(value) || !Number.isFinite(numericValue) || numericValue <= 0 || numericValue > 100) { + context.addIssue({ code: 'custom', message: 'Enter a percentage greater than 0 and no more than 100, with up to 4 decimal places.', path: ['value'] }); + } +}); +export type ProjectBudgetCardFormData = Infer; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts index 3ec0186c02..3c1cb1494a 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts @@ -7,6 +7,11 @@ export enum StackStatus { Discarded = "discarded", } +export enum ProjectIngestLimitType { + Fixed = 0, + PercentOfOrganizationLimit = 1, +} + export enum BillingStatus { Trialing = 0, Active = 1, @@ -130,6 +135,7 @@ export interface NewProject { organization_id: string; name: string; delete_bot_data_enabled: boolean; + ingest_limit?: null | ProjectIngestLimit; promoted_tabs?: string[] | null; } @@ -295,6 +301,15 @@ export interface OAuthTokenResponse { resource?: null | string; } +export interface OrganizationBudgetAlertSettings { + enabled: boolean; + /** + * Percentage thresholds of the organization's effective monthly event allowance. + * Example: [50, 80, 90]. + */ + thresholds: number[]; +} + export interface PersistentEvent { /** * Unique id that identifies an event. @@ -383,6 +398,14 @@ export interface ProblemDetails { instance?: null | string; } +export interface ProjectIngestLimit { + type: ProjectIngestLimitType; + /** @format int32 */ + fixed_limit?: null | number; + /** @format double */ + percent_of_organization_limit?: null | number; +} + export interface ResetPasswordModel { password_reset_token: string; password: string; @@ -486,11 +509,18 @@ export interface UpdateEvent { description?: null | string; } +/** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ +export interface UpdateOrganization { + name: string; + budget_alert_settings?: null | OrganizationBudgetAlertSettings; +} + /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ export interface UpdateProject { name: string; delete_bot_data_enabled: boolean; - promoted_tabs?: string[] | null; + ingest_limit?: null | ProjectIngestLimit; + promoted_tabs?: null | string[]; } /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ @@ -502,7 +532,7 @@ export interface UpdateSavedView { slug?: null | string; filter_definitions?: null | string; columns?: null | Record; - column_order?: string[] | null; + column_order?: null | string[]; show_stats?: null | boolean; show_chart?: null | boolean; } @@ -688,6 +718,7 @@ export interface ViewOrganization { is_throttled: boolean; is_over_monthly_limit: boolean; is_over_request_limit: boolean; + budget_alert_settings?: null | OrganizationBudgetAlertSettings; } export interface ViewProject { @@ -709,6 +740,12 @@ export interface ViewProject { event_count: number; has_premium_features: boolean; has_slack_integration: boolean; + ingest_limit?: null | ProjectIngestLimit; + /** @format int32 */ + effective_ingest_limit?: null | number; + is_smart_throttled: boolean; + /** @format double */ + smart_throttle_sample_rate?: null | number; usage_hours: UsageHourInfo[]; usage: UsageInfo[]; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts index c2cd06eb56..9c8a1a5425 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts @@ -27,6 +27,7 @@ export const StackStatusSchema = zodEnum([ "ignored", "discarded", ]); +export const ProjectIngestLimitTypeSchema = union([literal(0), literal(1)]); export const BillingStatusSchema = union([ literal(0), literal(1), @@ -91,8 +92,10 @@ export const CountResultSchema = object({ aggregations: record( string(), lazy(() => IAggregateSchema), - ), - data: record(string(), unknown()).nullable(), + ) + .nullable() + .optional(), + data: record(string(), unknown()).nullable().optional(), }); export type CountResultFormData = Infer; @@ -169,6 +172,9 @@ export const NewProjectSchema = object({ .regex(/^[a-fA-F0-9]{24}$/, "Organization id has invalid format"), name: string().min(1, "Name is required"), delete_bot_data_enabled: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), promoted_tabs: array(string()).nullable().optional(), }); export type NewProjectFormData = Infer; @@ -422,6 +428,14 @@ export const OAuthTokenResponseSchema = object({ }); export type OAuthTokenResponseFormData = Infer; +export const OrganizationBudgetAlertSettingsSchema = object({ + enabled: boolean(), + thresholds: array(number()), +}); +export type OrganizationBudgetAlertSettingsFormData = Infer< + typeof OrganizationBudgetAlertSettingsSchema +>; + export const PersistentEventSchema = object({ id: string() .length(24, "Id must be exactly 24 characters") @@ -441,7 +455,8 @@ export const PersistentEventSchema = object({ type: string() .min(1, "Type is required") .max(100, "Type must be at most 100 characters") - .nullable(), + .nullable() + .optional(), source: string() .min(1, "Source is required") .max(2000, "Source must be at most 2000 characters") @@ -473,7 +488,7 @@ export const PredefinedSavedViewDefinitionSchema = object({ filter: string().min(1, "Filter is required").nullable().optional(), time: string().min(1, "Time is required").nullable().optional(), sort: string().min(1, "Sort is required").nullable().optional(), - filterDefinitions: unknown().optional(), + filterDefinitions: unknown().nullable().optional(), columns: record(string(), boolean()).nullable().optional(), columnOrder: array(string()).nullable().optional(), showStats: boolean().nullable().optional(), @@ -492,6 +507,13 @@ export const ProblemDetailsSchema = object({ }); export type ProblemDetailsFormData = Infer; +export const ProjectIngestLimitSchema = object({ + type: ProjectIngestLimitTypeSchema, + fixed_limit: int32().nullable().optional(), + percent_of_organization_limit: number().nullable().optional(), +}); +export type ProjectIngestLimitFormData = Infer; + export const ResetPasswordModelSchema = object({ password_reset_token: string().length( 40, @@ -554,7 +576,7 @@ export const StackSchema = object({ export type StackFormData = Infer; export const StringValueFromBodySchema = object({ - value: string().min(1, "Value is required").nullable(), + value: string().min(1, "Value is required").nullable().optional(), }); export type StringValueFromBodyFormData = Infer< typeof StringValueFromBodySchema @@ -578,9 +600,20 @@ export const UpdateEventSchema = object({ }); export type UpdateEventFormData = Infer; +export const UpdateOrganizationSchema = object({ + name: string().min(1, "Name is required").optional(), + budget_alert_settings: lazy(() => OrganizationBudgetAlertSettingsSchema) + .nullable() + .optional(), +}); +export type UpdateOrganizationFormData = Infer; + export const UpdateProjectSchema = object({ name: string().min(1, "Name is required").optional(), delete_bot_data_enabled: boolean().optional(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), promoted_tabs: array(string()).nullable().optional(), }); export type UpdateProjectFormData = Infer; @@ -768,6 +801,9 @@ export const ViewOrganizationSchema = object({ is_throttled: boolean(), is_over_monthly_limit: boolean(), is_over_request_limit: boolean(), + budget_alert_settings: lazy(() => OrganizationBudgetAlertSettingsSchema) + .nullable() + .optional(), }); export type ViewOrganizationFormData = Infer; @@ -789,6 +825,12 @@ export const ViewProjectSchema = object({ event_count: int(), has_premium_features: boolean(), has_slack_integration: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), + effective_ingest_limit: int32().nullable().optional(), + is_smart_throttled: boolean(), + smart_throttle_sample_rate: number().nullable().optional(), usage_hours: array(lazy(() => UsageHourInfoSchema)), usage: array(lazy(() => UsageInfoSchema)), }); diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte index 0c42001cd2..d1157dd7d4 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte @@ -10,6 +10,7 @@ import { env } from '$env/dynamic/public'; import { ChangePlanDialog } from '$features/billing'; import { getOrganizationQuery } from '$features/organizations/api.svelte'; + import OrganizationBudgetAlertCard from '$features/organizations/components/organization-budget-alert-card.svelte'; import { getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; import { formatDateLabel, formatLongDate } from '$shared/dates'; import { scaleUtc } from 'd3-scale'; @@ -139,6 +140,12 @@ {/if} +{#if organizationQuery.data} + {#key organizationQuery.data.id} + + {/key} +{/if} + {#if changePlanDialogOpen && organizationQuery.data} (changePlanDialogOpen = false)} /> {/if} diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte index 27f5590cde..94dcc84ab4 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte @@ -11,8 +11,9 @@ import { ChangePlanDialog } from '$features/billing'; import { getOrganizationQuery } from '$features/organizations/api.svelte'; import { organization } from '$features/organizations/context.svelte'; - import { getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; + import { getEffectiveEventLimit, getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; import { getProjectQuery } from '$features/projects/api.svelte'; + import ProjectIngestLimitCard from '$features/projects/components/project-ingest-limit-card.svelte'; import { formatDateLabel, formatLongDate } from '$shared/dates'; import { scaleUtc } from 'd3-scale'; import { curveNatural } from 'd3-shape'; @@ -32,6 +33,7 @@ const hasMonthlyUsage = $derived((organizationQuery.data?.max_events_per_month ?? 0) > 0); const canChangePlan = $derived(organizationQuery.isSuccess && !!env.PUBLIC_STRIPE_PUBLISHABLE_KEY); const remainingEventLimit = $derived(getRemainingEventLimit(organizationQuery.data)); + const organizationLimit = $derived(getEffectiveEventLimit(organizationQuery.data)); const nextBillingDate = $derived(getNextBillingDateUtc(organizationQuery.data)); const projectQuery = getProjectQuery({ @@ -44,15 +46,15 @@ let changePlanDialogOpen = $state(false); - const chartConfig = { + const chartConfig = $derived({ blocked: { color: 'var(--chart-2)', label: 'Blocked' }, deleted: { color: 'var(--chart-7)', label: 'Deleted' }, discarded: { color: 'var(--chart-3)', label: 'Discarded' }, - limit: { color: 'var(--chart-6)', label: 'Limit' }, + limit: { color: 'var(--chart-6)', label: projectQuery.data?.effective_ingest_limit == null ? 'Organization limit' : 'Project limit' }, org_total: { color: 'var(--chart-5)', label: 'Total in Organization' }, too_big: { color: 'var(--chart-4)', label: 'Too Big' }, total: { color: 'var(--chart-1)', label: 'Total' } - } satisfies Chart.ChartConfig; + } satisfies Chart.ChartConfig); const chartData = $derived.by(() => { const project = projectQuery.data; @@ -76,7 +78,7 @@ date: new Date(projItem.date), deleted: projItem.deleted, discarded: projItem.discarded, - limit: orgItem?.limit || 0, + limit: project.effective_ingest_limit ?? orgItem?.limit ?? 0, org_total: orgItem?.total || 0, too_big: projItem.too_big, total: projItem.total @@ -84,7 +86,7 @@ }); }); - const series = [ + const series = $derived([ { key: 'org_total', ...chartConfig.org_total }, { key: 'total', ...chartConfig.total }, { key: 'discarded', ...chartConfig.discarded }, @@ -99,7 +101,7 @@ line: { class: '[stroke-dasharray:4]' } } } - ]; + ]);
@@ -168,6 +170,12 @@ {/if}
+{#if projectQuery.data && organizationQuery.data} + {#key projectQuery.data.id} + + {/key} +{/if} + {#if changePlanDialogOpen && organizationQuery.data} (changePlanDialogOpen = false)} /> {/if} diff --git a/src/Exceptionless.Web/Mapping/OrganizationMapper.cs b/src/Exceptionless.Web/Mapping/OrganizationMapper.cs index 69b0e27556..4b4fa164e0 100644 --- a/src/Exceptionless.Web/Mapping/OrganizationMapper.cs +++ b/src/Exceptionless.Web/Mapping/OrganizationMapper.cs @@ -20,6 +20,7 @@ public OrganizationMapper(TimeProvider timeProvider) } public partial Organization MapToOrganization(NewOrganization source); + public partial Organization MapToOrganization(UpdateOrganization source); [MapperIgnoreTarget(nameof(ViewOrganization.IsOverMonthlyLimit))] [MapperIgnoreTarget(nameof(ViewOrganization.IsOverRequestLimit))] diff --git a/src/Exceptionless.Web/Mapping/ProjectMapper.cs b/src/Exceptionless.Web/Mapping/ProjectMapper.cs index 6d69ef583d..faabf4891d 100644 --- a/src/Exceptionless.Web/Mapping/ProjectMapper.cs +++ b/src/Exceptionless.Web/Mapping/ProjectMapper.cs @@ -18,6 +18,9 @@ public partial class ProjectMapper [MapperIgnoreTarget(nameof(ViewProject.OrganizationName))] [MapperIgnoreTarget(nameof(ViewProject.StackCount))] [MapperIgnoreTarget(nameof(ViewProject.EventCount))] + [MapperIgnoreTarget(nameof(ViewProject.EffectiveIngestLimit))] + [MapperIgnoreTarget(nameof(ViewProject.IsSmartThrottled))] + [MapperIgnoreTarget(nameof(ViewProject.SmartThrottleSampleRate))] private partial ViewProject MapToViewProjectCore(Project source); public ViewProject MapToViewProject(Project source) diff --git a/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs b/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs new file mode 100644 index 0000000000..dd1946a773 --- /dev/null +++ b/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs @@ -0,0 +1,12 @@ +using System.ComponentModel.DataAnnotations; +using Exceptionless.Core.Models; + +namespace Exceptionless.Web.Models; + +public record UpdateOrganization +{ + [Required] + public string Name { get; set; } = null!; + + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} diff --git a/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs b/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs index 90bdd3fffd..f3db6838af 100644 --- a/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs +++ b/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs @@ -46,6 +46,7 @@ public record ViewOrganization : IIdentity, IData, IHaveDates public bool IsThrottled { get; set; } public bool IsOverMonthlyLimit { get; set; } public bool IsOverRequestLimit { get; set; } + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } } public static class ViewOrganizationExtensions diff --git a/src/Exceptionless.Web/Models/Project/UpdateProject.cs b/src/Exceptionless.Web/Models/Project/UpdateProject.cs index fe76403b6e..73096d0b2b 100644 --- a/src/Exceptionless.Web/Models/Project/UpdateProject.cs +++ b/src/Exceptionless.Web/Models/Project/UpdateProject.cs @@ -1,8 +1,11 @@ -namespace Exceptionless.Web.Models; +using Exceptionless.Core.Models; + +namespace Exceptionless.Web.Models; public record UpdateProject { public string Name { get; set; } = null!; public bool DeleteBotDataEnabled { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } public List? PromotedTabs { get; set; } } diff --git a/src/Exceptionless.Web/Models/Project/ViewProject.cs b/src/Exceptionless.Web/Models/Project/ViewProject.cs index 52f5576e9a..26e1204abd 100644 --- a/src/Exceptionless.Web/Models/Project/ViewProject.cs +++ b/src/Exceptionless.Web/Models/Project/ViewProject.cs @@ -23,6 +23,10 @@ public class ViewProject : IIdentity, IData, IHaveCreatedDate public long EventCount { get; set; } public bool HasPremiumFeatures { get; set; } public bool HasSlackIntegration { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } + public int? EffectiveIngestLimit { get; set; } + public bool IsSmartThrottled { get; set; } + public double? SmartThrottleSampleRate { get; set; } public ICollection UsageHours { get; set; } = new SortedSet(Comparer.Create((a, b) => a.Date.CompareTo(b.Date))); public ICollection Usage { get; set; } = new SortedSet(Comparer.Create((a, b) => a.Date.CompareTo(b.Date))); } diff --git a/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs b/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs index 953bc61b6e..9117bdb984 100644 --- a/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs +++ b/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs @@ -14,18 +14,18 @@ public class DeltaSchemaTransformer : IOpenApiSchemaTransformer { private static readonly NullabilityInfoContext NullabilityContext = new(); - public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext context, CancellationToken cancellationToken) + public async Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext context, CancellationToken cancellationToken) { var type = context.JsonTypeInfo.Type; // Check if this is a Delta type if (!IsDeltaType(type)) - return Task.CompletedTask; + return; // Get the inner type T from Delta var innerType = type.GetGenericArguments().FirstOrDefault(); if (innerType is null) - return Task.CompletedTask; + return; // Set the type to object schema.Type = JsonSchemaType.Object; @@ -44,11 +44,29 @@ public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext .Where(p => p.CanRead && p.CanWrite)) { bool isNullable = IsPropertyNullable(property); - var propertySchema = CreateSchemaForType(property.PropertyType, isNullable); + var generatedSchema = await context.GetOrCreateSchemaAsync(property.PropertyType, cancellationToken: cancellationToken); + DataAnnotationHelper.ApplyToSchema(generatedSchema, property); + ApplyArrayAnnotations(generatedSchema, property); - // Apply data annotations from the inner type's property - DataAnnotationHelper.ApplyToSchema(propertySchema, property); - ApplyArrayAnnotations(propertySchema, property); + OpenApiSchema propertySchema; + + if (isNullable && RequiresNullableWrapper(property.PropertyType)) + { + propertySchema = new OpenApiSchema + { + OneOf = + [ + new OpenApiSchema { Type = JsonSchemaType.Null }, + generatedSchema + ] + }; + } + else + { + propertySchema = generatedSchema; + if (isNullable) + propertySchema.Type = propertySchema.Type.GetValueOrDefault() | JsonSchemaType.Null; + } string propertyName = property.Name.ToLowerUnderscoredWords(); schema.Properties[propertyName] = propertySchema; @@ -56,8 +74,6 @@ public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext // Ensure no required array - all properties are optional for PATCH schema.Required = null; - - return Task.CompletedTask; } private static bool IsDeltaType(Type type) @@ -84,76 +100,10 @@ private static bool IsPropertyNullable(PropertyInfo property) } } - private static OpenApiSchema CreateSchemaForType(Type type, bool isNullable) + private static bool RequiresNullableWrapper(Type type) { - var schema = new OpenApiSchema(); - JsonSchemaType schemaType = default; - - // Handle nullable value types (int?, DateTime?, etc.) - var underlyingType = Nullable.GetUnderlyingType(type); - if (underlyingType is not null) - { - type = underlyingType; - isNullable = true; - } - - // Add null type if nullable - if (isNullable) - { - schemaType |= JsonSchemaType.Null; - } - - if (type == typeof(string)) - { - schemaType |= JsonSchemaType.String; - } - else if (type == typeof(bool)) - { - schemaType |= JsonSchemaType.Boolean; - } - else if (type == typeof(int) || type == typeof(long) || type == typeof(short) || type == typeof(byte)) - { - schemaType |= JsonSchemaType.Integer; - } - else if (type == typeof(double) || type == typeof(float) || type == typeof(decimal)) - { - schemaType |= JsonSchemaType.Number; - } - else if (type == typeof(DateTime) || type == typeof(DateTimeOffset)) - { - schemaType |= JsonSchemaType.String; - schema.Format = "date-time"; - } - else if (type == typeof(Guid)) - { - schemaType |= JsonSchemaType.String; - schema.Format = "uuid"; - } - else if (type.IsEnum) - { - schemaType |= JsonSchemaType.String; - } - else if (type.IsGenericType && type.GetInterfaces().Concat([type]).Any(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IDictionary<,>))) - { - schemaType |= JsonSchemaType.Object; - var valueType = type.GetGenericArguments().ElementAtOrDefault(1); - if (valueType is not null) - { - schema.AdditionalProperties = CreateSchemaForType(valueType, false); - } - } - else if (TryGetEnumerableElementType(type, out var elementType)) - { - schemaType |= JsonSchemaType.Array; - schema.Items = CreateSchemaForType(elementType, false); - } - else - { - schemaType = JsonSchemaType.Object; - } - - schema.Type = schemaType; - return schema; + type = Nullable.GetUnderlyingType(type) ?? type; + return type != typeof(string) && !type.IsValueType; } private static void ApplyArrayAnnotations(OpenApiSchema schema, PropertyInfo property) @@ -169,26 +119,4 @@ private static void ApplyArrayAnnotations(OpenApiSchema schema, PropertyInfo pro schema.MaxItems = maxLength.Length; } } - - private static bool TryGetEnumerableElementType(Type type, out Type elementType) - { - if (type.IsArray) - { - elementType = type.GetElementType() ?? typeof(object); - return true; - } - - if (type == typeof(string) || !typeof(System.Collections.IEnumerable).IsAssignableFrom(type)) - { - elementType = typeof(object); - return false; - } - - var enumerableType = type.IsGenericType && type.GetGenericTypeDefinition() == typeof(IEnumerable<>) - ? type - : type.GetInterfaces().FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>)); - - elementType = enumerableType?.GetGenericArguments()[0] ?? typeof(object); - return true; - } } diff --git a/tests/Exceptionless.Tests/Api/Data/openapi.json b/tests/Exceptionless.Tests/Api/Data/openapi.json index 68ec2364ce..e4e676697a 100644 --- a/tests/Exceptionless.Tests/Api/Data/openapi.json +++ b/tests/Exceptionless.Tests/Api/Data/openapi.json @@ -5766,12 +5766,12 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } }, "application/*\u002Bjson": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } } }, @@ -5842,12 +5842,12 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } }, "application/*\u002Bjson": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } } }, @@ -11133,6 +11133,16 @@ "delete_bot_data_enabled": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, "promoted_tabs": { "type": [ "null", @@ -11817,6 +11827,27 @@ } } }, + "OrganizationBudgetAlertSettings": { + "required": [ + "enabled", + "thresholds" + ], + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "thresholds": { + "uniqueItems": true, + "type": "array", + "items": { + "type": "integer", + "format": "int32" + }, + "description": "Percentage thresholds of the organization\u0027s effective monthly event allowance.\nExample: [50, 80, 90]." + } + } + }, "PersistentEvent": { "required": [ "organization_id", @@ -12075,6 +12106,42 @@ } } }, + "ProjectIngestLimit": { + "required": [ + "type" + ], + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/ProjectIngestLimitType" + }, + "fixed_limit": { + "type": [ + "null", + "integer" + ], + "format": "int32" + }, + "percent_of_organization_limit": { + "type": [ + "null", + "number" + ], + "format": "double" + } + } + }, + "ProjectIngestLimitType": { + "enum": [ + 0, + 1 + ], + "type": "integer", + "x-enumNames": [ + "Fixed", + "PercentOfOrganizationLimit" + ] + }, "ResetPasswordModel": { "required": [ "password_reset_token", @@ -12354,6 +12421,25 @@ }, "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." }, + "UpdateOrganization": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "budget_alert_settings": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/OrganizationBudgetAlertSettings" + } + ] + } + }, + "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." + }, "UpdateProject": { "type": "object", "properties": { @@ -12363,14 +12449,28 @@ "delete_bot_data_enabled": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, "promoted_tabs": { - "type": [ - "null", - "array" - ], - "items": { - "type": "string" - } + "oneOf": [ + { + "type": "null" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] } }, "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." @@ -12415,23 +12515,31 @@ ] }, "columns": { - "type": [ - "null", - "object" - ], - "additionalProperties": { - "type": "boolean" - } + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "additionalProperties": { + "type": "boolean" + } + } + ] }, "column_order": { - "maxItems": 50, - "type": [ - "null", - "array" - ], - "items": { - "type": "string" - } + "oneOf": [ + { + "type": "null" + }, + { + "maxItems": 50, + "type": "array", + "items": { + "type": "string" + } + } + ] }, "show_stats": { "type": [ @@ -13068,6 +13176,16 @@ }, "is_over_request_limit": { "type": "boolean" + }, + "budget_alert_settings": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/OrganizationBudgetAlertSettings" + } + ] } } }, @@ -13084,6 +13202,7 @@ "event_count", "has_premium_features", "has_slack_integration", + "is_smart_throttled", "usage_hours", "usage" ], @@ -13147,6 +13266,33 @@ "has_slack_integration": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, + "effective_ingest_limit": { + "type": [ + "null", + "integer" + ], + "format": "int32" + }, + "is_smart_throttled": { + "type": "boolean" + }, + "smart_throttle_sample_rate": { + "type": [ + "null", + "number" + ], + "format": "double" + }, "usage_hours": { "type": "array", "items": { diff --git a/tests/Exceptionless.Tests/Api/Endpoints/ContactEndpointTests.cs b/tests/Exceptionless.Tests/Api/Endpoints/ContactEndpointTests.cs index 21989facec..5c4cc4f62f 100644 --- a/tests/Exceptionless.Tests/Api/Endpoints/ContactEndpointTests.cs +++ b/tests/Exceptionless.Tests/Api/Endpoints/ContactEndpointTests.cs @@ -181,6 +181,16 @@ public Task SendOrganizationNoticeAsync(User user, Organization organization, bo return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + public Task SendOrganizationPaymentFailedAsync(User owner, Organization organization) { return Task.CompletedTask; diff --git a/tests/Exceptionless.Tests/Api/Endpoints/OrganizationEndpointTests.cs b/tests/Exceptionless.Tests/Api/Endpoints/OrganizationEndpointTests.cs index 4fddfae77a..aa72e99cb1 100644 --- a/tests/Exceptionless.Tests/Api/Endpoints/OrganizationEndpointTests.cs +++ b/tests/Exceptionless.Tests/Api/Endpoints/OrganizationEndpointTests.cs @@ -2,6 +2,7 @@ using Exceptionless.Core; using Exceptionless.Core.Billing; using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; using Exceptionless.Core.Models; using Exceptionless.Core.Models.Billing; using Exceptionless.Core.Repositories; @@ -12,6 +13,8 @@ using Exceptionless.Web.Utility; using Foundatio.Repositories; using Foundatio.Repositories.Utility; +using Foundatio.AsyncEx; +using Foundatio.Messaging; using Microsoft.Extensions.DependencyInjection; using Stripe; using Xunit; @@ -1945,6 +1948,115 @@ public async Task PatchAsync_EmptyJsonBody_ReturnsOriginalOrganizationUnchanged( Assert.Equal(originalOrganization.UpdatedUtc, updated.UpdatedUtc); } + [Fact] + public async Task PatchAsync_SameNameWithBudgetAlertSettings_DoesNotRejectAsDuplicate() + { + // Regression: If PATCH sends the org's current name alongside budget_alert_settings, + // IsOrganizationNameAvailableInternalAsync previously found the org itself and returned + // false ("already exists"), rejecting an idempotent name update. + var originalOrg = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(originalOrg); + originalOrg.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(originalOrg, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content(new { name = originalOrg.Name, budget_alert_settings = new { enabled = true, thresholds = new[] { 75 } } }) + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Equal(originalOrg.Name, updated.Name); + Assert.NotNull(updated.BudgetAlertSettings); + Assert.True(updated.BudgetAlertSettings.Enabled); + } + + [Fact] + public async Task PatchAsync_BudgetAlertSettingsNull_ClearsSettings() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = [50] }; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", organization.Id) + .Content("""{"budget_alert_settings":null}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Null(updated.BudgetAlertSettings); + } + + [Theory] + [InlineData("{\"budget_alert_settings\":{\"enabled\":false,\"thresholds\":[100]}}")] + [InlineData("{\"budget_alert_settings\":{\"enabled\":true,\"thresholds\":[]}}")] + public async Task PatchAsync_InvalidBudgetAlertSettings_ReturnsValidationError(string content) + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content(content, "application/json") + .StatusCodeShouldBeUnprocessableEntity() + ); + } + + [Fact] + public Task PatchAsync_NullBudgetAlertThresholds_ReturnsBadRequest() + { + return SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content("""{"budget_alert_settings":{"enabled":true,"thresholds":null}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } + + [Fact] + public async Task PatchAsync_EnablingAlertAboveCurrentUsage_PublishesOnceImmediately() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + organization.BudgetAlertSettings = null; + organization.GetCurrentUsage(TimeProvider).Total = 600; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var messageBus = GetService(); + var countdown = new AsyncCountdownEvent(1); + OrganizationBudgetAlert? alert = null; + await messageBus.SubscribeAsync(message => + { + alert = message; + countdown.Signal(); + }, TestCancellationToken); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", organization.Id) + .Content("""{"budget_alert_settings":{"enabled":true,"thresholds":[50]}}""", "application/json") + .StatusCodeShouldBeOk() + ); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.NotNull(alert); + Assert.Equal(50, alert.Threshold); + Assert.Equal(600, alert.CurrentEventCount); + } + private static MultipartFormDataContent CreateProfileImageContent() { byte[] bytes = new byte[256 * 1024]; diff --git a/tests/Exceptionless.Tests/Api/Endpoints/ProjectEndpointTests.cs b/tests/Exceptionless.Tests/Api/Endpoints/ProjectEndpointTests.cs index 00516ae49d..0e49af8aac 100644 --- a/tests/Exceptionless.Tests/Api/Endpoints/ProjectEndpointTests.cs +++ b/tests/Exceptionless.Tests/Api/Endpoints/ProjectEndpointTests.cs @@ -1634,4 +1634,147 @@ public async Task CanGetProjectListStats() Assert.Equal(newStacks.Count, project.StackCount); Assert.Equal(newEvents.Count, project.EventCount); } + + [Fact] + public async Task PatchAsync_WithFixedIngestLimit_SavesCorrectly() + { + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":500}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(ProjectIngestLimitType.Fixed, updated.IngestLimit.Type); + Assert.Equal(500, updated.IngestLimit.FixedLimit); + + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + Assert.NotNull(project.IngestLimit); + Assert.Equal(ProjectIngestLimitType.Fixed, project.IngestLimit.Type); + Assert.Equal(500, project.IngestLimit.FixedLimit); + } + + [Fact] + public async Task PatchAsync_WithPercentageIngestLimit_SavesCorrectly() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":1,"percent_of_organization_limit":75.0}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(ProjectIngestLimitType.PercentOfOrganizationLimit, updated.IngestLimit.Type); + Assert.Equal(75.0m, updated.IngestLimit.PercentOfOrganizationLimit); + + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + Assert.NotNull(project.IngestLimit); + Assert.Equal(75.0m, project.IngestLimit.PercentOfOrganizationLimit); + } + + [Fact] + public async Task PatchAsync_RemoveIngestLimit_SavesNull() + { + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 }; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":null}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Null(updated.IngestLimit); + + var saved = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(saved); + Assert.Null(saved.IngestLimit); + } + + [Fact] + public async Task PatchAsync_OnlyIngestLimit_NoNameValidationError() + { + var original = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(original); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":200}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Equal(original.Name, updated.Name); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(200, updated.IngestLimit.FixedLimit); + } + + [Theory] + [InlineData("{\"ingest_limit\":{\"type\":0,\"fixed_limit\":0}}")] + [InlineData("{\"ingest_limit\":{\"type\":1,\"percent_of_organization_limit\":0}}")] + [InlineData("{\"ingest_limit\":{\"type\":1,\"percent_of_organization_limit\":100.1}}")] + public async Task PatchAsync_InvalidIngestLimit_ReturnsValidationError(string content) + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content(content, "application/json") + .StatusCodeShouldBeUnprocessableEntity() + ); + } + + [Fact] + public Task PatchAsync_FractionalFixedIngestLimit_ReturnsBadRequest() + { + return SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":1.5}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } + + [Fact] + public async Task PatchAsync_PercentageIngestLimitForUnlimitedOrganization_ReturnsValidationError() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = -1; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":1,"percent_of_organization_limit":50}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } } diff --git a/tests/Exceptionless.Tests/Api/Handlers/ContactHandlerTests.cs b/tests/Exceptionless.Tests/Api/Handlers/ContactHandlerTests.cs index 0b15a79b36..163049facc 100644 --- a/tests/Exceptionless.Tests/Api/Handlers/ContactHandlerTests.cs +++ b/tests/Exceptionless.Tests/Api/Handlers/ContactHandlerTests.cs @@ -117,6 +117,16 @@ public Task SendOrganizationNoticeAsync(User user, Organization organization, bo return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + public Task SendOrganizationPaymentFailedAsync(User owner, Organization organization) { return Task.CompletedTask; diff --git a/tests/Exceptionless.Tests/Api/OpenApiSnapshotTests.cs b/tests/Exceptionless.Tests/Api/OpenApiSnapshotTests.cs index 9561adc053..69645ab9c3 100644 --- a/tests/Exceptionless.Tests/Api/OpenApiSnapshotTests.cs +++ b/tests/Exceptionless.Tests/Api/OpenApiSnapshotTests.cs @@ -125,8 +125,8 @@ public async Task GetOpenApiJson_MigratedContracts_PreserveControllerMetadata() AssertArrayResponseSchema(paths, "/api/v2/projects/{projectId}/webhooks", "WebHook"); AssertRequiredJsonRequestBody(paths, "/api/v1/error/{id}", "patch", "UpdateEvent"); - AssertRequiredJsonRequestBody(paths, "/api/v2/organizations/{id}", "patch", "NewOrganization"); - AssertRequiredJsonRequestBody(paths, "/api/v2/organizations/{id}", "put", "NewOrganization"); + AssertRequiredJsonRequestBody(paths, "/api/v2/organizations/{id}", "patch", "UpdateOrganization"); + AssertRequiredJsonRequestBody(paths, "/api/v2/organizations/{id}", "put", "UpdateOrganization"); AssertRequiredJsonRequestBody(paths, "/api/v2/projects/{id}", "patch", "UpdateProject"); AssertRequiredJsonRequestBody(paths, "/api/v2/projects/{id}", "put", "UpdateProject"); AssertRequiredJsonRequestBody(paths, "/api/v2/tokens/{id}", "patch", "UpdateToken"); @@ -195,6 +195,26 @@ public async Task GetOpenApiJson_MigratedContracts_PreserveControllerMetadata() } } + [Fact] + public async Task GetOpenApiJson_DeltaNullableComplexProperties_PreserveSchemasAndAnnotations() + { + using var document = await GetOpenApiDocumentAsync(); + var schemas = document.RootElement.GetProperty("components").GetProperty("schemas"); + + AssertNullableReference(schemas, "UpdateOrganization", "budget_alert_settings", "OrganizationBudgetAlertSettings"); + AssertNullableReference(schemas, "UpdateProject", "ingest_limit", "ProjectIngestLimit"); + + var columnOrderAlternatives = schemas + .GetProperty("UpdateSavedView") + .GetProperty("properties") + .GetProperty("column_order") + .GetProperty("oneOf") + .EnumerateArray(); + var columnOrderArray = Assert.Single(columnOrderAlternatives, alternative => + alternative.TryGetProperty("type", out var type) && type.GetString() == "array"); + Assert.Equal(50, columnOrderArray.GetProperty("maxItems").GetInt32()); + } + private async Task GetOpenApiDocumentAsync() { string json = await GetOpenApiJsonAsync(); @@ -271,4 +291,20 @@ private static void AssertRequiredJsonRequestBody(JsonElement paths, string path foreach (var mediaType in content.EnumerateObject()) Assert.Equal($"#/components/schemas/{expectedSchema}", mediaType.Value.GetProperty("schema").GetProperty("$ref").GetString()); } + + private static void AssertNullableReference(JsonElement schemas, string schemaName, string propertyName, string referenceName) + { + Assert.True(schemas.TryGetProperty(referenceName, out _), $"Expected referenced schema '{referenceName}'."); + + var propertySchema = schemas + .GetProperty(schemaName) + .GetProperty("properties") + .GetProperty(propertyName); + var alternatives = propertySchema.GetProperty("oneOf").EnumerateArray().ToArray(); + + Assert.Contains(alternatives, alternative => alternative.TryGetProperty("type", out var type) && type.GetString() == "null"); + Assert.Contains(alternatives, alternative => + alternative.TryGetProperty("$ref", out var reference) && + reference.GetString() == $"#/components/schemas/{referenceName}"); + } } diff --git a/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs b/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs index fb07d885e1..5a86695711 100644 --- a/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs +++ b/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs @@ -268,4 +268,73 @@ private PersistentEvent GenerateEvent(DateTimeOffset? occurrenceDate = null, str occurrenceDate ??= DateTimeOffset.Now; return _eventData.GenerateEvent(projectId: TestConstants.ProjectId, organizationId: TestConstants.OrganizationId, generateTags: false, generateData: false, occurrenceDate: occurrenceDate, userIdentity: userIdentity, type: type, source: source, sessionId: sessionId); } + + [Fact] + public async Task CanRunJob_WhenProjectOverLimit_BlocksEvents() + { + var project = await _projectRepository.GetByIdAsync(TestConstants.ProjectId); + Assert.NotNull(project); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 1 }; + project.GetCurrentUsage(TimeProvider).Total = 1; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + Assert.NotNull(await EnqueueEventPostAsync(GenerateEvent())); + Assert.Equal(1, (await _eventQueue.GetQueueStatsAsync()).Enqueued); + + var result = await _job.RunAsync(TestCancellationToken); + Assert.True(result.IsSuccess); + + await RefreshDataAsync(); + Assert.Equal(0, await _eventRepository.CountAsync()); + + var usage = await _usageService.GetUsageAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.Equal(1, usage.CurrentUsage.Blocked); + } + + [Fact] + public async Task CanRunJob_WhenSmartThrottleApplied_BlocksEventsOnce() + { + var organization = await _organizationRepository.GetByIdAsync(TestConstants.OrganizationId); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1_000_000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var project = await _projectRepository.GetByIdAsync(TestConstants.ProjectId); + Assert.NotNull(project); + await _projectRepository.AddAsync(_projectData.GenerateProject(generateId: true, organizationId: organization.Id), o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(_projectData.GenerateProject(generateId: true, organizationId: organization.Id), o => o.ImmediateConsistency().Cache()); + var utcNow = TimeProvider.GetUtcNow().UtcDateTime; + var endOfMonth = new DateTime(utcNow.Year, utcNow.Month, 1, 0, 0, 0, DateTimeKind.Utc).AddMonths(1); + double windowsLeft = Math.Max(1, Math.Ceiling((endOfMonth - utcNow).TotalMinutes / 5)); + int currentWindowSpike = (int)Math.Floor(organization.MaxEventsPerMonth / windowsLeft * 10 * 0.9); + await _usageService.IncrementTotalAsync(organization, project.Id, currentWindowSpike); + + var throttleResult = await _usageService.GetSmartThrottleRateAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.True(throttleResult.IsThrottled); + Assert.Equal(0.05, throttleResult.SampleRate); + + var events = Enumerable.Range(0, 100).Select(index => GenerateEvent(source: $"source-{index}")).ToList(); + Assert.NotNull(await EnqueueEventPostAsync(events)); + + var result = await _job.RunAsync(TestCancellationToken); + Assert.True(result.IsSuccess); + + var usage = await _usageService.GetUsageAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.True(usage.CurrentUsage.Blocked > 0); + Assert.Equal(0, usage.CurrentUsage.Discarded); + + await RefreshDataAsync(); + Assert.InRange(await _eventRepository.CountAsync(), 1, 20); + } + + [Fact] + public void SmartThrottleSelection_SingleEventPosts_UsesStableFivePercentSample() + { + var persistentEvent = GenerateEvent(); + + int accepted = Enumerable.Range(0, 1_000) + .Count(index => EventPostsJob.IsSelectedForSmartThrottle($"post-{index}", persistentEvent, 0)); + + Assert.InRange(accepted, 30, 70); + } } diff --git a/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs b/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs new file mode 100644 index 0000000000..93b3a89c27 --- /dev/null +++ b/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs @@ -0,0 +1,242 @@ +using Exceptionless.Core.Billing; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Jobs.WorkItemHandlers; +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.DateTimeExtensions; +using Exceptionless.Tests.Mail; +using Exceptionless.Tests.Extensions; +using Exceptionless.Tests.Utility; +using Foundatio.Jobs; +using Foundatio.Repositories; +using Microsoft.Extensions.DependencyInjection; +using System.Text.Json; +using Xunit; + +namespace Exceptionless.Tests.Jobs.WorkItemHandlers; + +public class UsageBudgetNotificationWorkItemHandlerTests : IntegrationTestsBase +{ + private const string OrganizationId = "664ec4c1f12e4f2b7a0d3101"; + private const string ProjectId = "664ec4c1f12e4f2b7a0d3201"; + private const string UserId = "664ec4c1f12e4f2b7a0d3301"; + + public UsageBudgetNotificationWorkItemHandlerTests(ITestOutputHelper output, AppWebHostFactory factory) : base(output, factory) { } + + private CountingMailer Mailer => GetService(); + private IOrganizationRepository OrganizationRepository => GetService(); + private IProjectRepository ProjectRepository => GetService(); + private IUserRepository UserRepository => GetService(); + private UsageService UsageService => GetService(); + + protected override void RegisterServices(IServiceCollection services) + { + base.RegisterServices(services); + services.AddSingleton(); + services.ReplaceSingleton(serviceProvider => serviceProvider.GetRequiredService()); + } + + protected override async Task ResetDataAsync() + { + await base.ResetDataAsync(); + Mailer.Reset(); + + var billingManager = GetService(); + var billingPlans = GetService(); + var organization = GetService().GenerateOrganization(billingManager, billingPlans, id: OrganizationId, name: "Budget Organization", plan: billingPlans.ExtraLargePlan); + organization.MaxEventsPerMonth = 1_000_000; + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = [50] }; + await OrganizationRepository.AddAsync(organization, options => options.ImmediateConsistency().Cache()); + + var projectData = GetService(); + await ProjectRepository.AddAsync([ + projectData.GenerateProject(id: ProjectId, organizationId: OrganizationId, name: "Noisy Project"), + projectData.GenerateProject(generateId: true, organizationId: OrganizationId, name: "Quiet Project"), + projectData.GenerateProject(generateId: true, organizationId: OrganizationId, name: "Another Project") + ], options => options.ImmediateConsistency().Cache()); + + var user = GetService().GenerateUser(id: UserId, organizationId: OrganizationId, emailAddress: "budget-owner@example.org"); + user.IsEmailAddressVerified = true; + user.EmailNotificationsEnabled = true; + await UserRepository.AddAsync(user, options => options.ImmediateConsistency().Cache()); + } + + [Fact] + public async Task OrganizationHandler_CurrentCrossing_SendsCurrentUsageAndPlan() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + Assert.NotNull(organization); + organization.GetCurrentUsage(TimeProvider).Total = 600_000; + await OrganizationRepository.SaveAsync(organization, options => options.ImmediateConsistency().Cache()); + + await HandleAsync(GetService(), new OrganizationBudgetAlertWorkItem + { + OrganizationId = OrganizationId, + Threshold = 50, + ThresholdEventCount = 1, + CurrentEventCount = 1, + EventLimit = 1 + }); + + var call = Assert.Single(Mailer.OrganizationBudgetAlertCalls); + Assert.Equal(500_000, call.ThresholdEventCount); + Assert.Equal(600_000, call.CurrentEventCount); + Assert.Equal(1_000_000, call.EventLimit); + } + + [Fact] + public void UsageNotificationContracts_LegacyPayloadWithoutPeriod_Deserializes() + { + const string organizationPayload = """{"OrganizationId":"organization","Threshold":50,"ThresholdEventCount":500,"CurrentEventCount":600,"EventLimit":1000}"""; + const string projectPayload = """{"OrganizationId":"organization","ProjectId":"project","SampleRate":0.05,"CurrentEventCount":600,"EventLimit":1000}"""; + + Assert.Equal(0, JsonSerializer.Deserialize(organizationPayload)!.UsagePeriod); + Assert.Equal(0, JsonSerializer.Deserialize(organizationPayload)!.UsagePeriod); + Assert.Equal(0, JsonSerializer.Deserialize(projectPayload)!.UsagePeriod); + Assert.Equal(0, JsonSerializer.Deserialize(projectPayload)!.UsagePeriod); + } + + [Fact] + public async Task OrganizationHandler_DisabledAfterQueue_SuppressesStaleEmail() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + Assert.NotNull(organization); + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = false, Thresholds = [50] }; + await OrganizationRepository.SaveAsync(organization, options => options.ImmediateConsistency().Cache()); + + await HandleAsync(GetService(), new OrganizationBudgetAlertWorkItem + { + OrganizationId = OrganizationId, + Threshold = 50, + ThresholdEventCount = 500_000, + CurrentEventCount = 600_000, + EventLimit = 1_000_000, + UsagePeriod = TimeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }); + + Assert.Empty(Mailer.OrganizationBudgetAlertCalls); + } + + [Fact] + public async Task ProjectHandler_CurrentThrottle_SendsProjectUsageAndFairShare() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + var project = await ProjectRepository.GetByIdAsync(ProjectId); + Assert.NotNull(organization); + Assert.NotNull(project); + + int spike = GetCurrentWindowSpike(organization.MaxEventsPerMonth); + await UsageService.IncrementTotalAsync(organization, project.Id, spike); + Assert.True((await UsageService.GetEventIngestAllowanceAsync(organization, project)).SmartThrottle.IsThrottled); + + await HandleAsync(GetService(), new ProjectSmartThrottleWorkItem + { + OrganizationId = OrganizationId, + ProjectId = ProjectId, + SampleRate = 0.05, + CurrentEventCount = 1, + EventLimit = 1, + UsagePeriod = TimeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }); + + var call = Assert.Single(Mailer.ProjectThrottleCalls); + Assert.Equal(spike, call.CurrentEventCount); + Assert.Equal(333_333, call.EventLimit); + Assert.Equal(0.05, call.SampleRate); + } + + [Fact] + public async Task ProjectHandler_ThrottleExpiredBeforeHandling_SuppressesStaleEmail() + { + await HandleAsync(GetService(), new ProjectSmartThrottleWorkItem + { + OrganizationId = OrganizationId, + ProjectId = ProjectId, + SampleRate = 0.05, + CurrentEventCount = 1, + EventLimit = 1, + UsagePeriod = TimeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }); + + Assert.Empty(Mailer.ProjectThrottleCalls); + } + + [Fact] + public async Task OrganizationHandler_PartialFailureRetry_SendsEachRecipientOnce() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + Assert.NotNull(organization); + organization.GetCurrentUsage(TimeProvider).Total = 600_000; + await OrganizationRepository.SaveAsync(organization, options => options.ImmediateConsistency().Cache()); + + var secondUser = GetService().GenerateUser(generateId: true, organizationId: OrganizationId, emailAddress: "second-owner@example.org"); + secondUser.IsEmailAddressVerified = true; + secondUser.EmailNotificationsEnabled = true; + await UserRepository.AddAsync(secondUser, options => options.ImmediateConsistency().Cache()); + + var workItem = new OrganizationBudgetAlertWorkItem + { + OrganizationId = OrganizationId, + Threshold = 50, + ThresholdEventCount = 500_000, + CurrentEventCount = 600_000, + EventLimit = 1_000_000, + UsagePeriod = TimeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }; + Mailer.ThrowOnOrganizationBudgetAlertAttempt = 2; + + await Assert.ThrowsAsync(() => HandleAsync(GetService(), workItem)); + Mailer.ThrowOnOrganizationBudgetAlertAttempt = null; + await HandleAsync(GetService(), workItem); + + Assert.Equal(2, Mailer.OrganizationBudgetAlertCalls.Count); + Assert.Equal(2, Mailer.OrganizationBudgetAlertCalls.Select(call => call.UserId).Distinct().Count()); + } + + [Fact] + public async Task ProjectHandler_DuplicateDelivery_SendsRecipientOnce() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + var project = await ProjectRepository.GetByIdAsync(ProjectId); + Assert.NotNull(organization); + Assert.NotNull(project); + int spike = GetCurrentWindowSpike(organization.MaxEventsPerMonth); + await UsageService.IncrementTotalAsync(organization, project.Id, spike); + Assert.True((await UsageService.GetEventIngestAllowanceAsync(organization, project)).SmartThrottle.IsThrottled); + + var workItem = new ProjectSmartThrottleWorkItem + { + OrganizationId = OrganizationId, + ProjectId = ProjectId, + SampleRate = 0.05, + CurrentEventCount = spike, + EventLimit = 333_333, + UsagePeriod = TimeProvider.GetUtcNow().UtcDateTime.StartOfMonth().ToEpoch() + }; + + await HandleAsync(GetService(), workItem); + await HandleAsync(GetService(), workItem); + + Assert.Single(Mailer.ProjectThrottleCalls); + } + + private int GetCurrentWindowSpike(int maxEventsPerMonth) + { + var utcNow = TimeProvider.GetUtcNow().UtcDateTime; + var endOfMonth = new DateTime(utcNow.Year, utcNow.Month, 1, 0, 0, 0, DateTimeKind.Utc).AddMonths(1); + double windowsLeft = Math.Max(1, Math.Ceiling((endOfMonth - utcNow).TotalMinutes / 5)); + return (int)Math.Floor(maxEventsPerMonth / windowsLeft * 10 * 0.9); + } + + private static async Task HandleAsync(WorkItemHandlerBase handler, object workItem) + { + await using var workItemLock = await handler.GetWorkItemLockAsync(workItem, TestContext.Current.CancellationToken); + Assert.NotNull(workItemLock); + var context = new WorkItemContext(workItem, "test-job", workItemLock, TestContext.Current.CancellationToken, static (_, _) => Task.CompletedTask); + await handler.HandleItemAsync(context); + } +} diff --git a/tests/Exceptionless.Tests/Mail/CountingMailer.cs b/tests/Exceptionless.Tests/Mail/CountingMailer.cs index 2967a84886..5c8d039891 100644 --- a/tests/Exceptionless.Tests/Mail/CountingMailer.cs +++ b/tests/Exceptionless.Tests/Mail/CountingMailer.cs @@ -6,16 +6,20 @@ namespace Exceptionless.Tests.Mail; public class CountingMailer : IMailer { private int _organizationNoticeCount; + private int _organizationBudgetAlertAttemptCount; public int OrganizationNoticeCount => _organizationNoticeCount; public List OrganizationNoticeCalls { get; } = []; + public List OrganizationBudgetAlertCalls { get; } = []; + public List ProjectThrottleCalls { get; } = []; /// /// When true, throws instead of recording a call. /// Reset by . /// public bool ShouldThrow { get; set; } + public int? ThrowOnOrganizationBudgetAlertAttempt { get; set; } public Task SendContactRequestAsync(string name, string emailAddress, string? company, string? subject, string message, string? clientIpAddress, string? userAgent, string? referrer) { @@ -55,6 +59,28 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + int attempt = Interlocked.Increment(ref _organizationBudgetAlertAttemptCount); + if (ThrowOnOrganizationBudgetAlertAttempt == attempt) + throw new InvalidOperationException("Simulated budget alert mailer failure."); + + lock (OrganizationBudgetAlertCalls) + { + OrganizationBudgetAlertCalls.Add(new OrganizationBudgetAlertCall(user.Id, organization.Id, threshold, thresholdEventCount, currentEventCount, eventLimit)); + } + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + lock (ProjectThrottleCalls) + { + ProjectThrottleCalls.Add(new ProjectThrottleCall(user.Id, organization.Id, project.Id, sampleRate, currentEventCount, eventLimit)); + } + return Task.CompletedTask; + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { return Task.CompletedTask; @@ -73,12 +99,24 @@ public Task SendUserPasswordResetAsync(User user) public void Reset() { Interlocked.Exchange(ref _organizationNoticeCount, 0); + Interlocked.Exchange(ref _organizationBudgetAlertAttemptCount, 0); lock (OrganizationNoticeCalls) { OrganizationNoticeCalls.Clear(); } + lock (OrganizationBudgetAlertCalls) + { + OrganizationBudgetAlertCalls.Clear(); + } + lock (ProjectThrottleCalls) + { + ProjectThrottleCalls.Clear(); + } ShouldThrow = false; + ThrowOnOrganizationBudgetAlertAttempt = null; } } public record OrganizationNoticeCall(string UserId, string OrganizationId, bool IsOverMonthlyLimit, bool IsOverHourlyLimit); +public record OrganizationBudgetAlertCall(string UserId, string OrganizationId, int Threshold, int ThresholdEventCount, int CurrentEventCount, int EventLimit); +public record ProjectThrottleCall(string UserId, string OrganizationId, string ProjectId, double SampleRate, int CurrentEventCount, int EventLimit); diff --git a/tests/Exceptionless.Tests/Mail/NullMailer.cs b/tests/Exceptionless.Tests/Mail/NullMailer.cs index 7ce54fcde6..86b854c649 100644 --- a/tests/Exceptionless.Tests/Mail/NullMailer.cs +++ b/tests/Exceptionless.Tests/Mail/NullMailer.cs @@ -35,6 +35,16 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { return Task.CompletedTask; diff --git a/tests/Exceptionless.Tests/Serializer/Models/OrganizationSerializerTests.cs b/tests/Exceptionless.Tests/Serializer/Models/OrganizationSerializerTests.cs index f18ea0b28c..a916a3ae05 100644 --- a/tests/Exceptionless.Tests/Serializer/Models/OrganizationSerializerTests.cs +++ b/tests/Exceptionless.Tests/Serializer/Models/OrganizationSerializerTests.cs @@ -137,4 +137,41 @@ public void Deserialize_SnakeCaseJson_ParsesCorrectly() Assert.Equal("EX_SMALL", result.PlanId); Assert.Equal(10000, result.MaxEventsPerMonth); } + + [Fact] + public void Deserialize_LegacyJsonWithoutBudgetSettings_DefaultsToNull() + { + var result = _serializer.Deserialize("""{"id":"550000000000000000000005","name":"Legacy"}"""); + + Assert.NotNull(result); + Assert.Null(result.BudgetAlertSettings); + } + + [Fact] + public void RoundTrip_WithBudgetSettings_PreservesSnakeCaseValues() + { + var organization = new Organization + { + Id = "550000000000000000000006", + Name = "Budgeted", + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = [50, 80] } + }; + + string json = _serializer.SerializeToString(organization); + var result = _serializer.Deserialize(json); + + Assert.Contains("\"budget_alert_settings\"", json); + Assert.NotNull(result?.BudgetAlertSettings); + Assert.True(result.BudgetAlertSettings.Enabled); + Assert.Equal([50, 80], result.BudgetAlertSettings.Thresholds); + } + + [Fact] + public void RoundTrip_WithNullBudgetSettings_OmitsPersistedField() + { + string json = _serializer.SerializeToString(new Organization { Id = "550000000000000000000007", Name = "Default" }); + + Assert.DoesNotContain("budget_alert_settings", json); + Assert.Null(_serializer.Deserialize(json)?.BudgetAlertSettings); + } } diff --git a/tests/Exceptionless.Tests/Serializer/Models/ProjectSerializerTests.cs b/tests/Exceptionless.Tests/Serializer/Models/ProjectSerializerTests.cs index b512a647f6..be5a604e79 100644 --- a/tests/Exceptionless.Tests/Serializer/Models/ProjectSerializerTests.cs +++ b/tests/Exceptionless.Tests/Serializer/Models/ProjectSerializerTests.cs @@ -69,4 +69,50 @@ public void Deserialize_RoundTrip_PreservesBasicProperties() Assert.Equal("My Project", deserialized.Name); Assert.True(deserialized.IsConfigured); } + + [Fact] + public void Deserialize_LegacyJsonWithoutIngestLimit_DefaultsToNull() + { + var project = _serializer.Deserialize("""{"id":"proj-legacy","organization_id":"org1","name":"Legacy"}"""); + + Assert.NotNull(project); + Assert.Null(project.IngestLimit); + } + + public static TheoryData IngestLimits => new() + { + { ProjectIngestLimitType.Fixed, 100, null }, + { ProjectIngestLimitType.PercentOfOrganizationLimit, null, 8.3m } + }; + + [Theory] + [MemberData(nameof(IngestLimits))] + public void RoundTrip_WithIngestLimit_PreservesSnakeCaseValues(ProjectIngestLimitType type, int? fixedLimit, decimal? percentage) + { + var project = new Project + { + Id = "proj-budget", + OrganizationId = "org1", + Name = "Budgeted", + IngestLimit = new ProjectIngestLimit { Type = type, FixedLimit = fixedLimit, PercentOfOrganizationLimit = percentage } + }; + + string json = _serializer.SerializeToString(project); + var result = _serializer.Deserialize(json); + + Assert.Contains("\"ingest_limit\"", json); + Assert.NotNull(result?.IngestLimit); + Assert.Equal(type, result.IngestLimit.Type); + Assert.Equal(fixedLimit, result.IngestLimit.FixedLimit); + Assert.Equal(percentage, result.IngestLimit.PercentOfOrganizationLimit); + } + + [Fact] + public void RoundTrip_WithNullIngestLimit_OmitsPersistedField() + { + string json = _serializer.SerializeToString(new Project { Id = "proj-default", OrganizationId = "org1", Name = "Default" }); + + Assert.DoesNotContain("ingest_limit", json); + Assert.Null(_serializer.Deserialize(json)?.IngestLimit); + } } diff --git a/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs b/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs new file mode 100644 index 0000000000..2460891ec6 --- /dev/null +++ b/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs @@ -0,0 +1,681 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using Exceptionless.Core.Billing; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.Tests.Extensions; +using Foundatio.AsyncEx; +using Foundatio.Caching; +using Foundatio.Messaging; +using Foundatio.Repositories; +using Xunit; +using LogLevel = Microsoft.Extensions.Logging.LogLevel; + +namespace Exceptionless.Tests.Services; + +public sealed partial class UsageServiceTests +{ + [Fact] + public async Task GetSmartThrottleRateAsync_OrganizationNotFound_ReturnsNoThrottle() + { + // Regression test: When the organization does not exist, GetMaxEventsPerMonthAsync + // returns 0 (default). GetSmartThrottleRateAsync must not divide by zero. + string nonExistentOrgId = "000000000000000000000099"; + string nonExistentProjectId = "000000000000000000000098"; + + // Act - this would divide by zero before the fix (maxEventsPerMonth=0 returned for missing org) + var result = await _usageService.GetSmartThrottleRateAsync(nonExistentOrgId, nonExistentProjectId); + + // Assert - should return NoThrottle since maxEventsPerMonth <= 0 means unlimited/invalid + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + // ── GetEventIngestAllowanceAsync ──────────────────────────────────────── + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithNoIngestLimit_ReturnsOrgLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.True(result.EventsLeft > 0); + Assert.Equal(1.0, result.SampleRate); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedIngestLimit_BelowOrgLimit_ReturnsFixed() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(100, result.EventsLeft); + Assert.Equal(100, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedIngestLimit_AboveOrgLimit_ReturnsOrgLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 10000 } + }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 749); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(1, result.EventsLeft); + Assert.Equal(750, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithPercentageIngestLimit_50Percent_ReturnsCorrectLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 50 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(500, result.EventsLeft); + Assert.Equal(500, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithInvalidPersistedPercentage_IsInactive() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks + }, o => o.ImmediateConsistency().Cache()); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 150 }; + + var result = await _usageService.GetEventIngestAllowanceAsync(organization, project); + + Assert.Equal(-1, result.EffectiveProjectLimit); + Assert.True(result.EventsLeft > 0); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedLimit_OverLimit_IsOverProjectLimitTrue() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + project.GetCurrentUsage(TimeProvider).Total = 100; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(0, result.EventsLeft); + Assert.True(result.IsOverProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.Equal(100, result.EffectiveProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Organization_OverLimit_IsOverOrgLimitTrue() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 750); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(0, result.EventsLeft); + Assert.True(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_UnlimitedOrg_WithFixedProjectLimit_ReturnsFixed() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = _plans.UnlimitedPlan.MaxEventsPerMonth, PlanId = _plans.UnlimitedPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 500 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(500, result.EventsLeft); + Assert.Equal(500, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + // ── GetSmartThrottleRateAsync ──────────────────────────────────────────── + + [Fact] + public async Task GetSmartThrottleRateAsync_BelowThreshold_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 600; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + project1.GetCurrentUsage(TimeProvider).Total = 600; + await _projectRepository.SaveAsync(project1, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_SingleProjectInOrg_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + project.GetCurrentUsage(TimeProvider).Total = 900; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_NoOrgUsage_ReturnsNoThrottle() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_NoProjectUsage_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_AboveThreshold_FairShare_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + // fairShare = 1000/2 = 500; projectTotal = 900 → fairShareRatio = 1.8 ≤ 2.0 + project1.GetCurrentUsage(TimeProvider).Total = 900; + await _projectRepository.SaveAsync(project1, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_CurrentWindowSpike_UsesFixedFivePercentSample() + { + var organization = await _organizationRepository.AddAsync(new Organization + { + Name = "Test", + MaxEventsPerMonth = 1_000_000, + PlanId = _plans.ExtraLargePlan.Id + }, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P3", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization, project1.Id, 1_900); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.True(result.IsThrottled); + Assert.Equal(0.05, result.SampleRate); + Assert.Equal(1_900, result.CurrentProjectUsage); + Assert.Equal(333_333, result.FairShareLimit); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_HighMonthlyUsageWithoutCurrentSpike_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1_000_000, PlanId = _plans.ExtraLargePlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900_000; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }; + project1.GetCurrentUsage(TimeProvider).Total = 900_000; + project1 = await _projectRepository.AddAsync(project1, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_UnlimitedOrg_ReturnsNoThrottle() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = _plans.UnlimitedPlan.MaxEventsPerMonth, PlanId = _plans.UnlimitedPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + // ── Budget Alert Threshold tests ───────────────────────────────────────── + + [Fact] + public async Task IncrementTotalAsync_CrossingThreshold_PublishesAlertMessage() + { + var messageBus = GetService(); + var countdown = new AsyncCountdownEvent(1); + OrganizationBudgetAlert? alert = null; + await messageBus.SubscribeAsync(a => + { + alert = a; + countdown.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 15); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.NotNull(alert); + Assert.Equal(organization.Id, alert.OrganizationId); + Assert.Equal(50, alert.Threshold); + Assert.Equal(500, alert.ThresholdEventCount); + Assert.Equal(1000, alert.EventLimit); + } + + [Fact] + public async Task IncrementTotalAsync_Threshold_NotDuplicated_WhenCrossedTwice() + { + var messageBus = GetService(); + int alertCount = 0; + var firstAlert = new AsyncCountdownEvent(1); + await messageBus.SubscribeAsync(a => + { + int count = Interlocked.Increment(ref alertCount); + if (count == 1) firstAlert.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 15); + await firstAlert.WaitAsync(TimeSpan.FromSeconds(5)); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 20); + await Task.Delay(200, TestCancellationToken); + + Assert.Equal(1, alertCount); + } + + [Fact] + public async Task CheckBudgetAlertThresholdsAsync_ConcurrentCalls_PublishesAlertExactlyOnce() + { + // Regression: non-atomic get-then-set allowed two concurrent workers to both see the + // key absent and both publish. The fix uses AddAsync (atomic) so only the worker + // whose increment returns 1 sends the alert. + var messageBus = GetService(); + int alertCount = 0; + var firstAlert = new AsyncCountdownEvent(1); + await messageBus.SubscribeAsync(_ => + { + int count = Interlocked.Increment(ref alertCount); + if (count == 1) firstAlert.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + // Simulate concurrent workers by calling GetEventsLeftAsync + IncrementTotalAsync + // concurrently. Both workers see usage crossing 50%; only one should publish the alert. + await _usageService.GetEventsLeftAsync(organization.Id); + var tasks = Enumerable.Range(0, 5) + .Select(_ => _usageService.IncrementTotalAsync(organization.Id, project.Id, 15)) + .ToArray(); + await Task.WhenAll(tasks); + + // Wait for the first (and only) alert, then give extra time for any duplicates. + await firstAlert.WaitAsync(TimeSpan.FromSeconds(5)); + await Task.Delay(300, TestCancellationToken); + + Assert.Equal(1, alertCount); + } + + [Fact] + public async Task IncrementTotalAsync_DisabledAlerts_DoNotPublishMessages() + { + var messageBus = GetService(); + int alertCount = 0; + await messageBus.SubscribeAsync(_ => { Interlocked.Increment(ref alertCount); }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = false, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 100); + + await Task.Delay(200, TestCancellationToken); + Assert.Equal(0, alertCount); + } + + [Fact] + public async Task IncrementTotalAsync_AllThresholds_AllFireOnce() + { + var messageBus = GetService(); + var firedThresholds = new System.Collections.Concurrent.ConcurrentBag(); + var countdown = new AsyncCountdownEvent(3); + await messageBus.SubscribeAsync(a => + { + firedThresholds.Add(a.Threshold); + countdown.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50, 80, 90 } } + }; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 950); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.Equal(3, firedThresholds.Count); + Assert.Contains(50, firedThresholds); + Assert.Contains(80, firedThresholds); + Assert.Contains(90, firedThresholds); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_SmallPercentageLimit_UsesRoundUp_NotZero() + { + // Regression: 1% of 50 events floored to 0 with (int)cast, making the project + // permanently over-budget on the first event. Math.Ceiling must be used. + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 50, PlanId = _plans.SmallPlan.Id }; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 1 } + }, o => o.ImmediateConsistency().Cache()); + + // 1% of 50 = 0.5 → must ceil to 1, not floor to 0 + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + Assert.True(result.EffectiveProjectLimit >= 1, $"EffectiveProjectLimit was {result.EffectiveProjectLimit}, expected >= 1 (floor bug would produce 0)"); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_DecimalPercentage_UsesExactCeiling() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 3000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 1.1m } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization, project); + + Assert.Equal(33, result.EffectiveProjectLimit); + } + + [Fact] + public void GetBudgetThresholdEventCount_ExactPercentage_DoesNotRoundPastInteger() + { + Assert.Equal(210, UsageService.GetBudgetThresholdEventCount(3000, 7)); + Assert.Equal(5250, UsageService.GetBudgetThresholdEventCount(75000, 7)); + } + + [Fact] + public async Task ReserveEventIngestAsync_ConcurrentProjectCapReservations_DoNotExceedRemainingLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + await _usageService.IncrementTotalAsync(organization, project.Id, 90); + + var candidates = Enumerable.Range(0, 10).Select(index => new EventIngestCandidate(index, (ulong)index)).ToArray(); + var reservations = await Task.WhenAll(Enumerable.Range(0, 10).Select(index => + _usageService.ReserveEventIngestAsync(organization, project, $"reservation-{index}", candidates, TestCancellationToken))); + + Assert.Equal(10, reservations.Sum(reservation => reservation.ReservedCount)); + + await Task.WhenAll(reservations.Select(_usageService.ReleaseEventIngestReservationAsync)); + } + + [Fact] + public async Task CompleteEventIngestReservationAsync_ReleasesUnprocessedCapacity() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + await _usageService.IncrementTotalAsync(organization, project.Id, 90); + var candidates = Enumerable.Range(0, 10).Select(index => new EventIngestCandidate(index, (ulong)index)).ToArray(); + var first = await _usageService.ReserveEventIngestAsync(organization, project, "first", candidates, TestCancellationToken); + + await _usageService.CompleteEventIngestReservationAsync(first, organization, 4); + var second = await _usageService.ReserveEventIngestAsync(organization, project, "second", candidates, TestCancellationToken); + + Assert.Equal(6, second.ReservedCount); + await _usageService.ReleaseEventIngestReservationAsync(second); + } + + [Fact] + public async Task ReserveEventIngestAsync_AfterRelease_RecreatesReservationWithoutDoubleDecrement() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Retry", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Retry", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 10 } + }, o => o.ImmediateConsistency().Cache()); + var candidates = Enumerable.Range(0, 10).Select(index => new EventIngestCandidate(index, (ulong)index)).ToArray(); + var first = await _usageService.ReserveEventIngestAsync(organization, project, "retry", candidates, TestCancellationToken); + await _usageService.ReleaseEventIngestReservationAsync(first); + + var retried = await _usageService.ReserveEventIngestAsync(organization, project, "retry", candidates, TestCancellationToken); + var competing = await _usageService.ReserveEventIngestAsync(organization, project, "competing", candidates, TestCancellationToken); + + Assert.Equal(10, retried.ReservedCount); + Assert.Equal(0, competing.ReservedCount); + await _usageService.ReleaseEventIngestReservationAsync(retried); + await _usageService.CompleteEventIngestReservationAsync(competing, organization, 0); + } + + [Fact] + public async Task ReserveEventIngestAsync_AfterCompletion_ReturnsCompletedTombstone() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Completed", MaxEventsPerMonth = 100, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Completed", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var candidates = Enumerable.Range(0, 2).Select(index => new EventIngestCandidate(index, (ulong)index)).ToArray(); + var reservation = await _usageService.ReserveEventIngestAsync(organization, project, "completed", candidates, TestCancellationToken); + await _usageService.CompleteEventIngestReservationAsync(reservation, organization, 2); + + var retry = await _usageService.ReserveEventIngestAsync(organization, project, "completed", candidates, TestCancellationToken); + + Assert.True(retry.IsCompleted); + Assert.Equal(2, retry.ProcessedCount); + } + + [Fact] + public async Task ReserveEventIngestAsync_ColdBucketNoisyBatch_IsSampledImmediately() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1_000_000, PlanId = _plans.ExtraLargePlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Noisy", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "Quiet", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "Other", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + int spike = 2000; + var candidates = Enumerable.Range(0, spike) + .Select(index => new EventIngestCandidate(index, (ulong)index * 7919UL)) + .ToArray(); + + var reservation = await _usageService.ReserveEventIngestAsync(organization, project, "cold-bucket", candidates, TestCancellationToken); + + Assert.True(reservation.SmartThrottle.IsThrottled); + Assert.InRange(reservation.ReservedCount, 60, 140); + await _usageService.ReleaseEventIngestReservationAsync(reservation); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_FirstMinutesOfMonth_IgnoresPreviousMonthBucket() + { + TimeProvider.SetUtcNow(new DateTimeOffset(2026, 7, 31, 23, 59, 0, TimeSpan.Zero)); + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _usageService.IncrementTotalAsync(organization, project.Id, 900); + TimeProvider.Advance(TimeSpan.FromMinutes(3)); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization, project); + + Assert.Equal(1000, result.EventsLeft); + } + + [Fact] + public async Task CompleteEventIngestReservationAsync_AfterMonthBoundary_ChargesCurrentPeriodWithoutReleasingCapacity() + { + TimeProvider.SetUtcNow(new DateTimeOffset(2026, 7, 31, 23, 59, 0, TimeSpan.Zero)); + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Boundary", MaxEventsPerMonth = 100, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Boundary", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var candidates = Enumerable.Range(0, 10).Select(index => new EventIngestCandidate(index, (ulong)index)).ToArray(); + var reservation = await _usageService.ReserveEventIngestAsync(organization, project, "month-boundary", candidates, TestCancellationToken); + + TimeProvider.Advance(TimeSpan.FromMinutes(2)); + await _usageService.CompleteEventIngestReservationAsync(reservation, organization, 10); + + Assert.Equal(90, (await _usageService.GetEventIngestAllowanceAsync(organization, project)).EventsLeft); + } + + [Fact] + public async Task CompleteEventIngestReservationAsync_MissingState_FailsClosed() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Missing", MaxEventsPerMonth = 100, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Missing", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var reservation = await _usageService.ReserveEventIngestAsync(organization, project, "missing-state", [new EventIngestCandidate(0, 0)], TestCancellationToken); + await GetService().RemoveAsync($"usage:ingest-reservation:{{{organization.Id}}}:missing-state"); + + await Assert.ThrowsAsync(() => _usageService.CompleteEventIngestReservationAsync(reservation, organization, 1)); + } +} diff --git a/tests/Exceptionless.Tests/Services/UsageServiceTests.cs b/tests/Exceptionless.Tests/Services/UsageServiceTests.cs index 8a54c6cbd7..4be2f70b97 100644 --- a/tests/Exceptionless.Tests/Services/UsageServiceTests.cs +++ b/tests/Exceptionless.Tests/Services/UsageServiceTests.cs @@ -1,3 +1,4 @@ +using System.Collections.Concurrent; using System.Diagnostics; using Exceptionless.Core.Billing; using Exceptionless.Core.Extensions; @@ -14,7 +15,7 @@ namespace Exceptionless.Tests.Services; -public sealed class UsageServiceTests : IntegrationTestsBase +public sealed partial class UsageServiceTests : IntegrationTestsBase { private readonly IOrganizationRepository _organizationRepository; private readonly IProjectRepository _projectRepository; @@ -655,4 +656,5 @@ public async Task RunBenchmarkAsync() sw.Stop(); _logger.LogInformation("Time: {Duration:g}, Avg: ({AverageTickDuration:g}ticks | {AverageDuration}ms)", sw.Elapsed, sw.ElapsedTicks / iterations, sw.ElapsedMilliseconds / iterations); } + } diff --git a/tests/Exceptionless.Tests/Validation/ProjectValidatorTests.cs b/tests/Exceptionless.Tests/Validation/ProjectValidatorTests.cs index 9c5df0271c..2989a571ce 100644 --- a/tests/Exceptionless.Tests/Validation/ProjectValidatorTests.cs +++ b/tests/Exceptionless.Tests/Validation/ProjectValidatorTests.cs @@ -135,6 +135,24 @@ public async Task Validate_WhenProjectIsValid_ReturnsSuccess() Assert.True(isValid); } + [Theory] + [InlineData("0.0001", true)] + [InlineData("8.3", true)] + [InlineData("0.00001", false)] + public async Task Validate_PercentageIngestLimit_UsesBoundedPrecision(string value, bool expectedValid) + { + var project = CreateValidProject(); + project.IngestLimit = new ProjectIngestLimit + { + Type = ProjectIngestLimitType.PercentOfOrganizationLimit, + PercentOfOrganizationLimit = Decimal.Parse(value, System.Globalization.CultureInfo.InvariantCulture) + }; + + var (isValid, _) = await _validator.ValidateAsync(project); + + Assert.Equal(expectedValid, isValid); + } + [Fact] public async Task ValidateAsync_WhenProjectIsValid_ReturnsSuccess() { diff --git a/tests/http/organizations.http b/tests/http/organizations.http index 09119dbf67..22dc921876 100644 --- a/tests/http/organizations.http +++ b/tests/http/organizations.http @@ -71,6 +71,20 @@ Content-Type: application/json { "name": "My updated organization" } +### Enable Budget Alerts +PATCH {{apiUrl}}/organizations/{{organizationId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "budget_alert_settings": { "enabled": true, "thresholds": [50, 80, 90] } } + +### Clear Budget Alert Settings +PATCH {{apiUrl}}/organizations/{{organizationId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "budget_alert_settings": null } + ### Upload Icon # Replace {{iconFile}} with a local PNG, JPEG, GIF, or WebP image. POST {{apiUrl}}/organizations/{{organizationId}}/icon diff --git a/tests/http/projects.http b/tests/http/projects.http index 45bbcbf6df..36f523dcd7 100644 --- a/tests/http/projects.http +++ b/tests/http/projects.http @@ -153,6 +153,27 @@ Content-Type: application/json { "Name": "My Updated Project" } +### Set Fixed Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": { "type": 0, "fixed_limit": 20000 } } + +### Set Percentage Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": { "type": 1, "percent_of_organization_limit": 50 } } + +### Clear Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": null } + ### Delete DELETE {{apiUrl}}/projects/{{projectId}} Authorization: Bearer {{token}}