[skyfallwastaken]

Summary of the problem

Several heartbeat queries read far more rows than they need to:

• The Sailor's Log poll (time >= cutoff OR (direct_entry AND created_at >= cutoff)) has no usable index for the created_at branch, so it scans the whole heartbeats table. It is the slowest query in production by total time (about 47 s mean in pg_stat_statements).
• Admin alt candidates and IP/machine pairs time out at 15 s once a user has a lot of history, because the correlated exclusion EXISTS blocks parallel aggregation and the planner badly misestimates the machine/IP group count.
• Dashboard filter options and goal targets run SELECT DISTINCT over every heartbeat a user has (about 1 s for very active users).
• The footer's active users graph sorts every heartbeat from the last day, and every page reads HeartbeatCountsJob twice.
• The leaderboard API loads email addresses one user at a time (4,206 queries for the daily board).

Describe your changes

• Migrations
  • ndistinct statistics on (user_id, machine, ip_address) and (machine, user_id) so the admin machine/IP queries get parallel aggregation. They take effect at the next ANALYZE heartbeats (autovacuum's or a manual one); the migration does not run it so deploys stay fast. schema.rb does not dump CREATE STATISTICS, so databases built with db:schema:load will not have them (this only affects plans).
  • Drops index_heartbeats_on_user_id_with_ip (15 GB, last scanned 2026-09-02 after the geocoding job was removed) and idx_heartbeats_lb_eligible_user_time (10 GB, last scanned 2026-10-25 when leaderboards started counting AI coding, so its predicate no longer matches).
• Admin machine/IP queries aggregate once using a new HeartbeatExclusion::HIDDEN_JOIN_SQL (a LEFT JOIN on the same match conditions), a materialised CTE for the self join and a window count for shared machines. MATCHED_SQL, VISIBLE_SQL and HIDDEN_SQL are unchanged.
• Heartbeat.distinct_values(column): a recursive loose index scan built from the relation's own SQL, so visibility, deletion and archived project filters still apply. It is used for dashboard filter options, the rollup refresh payload and goal targets.
• Footer: the active users graph counts distinct (hour, user) pairs with integer hour arithmetic that gives the same hours as the previous EXTRACT form, and the footer reads HeartbeatCountsJob once.
• Leaderboard API preloads users' email addresses in id order, so display_name and avatar_url resolve without N+1 queries. The JSON is unchanged.
• Sailor's Log poll bounds the "received recently" branch by id. Ids follow arrival order, so the job first finds the newest heartbeat received a minute before the cutoff with a short backward scan of the primary key, then only considers ids above it. The poll becomes a bitmap OR of the time index and the primary key, with no new index. In production the last 2M heartbeats were never more than 1.7 s out of created_at order, so the minute of slack is generous.
• Stats summary computes the language and project durations with load_async while the total runs. The async executor is enabled in production.rb only, because in tests it deadlocks with transactional fixtures (Deadlock in ConnectionPool#checkout when the pinned connection is replaced concurrently (transactional system tests + Action Cable) rails/rails#58478). Outside an executor session (such as rails console) or inside a transaction the query runs in the foreground instead. It uses up to 4 extra connections per process, which fits within the pool's RAILS_MAX_THREADS + 8.

Responses are unchanged. Both versions were compared on every endpoint across user sizes, time ranges and project/language/editor/OS/category filters, and after fresh writes. The only differences are the admin alt candidates and IP pairs lists. They use LIMIT without ORDER BY, so which of the tied rows come back already depends on the plan. Every returned row is in the same full set.

index_heartbeats_on_project (covered by index_heartbeats_on_project_and_time) and index_heartbeats_on_category_and_time (not scanned in production since 2026-10-13) look droppable too but are left for a separate decision.

Screenshots / Media

Production-shaped synthetic data (9.6M heartbeats; users sized at the production p50, p90, p99 and p99.9) on PostgreSQL 18 with 16 vCPU and 32 GB. Rails in production mode, p50 ms with the Rails cache disabled unless noted.

Request	main	This PR
Sailor's Log poll query, production (EXPLAIN ANALYZE)	about 47,000 (mean)	23
Admin alt candidates	16,281 (timeout)	1,016
Admin IP/machine pairs	16,963 (timeout)	950
Admin shared machines	3,274	832
Filter options, very active user	950 to 1,050	18 to 31
Goals settings page, very active user	987	223
Goals settings page, very active user (cache on)	432	25
Active users graph (cache miss)	500	100
Typical page (footer stats not cached)	560 to 600	180 to 210
v1 leaderboard daily	3,405	402
v1 leaderboard weekly	8,759	1,116
v1 user stats with features, very active user, all time	1,528	683
Single heartbeat push	20.1	18.6

Across 1,887 cases (every endpoint by user size, time range and filter), the geometric mean of the p50 speedup is 1.57x with caches off and 1.13x with them on.

AI Usage

• Model: Claude Opus 5.5 (anthropic/claude-opus-5-5) in Amp, default reasoning effort.
• Threads: coordinating thread, Postgres tuning thread and an independent verification thread.
• The human sent 9 messages in the coordinating thread. They critiqued the work: they asked whether the time is dominated by IO rather than the framework, questioned a claimed dashboard speedup given that the default dashboard is served from rollups, ruled out ClickHouse, asked for this Postgres-only PR and asked for an independent verification, which led to removing a BRIN index that made the Sailor's Log poll slower; they then asked why the poll does not just use time, which led to the id floor.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; the latest test change still matches the notification lookup URL.

Summary

Speeds up heartbeat queries for admin tools, dashboards, leaderboards, footer counts and Sailor's Log notifications.

• The Sailor’s Log poll bounds its recent direct-entry search by heartbeat ID.
• Admin machine views build their results from shared heartbeat aggregates.
• Dashboard and goal choices find distinct heartbeat values without scanning every row.
• The footer builds its active-user graph and heartbeat counts with less repeated work.
• Leaderboard entries load users’ email addresses together.
• Wakatime stats can calculate grouped durations while the total runs.
• Two unused heartbeat indexes are removed from the database.

_{Reviews (4) · Last reviewed commit: "Anchor and escape the Cachet stub URL in..." · Reviewed by Greptile}