Adding the runs_jobs_detail_aggregation endpoint by tmikula-dev · Pull Request #243 · AbsaOSS/EventGate

tmikula-dev · 2026-10-20T13:47:38Z

Overview

This pull request introduces support for window-wide aggregation queries alongside existing paginated named queries, with a focus on the new runs_jobs_detail_aggregation endpoint. It updates the OpenAPI spec, handler validation, query registry, and reader logic to support aggregation queries with a new response shape, and introduces new data models for typed aggregation results.

Release Notes

Added a new runs_jobs_detail_aggregation named-query endpoint providing window-wide summary aggregates plus hourly/daily timeline buckets for runs/jobs data.

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…regated-query' into feature/116-adding-dashboard-aggregated-query

oto-macenauer-absa

solid piece of work, thx, just minor comments and questions

oto-macenauer-absa · 2026-10-23T13:26:29Z

+
+-- name: get_runs_jobs_timeline_hourly(ts_start, ts_end)
+-- Hourly timeline buckets (non-empty buckets only) for runs_jobs_detail, oldest first.
+SELECT (EXTRACT(EPOCH FROM date_trunc('hour', to_timestamp(r.timestamp_start / 1000.0))) * 1000)::bigint


check if this is UTC safe (no time zone drift)

Both hourly and daily SQL buckets now use date_trunc(..., 'UTC'). See changes in commit: 4b521db

oto-macenauer-absa · 2026-10-23T13:29:59Z

+                if db_cursor.description is None:
+                    raise RuntimeError("Timeline query returned no result description.")
+                timeline_cols = [desc[0] for desc in db_cursor.description]
+                timeline_rows = [dict(zip(timeline_cols, row, strict=True)) for row in db_cursor.fetchall()]


timeline and summary are two separate reads, I suppose the risk is minimal but there might be inserts in between

Totally agree, please see the solution + the integration test in commit: 1707265

oto-macenauer-absa · 2026-10-23T13:36:30Z

+        timestamp_start = parsed_body.get("timestamp_start")
+        timestamp_end = parsed_body.get("timestamp_end")
+
+        int_fields = (


add validation for timestamp_start <= timestamp_end

Sure thing, I added the validation to the solution. See commit: 513d8a0

oto-macenauer-absa · 2026-10-23T14:03:41Z

  echo "### pip installs ###" && \
  echo "###################" && \
-    # requirements.txt pins the version of confluent-kafka.
+    # requirements.txt pins the version of confluent-kafka. \


comments don't require newlines:)

my IDE actually complained about this, possibly because it's within the RUN command (but this docker was working and the comments did not have it, so you might be wrong and IDE is false alarm)

but actually I fixed this Dockerfile in some past PR so correcting the base should result in Tobias not touching this file even IIRC

My IDE shouts about it, so I wanted to make it happy :D But this comment is solved simply by merging the master to this dev branch. Approach of having newlines is also implemented in one past PR, which Ladislav is talking about.

lsulak · 2026-10-26T16:16:13Z

+        self._require_connection_config()

        limit = max(1, min(limit, POSTGRES_MAX_LIMIT))
        now_ms = int(time.time() * 1000)


perhaps write a comment that this is time since epoch for clarity?

Sure, I did add the comment to all three places where current time in ms is established. See commit: d17382a

lsulak · 2026-10-26T16:27:27Z

+ INNER JOIN public_cps_za_runs r ON j.event_id = r.event_id
+ WHERE r.timestamp_start >= :ts_start AND r.timestamp_start <= :ts_end
+ GROUP BY bucket_start
+ ORDER BY bucket_start ASC;


I'm thinking that maybe there is a value in adding one more parameter - source_app for filtering. It can be NULL by default, meaning ALL. This can be implemented in many ways, a first quick idea could be using CASE statement in the where clause

this comment applies to pretty much all queries in this file

(further segmentation of some of these queries could be based on tenant_id, but maybe let's wait for further requirements)

I did implement it here in commit: d17382a. There is also implementation of second filter that was suggested by @lukasmatta in his comment below: #243 (comment). Please see the implementation, if you find it sufficient. The solution is done by SQL expression instead of CASE.

lsulak

Reviewed most of it. Good job, but I noticed several things to improve. Also, when do you anticipate we should remove the stats named queries? They are basically a duplicates of the other ones in named_queries.sql - my recommendation is either now & a ticket (probably preferable to keep PRs small scoped)

lsulak · 2026-10-26T16:31:11Z

+ INNER JOIN public_cps_za_runs r ON j.event_id = r.event_id
+ WHERE r.timestamp_start >= :ts_start AND r.timestamp_start <= :ts_end
+ GROUP BY bucket_start
+ ORDER BY bucket_start ASC;


this comment applies to pretty much all queries in this file

lsulak · 2026-10-26T16:31:57Z

+           AS bucket_start,
+       COUNT(*) AS total,
+       COUNT(*) FILTER (WHERE j.status = 'succeeded') AS succeeded,
+       COUNT(*) FILTER (WHERE j.status = 'failed') AS failed


for the sake of completeness, the topic has also these two statuses available:

"killed", "skipped"

I would advise to add them here and the other query in this file

I added that in commit: d17382a. I am little lost in some of your notes, so please if I am missing something. Shout:) Thank you!

lsulak · 2026-10-26T16:35:10Z

+ INNER JOIN public_cps_za_runs r ON j.event_id = r.event_id
+ WHERE r.timestamp_start >= :ts_start AND r.timestamp_start <= :ts_end
+ GROUP BY bucket_start
+ ORDER BY bucket_start ASC;


(further segmentation of some of these queries could be based on tenant_id, but maybe let's wait for further requirements)

lsulak · 2026-10-26T16:39:36Z

+    return _build_json_response(
+        200,
+        {
+            "success": True,


hmm this is a bit weird contract, success is quite visible from the status code, and then also status code is there twice. Is there any situation when it differentiates?

I completely see what you are saying. However this contract was established in the past. And this PR is only a refactor - not creating anything new. So modification in this PR would change the established client contract and that is IMO not a current scope of this task. We can discuss it further and create from this comment a follow-up GH issue. What do you think?

lsulak · 2026-10-26T16:48:21Z

+    timeline_daily_sql_key: str
+
+
+SUPPORTED_AGGREGATION_QUERIES: dict[str, AggregationQuery] = {


hmm this really looks comlicated, the whole file. Isn't there already some native aiosql capability to list supported queries? Also, maybe it makes sense to split the existing queries into 2 files, i.e. 2 categories - like job detail type queries and then analytical / aggregation like queries

Update: I could not find it quickly in the aiosql docs so I just opened aiosql code and found this as a public property that you can hopefully use: available_queries against your query, for example when you are doing this: aiosql.from_path(_SQL_DIR, "psycopg2"))

Also the naming is not really explanatory. This is a typical SW development process - we started with named queries, then created data structures and algorithms (loading process, validation etc) around that, and then there was a new type, more analytical queries, and now there are new data structures and algorithms around that - but basically everything is a named query, right? So the names are not descriptive enough. This is technical debt. Not to mention that stats.sql are now duplicated.

I think that this PR is a good time to reduce tech debt, not add much more to it

I tried to make naming conventions better to current query state. See following commit: 4efa108.

I am also implementing suggested approach of aiosql available_queries. Thanks for raising that! But I want to have that in separate PR - I am about to build that on top of this PR. Following PR: #251. The solution is robust and solves several issues that we would face in the future, but for current solution maybe too complex. Share your thoughts please on the added link there.

# Conflicts: # Dockerfile

lukasmatta · 2026-11-03T12:29:07Z



+@dataclass(frozen=True)
+class AggregationQueryParams:


At some point, we may want to support filtering by tenant and source (and possibly other parameters as requirements emerge). If it’s a simple change, we can add it within this PR... otherwise, it can done in a separate PR.

Please see my response to similar comment from @lsulak: #243 (comment). The implementation is done in commit: d17382a.

… snapshot

tmikula-dev · 2026-11-03T15:59:56Z

Reviewed most of it. Good job, but I noticed several things to improve. Also, when do you anticipate we should remove the stats named queries? They are basically a duplicates of the other ones in named_queries.sql - my recommendation is either now & a ticket (probably preferable to keep PRs small scoped)

I completely feel, what you are saying or what I feel from the review points. I tried to make the solution more "friendly" by renaming the queries as described in the comment responses. But you are also suggesting some things, that are more on a refactor level or ready to be deeper discussion. If we want to start some better refactored, modifying some contracts etc lets move to follow-up issue and close this one (if possible and good enough) as a v1. Please share your thoughts.

… ones.

feat(aggregation_endpoint): Adding the runs_jobs_detail_aggregation e…

2d58812

…ndpoint

tmikula-dev self-assigned this Oct 20, 2026

tmikula-dev added the enhancement New feature or request label Oct 20, 2026

tmikula-dev requested review from lsulak, oto-macenauer-absa and petr-pokorny-absa as code owners October 20, 2026 13:47

Merge branch 'master' into feature/116-adding-dashboard-aggregated-query

51a2d47

tmikula-dev linked an issue Oct 20, 2026 that may be closed by this pull request

EventStats: compiled queries endpoint for server-side aggregations #116

Open

tmikula-dev added 3 commits October 20, 2026 17:07

Fixing the Dockerfile librdkafka.

b687ffd

Merge remote-tracking branch 'origin/feature/116-adding-dashboard-agg…

8af1d15

…regated-query' into feature/116-adding-dashboard-aggregated-query

Error message typo.

3c9b3bd

oto-macenauer-absa reviewed Oct 23, 2026

View reviewed changes

lsulak reviewed Oct 26, 2026

View reviewed changes

tmikula-dev added 3 commits November 3, 2026 11:15

Merge branch 'master' into feature/116-adding-dashboard-aggregated-query

eed0e59

# Conflicts: # Dockerfile

Implementing review comments: Timestamp validation

513d8a0

Implementing review comments: UTC safety

4b521db

lukasmatta reviewed Nov 3, 2026

View reviewed changes

tmikula-dev added 2 commits November 3, 2026 14:53

Implementing review comments: Summary and timeline queries within one…

1707265

… snapshot

Implementing review comments mentioned by lsulak

d17382a

Implementing review comments: Renaming PaginatedQueries to JobsDetail…

4efa108

… ones.

tmikula-dev added this pull request to stack #252 November 4, 2026 11:06

		timeline_daily_sql_key: str


		SUPPORTED_AGGREGATION_QUERIES: dict[str, AggregationQuery] = {



		@dataclass(frozen=True)
		class AggregationQueryParams:

Conversation

tmikula-dev commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Release Notes

Related

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oto-macenauer-absa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmikula-dev Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmikula-dev Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmikula-dev Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lsulak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lsulak Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmikula-dev Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmikula-dev commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

tmikula-dev Nov 3, 2026 •

edited

Loading

tmikula-dev Nov 3, 2026 •

edited

Loading

tmikula-dev Nov 3, 2026 •

edited

Loading

lsulak Oct 26, 2026 •

edited

Loading

tmikula-dev Nov 4, 2026 •

edited

Loading

tmikula-dev commented Nov 3, 2026 •

edited

Loading