Skip to content

feat: let SQL query dashboard filters use an optional source - #3691

Open
karl-power wants to merge 3 commits into
sql-dashboard-variables-appfrom
sql-dashboard-variables-source
Open

karl-power wants to merge 3 commits into
sql-dashboard-variables-appfrom
sql-dashboard-variables-source

Conversation

@karl-power

@karl-power karl-power commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Lets SQL query dashboard filters pick an optional source, so their SQL can use $__sourceTable and $__filters like raw SQL tiles. Follows up on review feedback on #3674.

  • Adds an optional source to the filter schema. Without one, $__sourceTable and $__filters are still rejected.
  • $__filters expands to the dashboard's broadcast selections for that source, via a new renderSourceFiltersToSql wrapper around the existing renderFiltersToSql.
  • The editor adds a clearable Source picker beside Connection. Autocomplete, validation and Preview values follow the selected source.
  • Moves resolveConnectionSourceSync from RawSqlChartEditor.tsx to utils/connectionSourceSync.ts so the filter editor can reuse it.
  • The API rejects missing or PromQL sources. The external API and MCP use sourceId and also check that the source is on the filter's connection.
  • Export and import map the source by name.

Screenshots or video

Screenshot 2026-11-17 at 14 16 13

How to reproduce

Starting point: yarn dev with a "Local ClickHouse" connection and a Logs source on default.otel_logs.

On main:

  1. Add a SQL query values filter with SELECT DISTINCT ServiceName FROM $__sourceTable. The editor rejects it.

On this branch:

  1. Add a Queried values filter Severity on Logs with expression SeverityText.
  2. Add a SQL query values filter Service with connection "Local ClickHouse", Source Logs, variable svc, and:
    SELECT DISTINCT ServiceName FROM $__sourceTable
    WHERE $__timeFilter(Timestamp) AND $__filters
    ORDER BY 1
  3. Save. The Service dropdown lists services.
  4. Pick a value in Severity. Service narrows to matching services.
  5. Edit Service and clear Source. Saving is blocked with "SQL uses $__sourceTable and $__filters but no source is selected".

References

Add an optional source to SQL query filters so their SQL can use
$__sourceTable and $__filters, as raw SQL tiles can.

- Add `source` to the filter schema and expand $__sourceTable and
  $__filters when it is set; without one, both are still rejected
- Expand $__filters to the dashboard's broadcast selections for that
  source, in both the dropdown and the editor preview
- Add a clearable source picker next to the connection picker, limited to
  Log, Trace, Session and Metric sources on the chosen connection
- Move resolveConnectionSourceSync out of the raw SQL chart editor into a
  shared util so the filter editor can use it
- Check on save that the source exists, is a supported kind, and is on
  the filter's connection
- Read and write it as `sourceId` in the external API and MCP, and map it
  by name on dashboard export and import

Refs HDX-5524
@karl-power
karl-power requested a review from a team as a code owner November 17, 2026 13:02
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 17, 2026 2:14pm UTC
hyperdx-storybook Ready Ready Preview Nov 17, 2026 2:14pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 4ca7b11

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 17, 2026 13:03
id => ({ id, kinds }),
);
}
if (isSqlQueryFilter(filter) && filter.source) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Internal API and dashboard import accept a SQL filter source on a different connection from the filter

The external API now rejects this case through getSourceConnectionMismatches, but the internal API does not. getFilterSourceRefs and validateFilterSources only check the source's kind. The import mapping schema (DBDashboardImportPage.tsx buildMappingFormSchema, around L579) maps filterConnectionMappings and filterSourceMappings separately. So an import can pair connection A with a source on connection B. The values query then runs on A against B's $__sourceTable, and $__filters resolves column metadata against B. Fix: in validateFilterSources, reject a SQL_QUERY filter whose source.connection !== filter.connection. Also add an import-schema issue on filterSourceMappings.${idx} when the mapped source's connection differs from the mapped connection.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 81d2f9c: validateFilterSources now rejects a SQL query filter whose source is on another connection. Imports save through this endpoint, so they're covered too. I've left out the extra import-form check, since the server already blocks the mismatched save.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 5 minor

5 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

2 new clones introduced by this PR.

  • packages/app/src/components/ChartEditor/RawSqlChartEditor.tsx:147-159 ↔ packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:102-119 — 13 lines
  • packages/app/src/hooks/usePromqlLabelFilterValues.tsx:127-150 ↔ packages/app/src/hooks/useSqlQueryFilterValues.tsx:164-187 — 24 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 110 in total (1.19%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 468 passed • 1 skipped • 1187s

Status Count
✅ Passed 468
❌ Failed 0
⚠️ Flaky 0
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Additional context: touches the query rendering engine lightly (47 lines, under the 150-line bar for Tier 4)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 24
  • Production lines changed: 848 (+ 685 in test files, excluded from tier calculation)
  • Critical-path lines changed: 74
  • Branch: sql-dashboard-variables-source
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Comment thread packages/common-utils/src/filters.ts Resolved
@karl-power
karl-power requested a review from pulpdrew November 17, 2026 14:11
];

/** The source kinds a SQL_QUERY filter can use, the same as raw SQL tiles. */
export const SQL_QUERY_FILTER_SOURCE_KINDS: SourceKind[] = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — SQL_QUERY_FILTER_SOURCE_KINDS duplicates QUERY_EXPRESSION_FILTER_SOURCE_KINDS

The new array is identical (Log, Trace, Session, Metric) to QUERY_EXPRESSION_FILTER_SOURCE_KINDS a few lines above in the same file. That gives two lists to keep in sync by hand, which breaks the REQUIRED DRY rule. Alias it (export const SQL_QUERY_FILTER_SOURCE_KINDS = QUERY_EXPRESSION_FILTER_SOURCE_KINDS), or derive both from one constant such as a shared NON_PROMQL_SOURCE_KINDS, with a doc comment.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 it would be good to re-use if reasonable

// The source is optional, so drop it rather than export an empty name
// the template schema would reject.
if (filter.source) {
filter.source = getSourceName(filter.source) || undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Export silently drops a deleted source, so the imported SQL filter can't be remapped

When the source is gone, the filter is exported with source: undefined but its sqlTemplate still uses $__sourceTable/$__filters. On import, isSourceBackedTemplateFilter is false, so there is no 'Data source' mapping row, and validateDashboardFilterQueries only shows a non-blocking warning. The dashboard imports with a filter that always errors with 'no source is selected'. Export a placeholder name, or keep a marker that the filter had a source, so the import UI shows a mapping row and requires one.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* Returns an error message if a SQL_QUERY filter's source is a kind its
* $__sourceTable macro can't expand (PromQL), or null.
*/
function getSqlQueryFilterSourceKindError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — SQL filter source rules are implemented twice, once per API

getSqlQueryFilterSourceKindError plus the new filter branch in getSourceConnectionMismatches restate what getFilterSourceRefs/validateFilterSources in packages/api/src/routers/api/dashboards.ts already enforce: an allowed kind and the same connection. The internal version also skips unchanged refs, which the external one doesn't, so the two can already disagree when a source has since been deleted or changed. Move the kind and connection check into one shared helper (for example in packages/api/src/utils/) that both routers call.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

await user.click(screen.getByRole('button', { name: 'Save' }));

expect(
screen.queryByText(/no source is selected|cannot be used/),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Absence assertion runs before async validation, so 'accepts the source macros' can't fail

handleSubmit validates asynchronously, and queryByText(...) runs right after the click, before any error could render. The test would still pass if the form rejected $__sourceTable with a source selected. Assert on a positive outcome instead: pass an onSubmit spy to the Harness and await waitFor(() => expect(onSubmit).toHaveBeenCalled()), or flush the validation first, as the sibling test does with findByText.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

stretches it to the full width. */}
<div
data-testid="filter-sql-source-select"
style={{ display: 'grid' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Raw inline style used where a Mantine style prop exists

<div style={{ display: 'grid' }}> breaks the convention 'Prefer individual Mantine style props over raw styles'. Use <Box display="grid" data-testid="filter-sql-source-select"> instead.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. No P0/P1 ship-blockers. Security, correctness, API-contract, and performance reviewers returned clean, and both fixes the author flagged in prior comments (validateFilterSources cross-connection rejection; the import mapping source/connection guard) were independently verified present and correct. The $__filters/$__sourceTable expansion reuses the existing raw-SQL-tile render path, so no new injection surface is introduced, and source/connection validation is team-scoped on both the internal and external APIs.

🟡 P2 -- recommended

  • packages/common-utils/src/core/renderChartConfig.ts:2670 -- renderSourceFiltersToSql has no end-to-end test; its per-source-kind branches (log/trace implicitColumnExpression/useTextIndexForImplicitColumn, log bodyExpression) are only exercised through a mock at every call site, so wrong $__filters SQL for implicit-column or full-text broadcast conditions would go uncaught.
    • Fix: Add a unit test that calls renderSourceFiltersToSql with real Metadata and a broadcast filter per source kind, asserting the produced condition string.
  • packages/common-utils/src/core/utils.ts:964 -- when a source-backed SQL query filter is exported after its source is deleted, convertToDashboardTemplate sets source to undefined while the sqlTemplate keeps $__sourceTable/$__filters; on import the filter is no longer source-backed so no mapping row renders, leaving an un-repairable filter that errors at query time.
    • Fix: At export, when a SQL query filter's source cannot be resolved but its template uses source macros, retain a mapping row (or surface an export-time warning) so import can re-attach a source.
    • adversarial, correctness
🔵 P3 nitpicks (4)
  • packages/app/src/DBDashboardImportPage.tsx:885 -- import source auto-propagation keys only on source name, so two SQL query filters sharing a source name but mapped to different connections get the same source id written to both, tripping the "Select a source on this filter's connection" validation on a field the user never touched (submit is correctly blocked, so no corruption).
    • Fix: Scope the source-name propagation to filters whose mapped connection matches the propagated source's connection.
  • packages/common-utils/src/filters.ts:1067 -- SQL_QUERY_FILTER_SOURCE_KINDS is a byte-for-byte duplicate of QUERY_EXPRESSION_FILTER_SOURCE_KINDS with no cross-reference, so editing one silently diverges from the other.
    • Fix: Derive both from a shared base constant, or add a comment on each noting they intentionally mirror the raw-SQL-tile kinds.
  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:82 -- the usePrevious + resolveConnectionSourceSync + setValue effect boilerplate is copied verbatim from RawSqlChartEditor.tsx (flagged by jscpd); only the pure decision function was extracted.
    • Fix: Wrap the shared effect in a useConnectionSourceSync({source, connection, sources, setValue}) hook used by both call sites.
    • maintainability, previous-comments
  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:352 -- the file grew from 251 to 352 lines, past the project's documented 300-line component guideline (CLAUDE.md Key Principles).
    • Fix: Extract the Connection/Source selection subsection into a child component.

Reviewers (11): correctness, security, adversarial, kieran-typescript, api-contract, testing, maintainability, performance, previous-comments, project-standards, agent-native.

Testing gaps:

  • No test re-renders useSqlQueryFilterValues with a changed broadcast selection to confirm the react-query key changes and the dropdown refetches rather than serving stale cached rows — the two-phase render's correctness hinges on this and is currently only reasoned about (flagged by correctness, adversarial, testing, performance).
  • No server-side test asserts a SQL query filter whose source belongs to another team is rejected by validateFilterSources (only same-team connection-mismatch appears covered).

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 4ca7b11f Deployed Nov 17, 2026 by vercel[bot]
Preview – hyperdx-storybook — 4ca7b11f Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants