Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/sql-query-dashboard-filter-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@hyperdx/common-utils': patch
'@hyperdx/app': patch
'@hyperdx/api': patch
---

feat: let SQL query dashboard filters use an optional source

A SQL query filter can now pick a source on its connection. With one, its SQL can use `$__sourceTable` and `$__filters`, which applies the dashboard's filter selections for that source, the same way raw SQL tiles do. The external API and MCP read and write it as `sourceId`.
7 changes: 6 additions & 1 deletion packages/api/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -4295,10 +4295,15 @@
"description": "ID of the ClickHouse connection the query runs on.",
"example": "65f5e4a3b9e77c001a567890"
},
"sourceId": {
"type": "string",
"description": "Optional Log, Trace, Session, or Metric source on the same\nconnection. Lets `sqlTemplate` use `$__sourceTable` and\n`$__filters`.\n",
"example": "65f5e4a3b9e77c001a345678"
},
"sqlTemplate": {
"type": "string",
"minLength": 1,
"description": "Query whose first column populates the dropdown. May use the time\nmacros and other filters' variables, but not `$__sourceTable` or\n`$__filters`, since the filter has no source.\n",
"description": "Query whose first column populates the dropdown. May use the time\nmacros and other filters' variables. `$__sourceTable` and\n`$__filters` need a `sourceId`; `$__filters` expands to the\ndashboard's broadcast filter selections for that source.\n",
"example": "SELECT DISTINCT env FROM default.envs ORDER BY env"
},
"isBroadcastEnabled": {
Expand Down
16 changes: 14 additions & 2 deletions packages/api/src/mcp/__tests__/dashboards/validation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -936,9 +936,10 @@ describe('getFilterVariableWarnings', () => {
});

describe('SQL query filters', () => {
const sqlFilter = (sqlTemplate: string) => ({
const sqlFilter = (sqlTemplate: string, sourceId?: string) => ({
name: 'Environment',
sqlTemplate,
sourceId,
isVariableEnabled: true,
variableName: 'env',
});
Expand Down Expand Up @@ -968,10 +969,21 @@ describe('getFilterVariableWarnings', () => {
sqlFilter('SELECT env FROM $__sourceTable'),
]);
expect(warnings).toEqual([
'Filter "Environment": SQL query filters have no source, so $__sourceTable cannot be used',
'Filter "Environment": SQL uses $__sourceTable but no source is selected',
]);
});

it('accepts source macros when the filter has a source', () => {
expect(
getFilterVariableWarnings([
sqlFilter(
'SELECT env FROM $__sourceTable WHERE $__filters',
'65f5e4a3b9e77c001a345678',
),
]),
).toEqual([]);
});

it.each([
['a source macro', 'SELECT env FROM $__sourceTable(gauge'],
['a time macro', 'SELECT env FROM envs WHERE $__timeFilter(ts'],
Expand Down
12 changes: 9 additions & 3 deletions packages/api/src/mcp/prompts/dashboards/content.ts
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ Dashboards open with a 15-minute default window. There is no dashboard-level fie
- Using $__filter or $__conditionalAll in a Lucene where (they are matched as literal text; switch to whereLanguage: "sql", or write ServiceName:$var).
- Chaining a filter's where off a filter that is not variable-enabled. Only isVariableEnabled: true publishes the $variableName token; without it the reference resolves to nothing and the downstream dropdown is unscoped or empty.
- Referencing a filter's OWN variable in its where. That narrows its dropdown to the values already picked, so the rest of the options vanish from a multi-select. Reference a sibling filter's variable instead.
- Mixing filter-type fields: expression/sourceId/where belong to QUERY_EXPRESSION filters, options to STATIC_LIST filters, connectionId/sqlTemplate to SQL_QUERY filters.
- Mixing filter-type fields: expression/sourceId/where belong to QUERY_EXPRESSION filters, options to STATIC_LIST filters, connectionId/sqlTemplate (and an optional sourceId) to SQL_QUERY filters.
- Expecting a STATIC_LIST filter to broadcast. It cannot; a tile must reference its $variableName (via $__filter with the column passed explicitly) or nothing changes when the user picks a value.`;
}

Expand Down Expand Up @@ -1208,10 +1208,11 @@ Example (static list):
]

SQL query filter shape:
{ type: "SQL_QUERY", name, connectionId, sqlTemplate, variableName? }
{ type: "SQL_QUERY", name, connectionId, sourceId?, sqlTemplate, variableName? }

connectionId ClickHouse connection the query runs on (from clickstack_list_sources).
sqlTemplate SELECT whose FIRST column becomes the dropdown values, cast to string with nulls and duplicates dropped, in row order (add ORDER BY). May use the time macros and other filters' variables; cannot use $__sourceTable or $__filters, because the filter has no source.
sourceId Optional log, trace, session, or metric source on that connection. Needed for $__sourceTable and $__filters; $__filters expands to the dashboard's broadcast filter selections for that source.
sqlTemplate SELECT whose FIRST column becomes the dropdown values, cast to string with nulls and duplicates dropped, in row order (add ORDER BY). May use the time macros and other filters' variables, plus $__sourceTable and $__filters when sourceId is set.

Use it when the values live in a lookup table, dictionary, or join that no single source column covers; otherwise prefer QUERY_EXPRESSION. Like a static filter it is variable-only, so tiles pass the column explicitly: $__filter(<expression>, $<variableName>).

Expand All @@ -1220,6 +1221,11 @@ Example (SQL query, narrowed by another filter's variable):
{ type: "SQL_QUERY", name: "Environment", connectionId: "<connection-id>", sqlTemplate: "SELECT DISTINCT env FROM default.envs WHERE $__filter(cloud, $cloud) ORDER BY env", variableName: "env" }
]

Example (SQL query on a source, narrowed by the dashboard's broadcast filters):
filters: [
{ type: "SQL_QUERY", name: "Host", connectionId: "<connection-id>", sourceId: "<log-source-id>", sqlTemplate: "SELECT DISTINCT ResourceAttributes['host.name'] FROM $__sourceTable WHERE $__timeFilter(TimestampTime) AND $__filters ORDER BY 1", variableName: "host" }
]

Example (broadcast to every tile, the common case):
filters: [
{ type: "QUERY_EXPRESSION", name: "Service", expression: "ServiceName", sourceId: "<trace-source-id>" }
Expand Down
12 changes: 10 additions & 2 deletions packages/api/src/mcp/tools/dashboards/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1777,13 +1777,21 @@ const mcpSqlQueryFilterSchema = z
connectionId: objectIdSchema.describe(
'ClickHouse connection the query runs on. Get IDs from clickstack_list_sources.',
),
sourceId: objectIdSchema
.optional()
.describe(
'Optional log, trace, session, or metric source on the same connection. ' +
'Lets sqlTemplate use $__sourceTable and $__filters. Get IDs from ' +
'clickstack_list_sources.',
),
sqlTemplate: SqlQueryDashboardFilterSchema.shape.sqlTemplate.describe(
'ClickHouse SELECT whose FIRST column becomes the dropdown values (cast to ' +
'string; nulls and duplicates dropped; row order kept, so add ORDER BY). ' +
'May use the time macros ($__timeFilter(<column>), $__fromTime, $__toTime) ' +
"and other filters' variables, ideally as $__filter(<expression>, $<variableName>). " +
'Cannot use $__sourceTable or $__filters: the filter has no source, so name ' +
'tables explicitly. Example: "SELECT DISTINCT env FROM default.envs ORDER BY env".',
'$__sourceTable and $__filters need a sourceId; $__filters expands to the ' +
"dashboard's broadcast filter selections for that source. Without a sourceId, " +
'name tables explicitly. Example: "SELECT DISTINCT env FROM default.envs ORDER BY env".',
),
})
.describe(
Expand Down
10 changes: 8 additions & 2 deletions packages/api/src/mcp/tools/dashboards/validation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,9 @@ export function getRawSqlTileMacroWarnings(
*/
export type FilterForVariableValidation = FilterForVariableDeclaration &
Partial<Pick<QueryExpressionDashboardFilter, 'where' | 'whereLanguage'>> &
Partial<Pick<SqlQueryDashboardFilter, 'sqlTemplate'>>;
Partial<Pick<SqlQueryDashboardFilter, 'sqlTemplate'>> & {
sourceId?: string;
};

/**
* Non-blocking checks on the dashboard variables references in a filter's `where`.
Expand Down Expand Up @@ -221,7 +223,11 @@ export function getFilterVariableWarnings(

if (sqlTemplate) {
// Also reports variable macro errors, which `result` already holds.
const detail = validateSqlQueryFilterTemplate(sqlTemplate, variables);
// Source tables aren't loaded here, so a source only allows its macros.
const detail = validateSqlQueryFilterTemplate(sqlTemplate, {
variables,
sourceTables: filter.sourceId ? 'unknown' : undefined,
});
if (detail && !issues.includes(detail)) issues.push(detail);
}

Expand Down
92 changes: 92 additions & 0 deletions packages/api/src/routers/api/__tests__/dashboard.int.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1966,6 +1966,98 @@ describe('dashboard router', () => {

expect(updated.body.filters).toEqual([moved]);
});

const createSqlFilterSource = (kind: SourceKind, teamId = team._id) =>
Source.create({
kind,
name: `Test ${kind} source`,
team: teamId,
connection: connectionId,
from: { databaseName: 'test_db', tableName: `${kind}_table` },
timestampValueExpression: 'timestamp',
defaultTableSelectExpression: 'body',
});

it('persists a SQL query filter with a source', async () => {
const source = await createSqlFilterSource(SourceKind.Log);
const filter = makeSqlFilter({
source: source._id.toString(),
sqlTemplate:
'SELECT DISTINCT ServiceName FROM $__sourceTable WHERE $__filters',
});

const created = await agent
.post('/dashboards')
.send({ ...MOCK_DASHBOARD, filters: [filter] })
.expect(200);

expect(created.body.filters).toEqual([filter]);
});

it('rejects a source that does not exist', async () => {
const source = new Types.ObjectId().toString();

const response = await agent
.post('/dashboards')
.send({ ...MOCK_DASHBOARD, filters: [makeSqlFilter({ source })] })
.expect(400);

expect(response.body.message).toContain(source);
});

it("rejects another team's source", async () => {
const source = await createSqlFilterSource(
SourceKind.Log,
new Types.ObjectId(),
);

await agent
.post('/dashboards')
.send({
...MOCK_DASHBOARD,
filters: [makeSqlFilter({ source: source._id.toString() })],
})
.expect(400);
});

it('rejects a source on another connection', async () => {
const source = await createSqlFilterSource(SourceKind.Log);
const otherConnection = await Connection.create({
team: team._id,
name: 'Other Connection',
host: 'http://localhost:8123',
username: 'default',
});

const response = await agent
.post('/dashboards')
.send({
...MOCK_DASHBOARD,
filters: [
makeSqlFilter({
source: source._id.toString(),
connection: otherConnection._id.toString(),
}),
],
})
.expect(400);

expect(response.body.message).toContain("not on the filter's connection");
});

it('rejects a PromQL source', async () => {
const source = await createSqlFilterSource(SourceKind.Promql);

const response = await agent
.post('/dashboards')
.send({
...MOCK_DASHBOARD,
filters: [makeSqlFilter({ source: source._id.toString() })],
})
.expect(400);

expect(response.body.message).toContain(source._id.toString());
});
});

describe('ad hoc filters', () => {
Expand Down
54 changes: 42 additions & 12 deletions packages/api/src/routers/api/dashboards.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
isAdhocFilter,
isPrometheusLabelFilter,
isSqlQueryFilter,
SQL_QUERY_FILTER_SOURCE_KINDS,
} from '@hyperdx/common-utils/dist/filters';
import {
DashboardFilter,
Expand Down Expand Up @@ -66,10 +67,15 @@ const addFilterIssues = (
validateDashboardFilterOptionUniqueness(data.filters ?? [], ctx);
};

type FilterSourceRef = {
id: string;
kinds: SourceKind[];
/** The connection the source must be on, when the filter has its own. */
connection?: string;
};

/** The sources a filter reads from, each with the kinds it must be. */
function getFilterSourceRefs(
filter: DashboardFilter,
): { id: string; kinds: SourceKind[] }[] {
function getFilterSourceRefs(filter: DashboardFilter): FilterSourceRef[] {
if (isPrometheusLabelFilter(filter)) {
return [{ id: filter.source, kinds: [SourceKind.Promql] }];
}
Expand All @@ -79,13 +85,23 @@ function getFilterSourceRefs(
id => ({ id, kinds }),
);
}
if (isSqlQueryFilter(filter) && filter.source) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Internal API and dashboard import accept a SQL filter source on a different connection from the filter

The external API now rejects this case through getSourceConnectionMismatches, but the internal API does not. getFilterSourceRefs and validateFilterSources only check the source's kind. The import mapping schema (DBDashboardImportPage.tsx buildMappingFormSchema, around L579) maps filterConnectionMappings and filterSourceMappings separately. So an import can pair connection A with a source on connection B. The values query then runs on A against B's $__sourceTable, and $__filters resolves column metadata against B. Fix: in validateFilterSources, reject a SQL_QUERY filter whose source.connection !== filter.connection. Also add an import-schema issue on filterSourceMappings.${idx} when the mapped source's connection differs from the mapped connection.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 81d2f9c: validateFilterSources now rejects a SQL query filter whose source is on another connection. Imports save through this endpoint, so they're covered too. I've left out the extra import-form check, since the server already blocks the mismatched save.

return [
{
id: filter.source,
kinds: SQL_QUERY_FILTER_SOURCE_KINDS,
connection: filter.connection,
},
];
}
return [];
}

/**
* Rejects PROMETHEUS_LABEL and ADHOC filters that reference a source that is
* missing or of the wrong kind. Resolving these filters needs a live source of
* the right kind, unlike other filter types. References the stored filter with
* Rejects PROMETHEUS_LABEL, ADHOC and SQL_QUERY filters that reference a source
* that is missing or of the wrong kind, and SQL_QUERY filters whose source is
* on another connection. Resolving these filters needs a live source of the
* right kind, unlike other filter types. References the stored filter with
* the same ID already made are skipped, so a since-deleted source doesn't block
* unrelated saves. Returns an error message, or null when there is nothing to
* reject.
Expand All @@ -95,8 +111,8 @@ async function validateFilterSources(
filters: DashboardFilter[] = [],
existingFilters: DashboardFilter[] = [],
): Promise<string | null> {
const refKey = ({ id, kinds }: { id: string; kinds: SourceKind[] }) =>
`${id}:${kinds.join(',')}`;
const refKey = ({ id, kinds, connection }: FilterSourceRef) =>
`${id}:${kinds.join(',')}:${connection ?? ''}`;
const existingRefsById = new Map(
existingFilters.map(filter => [
filter.id,
Expand All @@ -111,17 +127,31 @@ async function validateFilterSources(
if (newRefs.length === 0) return null;

const sources = await getSources(teamId.toString());
const kindById = new Map(sources.map(s => [s._id.toString(), s.kind]));
const sourceById = new Map(sources.map(s => [s._id.toString(), s]));
const invalid = new Set(
newRefs
.filter(({ id, kinds }) => {
const kind = kindById.get(id);
const kind = sourceById.get(id)?.kind;
return kind === undefined || !kinds.includes(kind);
})
.map(({ id }) => id),
);
if (invalid.size === 0) return null;
return `The following filter source IDs do not exist or are not a kind of source the filter supports: ${[...invalid].join(', ')}`;
if (invalid.size > 0) {
return `The following filter source IDs do not exist or are not a kind of source the filter supports: ${[...invalid].join(', ')}`;
}

// The query runs on the filter's connection, so its source must be there too.
const mismatched = new Set(
newRefs
.filter(
({ id, connection }) =>
connection !== undefined &&
sourceById.get(id)?.connection.toString() !== connection,
)
.map(({ id }) => id),
);
if (mismatched.size === 0) return null;
return `The following filter source IDs are not on the filter's connection: ${[...mismatched].join(', ')}`;
}

/**
Expand Down
Loading
Loading