fix: backtick-escape column names in filter queries

[pulpdrew]

Summary

Fixes [HDX-4585]: Column names containing special characters (e.g. hyphens like log-status, az-id, flow-direction) were interpolated as bare identifiers in filter/key-value queries, causing ClickHouse to misparse them. For example, log-status was interpreted as log minus status.

Expressions are now backtick-quoted using SqlString.escapeId prior to being queried as filter keys:

1. LogStatus --> LogStatus
2. log-status --> `log-status`
3. log.status --> `log`.`status`
4. ResourceAttributes['log.status'] --> `ResourceAttributes`['log.status']

This has been fixed both when querying for filter key/values and when generating query conditions based on selected filters.

Testing

I've added a fairly comprehensive set of E2E tests testing the following cases

• Search page filters
  • Filter values load for the following columns: ServiceName, ResourceAttributes['key.subKey.subSubKey'], __hdx_materialized_k8s.cluster.name, and service-name
  • Filter values can be included and excluded for each of those columns
• Dashboard filters
  • Filter values load for the following columns: ServiceName, ResourceAttributes['key.subKey.subSubKey'], __hdx_materialized_k8s.cluster.name, ResourceAttributesJSON.key.subKey.subSubKey and service-name
  • Filter values can be included and excluded for each of those columns
  • Additionally, all of the above filters can be defined by the user using a non-backticked OR pre-backticked expression (i.e. the user types `service-name` into the filter expression input, or just service-name).

Linear Issue: Closes HDX-4585

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jun 18, 2026 9:44pm
hyperdx-storybook	Ready	Preview, Comment	Jun 18, 2026 9:44pm

[changeset-bot]

🦋 Changeset detected

Latest commit: 76b889d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Patch
@hyperdx/app	Patch
@hyperdx/api	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[greptile-apps]

Greptile Summary

This PR fixes ClickHouse identifier escaping for filter keys containing special characters (hyphens, dots-in-column-names), which were previously interpolated verbatim, causing ClickHouse to misparse them as arithmetic expressions. The fix introduces a new toQuotedClickHouseKeyExpression helper and escapeFilterKeysForSql function that conditionally backtick-quote column identifiers at SQL-generation time while keeping the persisted filter state (URL/localStorage) free of quoting.

• A new escapeFilterKeysForSql function in searchFilters.tsx rewrites IN, NOT IN, and BETWEEN filter conditions to backtick-quote non-standard identifiers before they reach ClickHouse; quoting is schema-aware via a knownColumns set that prevents dotted materialized column names from being mis-parsed as Map/JSON sub-path access.
• useDashboardFilters.getFilterQueriesForSource was fixed to apply { stringifyKeys: true }, making it consistent with queriesForExistingFilters (previously the two code paths diverged, causing scoped filter queries to omit the toString() wrapping).
• The "Load more" filter values path in DBSearchPageFilters.tsx is updated to escape keys and correctly map SQL-quoted keys back to UI-display keys.

Confidence Score: 5/5

Safe to merge. The core escaping logic is correct, well-tested (unit + E2E), and the design cleanly separates the persisted form (unquoted URL state) from the SQL-generation form (quoted at query time).

The changes are tightly scoped: new quoting helpers are covered by unit tests, the dashboard getFilterQueriesForSource bug is a one-line fix that aligns two previously-divergent code paths, and the E2E suite validates the full round-trip for all key shapes including hyphens, dots-in-column-names, and Map bracket access. The one gap identified — toString(...) keys falling through to incorrect segment quoting — cannot be reached through any current call path and is a hardening suggestion only.

packages/app/src/components/DBSearchPageFilters/utils.ts — the toQuotedClickHouseKeyExpression function has a narrow toString(...) edge case worth closing; everything else in the file is solid.

Important Files Changed

Filename	Overview
packages/app/src/components/DBSearchPageFilters/utils.ts	Adds quoteIdentifierIfNeeded and toQuotedClickHouseKeyExpression helpers; handles bracket-form, backtick-form, plain column, and dotted JSON path keys correctly, but has a narrow gap for toString(...) expressions.
packages/app/src/searchFilters.tsx	New escapeFilterKeysForSql + rewriteFilterConditionKey correctly rewrites IN/NOT IN/BETWEEN conditions at query-generation time; parseQuery round-trip and idempotency look sound.
packages/app/src/hooks/useDashboardFilters.tsx	One-line fix: adds { stringifyKeys: true } to getFilterQueriesForSource so it is consistent with the queriesForExistingFilters path above it.
packages/app/src/DBSearchPage.tsx	Wires column metadata loading and escapeFilterKeysForSql into useSearchedConfigToChartConfig; chart config now waits for columns to load, which may add a brief additional loading state.
packages/app/src/components/DBSearchPageFilters.tsx	Adds knownColumns, escapedKeysToFetch, and sqlKeyToUiKey to correctly escape facet keys before querying and map results back to UI keys; also applies escaping in the Load more path.
packages/common-utils/src/tests/filters.test.ts	Added test section explicitly verifying verbatim (unescaped) key emission from filtersToQuery, documenting the new design contract cleanly.
packages/app/tests/e2e/features/filter-key-edge-cases.spec.ts	New E2E test covering the full spectrum of edge-case filter keys (hyphens, dots, map access, JSON paths) on both search and dashboard pages; clearly documents limitations (raw dashboard expressions expected to fail).

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User selects filter value in UI] --> B[setFilterValue stores unquoted key e.g. service-name]
    B --> C[filtersToQuery emits verbatim: service-name IN 'foo']
    C --> D[URL / persisted Filter array stores unquoted key]
    D --> E{Context}
    E -->|Search page| F[escapeFilterKeysForSql rewriteFilterConditionKey]
    F --> G[toQuotedClickHouseKeyExpression with knownColumns set]
    G --> H{Is key a known flat column?}
    H -->|Yes e.g. __hdx_materialized_k8s.cluster.name| I[Quote whole name as single backtick identifier]
    H -->|No - map bracket form| J[Quote only the root column name]
    H -->|No - plain needs quoting| K[Quote each dot segment]
    I --> L[ClickHouse query with escaped keys]
    J --> L
    K --> L
    E -->|Dashboard page| M[filtersToQuery with stringifyKeys=true]
    M --> N[toString wrapped key e.g. toString service-name]
    N --> O[ClickHouse query - user must pre-quote special identifiers]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[User selects filter value in UI] --> B[setFilterValue stores unquoted key e.g. service-name]
    B --> C[filtersToQuery emits verbatim: service-name IN 'foo']
    C --> D[URL / persisted Filter array stores unquoted key]
    D --> E{Context}
    E -->|Search page| F[escapeFilterKeysForSql rewriteFilterConditionKey]
    F --> G[toQuotedClickHouseKeyExpression with knownColumns set]
    G --> H{Is key a known flat column?}
    H -->|Yes e.g. __hdx_materialized_k8s.cluster.name| I[Quote whole name as single backtick identifier]
    H -->|No - map bracket form| J[Quote only the root column name]
    H -->|No - plain needs quoting| K[Quote each dot segment]
    I --> L[ClickHouse query with escaped keys]
    J --> L
    K --> L
    E -->|Dashboard page| M[filtersToQuery with stringifyKeys=true]
    M --> N[toString wrapped key e.g. toString service-name]
    N --> O[ClickHouse query - user must pre-quote special identifiers]

Loading

_{Reviews (9): Last reviewed commit: "fix: correct filter handling for columns..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

❌ 1 test failed • 208 passed • 3 skipped • 1483s

Status	Count
✅ Passed	208
❌ Failed	1
⚠️ Flaky	6
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →