Skip to content

Ingest SQL files: add graphify's sql extra (tree-sitter-sql) so .sql is parsed, not skipped #71

Description

@gangtao

Summary

The knowledge graph currently skips .sql files during ingest. For a
streaming-SQL company, a knowledge base that ignores SQL is a conspicuous gap —
the corpus (proton, docs, examples) is full of SQL that never makes it into the
graph, so the agent can't ground answers in it.

Root cause

graphify parses SQL via tree-sitter-sql, which it ships behind an optional
sql extra
— not part of its base install. tpk depends on graphify with only
the LLM extras:

# pyproject.toml:13
"graphifyy[claude,openai]>=0.9.38",

Since [sql] isn't installed, tree-sitter-sql is absent, and graphify extract skips .sql files (emitting a "skipping / unsupported" warning in its
output during the extract phase — visible with tpk ingest --verbose).

Verified against the installed graphifyy 0.9.38 metadata: it declares
tree-sitter-sql; extra == "sql" (alongside an all extra). We install
claude + openai only.

Impact

  • Timeplus is a streaming-SQL product; SQL is arguably the most important file
    type for this corpus, and it's the one being dropped.
  • The skip is silent unless you run --verbose — the ingest reports success and
    node/edge counts that simply omit SQL, so it's easy to miss.

Proposed fix

Add graphify's sql extra to the dependency:

"graphifyy[claude,openai,sql]>=0.9.38",

Then:

  • uv lock / uv sync to pull tree-sitter-sql.
  • Rebuild + publish the Docker images (deps are baked into deploy/docker/Dockerfile — both the app and all-in-one targets), since a stale image would still skip SQL.
  • Re-ingest so existing corpora pick up their SQL.

tree-sitter-sql is an AST parser, so this should work in the default
code-only extraction mode (no LLM key needed) — SQL files become
file/function-style nodes like other code. Worth confirming graphify emits
nodes for .sql once the extra is present, rather than needing semantic mode.

Acceptance criteria

  • .sql files are parsed during ingest (no "skipping SQL" warning), producing
    nodes/edges in the graph.
  • The Docker images (app + all-in-one) include tree-sitter-sql, so a
    fresh docker run timeplus/tpk:latest + tpk ingest indexes SQL.
  • A re-ingest of a SQL-containing repo shows a higher node/edge count than
    before, and the agent can cite .sql sources.
  • (Nice to have) a note in the README / ingest docs that SQL is indexed.

Notes

  • Consider whether other Timeplus-relevant file types are behind graphify extras
    too (e.g. terraform/HCL via tree-sitter-hcl) and worth adding while we're
    here — but SQL is the priority.
  • If tree-sitter-sql proves heavy or platform-fragile in the image build,
    fall back to documenting it as an opt-in extra rather than skipping silently.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions