Skip to content

feat(db): put all tpk streams under a dedicated tpk database (#58) - #59

Merged
gangtao merged 1 commit into
mainfrom
feat/tpk-database
Aug 17, 2026
Merged

feat(db): put all tpk streams under a dedicated tpk database (#58)#59
gangtao merged 1 commit into
mainfrom
feat/tpk-database

Conversation

@gangtao

@gangtao gangtao commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Closes #58.

What

All tpk streams now live under a dedicated database — tpk by default — instead of the server's default database. tpk creates it on startup and qualifies every stream as tpk.<name>.

Motivation: keep tpk's ~8 streams out of default, make them grantable/droppable as a unit, and avoid name collisions when tpk shares a Timeplus instance — most relevant for the new k8s app-only mode (#57) that connects to an already-running Timeplus Enterprise.

How

  • config: database() reader — TIMEPLUS_DATABASE env > [db].database file > tpk default (validated as an identifier), consistent with the feat(config): unify env/file config with env>file>default precedence (#54) #55 precedence system.
  • db.py: qualified(name, prefix) → <database>.<prefix><name>. ensure_schema runs CREATE DATABASE IF NOT EXISTS <db> first; all DDL and drop_schema qualify their names. get_client() deliberately stays unscoped (connects to the always-present default DB) — I verified that connecting with database=<x> eagerly fails if <x> doesn't exist yet, so scoping the session would create a bootstrap chicken-and-egg.
  • All call sites route through db.qualified(): tools, corpus, auth, audit, ingest, api, cli (status), transfer. TPK_STREAM_PREFIX still namespaces within the database.
  • Works on both backends (timeplusd MUTABLE STREAM + proton versioned_kv) — CREATE DATABASE and qualified names apply to both.

Migration (note in README)

Existing deployments' streams stay in default and are not moved automatically — after upgrade tpk reads from tpk (empty) until you re-ingest (tpk ingest) or carry data over with tpk export → tpk import. Setting TIMEPLUS_DATABASE=default keeps the old location.

Verification

  • Full unit suite green (107 passed; +3 new: qualified(), database() precedence, updated backend DDL assertions).
  • End-to-end on live proton: ensure_schema creates all 8 streams under tpk, and upsert/get/delete/list/drop_schema all work through the real code path (test-prefixed, cleaned up, existing data untouched).

🤖 Generated with Claude Code

@gangtao @claude
feat(db): put all tpk streams under a dedicated tpk database (#58)
09b0e61 
Every tpk stream (kg_nodes/kg_edges/kg_repos/kg_ingest_log/kg_users/
kg_roles/kg_sessions/chat_audit_log) is now created and queried under a
dedicated database — `tpk` by default — instead of the server's `default`
database. This keeps tpk's objects out of `default`, lets them be granted or
dropped as a unit, and avoids collisions when tpk shares a Timeplus instance
(e.g. the k8s app-only mode against an existing Timeplus Enterprise).

- config: add `database()` (env TIMEPLUS_DATABASE > [db].database > "tpk"),
  validated as an identifier.
- db: add `qualified(name, prefix)` -> `<database>.<prefix><name>`;
  ensure_schema runs `CREATE DATABASE IF NOT EXISTS` first; all DDL and
  drop_schema qualify names. get_client stays unscoped (connects to the
  always-present default DB) to avoid a bootstrap chicken-and-egg — connecting
  with database= fails if it doesn't exist yet.
- Route every stream identifier through db.qualified(): tools, corpus, auth,
  audit, ingest, api, cli (status), transfer. TPK_STREAM_PREFIX still
  namespaces within the database.
- tests: qualified()/database() precedence + updated backend DDL assertions.
  Verified end-to-end on live proton (create/upsert/read/delete/drop).
- docs: README Configuration matrix + a `tpk` database section with the
  migration note (existing `default` data isn't moved — re-ingest or
  export/import, or set TIMEPLUS_DATABASE=default); repos.toml/.env.example.

Migration: existing deployments' streams stay in `default`; after upgrade tpk
reads from `tpk` (empty) until re-ingested or export/import'd.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb
@gangtao
gangtao merged commit 1814024 into main Aug 17, 2026
2 checks passed
@gangtao gangtao mentioned this pull request Aug 17, 2026
Merged
gangtao added a commit that referenced this pull request Aug 17, 2026
@gangtao @claude
docs(k8s): app-only — account for the dedicated tpk database (#58/#59…
534f3d6 
…) (#61)

The k8s manifests (#56/#57) predate the change that moved all tpk streams
under a dedicated `tpk` database (#59). For app-only — which connects to an
externally-managed Timeplus Enterprise — the app now runs
`CREATE DATABASE IF NOT EXISTS tpk` on startup, so the pre-existing DB user
needs CREATE DATABASE (first run) plus read/write on that database. A
restricted user would otherwise fail at startup/ingest.

- app-only.yaml: header now states the app creates its own database on the
  existing server; add a `TIMEPLUS_DATABASE` env (default `tpk`) with the
  privilege note and the pre-create fallback for restricted users.
- deploy/k8s/README.md: add a "tpk database" note to the App-only section
  (CREATE DATABASE grant + pre-create SQL), and a TIMEPLUS_DATABASE row to the
  Configuration table. Notes that enterprise/allinone are unaffected (they
  provision a full-privilege tpk user).


Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
gangtao added a commit that referenced this pull request Aug 18, 2026
@gangtao @claude
feat(chat): per-user daily token budget for non-admin users (#62) (#63)
07f4158 
* docs(k8s): app-only — account for the dedicated `tpk` database (#58/#59)

The k8s manifests (#56/#57) predate the change that moved all tpk streams
under a dedicated `tpk` database (#59). For app-only — which connects to an
externally-managed Timeplus Enterprise — the app now runs
`CREATE DATABASE IF NOT EXISTS tpk` on startup, so the pre-existing DB user
needs CREATE DATABASE (first run) plus read/write on that database. A
restricted user would otherwise fail at startup/ingest.

- app-only.yaml: header now states the app creates its own database on the
  existing server; add a `TIMEPLUS_DATABASE` env (default `tpk`) with the
  privilege note and the pre-create fallback for restricted users.
- deploy/k8s/README.md: add a "tpk database" note to the App-only section
  (CREATE DATABASE grant + pre-create SQL), and a TIMEPLUS_DATABASE row to the
  Configuration table. Notes that enterprise/allinone are unaffected (they
  provision a full-privilege tpk user).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* feat(chat): per-user daily token budget for non-admin users (#62)

Operators can cap daily LLM token consumption per non-admin user so a single
user can't run up unbounded cost against the shared agent. admin (the reserved
super-role) is exempt. (Token budget only; the optional question cap from #62 is
deferred.)

- usage.py: append-only `chat_usage` stream (ts, username, tokens) written per
  turn independent of TPK_CHAT_AUDIT, plus DbUsage (used_today/record). Reads
  fail OPEN (unreachable store never blocks a user); writes are best-effort.
- db: create chat_usage in ensure_schema (+ drop_schema); add
  kg_roles.daily_token_limit (uint32) with the same _add_column_if_missing
  lazy migration as capabilities (existing roles -> 0 = unlimited).
- auth.Role + kg_roles carry daily_token_limit; api /roles create/update/list
  carry it (validated >= 0).
- config.daily_token_limit(): global fallback (env TPK_DAILY_TOKEN_LIMIT >
  [server].daily_token_limit > 0) applied when a role sets no limit of its own.
- server /chat: sum usage_metadata tokens across the turn's model calls; before
  running the agent, block a non-admin who is at/over budget with HTTP 429 +
  reset time (next UTC midnight). Enforcement is next-turn (a turn's cost is
  only known once it runs); the crossing turn completes. Metering + enforcement
  only active when a usage store is present (production), so unit tests that
  inject a fake agent are untouched.
- web: role editor gains a "Daily token budget per user" field (add + edit);
  chat surfaces the 429 with the server's friendly reset message.
- tests: usage helpers + fail-open, config precedence, enforcement (over/under/
  admin-bypass/no-store), and backend round-trips (role limit, daily count).
  Verified end-to-end on live timeplusd.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* feat(chat): show the user their daily token budget (used / left / reset)

Surface the budget proactively, not only on a 429. Adds GET /chat/usage
(chat-cap gated) returning {limited, used, limit, remaining, reset} for the
current user — limited:false for admins, unlimited roles, or when no budget is
enforced, so the UI shows no indicator. Extracts the effective-limit rule into
usage.effective_daily_limit (role's own limit, else the global fallback), reused
by /chat enforcement and the new endpoint.

Chat UI fetches /chat/usage on load and after each turn (a turn spends tokens;
the append-only read lags slightly so it refreshes again shortly after), and
shows "used / limit", tokens left, and the reset time in the header and empty
state for users who have a budget.

Tests: /chat/usage for a limited user, and unlimited for admin / no-store.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* fix(compose): pass TPK_DAILY_TOKEN_LIMIT through to the app container

The global daily-token-budget fallback was documented in .env.example but not
wired into the compose env, so setting it in .env had no effect. Pass it through
in both the DB+App and all-in-one files (a role's own daily_token_limit still
overrides it).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* feat(chat): per-user daily token budget override + admin UI (#62)

Adds a per-user daily token limit that overrides the role limit, so admins can
tune budgets for individual users, not just per role. Effective limit precedence
is now: user override > role limit > global fallback (0 = unlimited at each
level).

- db/auth: kg_users.daily_token_limit (uint32) with _add_column_if_missing
  migration (existing users -> 0 = inherit); User field + parse/upsert.
- usage.effective_daily_limit(user_limit, role, global) implements the
  precedence; /chat enforcement and /chat/usage both pass the user's override.
- api: AddUser/UpdateUser carry daily_token_limit (validated >= 0); /users list
  returns it; update leaves it unchanged when omitted.
- web Users tab: a "Daily tokens" column with a per-user number input (admins
  show "unlimited"), and a budget field in the Add-user modal.
- tests: precedence unit test, user-override enforcement + /chat/usage reflect
  it. Verified E2E on live timeplusd incl. the kg_users column migration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* feat(chat): default the global daily token budget to 500000 (#62)

Change the built-in fallback from 0 (unlimited) to 500000 tokens/user/day, so
non-admin users are capped by default. A per-user or role daily_token_limit
still overrides it, and setting TPK_DAILY_TOKEN_LIMIT / [server].daily_token_limit
to 0 restores unlimited-by-default.

Updates the README matrix, repos.toml / repos.container.toml comments,
.env.example, and the config precedence test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

* fix(chat): correct 0=inherit labeling + admin-only token budget (review #63)

Code review of #63 found two issues:

1. Role/user daily_token_limit of 0 means "inherit the next level down", not
   "unlimited" (only a 0 that reaches the global fallback is unlimited). The
   role-level comments, API error, effective_daily_limit docstring, and role UI
   labels wrongly said "0 = unlimited" — which, now that the global default is
   500000, would mislead an admin into thinking a role is uncapped when it's
   capped at 500000. Relabel role level to "0 = inherit global" everywhere;
   the /chat/usage docstring's "degrades to unlimited" corrected to "global
   default". (The per-user field was already labeled correctly.)

2. daily_token_limit bypassed the bounded-delegation guard: a non-admin
   users:manage delegate could set any manageable user's/role's budget to an
   arbitrary value, handing out unlimited LLM spend. Add _guard_token_limit —
   only an admin may change a budget; a delegate may still manage users/roles
   as long as it leaves the budget unchanged (403 otherwise).

Tests: role-0-inherits-global precedence; manager-cannot-set-budget (verified
against live timeplusd). Full suite green; web build + sanitize pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XPzYZXxyTdj5G25KoujpHb

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Put all tpk resources (streams, mutable streams) under a dedicated tpk database

1 participant

@gangtao