k8s kustomize, Redis caching, background/indexing scaling, Azure Blob, retention + connector reliability#46

Merged

rajivml merged 102 commits into

feature/darwinfrom

feature/redis-assistants-ux-background-scaling

Jun 3, 2026

Commits on Jun 3, 2026

docs: add Redis caching & scaling plan

rajivml
and
claude
committed
P1: Redis foundation + read-through KV cache

rajivml
and
claude
committed
P2: per-user request rate limiter on chat/query endpoints

rajivml
and
claude
committed
Persona list cache with explicit write-through invalidation

rajivml
and
claude
committed
Manage Assistants page UX overhaul

rajivml
and
claude
committed
Assistant Gallery page UX overhaul

rajivml
and
claude
committed
Add backend/scripts/seed_assistants.py for local UX testing

rajivml
and
claude
committed
Assistants UX polish: toggle highlight + gallery declutter

rajivml
and
claude
committed
Remove tools chip from Manage Assistants page

rajivml
and
claude
committed
Parameterize gallery grid column count (default 3)

rajivml
and
claude
committed
Show document-set names on assistant cards (was: count only)

rajivml
and
claude
committed
Gallery: user-controllable column count (segmented control, persists)

rajivml
and
claude
committed
Gallery: column picker as dropdown to match Sort

rajivml
and
claude
committed
Scale indexing via remote Dask scheduler topology

rajivml
and
claude
committed
docs: add MIGRATION.md covering Redis / bg-scaling / UX

rajivml
and
claude
committed
darwin-kubernetes: port split-background manifests + lock convention in AGENTS.md

rajivml
and
claude
committed
docs(MIGRATION.md): reflect the darwin-kubernetes port

rajivml
and
claude
committed
docs(MIGRATION.md): update base reference after add-claude merge

rajivml
and
claude
committed
Introduce k8s/ — kustomize-based manifests replacing darwin-kubernetes/

rajivml
and
claude
committed
k8s: opt-in Redis via kustomize component (local includes, prod doesn't)

rajivml
and
claude
committed
k8s: prod now uses in-cluster Redis (was: managed Redis); bump memory to 512MB

rajivml
and
claude
committed
k8s: move redis from optional component to base

rajivml
and
claude
committed
docs(k8s/README): drop migration narrative + darwin-kubernetes references

rajivml
and
claude
committed
docs(k8s/README): add instructions for applying optional/ manifests

rajivml
and
claude
committed
k8s: make optional manifests parameterized like base (component + logical images)

rajivml
and
claude
committed
k8s: fully parameterize background-scaling component (replicas, env, env-neutral)

rajivml
and
claude
committed
docs(k8s/README): explicit apply/preview/verify commands for background-scaling

rajivml
and
claude
committed
k8s: right-size background-scaling for a few-hundred-user deployment

rajivml
and
claude
committed
k8s: add slack-listener to background-scaling component

rajivml
and
claude
committed
k8s: consolidate background-scaling 6 deployments → 4

rajivml
and
claude
committed
k8s: disable MULTILINGUAL_QUERY_EXPANSION in prod overlay

rajivml
and
claude
committed
k8s: pin Vespa to 8.600.35 (was :latest → caused prod outage)

rajivml
and
claude
committed
k8s: add readiness probes to Service-backed Vespa nodes

rajivml
and
claude
committed
docs(AGENTS.md): add Critical fact §10 — never :latest for Vespa, pin the version

rajivml
and
claude
committed
k8s: add guarded-apply.sh — block unsafe Vespa version jumps at apply time

rajivml
and
claude
committed
k8s: add KEDA indexing-autoscale component (opt-in) for dask-worker

rajivml
and
claude
committed
k8s: add KEDA operator install manifest (optional/keda, own namespace)

rajivml
and
claude
committed
docs(k8s/README): document mandatory pod restarts after a config change

rajivml
and
claude
committed
k8s: add startup + readiness probes to api-server (own /health, not deps)

rajivml
and
claude
committed
docs(k8s/README): add "Verify Redis caching is working" runbook

rajivml
and
claude
committed
perf: fix N+1 in basic indexing-status (chat page load + folder create)

rajivml
and
claude
committed
perf(chat): optimistic folder create — no full refetch

rajivml
and
claude
committed
perf(chat): Redis-cache the connector indexing-status read

rajivml
and
claude
committed
k8s(vespa): group into base/vespa/ + ordered, health-gated upgrade script

rajivml
and
claude
committed
fix(chat): createFolder returns the new folder id (was undefined)

rajivml
and
claude
committed
polish(chat): tidy chat-row drag (compact ghost + no browser split-view)

rajivml
and
claude
committed
feat(chat): pending spinner on "Manage Assistants" navigation

rajivml
and
claude
committed
perf(db): Celery broker on Redis + env-driven Postgres pool sizing

rajivml
and
claude
committed
style(config): black-format app_configs.py (line-length 130)

rajivml
and
claude
committed
style: black-format Redis persona cache + rate-limit middleware

rajivml
and
claude
committed
perf(chat): per-user Redis cache for the document-set list

rajivml
and
claude
committed
refactor(chat): make doc-set cache global + MIT-scoped (no EE dependency)

rajivml
and
claude
committed
feat(analytics): chat adoption curve + per-user breakdown; chat retention 90d

rajivml
and
claude
committed
feat(analytics): durable per-user daily stats (leaderboard survives retention)

rajivml
and
claude
committed
feat(analytics): most-used assistants + approximate datasets-in-use

rajivml
and
claude
committed
refactor(analytics): split page into Overview / User Activity tabs

rajivml
and
claude
committed
k8s(background-scaling): drop vestigial PVCs from split deployments + clarify README

rajivml
and
claude
committed
fix(indexing): stream large connector files instead of loading into RAM

rajivml
and
claude
committed
feat(file-store): Azure Blob backend (bytes off Postgres), opt-in via FILE_STORE_TYPE

rajivml
and
claude
committed
feat(file-store): lobj→Blob migration script + k8s wiring + README

rajivml
and
claude
committed
test(file-store): big-file round-trip integration test for Azure Blob

rajivml
and
claude
committed
fix(file-store): clear error when azure-storage-blob is missing

rajivml
and
claude
committed
fix(chat): block send while a file is still uploading

rajivml
and
claude
committed
feat(chat): direct-to-Blob upload backend (SAS mint + confirm)

rajivml
and
claude
committed
feat(chat): direct-to-Blob upload from the browser + progress bar

rajivml
and
claude
committed
docs(file-store): direct-to-Blob chat upload flow + Storage CORS

rajivml
and
claude
committed
feat(chat): enforce chat-upload size + token limits

rajivml
and
claude
committed
feat(chat): FE size pre-check reads the configured limit (no hardcoded dup)

rajivml
and
claude
committed
k8s: unmount vestigial PVCs from api-server + background (keep the PVCs)

rajivml
and
claude
committed
fix(file-store): robust account name/key resolution for upload SAS

rajivml
and
claude
committed
fix(file-store): support SAS-based connection strings for direct upload

rajivml
and
claude
committed
refactor(file-store): direct upload requires account-key (drop broad-SAS reuse)

rajivml
and
claude
committed
fix(file-store): clear diagnostic when the connection string is malformed

rajivml
and
claude
committed
k8s: roll out split-background scaling to prod

rajivml
and
claude
committed
docs(k8s): direct-to-Blob upload, CORS, chat limits, account-key conn string

rajivml
and
claude
committed
k8s: cut prod file store over to Azure Blob

rajivml
and
claude
committed
k8s: bump dask-worker resources (req 1cpu/4Gi, limit 8Gi)

rajivml
and
claude
committed
k8s: remove legacy deployment/helm + deployment/kubernetes manifests

rajivml
and
claude
committed
k8s: remove legacy darwin-kubernetes manifests

rajivml
and
claude
committed
fix(k8s): right-size background-scaling memory (OOMKilled after split)

rajivml
and
claude
committed
k8s: decouple Vespa from app overlays (separate prod-vespa/local-vespa)

rajivml
and
claude
committed
fix(indexing): LIMIT 1 in get_last_attempt — was materializing full history

rajivml
and
claude
committed
k8s: enable index_attempt retention (30d, keep last 20 per cc-pair)

rajivml
and
claude
committed
k8s: dask-worker waits for scheduler before starting (env-agnostic)

rajivml
and
claude
committed
k8s: right-size indexer-scheduler to 512Mi/2Gi (LIMIT-1 fix verified)

rajivml
and
claude
committed
k8s(prod): bump backend vha-139->vha-140, web vha-71->vha-72

rajivml
and
claude
committed
perf(db): eliminate SQL over-materialization found in audit (no behavior change)

rajivml
and
claude
committed
feat(cc-pair): server-side pagination for index-attempt history (FE + BE)

rajivml
and
claude
committed
k8s(prod): bump backend vha-140->vha-141, web vha-72->vha-73

rajivml
and
claude
committed
perf(indexing): skip re-indexing unchanged docs via content hash + parallel chunk deletes

rajivml
and
claude
committed
obs(indexing): log per-batch count of docs skipped as unchanged

rajivml
and
claude
committed
fix(k8s): celery-worker --concurrency instead of --autoscale (threads pool)

rajivml
and
claude
committed
perf(web-connector): faster + far more resilient crawling

rajivml
and
claude
committed
fix(jira): place poll time-window before ORDER BY in JQL

rajivml
and
claude
committed
feat(jira): per-issue error tolerance, ID-based pruning, richer metadata

rajivml
and
claude
committed
k8s(prod): bump backend vha-141->vha-144, web vha-73->vha-75

rajivml
and
claude
committed
style: apply pre-commit (black / reorder-imports / prettier) to changed files

rajivml
and
claude
committed
style: apply pre-commit across branch files (black/reorder/autoflake/prettier)

rajivml
and
claude
committed
fix(slack): register InputType.PRUNE so Slack connectors can be pruned

rajivml
and
claude
committed
k8s(prod): bump backend vha-144->vha-146, web vha-75->vha-76

rajivml
and
claude
committed
k8s(prod): bump backend vha-146->vha-147, web vha-76->vha-77

rajivml
and
claude
committed
k8s: add build-deploy.sh (build/push/deploy/verify for backend+web)

rajivml
and
claude
committed