fix: fail fast HNSW index when embedding dimension exceeds 2000

[slayoffer]

Problem

ensure_vector_extension() and ensure_embedding_dimension() unconditionally create HNSW indexes at startup. pgvector's HNSW implementation has a hard limit of 2000 dimensions — any column with more dimensions causes a ProgramLimitExceeded error:

psycopg2.errors.ProgramLimitExceeded: column cannot have more than 2000 dimensions for hnsw index

[SQL: CREATE INDEX IF NOT EXISTS idx_memory_units_embedding
      ON public.memory_units
      USING hnsw (embedding vector_cosine_ops)
      WITH (m = 16, ef_construction = 64)]

This crashes the application at startup for anyone using high-dimensional embeddings like OpenAI text-embedding-3-large (3072 dimensions) with standard pgvector (not VChord).

This was introduced by #350 and #355 which added VChord support and the ensure_vector_extension() startup check.

Fix

Before creating an HNSW index, check the embedding column dimension. If it exceeds 2000, log a warning and skip index creation — falling back to sequential scan, which works correctly at any dimension.

Applied in two places:

1. ensure_vector_extension() — queries pg_attribute.atttypmod per table to get the embedding dimension
2. ensure_embedding_dimension() — uses the already-available required_dimension parameter

Context

• pgvector 0.8.x HNSW limit: 2000 dimensions (pgvector docs)
• Cloud SQL for PostgreSQL ships pgvector 0.8.1 which has this limit
• VChord (vchordrq) does not have this limitation — only the pgvector HNSW path is affected
• Sequential scan performance is acceptable for small-to-medium datasets; users needing indexed performance at >2000 dims should use VChord

Test plan

• Verified on Cloud SQL pgvector 0.8.1 with 3072-dim OpenAI embeddings — startup succeeds
• Log output shows: Skipping HNSW index on memory_units: embedding dimension 3072 exceeds pgvector HNSW limit of 2000 dims
• Verify HNSW index is still created for ≤2000-dim embeddings (e.g., local sentence-transformers 384-dim)

🤖 Generated with Claude Code

[nicoloboschi]

we should raise the exception if pgvector can't handle that model

[slayoffer]

Updated: now raises RuntimeError instead of silently skipping HNSW index creation when dimensions exceed 2000. Both code paths (ensure_embedding_dimension and ensure_vector_extension) raise with an actionable message suggesting pgvectorscale/DiskANN as alternatives.

[slayoffer]

Done — updated both ensure_embedding_dimension and ensure_vector_extension to raise RuntimeError when embedding dimensions exceed 2000. The error message suggests switching to pgvectorscale/DiskANN which now has native support after #378/#381.