Thread-safe mode with dj.Instance

[dimitri-yatsenko]

Summary

• Add dj.Instance(host, user, password) for isolated config + connection pairs
• Add thread-safe mode (DJ_THREAD_SAFE=true) that disables global singletons and forces explicit dj.Instance() usage
• Thread config through codec and hash_registry chains so all runtime config reads go through the connection, not global state
• Refactor dj.Schema from a function wrapper into a proper class subclassing _Schema, so isinstance, hasattr, etc. work naturally
• Add adapter.split_full_table_name() — inverse of quote_identifier() — for backend-portable table name parsing
• Move dependency graph queries (load_primary_keys_sql, load_foreign_keys_sql) into adapter methods
• Remove unsupported schema.code()/save() methods (untested, MySQL-specific, no callers)

Usage

# Isolated instance (new)
inst = dj.Instance(host="localhost", user="root", password="secret")
schema = inst.Schema("my_schema")

# Global pattern (unchanged)
schema = dj.Schema("my_schema")

# Force instance usage in concurrent apps
export DJ_THREAD_SAFE=true

Use cases

• Web servers: one Instance per request/tenant
• Testing: isolated databases per test
• Multi-database: connect to production and staging simultaneously
• Parallel pipelines: independent connections per worker

Design spec

See docs/design/thread-safe-mode.md for the full specification including global state audit.

Test plan

• Unit tests for Instance, _ConfigProxy, thread-safe guards (tests/unit/test_thread_safe.py)
• Lint passes (ruff-format clean, only pre-existing mypy errors)
• Integration tests pass (semantic matching, jobs, GC, schema)

🤖 Generated with Claude Code

[mweitzel]

Looks really good. A few issues:

• hardcoded mysql always used in schema.py in list_schemas
• warnings for missing global config even when DJ_THREAD_SAFE is true

Regarding this code section: https://github.com/datajoint/datajoint-python/blob/feature/thread-safe-mode/src/datajoint/dependencies.py#L171-L252

Since you're forking on "adapter" ..instead of this being a globally aware "only 2 adapters, mysql and postgres", could this call a method on a local dependency? Like a backend aware function. Similar to the adapter.list_tables_sql function, but instead a adapter.load_table_keys/schema/etc or the like.

[dimitri-yatsenko]

@mweitzel Thanks for the thorough review. Here's where we stand:

Already fixed (commit da43ed37):

• schemas.py — list_schemas(), list_tables(), and schema_exists() all route through adapter methods now.

Fixed in dca80526 (just pushed):

• dependencies.py:171-252 — Extracted the backend fork into adapter.load_primary_keys_sql() and adapter.load_foreign_keys_sql() methods. The dependency graph loader now calls through the adapter like everything else.

settings.py warning — _create_config() is also called by Instance() to get fresh defaults from env vars, not just at import time. The warning helps users who forgot their config file. We'll leave it for now but can revisit if it becomes noisy in practice.

All unit tests pass (243/243). Let us know if there's anything else.

[mweitzel]

Some string manipulation ins schemas.py remains mysql specific. Regardless of the adapter used, this is incompatible with postgres.

datajoint-python/src/datajoint/schemas.py

Line 611 in 34f744e

for d, t in (table_name.replace("`", "").split(".") for table_name in self.connection.dependencies.topo_sort())

[dimitri-yatsenko]

@mweitzel Good catch. Fixed in 54887815 — added adapter.split_full_table_name() as the inverse of quote_identifier(). MySQL implementation strips backticks, PostgreSQL strips double quotes.

Updated call sites:

• schemas.py:552 — code() method
• schemas.py:611 — list_tables() topological ordering
• declare.py:298 — foreign key reference table name parsing
• table.py:236 — lineage parent table name parsing

All 243 unit tests pass.