[psaesha]

Pull Request

Description

Every tabular file is copied into a table in data_db, which previews, charts and the data API read. Three problems came from how that was done:

• Two resources could share one table. Names were built from the resource name and dataset title, up to 110 characters. Postgres keeps only the first 63 bytes of a name, so the unique part at the end was usually cut off. Two resources with long, similar names then used the same table, and re-uploading one replaced the other's data. The other resource's preview, charts and data API then showed the wrong numbers.
• Re-uploads dropped the old table before the new one was ready. For a few seconds readers saw empty or half-written data. If the new file failed, the old data was lost for good.
• Deleting a resource or dataset never removed its table, so the data database kept growing.

What changes:

• Each upload gets its own table, named from its own row id: rdt_<32 hex>, 36 characters. Names can't collide and don't change when something is renamed.
• A re-upload builds the new table completely, then switches the resource over in one database commit, and only then drops the old table. Readers see either the complete old data or the complete new data. If anything fails, the old data stays and the half-built table is dropped.
• The newest file wins: if two uploads of one resource overlap, the slower one checks before switching whether a newer file has been saved since, and backs off. The download and the served data never disagree.
• Deletes clean up. Deleting a resource or dataset, from the API or the admin, drops its tables once the delete is saved. A cancelled delete keeps them.
• A one-off command, migrate_data_table_names, moves existing tables:
  • It renames short old names in place.
  • It rebuilds long ones from each resource's own file, since they may already hold another resource's data.
  • It lists tables nothing uses. These are dropped only from a report someone reviewed.
  • It's a dry run by default. The runbook is in api/utils/data_indexing_architecture.md.

No schema change and no migration file: only the generated table name changes. Old and new names work side by side, and rolling back the code is safe.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality): the migration command
• Documentation update
• Test improvements

Changes Made

• api/utils/data_indexing.py: index_resource_data now builds the table, switches in one commit (resource row locked, stored file re-checked), confirms the switch and cleans up. It refuses to run inside an outer transaction. The old create_table_for_resource is removed.
• api/utils/data_tables.py, api/utils/data_table_names.py: the helpers for each step, and the naming rule.
• api/models/Resource.py: a new ResourceDataTable row is named rdt_<own id>.
• api/signals/data_table_signals.py, registered in api/signals/__init__.py: drops a row's table after its delete commits. It keeps an old table another row still shares.
• api/management/commands/update_dataindex.py: no longer wraps each resource in a transaction. One failing column used to silently roll the whole switch back.
• api/management/commands/migrate_data_table_names.py + api/utils/data_table_migration.py: the migration command.
• tests/data_index/: tests run against a real Postgres when TEST_DATA_DB_URL is set (tests/test_settings.py, pytest.ini marker data_db).
• .github/workflows/django-tests.yml: a postgres:16 service, plus REQUIRE_DATA_DB_TESTS=1, so CI fails instead of quietly skipping those tests.
• .github/workflows/sdk-tests.yml: pinned to ubuntu-24.04. ubuntu-latest (26.04) has no Python 3.9, which failed the SDK job on fix(data-api): read tables whose column names contain a percent sign #295.
• api/utils/data_indexing_architecture.md (new), with a link from docs/dataset_data_api.md.

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes. With a local Postgres 16: tests/data_index, tests/schema/test_resource_permissions.py and tests/test_data_indexing_filters.py give 133 passed. Every new test was seen failing first for the expected reason.
• Manual: the layer-5 journey tests/journeys/data_index/reupload-and-delete.py is written but not run yet. It needs a running backend and a token.

Test Instructions

1. Start a test Postgres and set the URL:

   docker run -d --name dataspace-test-data-db -e POSTGRES_PASSWORD=postgres -p 55432:5432 postgres:16
   export TEST_DATA_DB_URL=postgres://postgres:postgres@localhost:55432/postgres
   

2. Run pytest tests/data_index. Without the URL these tests are skipped; with REQUIRE_DATA_DB_TESTS=1 and no URL the run fails.
3. After deploying to dev:
   • re-upload a CSV while polling GET /api/resources/<id>/data/; every read is the complete old or the complete new file;
   • delete the resource; its table is gone.
4. Migration:
   • On a copy of production, run python manage.py migrate_data_table_names --report dry.json.
   • Review the report: counts, shared tables, rows needing a manual fix, unverified tables, orphans.
   • Then follow the runbook in the architecture doc.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code. Each phase was also reviewed separately, and the whole diff once more at the end.
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• black and isort --profile black on the changed files. mypy (with the Django plugin) has no errors in the changed api/ files.
• Database migrations are included (if applicable): none needed

Additional Notes

Known and left for later (marked TODO in the code and the architecture doc):

• A failed re-upload still reports success. This was already true: the upload mutations ignore the indexing result. Now the old data stays served while the download serves the new file. The direction, pending a product discussion, is to enforce the database's limits at upload (for example, column names within 255 characters) and to flag problem columns where people add column descriptions.
• Two column headers that differ only after 63 bytes become one column, so that file never gets a table. Hindi headers hit this at about 21 characters. This predates this PR.

Rollout: deploy, then run the migration command off-peak as described in the runbook: dry run, review, --apply until it prints "nothing to do", and orphans dropped only from a reviewed report.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!