Repository navigation
Conversation
A green 'Tests skipped' job records it on the run, and Finalize Deploy still prunes images. main ignores the flag.
Run workflow on dev with 'rollback' ticked skips the build and runs the existing rollback job, which restores the previously live image. The run stays green when the rollback succeeds.
ci(deploy): SKIP_DEV_TESTS flag and a dev rollback button
The fallback https://api.civicdataspace.in does not resolve, so every download_url in DCAT/Croissant/Dublin Core exports was a dead link when PUBLIC_API_URL was unset. Use https://api.datakeep.civicdays.in, the prod API host, in settings, the export adapter and .env.example. Closes #238
fix(api): point PUBLIC_API_URL fallback at the live prod API
The export view caught every ValueError around export_dataset() and sent str(exc) back with a 400, so internal errors (crosswalk, dates, rdflib) reached anonymous users verbatim and were reported as client errors. Validate standard and format against export_options() before exporting and answer with a fixed message plus the options. Every other failure now falls to the logged generic 500. Closes #240
fix(api): stop returning exception text from metadata export
DVC never worked: uploaded files were never under DVC_REPO_PATH, so every DVC call raised and createMajorVersion always failed. Remove the manager, the setup command, version-change detection and the post_save signal. createMajorVersion now writes the ResourceVersion row directly inside transaction.atomic(), behind IsAuthenticated plus a check that the caller owns the dataset or may change it. BREAKING CHANGE: the setup_dvc management command and DVC_* settings are gone.
Media moves to S3 only, through django-storages. Two storages share one bucket under <STORAGE_ENV>/public and <STORAGE_ENV>/private: public assets get unsigned, cacheable URLs; private files get 5-minute presigned URLs. The default storage is private, so anything that falls back to it is never world-readable by accident. Empty access keys use the boto3 credential chain (ECS/EC2 task role). boto/urllib3 loggers are capped at WARNING because at DEBUG they log every signed request, Authorization header included. Tests swap all storages for InMemoryStorage and never reach S3.
Every FileField/ImageField now names a storage: logos, cover images, profile pictures and publication block files go to public storage; resource files and chart images go to private storage. The storages are callables, so the migrations record a function reference, not bucket settings. upload_to paths drop their "files/public/" and "files/" prefixes, which only repeated what MEDIA_ROOT used to add; the storages already put every key under <env>/public or <env>/private.
S3 has no FieldFile.path, so every reader that used it would raise NotImplementedError once media left local disk. - local_copy() yields a temporary local copy (keeping the extension) for pandas, libmagic and the YAML/XML parsers; data indexing, entry counts and MIME validation use it. - The lru_cache on the loaders is dropped: it keyed on temp paths that are deleted on exit, so it only held DataFrames in memory. - Downloads stream with FileResponse instead of reading whole objects. - Charts only check that the file exists; they read the indexed table. - Search documents index the storage URL instead of a rewritten disk path.
Links issued before the move pointed at /public/<name> on disk. That route now redirects to the file's public S3 URL, after stripping the old "files/public/" or "files/" prefix from the name. Only public files redirect. The old static() route also served resources and drafts whenever DEBUG was on; anything outside public storage, or a name that tries to traverse paths, is a 404 here.
Rows created on disk carry a "files/public/" or "files/" prefix that the S3 keys no longer use. For each stored name the command puts the file at the normalised key in the storage its field names, taken from the old S3 key or, with --source, from the old disk tree, then repoints the row. Dry run by default; --execute copies and renames. Safe to re-run (names already in place are skipped) and deletes nothing.
Regression tests for the local-disk/DVC to S3 migration: S3 storage options (unsigned public URLs, 5-minute presigned private URLs, private default), which storage each file field names, local_copy() and MIME checks on storage without paths, prefix-free upload names, the legacy /public redirect (public files only, no traversal), createMajorVersion's owner-only version write, and migrate_media_to_s3 (dry run, disk and old S3 sources, missing files, failed copies).
Replace the DVC references in SECURITY.md and AGENTS.md with the S3 storage layout and access rules, and add the no-co-author commit rule to AGENTS.md.
Agents kept tripping over setup the file never mentioned (.env and logs/ are needed before tests boot; pre-commit pins python3.11) and over S3-era rules that lived only in code review. Add a quick start, a commands table with the SDK test command, task checklists for mutations, model changes and bug fixes, a files-and-media section (no FieldFile.path, local_copy, FileResponse, public vs private storage), and git safety rules (self-contained commits, no AI co-author trailer, never stash the user's work). Existing rules are unchanged.
CivicDataLab's scrapers fill "Source identifier" on every dataset they upload (dataspace-scrapers/<scraper_id>/<dataset_key>) and read it back to recognise their own datasets, so a dataset is never created twice and their upload records can be rebuilt from DataSpace. The data migration creates the item, or switches it back on if an admin switched it off; reversing it deletes nothing. Tests: tests/test_source_identifier_migration.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fier feat(metadata): add source identifier dataset metadata item
…blications Add useCasesTable, collaborativesTable, aiModelsTable and publicationsTable, shaped like datasetsTable: client filters and sort options limited to an allowlist per module, limit/offset paging, the total after filters, and counts per status for the dashboard tabs (ignoring the status filter so tab labels stay stable). Each query shows the same rows as the module's existing list query, plus published (for AI models: public and active) rows when includePublic is set. The existing list queries are unchanged. StatusCount and status_counts() are shared in api/utils/qs_utils.py. No migration.
The tab counts in datasetsTable were an inline copy of what status_counts() in api/utils/qs_utils.py now does for the other four tables. Call the helper instead, so all five tables count statuses in one place. status_counts() takes an optional item_type so the dataset table keeps its DatasetStatusCount type; the GraphQL schema is unchanged.
feat(api): table queries for use cases, collaboratives, AI models, publications
The charts dashboard loads every built chart and chart image through getChartData and searches them in the browser. chartsTable returns one page of both kinds in a single list, with the total count and per-status counts, using the shared table helpers. Both tables are merged in one SQL UNION, so counting, sorting and slicing happen in the database; only the rows of the page are then loaded. Visibility and permission are the same as getChartData. Rows keep their types (TypeResourceChart, TypeResourceChartImage), so existing fragments work unchanged. getChartData is untouched.
Sorting and paging now go through the same helper as the other table queries. Only filtering stays per table, because Django cannot filter after a UNION. Drops the extra id tie-breaker, matching the others.
feat(api): add chartsTable with filters, sorting and pagination
This commit introduces a new Event model with fields for name, location, dates, and status, along with a corresponding GraphQL schema for creating, updating, and querying events. The schema includes input types for event creation and updates, as well as queries for retrieving events based on various filters. Additionally, utility functions for handling Luma embeds and event permissions are implemented to ensure proper validation and access control.
An event whose owner was deleted (user set to NULL) and has no organization was reported as individual by the API but not by Event.is_individual_event. Use the same check in both.
Each event gets a 12-character public_id; its link is the name words plus that id (urlSlug), so renames never break links and two events can share a name. eventBySlug finds the event by the id at the end of the link. The slug is no longer unique and the counter loop is gone. Also drops the 0055_event dependency on 0049_merge_publication_and_accessmodel, which does not exist.
saqibmanan
had a problem deploying
to
development
November 14, 2026 09:38 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Table names were built from the resource name and dataset title, and Postgres keeps only the first 63 bytes of a name, so two resources with long similar names shared one table and a re-upload of one replaced the other's data. Re-uploads also dropped the old table before the new one existed, so readers saw empty or half-written data and a failed re-upload lost it. Each upload now gets a table named rdt_<row id>. index_resource_data builds the new table in one data-database transaction, then switches the resource over in one main-database commit (resource row locked, stored file re-checked so the newest file wins, column rows rebuilt with errors escaping) and confirms the new row exists; anything short of that drops this build's table and keeps the old data. A post_delete receiver on ResourceDataTable drops the old table after commit, which also frees storage on resource, dataset and admin deletes; it leaves legacy tables two rows still share. update_dataindex no longer wraps each call in a transaction, and the flow refuses to run inside one. Tests run against a real Postgres when TEST_DATA_DB_URL is set; CI adds a postgres service and fails instead of skipping them. Plan: plans/claude/data-index-table-names.md (Phase 1)
Existing tables still carry title-based names, and two resources can share one (Postgres cut both names to the same 63 bytes), so one of them serves the other's data. This one-off command moves them to rdt_<row id> names. Short old names can't have collided and are renamed in place, holding the resource row and waiting at most 2 s for readers. Long ones are rebuilt from each resource's own file through the normal upload flow, oldest first and with no read-time limit; the shared table is dropped once its last row has moved. A long table whose file is missing or can no longer become a table is renamed and marked "unverified" in its comment if no other row uses it, and every later report lists it until the resource is re-uploaded. A rerun after a crash finishes half-done renames. It is a dry run by default. The dry run checks each stored file, so its report shows what --apply will do, which tables are shared and which rows need a person. Tables no row uses are listed with size and comment. They are dropped only with --apply --drop-orphans --from-report <reviewed report>, and only if still unused at that moment. The command refuses to run when the data database looks like the main one. Every row's outcome is logged as it happens, and an in-progress report is written before any change. The Layer 5 journey (re-upload while polling, racing re-uploads, delete) is added for on-demand runs. Plan: plans/claude/data-index-table-names.md (Phase 2)
Covers per-upload table naming, the build -> switch -> drop-after-commit flow, the delete receiver and its shared-table guard, the readers, the migrate_data_table_names command with its runbook and old naming schemes, the Postgres test setup, the failure model, and known limitations. SDK: no. The data API doc links to it. Plan: plans/claude/data-index-table-names.md (Phase 4)
Final QC found that stopping a command (Ctrl+C) while the old table's after-commit drop waited for a reader made the flow's clean-up drop the new table, though the resource already pointed at it. The clean-up now asks the database first: it drops the build's table only if its row isn't saved, and leaves the table for the orphan audit when it can't tell. That also covers a commit that went through but reported an error. The runbook now says a row that fails twice the same way needs a person. Plan: plans/claude/data-index-table-names.md (final QC)
Two headers that differ only after Postgres's 63-byte cut become one column, so the file never gets a table. This bug predates per-upload tables and is noted, not fixed, by user decision.
A re-upload whose rows can't be loaded still reports success, so the download serves the new file while previews, charts and the data API serve the old one. Marked pending a product decision: enforce the database's limits at upload and flag problem columns in the column-description UI.
ubuntu-latest is now Ubuntu 26.04, where setup-python has no Python 3.9 build. The 3.9 job failed at setup and fail-fast cancelled 3.10 and 3.11 (seen on #295). The SDK still declares 3.9 support, so pin the runner instead of dropping it.
feat(data-index): per-upload data tables, safe re-upload, delete clean-up
They were skipped while CivicDataSpace-test#103 was open; it closed on 2026-09-24 and the provider suite now passes under parallel load.
saqibmanan
had a problem deploying
to
development
November 14, 2026 18:30 — with
GitHub Actions
Failure
GitHub Actions
Failure
saqibmanan
had a problem deploying
to
development
November 14, 2026 20:17 — with
GitHub Actions
Failure
GitHub Actions
Failure
saqibmanan
had a problem deploying
to
development
November 15, 2026 05:57 — with
GitHub Actions
Failure
GitHub Actions
Failure
saqibmanan
had a problem deploying
to
development
November 15, 2026 06:52 — with
GitHub Actions
Failure
GitHub Actions
Failure
saqibmanan
had a problem deploying
to
development
November 15, 2026 08:05 — with
GitHub Actions
Failure
GitHub Actions
Failure
saqibmanan
had a problem deploying
to
development
November 15, 2026 09:09 — with
GitHub Actions
Failure
GitHub Actions
Failure
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Brief description of the changes introduced by this PR.
Type of Change
Please delete options that are not relevant.
Related Issues
Fixes #(issue number)
Closes #(issue number)
Relates to #(issue number)
Changes Made
Testing
Test Instructions
Please describe how reviewers can test your changes:
Screenshots (if applicable)
Add screenshots to help explain your changes.
Checklist
black .,isort ., andflake8and fixed any issuesAdditional Notes
Add any additional notes or context about the PR here.