Skip to content

feat: Add is_producer_url_unstable to feed schema and expose in API#1771

Merged
jcpitre merged 5 commits into
mainfrom
feat/unstable_url
Jul 24, 2026
Merged

feat: Add is_producer_url_unstable to feed schema and expose in API#1771
jcpitre merged 5 commits into
mainfrom
feat/unstable_url

Conversation

@jcpitre

@jcpitre jcpitre commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Closes #1766

  • Added is_producer_url_unstable to the DB schema (nullable boolean, default null).
  • Modified the populate scripts to accept the new flag.
  • Exposed the flag in the public API, in the /v1/feeds, /v1/feeds/{id} and /v1/search responses (via source_info).
  • Also exposed/handled it in the internal Operations API (source_info + update-request models).

The rule for updating the DB from the CSV file is:

  • An empty is_producer_url_unstable cell in the CSV is a no-change in the DB.
  • An explicit TRUE or FALSE value in the CSV overwrites the current value.
  • This matches the existing behavior of is_official (get_safe_boolean_value + a not-None guard).

Note: This PR should be merged before this PR from mobility-database-catalogs: Add unstable_producer_url field
#1547

Testing tips:

This PR includes new unit tests.
The whole chain (mobility-database-catalogs → mobility-feed-api) was tested somewhat, except the notification between the repos, which is outside of scope.
The test was:

  • In a branch in mobility-database-catalogs, add the flag to the JSON files:
    • "is_producer_url_unstable": "True", to one JSON file, and set it to "False" in another.
  • Run the Export catalogs to CSV workflow.
    • Note this workflow was changed so it can accept a file name other than sources.csv. The file is placed in the mdb-csv bucket (authentication required).
  • Run the Mobility Catalog Update workflow in this repo.
    • Note this workflow was modified to accept the file name that was used when creating the CSV.

Tested these cases:

Catalog JSON value Existing DB value Resulting DB value Note
True any true explicit value always wins
False any false explicit value always wins
absent / empty null null new/unknown feed stays unknown
absent / empty true true retained — empty does not clobber
absent / empty false false retained — empty does not clobber

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@jcpitre jcpitre changed the title Add is_producer_url_unstable to feed schema and expose in API (#1766) feat: Add is_producer_url_unstable to feed schema and expose in API (#1766) Jul 23, 2026
@jcpitre jcpitre changed the title feat: Add is_producer_url_unstable to feed schema and expose in API (#1766) feat: Add is_producer_url_unstable to feed schema and expose in API Jul 23, 2026
# Three-state flag from the catalog CSV ("True"/"False"/empty). An empty cell leaves the DB
# value untouched (same as is_official), so a re-import never wipes an operator-set value.
is_producer_url_unstable_from_csv = self.get_safe_boolean_value(row, "is_producer_url_unstable", None)
if is_producer_url_unstable_from_csv is not None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick]: Taking into consideration that None is a valid value and has a "meaning", this conditional doesn't allow setting the value back to None.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussion with @emmambd, we don't need to go back to None.

For clarity (and my own understanding), here is a trace of the value through each stage.

1. Export — JSON → CSV (export_to_csv.py)

JSON value CSV cell
true True
false False
null (empty)
field absent (empty)

An explicit null and a missing field are indistinguishable after export — both become an empty cell. The CSV never contains the literal string null.

2. Populate — CSV → DB (populate_db_gtfs.py)

  • DB default: NULL
  • Empty cell → no-op — the DB value is left unchanged (so NULL stays NULL, and an existing True/False is retained).
  • True / False → written to the DB (overwrites whatever was there).

3. Read — DB → API response

DB Python API JSON
NULL None null
true True true
false False false

Net behavior

A null or absent value in the JSON results in:

  • true / false if the DB value was already true/false, or
  • null if true/false was never set.

You can't go back to the null API response once it's set to True or False.

@jcpitre
jcpitre marked this pull request as ready for review July 23, 2026 20:50
@jcpitre
jcpitre merged commit 51e5b73 into main Jul 24, 2026
21 of 22 checks passed
@jcpitre
jcpitre deleted the feat/unstable_url branch July 24, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add unstable_producer_url property to schema and expose in API

2 participants