[jwrosewell]

Problem

The data update service differs from the data updates specification in four ways. These are items D1, D2, D4 and D6 of the list of gaps between the pipeline libraries and the specification.

1. If-Modified-Since is sent as 2026-11-03T10:30:00Z, which is not an HTTP date, so a server ignores it.
2. get_next_update exists on every data file but nothing calls it, so polling starts before the time new data is expected.
3. A data file kept in memory can only be refreshed by downloading it. There is no way to give it the bytes.
4. Almost nothing is logged. A 304 response is logged at level warn, and Logger.log accepts only debug, info, warning, error and critical, so that call raises ValueError: list.index(x): x not in list in the update thread (run against Logger in pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/logger.py). A connection failure raises out of the update thread as well, so no later check is scheduled.

What changed

All changes are in pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py.

D1, the header. Update polling says "the If-Modified-Since header SHOULD be set to the publication date of the existing data file. If the data file does not have a publication date then the file system last modified date/time can be used instead." The value is now written with email.utils.formatdate(..., usegmt=True), which gives the HTTP date of RFC 9110 section 5.6.7. A published date that cannot be read falls back to the file's last modified time. A data file with neither sends no header.

D2, waiting for the expected time. Automatic update via HTTP says "If the current data contains a data update expected timestamp, then polling will not start until after that time." check_next_update now calls get_next_update. When that time is in the future, the next check waits until then plus the random offset, as the .NET service does. Otherwise the polling interval plus the random offset applies as before. A plain date that is not later than today is the default of a data file that does not know, and is ignored.

D4, update from memory. Programmatic update says "For memory data sources, programmatic update MUST allow provision of the new memory data source." The new public method DataFileUpdateService.update_from_memory(data_file, data) is named as in .NET and Java (UpdateFromMemory, updateFromMemory). It refreshes a data file created with download=False from the bytes given, reports the result to the on_complete methods, and returns an UpdateStatus. If the engine cannot use the data, the data it had is kept and AUTO_UPDATE_REFRESH_FAILED is returned. A data file kept on disk raises ValueError, because a refresh would read the file and ignore the bytes. A data file whose engine has been closed returns AUTO_UPDATE_NOT_NEEDED.

D6, logging. The Logging section says "All major actions SHOULD be logged at the appropriate level" and gives the messages, and "These messages MUST also be consistent across languages". The service now logs these at info, with the engine type being the class name of the data file's flow element.

Action	Message
Update on start-up	Updating on start-up
File system watcher created	Creating file system watcher
Check begins	Checking for update
HTTP request sent	Checking for update from '<url>' for Engine '<Engine type>'
Download succeeded	Downloaded new data from '<url>' for Engine '<Engine type>'
Refresh called	Attempting to refresh Engine '<Engine type>' with new data
304 response	No data newer than <datetime> found at '<url>' for Engine '<Engine type>'

The 304 message is at info and not a warning. <datetime> is the HTTP date that was sent. The error table in the same section is followed at the levels it gives, with "Update will be attempted again later." added only when the data file has auto_update on. Connection failure, download failure (any status other than 200 or 304, with the status code as the detail) and a failed integrity check are warning. A failed refresh, a failed write and a failed copy over the existing file are error. The write and copy failures are now caught, reported to on_complete and followed by the next scheduled check, where before the exception ended the update thread.

The earlier messages of the service (for example "Access denied from ...") are replaced by the specification's text.

Left alone

The MD5 comparison still reads the content-md5 header directly and raises if it is missing, as before. Items D3 and D8 are already met in Python, and D7 and D9 are not required by the specification, so this draft leaves them alone.

Overlap with other drafts

No open draft touches datafile_update_service.py or the tests changed here (#53 changes usage sharing, #57 the performance profile). The release draft #67 was left alone.

How it was verified

New tests are in pipeline/fiftyone_pipeline_engines/tests/test_datafile_update_http.py. One existing test (testFileSystemWatcherKeepsWatchingAfterFailedRefresh) counted every log line and now counts only the error lines, as the watcher logs a new information line.

• Red, tests added to rc: run 42651987401, 15 of the new engine tests fail (the header format, the wait, update_from_memory missing, the log messages and levels, the connection failure raising).
• Green, same tests with the fix: run 42652597574, engines 27 passed, core 354 passed.
• Full build: FULLBUILD

The last release into main merged on 2026-11-04.

Outstanding

Nothing in this change. The .NET message reads "Updating on startup", which differs by a hyphen from the specification table. This draft follows the specification table exactly.