[jwrosewell]

The problem

pipeline/Gaps-vs-spec.md lists where the Java pipeline differed from the specification. Checked against the code on rc and against the data update page, four data update items are still true:

Item	What the code did
D1, no published date	DataUpdateServiceDefault sent the epoch as If-Modified-Since when the data file had no published date.
D3, update on start-up alone	OnPremiseAspectEngineBuilderBase.preCreateEngine registered the data file with the update service only when auto update was on, so updateOnStartup with autoUpdate off did nothing.
D6, logging	Only debug lines for most actions, and error text that differed from the specification's table.
D8, licence key	An engine with the Distributor's URL built without a licence key, then failed on every update.

The same checks found D2, D4 and the programmatic update already met, and D5, D7 and D9 not required by the text, so those are not touched. The usage sharing items (U1 to U5) wait for the record format decision and are not touched either.

What changes

Each change cites the specification sentence it follows.

1. D1. If-Modified-Since is the data file's published date, or its last modified time on disk when it has none. Update polling: "If the data file does not have a publication date then the file system last modified date/time can be used instead." An in-memory data file with no date still sends nothing.
2. D3. The builder registers the data file with the update service when auto update or update on start-up is on, as .NET does. Update on start-up: "This option can be used to bootstrap an Aspect Engine that does not yet have any local data." The service now creates the file system watcher only when auto update is on, because the master switch "is the master switch for all automatic update activity", file system watching included. Without that, registering for update on start-up alone would also have started the watcher.
3. D6. The info messages and the error messages in the Logging tables, with their text and their levels (the specification gives no level for the info tables, so they are info):
   • "Updating on start-up", "Creating file system watcher", "Checking for update", "Attempting to refresh Engine '...' with new data", "Checking for update from '...' for Engine '...'", "Downloaded new data from '...' for Engine '...'" and "No data newer than ... found at '...' for Engine '...'" (a 304 reply, which was debug).
   • Warnings for a failed connection, a failed download (HTTP 429, 403 and any other failing status included, as the table's rows say) and a failed integrity check, with "Update will be attempted again later." only when auto update is on and the update is not a manual one. A failed connection and a failing HTTP status were errors, and a failed integrity check logged nothing.
   • Errors for a failed write, a failed refresh (one error after the retries, where there were ten warnings), a missing temporary file path and a failed copy of the data file.
   • Not added: the row for a locked data file with "use temp file" off, because no code path matches it.
4. Engine name before the engine exists. Update on start-up runs before the engine is built, so its log lines said Engine 'No Engine'. The builder now gives the data file its engine's type name, taken from the builder's own name (DeviceDetectionHashEngineBuilder gives DeviceDetectionHashEngine), and the update service logs it until the engine is set. AspectEngineDataFileDefault gains getEngineTypeName and setEngineTypeName.
5. D8. The build throws PipelineConfigurationException when auto update or update on start-up is on, the update URL is the Distributor's and no licence key is set. The message names setAutoUpdate(false) or setDataUpdateOnStartup(false), whichever is on. Defaults: the licence key is "required before the Engine builds with autoUpdate or updateOnStartup on ... The build fails with a configuration error that names the option to turn off." A key is not required when the URL is not the Distributor's, which is how an update source that serves the file without one is configured. OnPremiseAspectEngineBuilderBase gains a protected isLicenseKeyRequired that returns false, and FiftyOneOnPremiseAspectEngineBuilderBase returns true for the Distributor's URL.

Behaviour that changes for callers

• A pipeline built from the free Lite data file with the defaults now fails to build, because auto update is on by default and no licence key is set. The caller turns auto update off with setAutoUpdate(false), as the specification's defaults say. The tests that built with the defaults now do this or give a key, and the examples already did.
• setDataUpdateOnStartup(true) now works with setAutoUpdate(false).
• The data update service no longer creates a file system watcher for a data file whose auto update is off.
• No public method, option or default value is removed or changed. The one addition is the protected isLicenseKeyRequired.

Other open drafts

No open draft changes a file this draft changes. The drafts #78, #63 and #77 are left alone.

How it was checked

Tests were written first and fail on rc.

• DataUpdateServiceLoggingTests is new. It reads each message and level from the specification and checks If-Modified-Since for a data file with and without a published date.
• DataUpdateServiceTests now expects no file system watcher when auto update is off, and a warning, not an error, for a failed connection or a 429.
• FiftyOneDegreesOnPremiseAspectEngineBuilderBaseTests checks registration with update on start-up alone, and the licence key refusal and its exceptions.

Run	Link	Result
Red, tests only	run	12 of 14 logging tests, 10 of 108 service tests and 3 of 9 builder tests fail on Ubuntu and Windows
Green, with the fix	run	all 14, 104 and 9 pass on Ubuntu and Windows
Red, HTTP statuses	run	3 new logging tests and the 429 test fail on Ubuntu and Windows
Green, HTTP statuses	run	all 17 logging tests and 104 service tests pass on Ubuntu and Windows
Red, engine name	run	The tests do not compile on rc without setEngineTypeName
Green, engine name	run	18 logging tests and 10 builder tests pass
Full pull request build	run	Every Ubuntu and Windows job (Java 8, 11, 17 and 21, and Ubuntu ARM) passed, the Selenium contract tests included. Every Setting 'PullRequestSha' line is 98f3a9a, the head. ExamplesWithoutSettingsTest for Device Detection (6 tests) and IP Intelligence (9 tests) passes, so the examples still run out of the box with the free Lite file after the licence key check

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Every exit path of downloadFile, checkForUpdateFromUrl, updatedFileAvailable and updateFromMemory was read after the change. The new temporary file path check returns AUTO_UPDATE_REFRESH_FAILED where a missing path used to throw a NullPointerException out of the timer thread.

What remains

• D2 and D4 need no change. D5, D7 and D9 are not required by the text, and the usage sharing items wait for the record format decision.
• The pipeline/Gaps-vs-spec.md file is not edited here. Its removal is for whoever closes the list.