[jwrosewell]

The problem

pipeline/Gaps-vs-spec.md lists where the Java pipeline differed from the specification. Checked against the code on rc and against the web integration page, these are still true:

Item	What the code does
W1, endpoints	PipelineFilter reads one init parameter (clientside-properties-enabled). The JavaScript path (51Degrees.core.js) and JSON path (51Degrees.core.json) are constants.
W2, optional elements	The Sequence Element and the Set Headers Element are always added when the configuration lacks them, with no way to prevent it.
5b, Sequence Element output	SequenceElement writes Evidence only and declares no properties. The specification lists Element Data as well.

Item 10 (Drift as a sum) is already fixed by the open draft #60, so it is not repeated here.

What changes

1. W1. The filter takes two new init parameters, javascript-path and json-path. Intercepted Urls: "These SHOULD be configurable, but the default values are: /51Degrees.core.js ... /51dpipeline/json". The JavaScript path default stays 51Degrees.core.js. The JSON path default becomes the specification's /51dpipeline/json, as .NET and Go use, and the JavaScript the filter serves calls it. The old path 51Degrees.core.json is still answered, whatever json-path says, so a site or a cached script that calls it keeps working. FiftyOneJSServiceCore.Default gains a constructor that takes the two paths, and the existing constructor uses the defaults. Constants.CORE_JSON_NAME stays, for the old path, and Constants.DEFAULT_JSON_PATH is new.
2. W2. The filter takes sequence-element-enabled and set-headers-element-enabled, both true by default. Pipeline configuration: "For all of these elements, there MUST be configuration options to allow users to prevent them being added (or disable them) if they wish to do so." A false value stops the element being added when the configuration lacks it, and an element named in the configuration stays. With client-side properties on and the Sequence Element left out the build is refused, naming the option, as .NET does, because the JSON Builder Element fails on every request without the sequence number (json-builder.md). clientside-properties-enabled works as before and still controls only the JSON Builder Element and the JavaScript Builder Element. StartupHelpers.buildFromConfiguration gains an overload with the JSON path and the two switches, and the existing overloads call it with the defaults.
3. 5b. SequenceElement also writes the Element Data session-id and sequence, under the key sequence, as .NET, Go and Python do (the page names no key), and declares them as properties. Sequence Element: the Element Data table lists both, and "Add both output values to Evidence as well" keeps the Evidence writes, which are unchanged. New class SequenceData. Process treats the two values on their own, so a session id that is not a non-empty string counts as absent and gets a new id, and a sequence counts only if it is a whole number or text of decimal digits that fits in an int with room to add one, and otherwise gives 1, with no error and no overflow. An empty session id counts as absent, which the page does not say outright and a specifications sentence is coming to state. An id of "0" is kept, a sequence with no session id counts up with a new id, and a session id with no sequence gives 1. The Evidence and the Element Data are written from the same values. Java used to cast the sequence with (int), which failed on text.
4. MVC integration. FiftyOneInterceptorConfig gains getJavaScriptPath, getJsonPath, getSequenceElementEnabled and getSetHeadersElementEnabled, as default methods so existing implementations keep working, and FiftyOneInterceptorConfigDefault gains the setters. The interceptor passes them to StartupHelpers and to the new FiftyOneJSService.setPaths, and FiftyOneInterceptor.enableClientsideProperties has an overload that maps the configured paths. The JSON default is the same new default as the filter's, the old path is answered and mapped too, and both elements are on.
5. Example comments. The comments in GettingStartedOnPrem.java, MetadataOnPrem.java and OfflineProcessing.java that said the Lite file has "limited accuracy" now say it has a small number of Properties, as required-examples.md says ("the Lite data file has a small number of Properties"), and so does the one in UpdateDataFile.java. These are comments only, so there is no test. A search found no other comment with that wording.
6. Usage sharing filter. EvidenceKeyFilterShareUsage.include returns false for server.http-request before any other rule, as .NET's filter does, so the whole web request that draft Accept no Evidence for usage sharing, and give it the whole web request #68 added is never a value to share. The review of Accept no Evidence for usage sharing, and give it the whole web request #68 asked for this.

Does 5b add a section to the JSON sent to browsers?

Yes. The JSON Builder Element skips only the element data keys cloud-response, json-builder and set-headers, so the Sequence Element's data now appears:

"sequence": { "session-id": "...", "sequence": 1 }

JsonBuilder_SequenceElementData_InJson in JsonBuilderTests runs a Sequence Element and a JSON Builder Element in one Pipeline and reads that section. The specification's exclusion list names the Set Headers Element, the Cloud Request Engine and the Usage Sharing Element, not the Sequence Element. Whether to exclude it, because the session id then reaches the page, is a decision for the maintainers, and the test records what happens today.

Release note

The JSON that 51Degrees client-side properties read is now served from /51dpipeline/json, the path the specification names. The old path /51Degrees.core.json still works, so no change is needed to a page, a cached script or a proxy rule. A site that wants another path sets json-path in the filter, or getJsonPath in the MVC configuration. The Sequence Element's data now appears in that JSON under sequence, and the filter and the MVC integration can leave the Sequence Element and the Set Headers Element out.

Behaviour that changes for callers

• The JSON is served from /51dpipeline/json, and the script the filter and the MVC integration serve calls that path. 51Degrees.core.json is still answered. A site that sets json-path keeps its choice.
• The JSON sent to browsers gains the sequence section above.
• A site whose pages call the MVC integration is unchanged unless it sets the new options.
• Nothing else. No public method, option or default is removed or changed. Added: SequenceData, the extra StartupHelpers.buildFromConfiguration overload, the extra FiftyOneJSServiceCore.Default constructor, the init parameter names in Constants, and the two properties on SequenceElement.

Other open drafts

• Accept no Evidence for usage sharing, and give it the whole web request #68 is merged. Its accepted Evidence filter in SequenceElement is kept, and this draft's toSequence replaces its getSequence. Cache the client-side endpoints' answers as the specification and RFC 9110 ask #67 also edits PipelineFilter, Constants and FiftyOneInterceptorConfig, so whichever merges second does the text merge.
• Let common-ci set the versions, find the bundle and collect the unit test reports #78, Give each package its own version and release only what changed #63 and Check the callback URL as the specification's six rules say #77 are left alone.

How it was checked

Tests were written first and fail on rc.

• PipelineFilterTests checks the elements the filter adds with each init parameter, and the callback endpoint.
• FiftyOneJSServiceCoreTests is new. It checks the default and configured paths. It does not compile on rc because it uses the new constructor, so the web module's tests fail to build there.
• SequenceElementTests is new, and JsonBuilderTests gains the JSON test above.
• FiftyOneInterceptorTests has four new tests for the MVC elements and paths, and NoSequence.xml is a configuration without a Sequence Element.

Run	Link	Result
Red, tests only	all tests, filter tests only	On Ubuntu and Windows: 3 of 3 SequenceElementTests, the new JsonBuilderTests test, and 3 of 9 PipelineFilterTests fail. In the first run the web module does not compile because of the new constructor, which the second run avoids by leaving FiftyOneJSServiceCoreTests out
Green	run	All pass on Ubuntu and Windows
Red, MVC tests	run	The MVC module does not compile on that head because the tests use the new options
Green, MVC	run	FiftyOneInterceptorTests (7), FiftyOneJSServiceCoreTests (4) and PipelineFilterTests (9) pass on Ubuntu and Windows
Red, key sequence	run	SequenceElementTests and the JsonBuilderTests test fail looking for the key
Green, key sequence	run	Pass on Ubuntu and Windows
Red, Sequence Element values	run	5 of 11 SequenceElementTests fail
Green, Sequence Element values	run	All 11 pass, with JsonBuilderTests and ShareUsageElementTests
Red, filter	run	webRequestIsNeverIncluded fails
Green, filter and merged Sequence Element	run	Pass on Java 8 and 21, with SequenceElementTests (11) and ShareUsageElementTests (22)
Earlier full build, Java 21 unit test	run	SessionIdAndSequenceTests expected an empty session id to be kept, and the Sequence Element now creates a new one. The test expects that now
Earlier full build	run	Failed on Ubuntu Java 21, where a constructor called an overridable method and the build treats that warning as an error. Fixed in a6303a4c
Red, JSON path	run	6 tests fail: the default endpoint, the served script, the old and new paths answered, and the MVC paths
Green, JSON path	run	Pass on Java 8 and Java 21
Red, refusal	run	The two refusal tests fail
Green, refusal	run	Pass on Java 8 and 21
Full pull request build, before the refusal and the merge of #81	run	Head 2d741ef, and every Setting 'PullRequestSha' line matches. The Ubuntu Java 11 job passes the unit tests and the example tests, then fails at the Selenium contract tests against the cloud and on-premise examples ("the first page must call the json endpoint at least once"), which stopped the other jobs. See "What remains"

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Every exit path of SequenceElement.processInternal was read after the change.

What remains

• The shared Selenium contract tests give Java the JSON path /51Degrees.core.json in their java descriptor (Examples/ExampleApps.cs in selenium-api-tests, read from main). With the new default the Java examples call /51dpipeline/json, so those tests fail in the full build ("the first page must call the json endpoint at least once") until the descriptor uses the default path. That change is in the other repository.
• The full build has to be run again with the whole matrix after the Selenium contract tests accept both JSON paths (selenium-api-tests#23, waiting to merge into main).
• Whether the JSON Builder Element should exclude sequence needs a decision.