[jwrosewell]

Problem

For the list of gaps between the pipeline libraries and the specification (the Java list pipeline/Gaps-vs-spec.md and the .NET issues #23 to #29), covering item W2, the Sequence Element order and item 5b. The Python PipelineBuilder differs from the web integration specification in three ways.

1. The Sequence, JSON Builder and JavaScript Builder elements share one switch (add_javascript_builder), and Set Headers has another (use_setheader_properties). The specification asks for a way to prevent each being added.
2. PipelineBuilder.build adds the Sequence Element after the elements the user added (pipelinebuilder.py, build).
3. The Sequence Element writes only Evidence. The Sequence Element specification lists the Element Data session-id and sequence.

What changed

Switches. Pipeline configuration says "For all of these elements, there MUST be configuration options to allow users to prevent them being added (or disable them) if they wish to do so." PipelineBuilder settings now include add_sequence_element, add_json_builder_element, add_javascript_builder_element and add_set_header_element, each defaulting to true. add_javascript_builder still adds or drops the Sequence, JSON Builder and JavaScript Builder elements together, and use_setheader_properties still adds or drops Set Headers, so no existing setting changes meaning. An element is added when its own switch and any combined switch it belongs to are both true. The JavaScript Builder reads the JSON Builder's output, so a pipeline that keeps the JavaScript Builder and drops the JSON Builder needs one added by the caller. The settings are listed in the builder's docstring. The specification does not name the options, it says only that there MUST be configuration options, so the names are mine and follow the existing add_javascript_builder and use_setheader_properties.

Order. The same section says the Sequence Element "MUST be present. If not, it MUST be added as the first element in the Pipeline." The JSON Builder and JavaScript Builder were already added last, in the order the section gives, and Set Headers follows them. build now puts the Sequence Element first. It also builds the element list afresh, so calling build twice no longer adds the extra elements twice to the builder's own list. A new method get_sequence_elements returns the Sequence Element. get_javascript_elements now returns the JSON Builder and JavaScript Builder only, because the Sequence Element can no longer be added in the same place.

Element Data (item 5b). The Sequence Element lists the Element Data session-id (string) and sequence (int), and its Process section says "Add both output values to Evidence as well". The element now writes both Element Data values, flowdata.get("sequence").get("session-id") and .get("sequence"), keeps its Evidence writes query.session-id and query.sequence, and declares the two as its properties. The specification does not give the element's data key. The key stays sequence, which .NET and Go use.

Does this add a sequence section to the JSON sent to browsers? Yes. The JSON Builder specification says "There MUST be a top-level entry for each Element Data" and excludes only Set Headers, the Cloud Request Engine and Usage Sharing. JSONBundlerElement no longer skips the Sequence Element, so the JSON has "sequence": {"session-id": ..., "sequence": ...}. The JSON is written into the script, so the JavaScript Builder now writes <, > and & in it as <, > and &. Without that, a session id such as </script> from the query string would end the script, now that the id is in the JSON. The script's own sessionId variable is still empty for an id the script cannot take, as before.

Two existing results change because the Sequence Element now declares an int property and runs first. FlowData.get_where("type", "int") returns one more entry, and testGetWhere now expects 2. The usage sharing record carries the sequence after the Sequence Element has added one, so for a callback with sequence 1 it shows 2 where it showed 1, and with no session id it now shows the generated session id where it was empty. test_session_and_sequence expects 2.

Session id and sequence are read independently. The Process section of the Sequence Element specification treats them separately, and the element now does the same as PHP, Node and .NET. A session id that is not a non-empty string is absent and gets a new id, and "0" is kept. A sequence counts only if it is a whole number from 1 to 2147483647 or text of ASCII decimal digits in that range, and otherwise gives 1. A sequence with no session id counts up with a new id, and a session id with no sequence gives 1. Evidence and Element Data are written from the same values. Before, a sequence with no session id was dropped, and text such as " 5 " or "+5" was accepted as a sequence. The JavaScript Builder's own handling of those (its get_sequence) is unchanged.

Native engines. The Sequence Element now runs before the Device Detection and IP Intelligence engines, so the engines see its integer query.sequence, which the native evidence map refuses (TypeError: Wrong number or type of arguments for overloaded function 'MapStringStringSwig___setitem__' in the first full build of this head). The two engines now pass evidence values as text.

Lite data file comments. The required examples specification says "the Lite data file has a small number of Properties". The four example comments that said "limited accuracy and capabilities" (gettingstarted_console.py, match_metrics.py, metadata_console.py and offlineprocessing.py under device-detection/fiftyone_devicedetection_examples) now say that. No other comment in the repository uses the wording.

Effect on existing code

Pipelines built with the default switches now have the Sequence Element at index 0, so code that reads pipeline.flow_elements[0] for an engine sees the Sequence Element instead. Looking an element up by key (pipeline.get_element("device"), pipeline.flow_elements_list[...]) is not affected. I changed the tests that read by position (test_devicedetection.py, test_automatic_data_file_updates.py and the Device Detection test_cloud.py) to look elements up by key. The Python sources and examples already did.

Overlap with other drafts

#63 changes sequenceelement.py (the key filter method, a few lines above my change) and jsonbundler.py, javascriptbuilder.py and setheaderelement.py (which I do not touch). #61 and #59 change web.py, which I do not touch. The release draft #67 was left alone. The data update draft #69 is separate and touches none of these files.

How it was verified

New tests are in pipeline/fiftyone_pipeline_core/tests/test_web_integration_elements.py.

• Red, switches, order and Element Data, tests added to rc: run 42651987401, 10 core tests fail.
• Red, JSON entry and properties: run 42669756342, testGetWhere and the two new Sequence tests fail, as does test_session_and_sequence, which expected the sequence from before Sequence ran first.
• Red, the native engine evidence: the first full build of the head with the JSON entry, run 42671208739, failed the Device Detection tests on Windows with the TypeError above.
• Red, the independent session id and sequence: run 42676825997, 4 new tests fail.
• Green, head 86872f4f: run 42678332577, usage sharing 34 passed, engines 9 passed, core all passed.
• Full build of head 86872f4fda5d02f5c0f87911b8af3c8048eda182: run 42678851099. All 11 non-macOS jobs passed, and every Setting 'PullRequestSha' line read (Configure, Ubuntu_Python_3.12 and Windows_Python_3.10) equals the head. The run was cancelled while its 5 macOS jobs were queued.
  macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

The last release into main merged on 2026-11-04.

Outstanding

• The specification names no options, so the four switch names are mine. Other languages may pick different names, and James decides whether one naming is used everywhere.
• W1 (configurable endpoints) is met in Python by design, as the application defines its routes.