[jwrosewell]

The problem

pipeline/Gaps-vs-spec.md lists where the Java usage sharing record differs from the specification. The Processing section of the Usage Sharing Element page says:

• "Where changes need to be made to the value, attributes are added to the XML element for that Evidence entry to let the reader know the value was modified."
• "escaped - Indicates that the original evidence value included invalid characters, which have been replaced with escaped ones."
• "truncated - Indicates that the original evidence value was too long. The value in the XML does not include all characters from the original."
• The snippet shows <PREFIX [escaped=true] [truncated=true] Name="FIELD">VALUE</PREFIX>, with SessionId, Sequence, DateSent, Version, Product, FlowElement, Language, LanguageVersion, ClientIP, ServerIP and Platform in that order.

Java marked an Evidence value that had invalid characters replaced with replaced="true", wrote Version and the server details before SessionId, Sequence and DateSent, replaced each invalid character with U+FFFD and cut values at 512 UTF-16 code units. The record rules for the escape and the limit are in the specifications draft #122, which says:

• "A character that XML 1.0 does not allow is replaced by the text \x followed by the code of the character as four lower-case hexadecimal digits, for example \x001f, and the element for that Evidence entry is marked escaped="true"."
• "A value longer than 10,000 characters is cut to 10,000 characters and the element for that Evidence entry is marked truncated="true". The limit counts Unicode code points, and the cut never splits a character, so a surrogate pair is never divided."

What changes

In ShareUsageElement.writeXmlData:

1. The attribute is escaped="true", where it was replaced="true", and truncated="true" is unchanged.
2. A character XML 1.0 does not allow is written as \x and four lower-case hexadecimal digits (\x001f), where U+FFFD replaced it. The characters XML 1.0 allows are kept, which is every character in the Char production of section 2.2: tab, line feed, carriage return, U+0020 to U+D7FF, U+E000 to U+FFFD and U+10000 to U+10FFFF. A surrogate on its own, U+FFFE and U+FFFF are written as escapes.
3. The limit is 10,000 characters, where it was 512 UTF-16 code units. ReplacedString counts characters with codePointCount and cuts at offsetByCodePoints, so a pair of surrogates is never split, and Constants.SHARE_USAGE_MAX_EVIDENCE_LENGTH is 10,000.
4. The elements are written in the order of the specification's snippet.

Not changed:

• There is no CDATA and no BadSchema element, as the page says.
• A backslash in a value is not escaped, so a value that contains the text \x001f reads the same as an escaped character. The decision above does not escape it.
• The version="1.1" attribute on <Devices> stays. The page asks only for a <Devices> root element and says nothing about attributes on it.
• The filter rules for which keys are shared, which draft Accept no Evidence for usage sharing, and give it the whole web request #68 changed and merged.

Behaviour that changes for the receiver

The attribute name replaced is gone and the same information is in escaped. Invalid characters arrive as \x escapes, values can be up to 10,000 characters, characters XML 1.0 allows (non-Latin text, tab, line feed, carriage return and more) arrive as they are, and the element order changes.

How it was checked

Tests were written first and fail on rc. ReplacedStringTest covers the escapes, the 10,000 limit (exactly 10,000 kept whole, 10,001 cut), a character made of a pair of surrogates at the limit kept whole with the next one cut, and a limit counted in characters (10,000 emoji kept, 10,001 cut). In ShareUsageElementTests, the bad-schema test now looks for escaped and the escape, and three tests are new: invalid characters give escaped and no replaced or truncated, a 10,001 character value gives truncated and no escaped with 10,000 characters kept, and the element order follows the snippet with no CDATA node and no BadSchema.

Run	Link	Result
Red, order and attribute	run	3 of 24 ShareUsageElementTests fail on Ubuntu and Windows
Green, order and attribute	run	All pass
Red, escapes and limit	run	13 of 17 ReplacedStringTest and the bad-schema test fail on Java 8 and 21
Green, escapes and limit	run	17 ReplacedStringTest and 25 ShareUsageElementTests pass on Java 8 and 21
Full pull request build	run	Every Ubuntu and Windows job (Java 8, 11, 17 and 21, and Ubuntu ARM) passed, the Selenium contract tests included. Every Setting 'PullRequestSha' line is 5a356b7, the head

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Every exit path of writeXmlData was read after the change.

Other open drafts

Draft #80 also changes the ReplacedString class in ShareUsageElement.java, to keep the characters XML 1.0 allows. This draft contains the same rule, so when both merge, the class from this draft is the one to keep. The merge needs a manual step in that class and in ReplacedStringTest.

What remains

Nothing for this record. The pipeline/Gaps-vs-spec.md file is not edited here.