Skip to content

Ask the appliance for its DTLS port before sweeping for it - #98

Merged
QuiteYellow merged 10 commits into
mainfrom
feat/directory-first-discovery
Sep 20, 2026
Merged

QuiteYellow merged 10 commits into
mainfrom
feat/directory-first-discovery

Conversation

@QuiteYellow

@QuiteYellow QuiteYellow commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Discovery now reads the device's own /oic/res directory first and takes the secure port from it. That resource is mandatory on every OCF device and must be reachable without DTLS, so one exchange yields the port. Two other fixes come with it: a probe bug that let a wrong port pass as proven, and a handshake collision that was reported as a fault.

Six commits, each standing alone. Validated on a dryer and an oven over several stop/start cycles.

The probe was counting the wrong port

probe_dtls_ports read "a reply arrived after dialling port N" as proof a DTLS server listens on N. An OCF stack binds its DTLS socket to port 0 and answers a first flight from that ephemeral port whatever port was addressed, and the probe's socket filters inbound on host only, so every port dialled looked live. Docker's bridge NAT had been discarding the mismatched replies and standing in for a correctness check the probe never had.

Selection now runs on the port a reply came from (responder_port). Measured port by port, exactly two ports answer on each appliance here: the standard CoAPS port 5684 and the ephemeral DTLS port, both replying from the ephemeral one. TizenRT's iotivity_1.2-rel fork, which these appliances run, binds 5684 as its secure multicast socket on INADDR_ANY (caipadapter.c:206 at pin e590f30ab), so a unicast ClientHello addressed there lands on that socket and is answered from the ephemeral secure one. The old code called those two separate listeners and reported 5684 for an appliance whose port is 49155.

Ask before sweeping

_resolve_port now runs pinned OCF_PORT → cached port → /oic/res on 5683 → band sweep, and every tier's answer goes through the same probe_dtls_ports gate, so an advertisement is only dialled once it has answered one. That matters: a live read here has returned sec: true, port: 0, because the stack binds port 0 and learns the assignment afterwards.

The band stays. 5683 is mandated only as the multicast listen port; it answers unicast because every OCF stack read here, classic included, wildcard-binds that socket, which is a strong convention with no clause behind it. And a device whose plaintext port is not 5683 has no directory for this tier to read. On the bench the fallback earned its keep unprompted: after one dropout the dryer's plaintext port stayed silent while its DTLS port recovered, and the sweep is what found it. Blocking the route to 5683 makes that path deterministic rather than opportunistic: both appliances log no_ocf_response after four attempts and resolve on the sweep, so a dead directory costs one bounded read and falls through.

_ports_from_links also now reads both advertised forms from one answer. Both appliances here are OIC 1.1, answer 4.06 to an OCF 1.0 Accept, and carry no eps key anywhere, so the real port arrives only as p.sec/port on the doxm link. The old split looked for eps on the first read and p.sec/port only on a second, filtered one, so every such device paid a wasted round trip. The doxm narrowing is unchanged and is the point: an eps URI is validated against the address the response came from, where a bare p.port integer carries no address, so it is read only from the link that has to describe the device's own secure endpoint. Nothing here has produced an eps entry, on hardware or in a report, so that path stays unexercised.

Say which tier chose the port

Four paths can pick it and only one logged anything. Each now names itself (configured OCF_PORT, cached from an earlier connect, advertised by /oic/res on 5683, found by sweeping the OCF band), because a swept port means the directory read got nothing out of that device, and that is the detail a bug report turns on.

A peer that opens its own handshake is not a fault

After the bridge has been away, the dryer answers its ClientHello with a ClientHello of its own, and OpenSSL, being the client, rejects it. That read as session error: session operation failed, a warning, an error_count bump and a backoff, for something that clears itself in about two seconds.

These appliances run TizenRT's iotivity_1.2-rel fork, which their own device log names. There, CAencryptSsl opens a session itself when it has a message for an endpoint it holds no peer for. That is how an appliance comes to dial the bridge: it still holds the fixed local port from the previous session. GetSslPeer keys the peer table on address and port with no role, so the bridge's ClientHello is stepped into that client-role context, and the failure path skips the alert for MBEDTLS_ERR_SSL_BAD_HS_CLIENT_HELLO and removes the peer. The device answers with silence and drops its half, which is why the next handshake succeeds. A per-record trace confirmed the prediction: the peer's ClientHello arrived and no alert came back.

connect() now raises PeerInitiatedHandshakeError when an inbound epoch-0 ClientHello preceded the failure, logging at info, and the bridge retries after 0.5s without touching error_count or growing the backoff.

Dropping the peer's ClientHello and carrying on was tried and rejected: the device sends nothing once it has consumed the bridge's, so recovery would still wait on OpenSSL's retransmit. It was armed on hardware for two cycles and never fired.

The demo bridge moves to host networking

Bridge NAT keys its conntrack entry on the port dialled, so a reply from any other port reads as an unrelated new inbound flow and the kernel drops it. That silently rules out the 5683 path, which is how standard-port firmware and non-Samsung OCF devices are reached. Symmetric DTLS to a known port traverses NAT fine, which is why this ran bridged for so long with the gap staying hidden. Safe only because of the responder-port fix, so the commit order matters.

Notes

  • 824 tests pass, 20 added. docs/api.md is regenerated; PeerInitiatedHandshakeError is in tools/api_contract.py because callers catch it by name to drive the retry.
  • docs/appliance-compatibility.md, docs/ocf-pki-laundry.md and docs/bridge-demo.md are corrected in the last commit. They told readers that a reply after dialling a port proved a listener on it, and described discovery as a sweep of the band. The compatibility page now also says that a silent 5683 is worth a second look, since the answer comes from another source port.
  • Hardware here is two OIC 1.1 appliances. The 5684 observation and the directory shape are n=2, and no standard-port or newer-PKI device has been tested.

Since the first push

Review of the branch found five things. The first three are fixed in 86ab57d, the fourth in a448a5b, the fifth in a3d6b5f.

The peer-initiated retry had no cap. An appliance that kept refusing drove a 2 Hz handshake loop for as long as it kept refusing, with error_count at zero throughout, so the health topic reported a healthy bridge. 200 consecutive collisions through the loop harness gave 200 immediate retries. Four still take the fast path, and past that a collision counts as a fault and takes the jittered backoff; a session that runs clears the count.

The fixed DTLS source port moves from 49700 to 26849. Host networking binds it on the host, and 49700 sits inside the default net.ipv4.ip_local_port_range of 32768-60999, where an unrelated process can hold it and the bind fails EADDRINUSE. Under bridge networking that bind was isolated in the container's namespace, which is why the old base was fine until this commit and is not after it.

A reply whose source port went unrecorded fell back to the port dialled. That fallback could not be reached through the real socket, and its only behaviour was to restore the rule this branch replaced, so it now reports unreachable. Two tests that selected on the dialled port went with it, each already having a counterpart written for the responder rule.

Both comments explaining the fixed DTLS source port credited RFC 6347 §4.2.8 eviction, where a re-handshake on the old 5-tuple makes the device discard a stale association that a device-side timer would otherwise reap. That mechanism is tinydtls's. The firmware these appliances run has neither the path nor the timer; what it has is a peer table with no cap, no idle timeout and no LRU, so the argument for one fixed port is the opposite shape. Reconnecting from a fresh ephemeral port each time leaves the old entry live and piles up mbedTLS contexts. Reconnecting over an entry the device still holds destroys the stale peer, since a ClientHello is not application data it can read, and that was measured at one extra ClientHello on the dryer and nothing on the oven. The dtls_session comment also credited the oven result to RT-OCF, which is not the stack that oven runs. The behaviour is unchanged either way.

The comment on the peer-initiated handshake cited IoTivity classic ca_adapter_net_ssl.c:1520 CAencryptSsl. 1520 is the upstream number, and in the fork these appliances run CAencryptSsl is at 2010 with its InitiateTlsHandshake call at 2040, so following that citation into the right tree landed hundreds of lines away. GetSslPeer (1074) and SSL_CHECK_FAIL (215, with the BAD_HS_CLIENT_HELLO test at 247) now carry numbers too. The family label was the other half of the problem: upstream sets VERIFY_REQUIRED and carries no ECDHE-ECDSA-AES128-GCM-SHA256, yet these appliances complete a session on an empty client certificate using exactly that suite, so upstream answers this question wrongly. Two mentions of the stack in this description are corrected the same way.

825 tests pass, which corrects the count above.

Since the rebase

Rebased onto main. The fork-naming pass merged there in the meantime reached the same dtls_session comment this branch rewrites, so the two collided; the resolution keeps this branch's text. That comment used to close on two corrections, and main has since made the RT-OCF half itself, so the note now states only the RFC 6347 §4.2.8 one.

Two comments this branch had left misplaced are tidied with it. The jitter note in run_forever sat above the immediate-retry branch, several lines away from the wait it describes. The new paragraph on the fixed source port ran straight into the one explaining why the base sits below 32768.

871 tests pass, 27 of them added here. The counts earlier in this description predate the rebase.

@QuiteYellow
QuiteYellow marked this pull request as ready for review September 20, 2026 15:16
probe_dtls_ports read a reply arriving after a ClientHello to port N as
proof that a DTLS server listens on N. An OCF stack binds its DTLS socket
to port 0 and answers a first flight from that ephemeral port whatever port
was addressed (RT-OCF rt_udp.c:152,164; iotivity-lite splits the same way
at ipadapter.c:465-467), and the probe's socket filters inbound on host
only. So every port dialled looked like a listener.

Docker's bridge NAT keys conntrack on the port dialled and had been
discarding the mismatched replies, standing in for a correctness check the
probe never had. Running the bridge on host networking exposed it in one
line: "multiple DTLS listeners answered; configure OCF_PORT", then a
confident "discovered DTLS port 5684" for an appliance whose DTLS port is
49155.

Measured port by port afterwards, exactly two ports answer on each
appliance: the standard CoAPS port 5684, which stock RT-OCF does not bind,
and the ephemeral DTLS port. Both reply from the ephemeral one. 5683, 5685,
40000 and the unused band ports stay silent, so the device is not answering
indiscriminately. The probe was counting 5684 and 49155 as two listeners.

Selection now runs on HostFilteredUdpSocket.observed_reply_port, which
already existed and was marked diagnostic only, and DtlsLivenessResult
carries it as responder_port. live_ports keeps its meaning, the ports
dialled that drew a reply, and responder_ports is the new proven set. Where
no source port was observed the old dialled-port selection still applies,
so nothing regresses on a platform that does not report it.

Validated on two appliances under host networking: correct ports, both
connected, no errors. The useful consequence is that one ClientHello to
5684 now returns the real DTLS port, including ports outside the
49152-49160 band the sweep covers.
Docker's bridge NAT keys its conntrack entry on the port dialled, so a
reply arriving from any other port reads as an unrelated new inbound flow.
Reverse-NAT needs a conntrack entry to match, so the kernel drops that
reply at the bridge. An OCF device answers plain-CoAP discovery on 5683 from its own
unicast ephemeral socket, so that whole path is unreachable from a bridged
container, and with it the standard-port firmware and the non-Samsung OCF
devices that path reaches. Symmetric traffic, DTLS to an already-known
port, traverses bridge NAT fine, which is why this ran bridged this long
without the gap showing.

Host mode was tried once before the probe was ready and broke discovery,
because the probe took the port it dialled as the listener. That is fixed
in the preceding commit, so host mode is safe now.

No ports are published: the bridge opens every connection itself, DTLS per
appliance and MQTT to the broker on 1883. The comment in the file carries
the reasoning, since the next person to tidy this will read network_mode as
laziness.
/oic/res discovery parsed eps on the unfiltered read and p.sec/port only on
a second /oic/res?rt=oic.r.doxm lookup. A device's spec generation decides
which form it emits, and the query has no say in it, so splitting them by
lookup cost a round trip to learn what the first answer had already said.

Both appliances here are OIC 1.1 (icv core.1.1.0), answer 4.06 to an OCF
1.0 Accept, and carry no eps key anywhere in /oic/res. Their real DTLS port
arrives as p.sec/port on the doxm link of the unfiltered directory: dryer
port 49155 in a 1727-byte representation, oven 49154 in 1629 bytes, both
with_eps 0. So the form the primary read looked for is one no device tested
here emits, and the fallback lookup fired every time. Nothing has yet
produced an eps entry for that path to parse, on hardware or in a report.

The extraction now reads both forms from whichever representation arrives.
The doxm narrowing is unchanged and is the point: an eps URI is validated
against the address the response came from, where a bare p.port integer
carries no address, so it is read only from the one link that structurally
has to describe this device's own secure endpoint. A sec-flagged port on any
other link stays unread, which a test now pins along with the rt and href
checks that establish the link.

eps ports lead the returned tuple, since they carry the stronger validation,
and _MAX_PORTS still caps the set. The filtered lookup stays for a device
whose unfiltered directory advertises nothing usable, where a smaller
representation gives the same parse a second chance.

An OIC 1.1 directory now resolves in one exchange, asserted end to end by
attempts == 1 and by no second request reaching the listener.
Port discovery swept a nine-port band plus 5684 and the descriptor default.
That is a guess at a layout, and it misses any device whose DTLS port sits
outside the band: vmonkey's TV at 51163 is one, and localthings#321 reads
like another.

The device already answers the question. /oic/res on the plaintext CoAP port
is the unauthenticated path every OCF device has to expose, it is mandatory
on all of them, and it names the secure port outright. On both appliances
here one such read returns the right port, 49155 and 49154, matching what a
ClientHello to 5684 selects. So _resolve_port now reads the directory first
and sweeps the band only if that yields nothing.

An advertisement stays a candidate and gets proven like any other: both
tiers go through the same probe_dtls_ports gate, so a directory serialised before the DTLS
bind, or one that has since moved, falls through to the band. That case is
real. RT-OCF binds the DTLS socket with port 0 and learns the assignment
through getsockname, and a live read here has returned sec: true, port: 0
while a later read of the same device gave a real port.

The band is kept for two reasons. 5683 is mandated only as the multicast
listen port, and it answers unicast because both reference stacks
wildcard-bind that socket, which is a strong convention with no clause
behind it. And a device whose plaintext port is not 5683 has no directory
for this tier to read.

A pinned OCF_PORT and a cached port both skip the read, since neither has
anything to learn from an advertisement. Tests cover the ordering, the
fall-through, a failed read, the two skips, and ambiguity inside the new
tier.
Four paths can pick the port: a pinned OCF_PORT, the cache from an earlier
connect, the device's own /oic/res advertisement, and the band sweep. Only
one of them logged anything. The port number on its own leaves the
useful part out: a swept port means the directory read got nothing out of
that device, and that is the reading a bug report turns on.

Every path now reports through one line. _accept_port logs the port with the
tier that produced it and caches what discovery proved, and the directory
read reports its own outcome: the ports it advertises, or the redacted
result, whose error code and attempt count separate a silent plaintext port
from a device that answered and advertised nothing usable. A cached port
going silent says so before rediscovering, where it used to fall through in
silence.

A pinned port is no longer written to the cache. It is re-read from config on
every connect and the cache is only consulted when nothing is pinned, so
storing it left _discovered_port meaning two different things.

Validated on both appliances, and the log answers the question it was added
for: "directory on 5683 advertises 49155", then "DTLS port 49155, advertised by
/oic/res on 5683", with the band never swept. The sample output
in docs/bridge-demo.md is that run, with the tier vocabulary spelled out
underneath.
The dryer answers the bridge's ClientHello with a ClientHello of its own
after the bridge has been away, and OpenSSL, being the client, rejects it. That
surfaced as "session error: session operation failed", a warning, and an
error_count bump, followed by a backoff, for behaviour that is expected and
clears itself in about two seconds.

Captured on the wire with a per-record trace:

  TX 160B  Handshake epoch=0 seq=0  ClientHello msg_seq=0 frag=0/135 of 135
  RX 123B from :49155  Handshake epoch=0 seq=1  ClientHello msg_seq=0 of 98
     -> dtls handshake failed at the TLS layer: unexpected message

The appliances run IoTivity classic, which their own device log names
(ocstack.c, cablockwisetransfer.c, ca_adapter_net_ssl.c, plus Samsung's
cloud_manager.c and friends). In that stack CAencryptSsl opens a session
itself when it has a message for an endpoint it holds no peer for, which is
how an appliance comes to dial the bridge: it still holds the fixed local
port from the previous session. GetSslPeer keys the peer table on address and port
with no role, so the bridge's own ClientHello is then stepped into that
client-role context, and the failure path skips the alert for
MBEDTLS_ERR_SSL_BAD_HS_CLIENT_HELLO and removes the peer. The device
therefore answers with silence and drops its half, which is why the next
handshake succeeds. The trace agrees: their ClientHello arrived and no alert
came back.

So the collision resolves itself, and the useful change is to say what it
was. connect() now raises PeerInitiatedHandshakeError when an inbound
epoch-0 ClientHello preceded the failure, logging at info, and the bridge retries after 0.5s without touching error_count or
growing the backoff. Nothing is wrong with the device or the network, and
the health topic keeps meaning what it says.

The detection reads only epoch 0, since a later epoch is encrypted and a
peer's first flight never appears there, and it walks every record because
flights arrive coalesced. PeerInitiatedHandshakeError subclasses
SessionError, so existing handling is unaffected.

Two things deliberately not done. Dropping the peer's ClientHello and
carrying on cannot work: the device sends nothing once it has consumed the
bridge's, so recovery would still wait on OpenSSL's retransmit. Accepting the inbound
handshake has no precedent in any OCF stack read here on UDP, and would
recreate the same role ambiguity.
The docs told readers that a reply after dialling a port meant a DTLS
listener was on it, and described port discovery as a sweep of the band.
Neither survives this branch: selection now runs on the reply's source
port, and the bridge asks /oic/res on 5683 before it sweeps.

Also notes that a silent 5683 is worth a second look, since the answer
arrives from another source port and a stateful firewall or a bridge NAT
will drop it while the device is answering.
Two follow-ups to the peer-initiated handshake classification, both
about what happens when the expected case stops being the actual one.

A refused handshake was retried at 0.5s with no cap, so an appliance
that kept refusing drove a 2 Hz handshake loop indefinitely while
error_count stayed at zero and the health topic reported a healthy
bridge. Measured with the loop harness: 200 consecutive collisions
gave 200 immediate retries. That is also the pattern that has been
observed to precede an appliance going silent for minutes. Four
consecutive collisions still take the fast path; past that a collision
counts as a fault and takes the jittered backoff. A session that runs
clears the count, so an appliance that collides once per outage keeps
the fast path for good.

The fixed DTLS source port moves from 49700 to 26849. Host networking
binds it on the host, and 49700 sits inside the default
net.ipv4.ip_local_port_range of 32768-60999, where an unrelated
process can hold it and the bind fails EADDRINUSE for one appliance
after a restart. Under bridge networking the bind was isolated in the
container's namespace, which is why the old base was safe before and
is not now. 26849-26864 was picked at random from the registered
range, carries no name in /etc/services, and is clear of the common
dev and IoT ports.

A reply whose source port went unrecorded no longer falls back to the
port dialled. The real socket records that port as it accepts the
datagram, so the fallback was unreachable. Its failure mode was the
problem: it restored the rule this work replaced, which answers
confidently and wrongly, since an OCF stack replies from its own port
whatever was addressed. It reports unreachable now. A regression in the
socket layer surfaces as a failed connect, where before it would have
opened a session against the wrong endpoint. Two tests that selected on
the dialled port go with it. Each already has a counterpart written for
the responder rule.

Three comments corrected alongside them. probe_dtls_ports selects on
the reply source port, so the note calling that port diagnostic no
longer describes what it does. The ClientHello scan reads only the
first handshake message in each record, so a coalesced one costs the
fast retry, and the docstring now says so. And reading both advertised
port forms from one answer also means eps is accepted from any link in
the filtered doxm lookup, which widens the source without widening
what is trusted.
Both comments explaining the fixed DTLS source port credited RFC 6347
section 4.2.8 eviction: a device that sees a re-handshake on the old
5-tuple treats the peer as rebooted and discards the stale association,
which otherwise lingers until a device-side timer reaps it.

That mechanism belongs to the tinydtls lineage. The firmware these
appliances run has no such path and no reaping timer. What it has is a
peer table with no cap, no idle timeout and no LRU, keyed on address and
port, so the real argument for one fixed port is the opposite shape: a
client that reconnects from a fresh ephemeral port every time leaves the
old entry live and piles up mbedTLS contexts on an embedded device.

Reconnecting over an entry the device still holds stays cheap, because a
ClientHello is not application data it can read, so it destroys the
stale peer. Measured on hardware: one extra ClientHello and about 1.1s
on the dryer, nothing on the oven.

The dtls_session comment also credited the oven result to RT-OCF, which
is not the stack that oven runs. Both now say "the firmware these
appliances run", naming a stack only where a source claim needs one.

Nothing about the behaviour changes. The fixed port stays, and the
oven's 2026-07-26 verification stands; what it stands on is different.
The comment explaining a peer-initiated handshake cited "IoTivity
classic ca_adapter_net_ssl.c:1520 CAencryptSsl". Both halves mislead.

1520 is the upstream number. These appliances run TizenRT's
iotivity_1.2-rel fork. There CAencryptSsl sits at 2010 and its
InitiateTlsHandshake call at 2040. Anyone who opened the tree the
comment is about landed hundreds of lines away in something unrelated.
The comment now also gives numbers for the two other symbols it leans
on, GetSslPeer at 1074 and SSL_CHECK_FAIL at 215, whose
BAD_HS_CLIENT_HELLO test is at 247.

Naming the family is the second problem, because upstream predicts
handshake behaviour these appliances do not show: it sets
VERIFY_REQUIRED, and it carries no ECDHE-ECDSA-AES128-GCM-SHA256, yet
the appliances complete a session on an empty client certificate, using
exactly that suite. Read upstream here and you get the wrong answer, so
the comment names the tree, and the pin.
@QuiteYellow
QuiteYellow force-pushed the feat/directory-first-discovery branch from a3d6b5f to 92a5624 Compare September 20, 2026 18:39
@QuiteYellow
QuiteYellow merged commit 16ae1b2 into main Sep 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant