Skip to content

netconf: classify mid-flight transport drops as ErrNotConnected - #481

Open
steiler wants to merge 4 commits into
mainfrom
netconf-midflight-eof-not-connected
Open

netconf: classify mid-flight transport drops as ErrNotConnected#481
steiler wants to merge 4 commits into
mainfrom
netconf-midflight-eof-not-connected

Conversation

@steiler

@steiler steiler commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

An NF restart during NETCONF edit-config (or the subsequent commit) returns a raw, unwrapped EOF-shaped error. No retry is triggered on the config-server side after the NF recovers, and status indicators (Target/Config/ConfigSet) incorrectly keep reporting Ready=True.

Context: sibling/predecessor work

This closes the residual gap left by #466 ("feature/not-connected-error", already merged to main), which introduced targettypes.ErrNotConnected and taught translateInternalToGrpcError to map it to codes.Unavailable, and updated the netconf target's two pre-flight connectivity checks (internalGet/Set guarded by t.Status().Err()) accordingly.

#466 did not update the three mid-flight transport-drop branches in pkg/datastore/target/netconf/nc.go, which still detected drops via a fragile strings.Contains(err.Error(), "EOF") check and returned the raw driver error unwrapped. That raw error propagated as codes.Unknown, which is exactly the case sdcio/config-server's isRecoverableTransactionError test table already anticipated as a data-server gap to close ("Non-Recoverable Unknown not connected (pre-fix data-server)").

Companion fix: sdcio/config-server#487 ("fix(config): correct Type stamped by updateConfigWithError's non-recoverable branch"). Either fix alone resolves the reported symptom for this specific NF-restart scenario; both are needed to fully close the gap — see that PR's Problem statement for why.

Pairs-with: sdcio/config-server#487

What changed

  • New isTransportError helper in pkg/datastore/target/netconf/transport_error.go: classifies a mid-flight transport drop via errors.Is(io.EOF)/errors.As(net.Error), with a substring fallback (EOF, broken pipe, connection reset, i/o timeout) for drivers that don't preserve that wrapping.
  • Normalized scrapligo's own opaque util.ErrConnectionError sentinel (what a real mid-flight EOF actually surfaces as, per tracing scrapligo v1.4.1's channel read loop) to io.EOF at the driver/scrapligo adapter boundary — see pkg/datastore/target/netconf/docs/adr/0001-normalize-driver-errors-at-adapter-boundary.md for why this lives at the driver boundary rather than in nc.go.
  • Deliberately excluded scrapligo's util.ErrTimeoutError from this classification — see docs/adr/0002-exclude-timeouts-from-transport-error-classification.md.
  • Replaced all three call sites' substring check (internalGet's GetConfig, setToDevice's EditConfig/Commit) with isTransportError, wrapping via a shared handleTransportError helper so the error flows through the exact same, already-reviewed translateInternalToGrpcErrorcodes.Unavailable path as the pre-flight checks.
  • Left the candidate Discard()-on-error path untouched for non-transport errors.
  • Added pkg/datastore/target/netconf/CONTEXT.md documenting that ErrNotConnected is intentionally reused for both pre-flight and mid-flight scenarios.

Testing

  • New unit tests: transport_error_test.go, nc_transport_error_test.go, driver/scrapligo/scrapligo_test.go.
  • go build ./..., go vet ./..., go test ./... all pass.

Out of scope

  • pkg/datastore/target/gnmi/ — the gNMI "Network Disconnect/Reconnect" path already works correctly.
  • Redesigning the reconnect() loop itself.
  • Retrying/re-driving the failed edit-config from data-server — that's a config-server responsibility.

Spec: .scratch/netconf-midflight-eof-not-connected/spec.md

Close the residual gap left by PR #466: the netconf target's three
mid-flight transport-drop branches (internalGet's GetConfig,
setToDevice's EditConfig and Commit) detected drops via a fragile
'EOF' substring check and returned the raw driver error, so it
crossed the wire as codes.Unknown instead of the already-established
codes.Unavailable/ErrNotConnected recoverable path.

- Add isTransportError in the netconf package: checks errors.Is(io.EOF),
  errors.As(net.Error), and a substring fallback (EOF, broken pipe,
  connection reset, i/o timeout) for drivers that don't preserve that
  wrapping.
- Normalize scrapligo's own opaque errConnectionError sentinel (which a
  real mid-flight EOF actually surfaces as, per tracing scrapligo
  v1.4.1's channel read loop) into io.EOF at the driver/scrapligo
  adapter boundary, so nc.go's isTransportError stays driver-agnostic
  and only scrapligo-specific code knows about scrapligo's sentinels.
- Replace all three call sites' substring check with isTransportError,
  and wrap the returned error with targettypes.ErrNotConnected via a
  shared handleTransportError helper, so translateInternalToGrpcError
  maps it to codes.Unavailable like the pre-flight checks already do.
- Leave the candidate Discard()-on-error path untouched for
  non-transport errors, per spec.

See .scratch/netconf-midflight-eof-not-connected/spec.md.
- ADR 0001: normalize driver-library errors (scrapligo's
  util.ErrConnectionError) to io.EOF at the driver/scrapligo adapter
  boundary, not in nc.go, so isTransportError stays driver-agnostic.
- ADR 0002: deliberately exclude scrapligo's util.ErrTimeoutError from
  transport-error classification, to avoid config-server retrying
  transactions against slow-but-alive devices.
- CONTEXT.md: document the netconf target and note that ErrNotConnected
  is intentionally reused for both the pre-flight (never connected) and
  mid-flight (transport dropped during an in-progress RPC) scenarios.
@steiler

steiler commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Companion fix is now up: sdcio/config-server#487 ("fix(config): correct Type stamped by updateConfigWithError's non-recoverable branch"). Fixes the independent control-plane bug where a non-recoverable TransactionSet failure silently loses its Config status update (wrong condition Type), so Ready stays stale True. Either fix alone resolves the reported NF-restart symptom; both are needed to fully close the gap.

@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.62500% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...store/target/netconf/driver/scrapligo/scrapligo.go 66.66% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alexandernorth alexandernorth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants