Skip to content

Data streaming IP addressing scheme docs #231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Tom-Willemsen wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Data streaming IP addressing scheme docs #231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a08f83
Data streaming IP addressing scheme docs
Tom-Willemsen Jul 22, 2026
773bc91
typo
Tom-Willemsen Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions doc/specific_iocs/datastreaming/Datastreaming-networking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Data streaming: networking

The data streaming system requires several items to be on the same network:
- `kafka_dae_control` (speaks UDP to streaming control board)
- `event_udp_to_kafka` (receives UDP from detector modules, converts and forwards them to Kafka)
- Streaming control board
- Control port which `kafka_dae_control` communicates with
- Status packet port which UDP headers come out of (which get processed by `event_udp_to_kafka`)
- Monitors
- Detectors

The current standard for network addressing is:
- `192.168.1.1` is the address of the streaming server, which will currently host both `kafka_dae_control` and `event_udp_to_kafka` processes

@FreddieAkeroyd FreddieAkeroyd Aug 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

192.168.1.1 and 192.168.1.2 are currently used by NDX/NDH - so the streaming computer in a cabin will be on its own 192 network separate to NDX/NDH?

@Tom-Willemsen Tom-Willemsen Aug 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was the idea yes - streaming private network will be totally distinct from the existing private network containing NDH/galil/beckhoff etc. I will try to make this clearer.

- It is possible, in future, that these two processes may run on different servers and therefore with different IPs
- `192.168.1.250` to be the streaming control board's control port
- `192.168.1.251` to be the streaming control board's status-packet port
- `192.168.1.252` to be streaming monitors
- The rest of the range to be streaming detectors

@rerpha rerpha Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before or after 250-252? guessing before, otherwise that's not many IPs for detector modules..?

@Tom-Willemsen Tom-Willemsen Aug 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha yeah, HRPD will have 80+ detector modules. SANDALS might be even worse. Will make this clearer...


@rerpha rerpha Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like a section in here about how our streaming software ie. kdaectrl + kdaediag interface with clients/the NDX - they probably need to be put on the site network, or also linked into the instrument private network with galils/beckhoffs on?

@Tom-Willemsen Tom-Willemsen Aug 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the intention was that the streaming server would be dual-networked:

  • one side to the streaming private network
  • other side to the main ISIS network

I don't currently have a use-case where the streaming server would need to connect to the instrument private network, but one might show up...

:::{note}
All of these IP addresses are on a private network which is **NOT** the same as the usual NDX/NDH private network.
This is a private network only used by the streaming server and streaming hardware. It therefore does not conflict with IP addresses on the NDX/NDH instrument private network.
:::
2 changes: 1 addition & 1 deletion doc/specific_iocs/datastreaming/Datastreaming-vetos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Data streaming - Vetoes
# Data streaming: Vetoes

Vetoes are essentially a way of telling either the detector acquisition hardware (with a `hard` veto) or downstream consumers of event data (`soft` veto) to ignore a set of events in a frame conditionally based on a signal (whether that's an internal state, electrical signal or a software-based condition).

Expand Down
2 changes: 1 addition & 1 deletion doc/specific_iocs/datastreaming/Datastreaming_hardware_requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Data streaming server hardware requirements
# Data streaming: server hardware requirements

This page aims to set a standard for the hardware requirements for the HRPD-X, SANDALS-II, WISH and VESUVIO instrument upgrades.

Expand Down
2 changes: 1 addition & 1 deletion doc/specific_iocs/datastreaming/Datastreaming_udp_packet_formats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Datastreaming: UDP packet formats
# Data streaming: UDP packet formats

This page describes the UDP packet format for instruments which stream UDP from detectors (e.g. HRPD-X).

Expand Down