From 9273338842ae74c5c801997a0597adea96f82d5c Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 5 Aug 2026 11:11:49 +0100 Subject: [PATCH 1/4] Don't write .raw files --- doc/conf.py | 1 + .../datastreaming/ADRs/008_isis_raw_files.md | 58 +++++++++++++++++++ 2 files changed, 59 insertions(+) create mode 100644 doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md diff --git a/doc/conf.py b/doc/conf.py index 5b81d2070..f133f91d5 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -116,4 +116,5 @@ # This inventory needs to be periodically updated (manually) if new pages are added or # page layouts changed. "sysadmin": ("https://shadow.nd.rl.ac.uk/ibex_sysadmin_manual/", "sysadmin_manual.inv"), + "mantid": ("https://docs.mantidproject.org/", None), } diff --git a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md new file mode 100644 index 000000000..7f660aff3 --- /dev/null +++ b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md @@ -0,0 +1,58 @@ +# 8 - Datastreaming filewriter will not write `.raw` files + +## Status + +Accepted. + +## Context + +The existing `ISISICP` data acquisition code can write both legacy ISIS `.raw` files and newer `.nxs` files. In the data streaming system, we need to choose which file outputs we will support. + +### Definitions +- **ISIS `.raw` file** - an ISIS-specific data format, written by previous versions of ISIS data-acquisition systems. See {external+mantid:doc}`Mantid documentation ` for further information about the format. `.raw` is a legacy ISIS-specific format, and can only ever support histogram-mode data. +- **`.nxs` file** - a cross-facility standard data format for neutron, x-ray, and muon science. See [NeXuS webpages](https://www.nexusformat.org/) for further details. In this document, when the term `.nxs` file is used, we refer to a `.nxs` file generated by the instrument data acquisition system, which may contain either histograms, event-mode data, or both. +- **Reduced or processed `.nxs` file** - Mantid can write results of data reduction and processing operations to a `.nxs` file. This file is distinct from the `.nxs` files generated by the instrument data acquisition system. The formats of these files are defined and managed by the Mantid team, and are **outside the scope of this ADR** as they have no direct dependence on the streaming system. + +### Consumers + +Various programs may consume file outputs from the data acquisition system. + +#### Mantid + +Mantid can read ISIS `.raw` files, and both histogram and event-mode `.nxs` files. As part of filewriter definition work, we are working with the Mantid team to ensure that our `.nxs` files will remain compatible with Mantid. Mantid therefore has no hard dependency on `.raw` files being generated. + +#### Gudrun + +GudRun can directly read both ISIS `.raw` files and histogram-mode `.nxs` files. To read event-mode `.nxs` files, the files are first put through a [`nexus-processor`](https://github.com/disorderedmaterials/np), which performs required event-based data manipulations and then outputs a histogram-mode `.nxs` file suitable for loading into GudRun. + +The current GudRun maintainer was consulted about `.raw` files, and stated that: +- SANDALS already uses `.nxs` files as standard, and does not use `.raw` files +- NIMROD currently uses `.raw` files, but "there isn't anything blocking us from moving to nxs" +- GEM & POLARIS may need some minor work to create definition files for GudRun: "Those instruments may need a little work as Gudrun requires a text-based definitions file when using NeXuS in order to understand the detector layout" + +#### JournalViewer + +JournalViewer [can load data and metadata from both `.nxs` and `.raw` files](https://github.com/disorderedmaterials/jv/blob/c646dc9c97f749d8c45c9c022a23a1dcb2ebc98d/src/jv_local.cpp#L114). It therefore has no hard dependency on `.raw` files being generated. + +#### OpenGENIE + +Some old analysis scripts may still use OpenGENIE, and not have finished migration to Mantid. + +OpenGENIE [supports both `.raw` and `.nxs` file formats](http://download.opengenie.org/doc/old_manuals/GENIEUserManual/gdai.htm). Analysis scripts using OpenGENIE should therefore be adaptable to work with `.nxs` files only. This may involve some changes to the analysis scripts in question. + +#### Technique-specific tools + +A large variety of technique-specific analysis code exists, for example Refl1D (reflectometry), SASView (SANS). These tools typically read the **processed** Nexus files emitted by Mantid, not the files emitted by the data acquisition system. These tools should therefore have no dependency on `.raw` files being emitted by the data acquisition system. + +## Decision + +The data streaming filewriter will **NOT** write `.raw` files, and will only support `.nxs` file output. All major downstream consumers have been verified to either already use `.nxs` files, or have a viable migration path towards using `.nxs` files. + +## Consequences + +- We will **not** develop a `.raw` file filewriter as part of the data streaming work. +- Only one file output format, `.nxs`, will be supported. This will reduce the maintenance burden on the filewriter. +- Programs which read the data produced by the data acquisition system will only have access to `.nxs` files +- If a program that currently reads `.raw` files only has been missed, the developers of that program will need to either: + - Modify their program to read `.nxs` files in addition/instead of `.raw` files + - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program. *For avoidance of doubt, we are not proposing to write such a program as part of the data streaming work*. From bd9f0ac492d9783c0ef59af360dca99e41ff66f0 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 5 Aug 2026 14:52:18 +0100 Subject: [PATCH 2/4] Add notes about use of .raw file alternate file streams for checksums --- doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md index 7f660aff3..eaab94dd0 100644 --- a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md +++ b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md @@ -44,6 +44,11 @@ OpenGENIE [supports both `.raw` and `.nxs` file formats](http://download.opengen A large variety of technique-specific analysis code exists, for example Refl1D (reflectometry), SASView (SANS). These tools typically read the **processed** Nexus files emitted by Mantid, not the files emitted by the data acquisition system. These tools should therefore have no dependency on `.raw` files being emitted by the data acquisition system. +### Archival process + +The current archival process attaches checksums of all files generated as part of a run, to an NTFS secondary +file stream **attached to the `.raw` file**. + ## Decision The data streaming filewriter will **NOT** write `.raw` files, and will only support `.nxs` file output. All major downstream consumers have been verified to either already use `.nxs` files, or have a viable migration path towards using `.nxs` files. @@ -56,3 +61,4 @@ The data streaming filewriter will **NOT** write `.raw` files, and will only sup - If a program that currently reads `.raw` files only has been missed, the developers of that program will need to either: - Modify their program to read `.nxs` files in addition/instead of `.raw` files - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program. *For avoidance of doubt, we are not proposing to write such a program as part of the data streaming work*. +- We will need to do work, captured in [datastreaming issue 85](https://github.com/ISISComputingGroup/DataStreaming/issues/85), to work out how to replace the functionality currently performed by the `checksum` NTFS alternate file stream attached to the `.raw` file. This may mean generating the checksums required for file-archiving in a different way. From f96484f93a7b2d6014ece5cc9e166d26f2567ca7 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 5 Aug 2026 17:08:21 +0100 Subject: [PATCH 3/4] Adjust for OpenGENIE details --- .../datastreaming/ADRs/008_isis_raw_files.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md index eaab94dd0..fad4d8cd2 100644 --- a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md +++ b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md @@ -38,7 +38,11 @@ JournalViewer [can load data and metadata from both `.nxs` and `.raw` files](htt Some old analysis scripts may still use OpenGENIE, and not have finished migration to Mantid. -OpenGENIE [supports both `.raw` and `.nxs` file formats](http://download.opengenie.org/doc/old_manuals/GENIEUserManual/gdai.htm). Analysis scripts using OpenGENIE should therefore be adaptable to work with `.nxs` files only. This may involve some changes to the analysis scripts in question. +OpenGENIE [supports reading both `.raw` and `.nxs` file formats](http://download.opengenie.org/doc/old_manuals/GENIEUserManual/gdai.htm), however many of its built in commands only work on workspaces created by reading `.raw` files. + +If instruments still require OpenGENIE analysis or reduction code, they will need to either: +- Migrate their code to a supported data reduction/analysis package, for example Mantid. +- Write a utility which reads in a `.nxs` file, and outputs a corresponding `.raw` formatted file for their analysis code to use (this converter would only need to 'convert' the `.raw` fields which are actually used by the corresponding analysis code). #### Technique-specific tools @@ -58,7 +62,7 @@ The data streaming filewriter will **NOT** write `.raw` files, and will only sup - We will **not** develop a `.raw` file filewriter as part of the data streaming work. - Only one file output format, `.nxs`, will be supported. This will reduce the maintenance burden on the filewriter. - Programs which read the data produced by the data acquisition system will only have access to `.nxs` files -- If a program that currently reads `.raw` files only has been missed, the developers of that program will need to either: +- For programs which are only capable of reading `.raw` files, the developers of that program will need to either: - Modify their program to read `.nxs` files in addition/instead of `.raw` files - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program. *For avoidance of doubt, we are not proposing to write such a program as part of the data streaming work*. - We will need to do work, captured in [datastreaming issue 85](https://github.com/ISISComputingGroup/DataStreaming/issues/85), to work out how to replace the functionality currently performed by the `checksum` NTFS alternate file stream attached to the `.raw` file. This may mean generating the checksums required for file-archiving in a different way. From d3798380780d1fb240e50f850c6511ac928b64d1 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 5 Aug 2026 20:43:06 +0100 Subject: [PATCH 4/4] Explicitly defer decision of whether to write a nxs-to-raw converter to MNeuData --- .../datastreaming/ADRs/008_isis_raw_files.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md index fad4d8cd2..a68720afb 100644 --- a/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md +++ b/doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md @@ -40,9 +40,11 @@ Some old analysis scripts may still use OpenGENIE, and not have finished migrati OpenGENIE [supports reading both `.raw` and `.nxs` file formats](http://download.opengenie.org/doc/old_manuals/GENIEUserManual/gdai.htm), however many of its built in commands only work on workspaces created by reading `.raw` files. -If instruments still require OpenGENIE analysis or reduction code, they will need to either: +If instruments still require OpenGENIE analysis or reduction code, there will be a need to either: - Migrate their code to a supported data reduction/analysis package, for example Mantid. -- Write a utility which reads in a `.nxs` file, and outputs a corresponding `.raw` formatted file for their analysis code to use (this converter would only need to 'convert' the `.raw` fields which are actually used by the corresponding analysis code). +- Write a utility which reads in a `.nxs` file, and outputs a corresponding `.raw` formatted file for their analysis code to use. + +OpenGENIE usage does not apply to the immediate-term target instruments for data streaming at the time of writing this document. #### Technique-specific tools @@ -62,7 +64,8 @@ The data streaming filewriter will **NOT** write `.raw` files, and will only sup - We will **not** develop a `.raw` file filewriter as part of the data streaming work. - Only one file output format, `.nxs`, will be supported. This will reduce the maintenance burden on the filewriter. - Programs which read the data produced by the data acquisition system will only have access to `.nxs` files -- For programs which are only capable of reading `.raw` files, the developers of that program will need to either: - - Modify their program to read `.nxs` files in addition/instead of `.raw` files - - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program. *For avoidance of doubt, we are not proposing to write such a program as part of the data streaming work*. +- For programs which are only capable of reading `.raw` files, for example OpenGENIE scripts, there will be a need to either: + - Modify their program to read `.nxs` files in addition/instead of `.raw` files, or migrate the program to a supported codebase such as Mantid. + - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program. + - The immediate-term target instruments at the time of writing this document use Mantid or GudRun for data reduction and analysis. Therefore, the decision of whether to write a nexus-to-raw converter or to convert these analysis scripts to something that can use `.nxs` files is **deferred to the MNeuData project**. - We will need to do work, captured in [datastreaming issue 85](https://github.com/ISISComputingGroup/DataStreaming/issues/85), to work out how to replace the functionality currently performed by the `checksum` NTFS alternate file stream attached to the `.raw` file. This may mean generating the checksums required for file-archiving in a different way.