DS ADR: don't write .raw files

doc/conf.py

@@ -116,4 +116,5 @@
     # This inventory needs to be periodically updated (manually) if new pages are added or
     # page layouts changed.
     "sysadmin": ("https://shadow.nd.rl.ac.uk/ibex_sysadmin_manual/", "sysadmin_manual.inv"),
+    "mantid": ("https://docs.mantidproject.org/", None),
 }

doc/specific_iocs/datastreaming/ADRs/008_isis_raw_files.md

@@ -0,0 +1,71 @@
+# 8 - Datastreaming filewriter will not write `.raw` files
+
+## Status
+
+Accepted.
+
+## Context
+
+The existing `ISISICP` data acquisition code can write both legacy ISIS `.raw` files and newer `.nxs` files. In the data streaming system, we need to choose which file outputs we will support.
+
+### Definitions
+- **ISIS `.raw` file** - an ISIS-specific data format, written by previous versions of ISIS data-acquisition systems. See {external+mantid:doc}`Mantid documentation <concepts/RAWFile>` for further information about the format. `.raw` is a legacy ISIS-specific format, and can only ever support histogram-mode data.
+- **`.nxs` file** - a cross-facility standard data format for neutron, x-ray, and muon science. See [NeXuS webpages](https://www.nexusformat.org/) for further details. In this document, when the term `.nxs` file is used, we refer to a `.nxs` file generated by the instrument data acquisition system, which may contain either histograms, event-mode data, or both.
+- **Reduced or processed `.nxs` file** - Mantid can write results of data reduction and processing operations to a `.nxs` file. This file is distinct from the `.nxs` files generated by the instrument data acquisition system. The formats of these files are defined and managed by the Mantid team, and are **outside the scope of this ADR** as they have no direct dependence on the streaming system.
+
+### Consumers
+
+Various programs may consume file outputs from the data acquisition system.
+
+#### Mantid
+
+Mantid can read ISIS `.raw` files, and both histogram and event-mode `.nxs` files. As part of filewriter definition work, we are working with the Mantid team to ensure that our `.nxs` files will remain compatible with Mantid. Mantid therefore has no hard dependency on `.raw` files being generated.
+
+#### Gudrun
+
+GudRun can directly read both ISIS `.raw` files and histogram-mode `.nxs` files. To read event-mode `.nxs` files, the files are first put through a [`nexus-processor`](https://github.com/disorderedmaterials/np), which performs required event-based data manipulations and then outputs a histogram-mode `.nxs` file suitable for loading into GudRun.
+
+The current GudRun maintainer was consulted about `.raw` files, and stated that:
+- SANDALS already uses `.nxs` files as standard, and does not use `.raw` files
+- NIMROD currently uses `.raw` files, but "there isn't anything blocking us from moving to nxs"
+- GEM & POLARIS may need some minor work to create definition files for GudRun: "Those instruments may need a little work as Gudrun requires a text-based definitions file when using NeXuS in order to understand the detector layout"
+
+#### JournalViewer
+
+JournalViewer [can load data and metadata from both `.nxs` and `.raw` files](https://github.com/disorderedmaterials/jv/blob/c646dc9c97f749d8c45c9c022a23a1dcb2ebc98d/src/jv_local.cpp#L114). It therefore has no hard dependency on `.raw` files being generated.
+
+#### OpenGENIE
+
+Some old analysis scripts may still use OpenGENIE, and not have finished migration to Mantid.
+
+OpenGENIE [supports reading both `.raw` and `.nxs` file formats](http://download.opengenie.org/doc/old_manuals/GENIEUserManual/gdai.htm), however many of its built in commands only work on workspaces created by reading `.raw` files.
+
+If instruments still require OpenGENIE analysis or reduction code, there will be a need to either:
+- Migrate their code to a supported data reduction/analysis package, for example Mantid.
+- Write a utility which reads in a `.nxs` file, and outputs a corresponding `.raw` formatted file for their analysis code to use.
+
+OpenGENIE usage does not apply to the immediate-term target instruments for data streaming at the time of writing this document.
+
+#### Technique-specific tools
+
+A large variety of technique-specific analysis code exists, for example Refl1D (reflectometry), SASView (SANS). These tools typically read the **processed** Nexus files emitted by Mantid, not the files emitted by the data acquisition system. These tools should therefore have no dependency on `.raw` files being emitted by the data acquisition system.
+
+### Archival process
+
+The current archival process attaches checksums of all files generated as part of a run, to an NTFS secondary
+file stream **attached to the `.raw` file**.
+
+## Decision
+
+The data streaming filewriter will **NOT** write `.raw` files, and will only support `.nxs` file output. All major downstream consumers have been verified to either already use `.nxs` files, or have a viable migration path towards using `.nxs` files.
+
+## Consequences
+
+- We will **not** develop a `.raw` file filewriter as part of the data streaming work.
+- Only one file output format, `.nxs`, will be supported. This will reduce the maintenance burden on the filewriter.
+- Programs which read the data produced by the data acquisition system will only have access to `.nxs` files
+- For programs which are only capable of reading `.raw` files, for example OpenGENIE scripts, there will be a need to either:
+  - Modify their program to read `.nxs` files in addition/instead of `.raw` files, or migrate the program to a supported codebase such as Mantid.
+  - Write a utility which reads in a `.nxs` file and writes out a corresponding `.raw` file for use by their program.
+  - The immediate-term target instruments at the time of writing this document use Mantid or GudRun for data reduction and analysis. Therefore, the decision of whether to write a nexus-to-raw converter or to convert these analysis scripts to something that can use `.nxs` files is **deferred to the MNeuData project**.
+- We will need to do work, captured in [datastreaming issue 85](https://github.com/ISISComputingGroup/DataStreaming/issues/85), to work out how to replace the functionality currently performed by the `checksum` NTFS alternate file stream attached to the `.raw` file. This may mean generating the checksums required for file-archiving in a different way.