8 - Datastreaming filewriter will not write .raw files

Status

Accepted.

Context

The existing ISISICP data acquisition code can write both legacy ISIS .raw files and newer .nxs files. In the data streaming system, we need to choose which file outputs we will support.

Definitions

  • ISIS .raw file - an ISIS-specific data format, written by previous versions of ISIS data-acquisition systems. See Mantid documentation for further information about the format. .raw is a legacy ISIS-specific format, and can only ever support histogram-mode data.

  • .nxs file - a cross-facility standard data format for neutron, x-ray, and muon science. See NeXuS webpages for further details. In this document, when the term .nxs file is used, we refer to a .nxs file generated by the instrument data acquisition system, which may contain either histograms, event-mode data, or both.

  • Reduced or processed .nxs file - Mantid can write results of data reduction and processing operations to a .nxs file. This file is distinct from the .nxs files generated by the instrument data acquisition system. The formats of these files are defined and managed by the Mantid team, and are outside the scope of this ADR as they have no direct dependence on the streaming system.

Consumers

Various programs may consume file outputs from the data acquisition system.

Mantid

Mantid can read ISIS .raw files, and both histogram and event-mode .nxs files. As part of filewriter definition work, we are working with the Mantid team to ensure that our .nxs files will remain compatible with Mantid. Mantid therefore has no hard dependency on .raw files being generated.

Gudrun

GudRun can directly read both ISIS .raw files and histogram-mode .nxs files. To read event-mode .nxs files, the files are first put through a nexus-processor, which performs required event-based data manipulations and then outputs a histogram-mode .nxs file suitable for loading into GudRun.

The current GudRun maintainer was consulted about .raw files, and stated that:

  • SANDALS already uses .nxs files as standard, and does not use .raw files

  • NIMROD currently uses .raw files, but “there isn’t anything blocking us from moving to nxs”

  • GEM & POLARIS may need some minor work to create definition files for GudRun: “Those instruments may need a little work as Gudrun requires a text-based definitions file when using NeXuS in order to understand the detector layout”

JournalViewer

JournalViewer can load data and metadata from both .nxs and .raw files. It therefore has no hard dependency on .raw files being generated.

OpenGENIE

Some old analysis scripts may still use OpenGENIE, and not have finished migration to Mantid.

OpenGENIE supports reading both .raw and .nxs file formats, however many of its built in commands only work on workspaces created by reading .raw files.

If instruments still require OpenGENIE analysis or reduction code, there will be a need to either:

  • Migrate their code to a supported data reduction/analysis package, for example Mantid.

  • Write a utility which reads in a .nxs file, and outputs a corresponding .raw formatted file for their analysis code to use.

OpenGENIE usage does not apply to the immediate-term target instruments for data streaming at the time of writing this document.

Technique-specific tools

A large variety of technique-specific analysis code exists, for example Refl1D (reflectometry), SASView (SANS). These tools typically read the processed Nexus files emitted by Mantid, not the files emitted by the data acquisition system. These tools should therefore have no dependency on .raw files being emitted by the data acquisition system.

Archival process

The current archival process attaches checksums of all files generated as part of a run, to an NTFS secondary file stream attached to the .raw file.

Decision

The data streaming filewriter will NOT write .raw files, and will only support .nxs file output. All major downstream consumers have been verified to either already use .nxs files, or have a viable migration path towards using .nxs files.

Consequences

  • We will not develop a .raw file filewriter as part of the data streaming work.

  • Only one file output format, .nxs, will be supported. This will reduce the maintenance burden on the filewriter.

  • Programs which read the data produced by the data acquisition system will only have access to .nxs files

  • For programs which are only capable of reading .raw files, for example OpenGENIE scripts, there will be a need to either:

    • Modify their program to read .nxs files in addition/instead of .raw files, or migrate the program to a supported codebase such as Mantid.

    • Write a utility which reads in a .nxs file and writes out a corresponding .raw file for use by their program.

    • The immediate-term target instruments at the time of writing this document use Mantid or GudRun for data reduction and analysis. Therefore, the decision of whether to write a nexus-to-raw converter or to convert these analysis scripts to something that can use .nxs files is deferred to the MNeuData project.

  • We will need to do work, captured in datastreaming issue 85, to work out how to replace the functionality currently performed by the checksum NTFS alternate file stream attached to the .raw file. This may mean generating the checksums required for file-archiving in a different way.