Using autoreduction

2023-01-09T16:23:38Z

Alex Buts: /* Autoreduction server and automatic data reduction */

== Autoreduction server and automatic data reduction ==

'''Autoreduction is currently disabled on inelastic instruments''', so this information is provided for references only

Autoreduction process listens to the events, produced by a machine, running the experiment and automatically reduces data, obtained from this experiment when the run is finished.

[https://users.facilities.rl.ac.uk/auth/?service=http://reduce.isis.cclrc.ac.uk&redirecturl=http://reduce.isis.cclrc.ac.uk Autoreduction Server] provides web-interface for changing and running reduction scripts.

The reduction scripts themselves can be found in ISIS network mounted on ISIS Unix machines in:

/archive/NDX[INSTRUMENT_NAME]/user/scripts/autoreduction/

or on Windows machines in:

\\isis\inst$\NDX[INSTRUMENT_NAME]\user\scripts\autoreduction

folders. Three files can be found(placed) there, namely ''reduce.py'', ''reduce_vars.py'' and ''[INSTRUMENT_NAME]Reduction.py''. Autoreduction automatically runs ''reduce.py'' script and processes ''reduce_vars.py'' as input variables. For ISIS inelastic instruments ''reduce.py'' script currently deploys ''[INSTRUMENT_NAME]Reduction.py'' script, which is the same script as the one, described in the [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing Sample Scripts] section of this WIKI, but renamed accordingly.

''reduce_vars.py'' can be automatically generated from ''[INSTRUMENT_NAME]Reduction.py'' if ''[INSTRUMENT_NAME]Reduction.py'' is run from Mantid, but autoreduction (or rather ''reduce.py'') runs ''[INSTRUMENT_NAME]Reduction.py'' overriding all variables from ''[INSTRUMENT_NAME]Reduction.py'' by variables from ''reduce_vars.py''.

Autoreduction server builds web interface on the basis of the variables found in '''reduce_vars.py''' file, and user can use Web-interface to change the values of these variables to run script with new values from autoreduction.

Reduction scripts

2023-01-09T16:22:08Z

Alex Buts: /* Reduction scripts used to process the experimental results */

== Reduction scripts used to process the experimental results ==

ISIS reduction scripts contain number of interconnected scripts which are the part of Mantid and can be located in ''Mantid Installation Folder/scripts/Inelastic/Direct/'' or see [https://github.com/mantidproject/mantid/tree/master/scripts/Inelastic/Direct Mantid distribution on the web]. These scripts set up and run various [http://docs.mantidproject.org/nightly/algorithms/index.html Mantid algorithms] to process input data according to the instrument related settings defined in [https://github.com/mantidproject/mantid/tree/master/instrument Instrument parameters files] and the settings, specific to a particular experiment. The later settings have to be done by instrument scientist and user locally in special scripts, allowing user to make these settings and run Mantid to reduce their data. An instrument specific examples of such scripts are part of Mantid continuous integration system and can be accessed on the web: (see, e.g. [https://github.com/mantidproject/mantid/blob/master/Testing/SystemTests/tests/analysis/ISIS_MariReduction.py MARI], [https://github.com/mantidproject/mantid/blob/master/Testing/SystemTests/tests/analysis/ISIS_MAPS_DGSReduction.py MAPS], [https://github.com/mantidproject/mantid/blob/master/Testing/SystemTests/tests/analysis/ISIS_MERLINReduction.py MERLIN] and [https://github.com/mantidproject/mantid/blob/master/Testing/SystemTests/tests/analysis/ISIS_LETReduction.py LET] or [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/ISIS_direct_reduction_script_description. their separate description] within this wiki)

Instrument scientists modify these scripts according to the requests of a particular experiment and Mantid provides special way to share these scripts through '''Mantid script repository''' configured at

File->Script Repository

field of Mantid GUI (see field pointed at as 1 on the picture below):

[[File:InstallScriptRepositorye.png|800px]]

The repository, installed locally according to the wizard invoked from this menu allows to download scripts located in the repository and share
the changes made to these files with other users of this repository. In addition to data exchange, ScriptRepository wizard configures Python path, to make repository's scripts runnable from Mantid. When Mantid starts, it updates its local copy of the '''Mantid script repository''' from the web. The progress of this operations reported to the Mantid log window (shown as the background of the fig above)
Default parameters of these scripts are part of Mantid and are described [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data#Instrument_Definition_Files in Instrument Definition Files] section of this Wiki.

Physically the '''Mantid script repository''' is located on [https://github.com/mantidproject/scriptrepository GitHub] and is normal Git repository, which is accessed through simplified Mantid GUI interface, but can also be operated by a person who has github account and is familiar with Git by accessing [https://github.com/mantidproject/scriptrepository GitScriptRepository] link and performing normal git operations.

[https://isis.analysis.stfc.ac.uk/workspaces/ IDAaaS workspaces] have Git repository cloned at

/opt/UserScripts/

folder, where the folder:

/opt/UserScripts/direct_inelastic/'''[InstrumentName]'''

contains scripts, used by excitation group. [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Automatic_user_configuration Automatic user configuration scripts] configure Mantid in user accounts to be able to run the scripts from this repository. In addition to that, configuration script processes special [https://github.com/mantidproject/scriptrepository/blob/master/direct_inelastic/MARI/USER_Files_description.xml USER_Files_description.xml] file, if one is found in particular instrument folder and copy all files, specified in this file to the RB folders, corresponding to the instrument.

To have write access to the repository through Mantid (and see what is actually there using Mantid GUI) users have to install '''Mantid script repository''' through Mantid (or Git) into their accounts.

[https://github.com/FITLinuxSupport/analysisscripts/blob/master/UpdateSVN Small configuration script] runs on IDAaaS every hour, and together with updating [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data Instrument Parameters repository], updates git repository, so all files changed or added by instrument scientists become available on all IDAaaS VM or local instrument machines within an hour. Members of excitation staff group can update the repository on request. See [[Automatic user configuration]] pages for detailed description of things, done to provide users with up-to-date data and parameters.

As '''Mantid script repository''' is a git repository, it has to obey the constrains imposed by GitHub. One of this constrains is the limitation on the size of the files allowed to be stored on GitHub. This does not allow to use Git repository for storing [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data Instrument files], necessary for data analysis, so local svn repository, described in [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data another chapter], has to be used instead.

[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]

2017-05-02T08:32:08Z

Alex Buts:

== ISIS Direct Inelastic Data Reduction ==

Inelastic data reduction is the process of transforming data obtained in inelastic scattering experiment into the data, describing inelastic cross-sections of the processes under investigation.

To process the data recorded by [http://www.isis.stfc.ac.uk/groups/excitations/excitations-4640.html Direct Inelastic Instruments] and obtain the desired cross-sections ISIS scientists use the [http://www.mantidproject.org/Main_Page Mantid-based] reduction program with a range of additional data and set-up parameters, tuning up this program to the particular inelastic instrument and experimental conditions. Users are also provided with access to the [http://isiscompute.nd.rl.ac.uk computing cluster], dedicated to help them in processing raw experimental data into the form, which has physical meaning.

From a user's point of view, the processing of their results should be simple and straightforward, namely:

# User logs into his or her [http://isiscompute.nd.rl.ac.uk/ isiscompute account] with logon name and password, obtained from the user office. In his home directory he finds a folder with the RB-name of the proposal, approved for the experiment.
# In this RB folder user can find a python script, necessary to process the results of the experiment, obtained from the instrument. User opens this script in Mantid, changes the run numbers to the numbers obtained in the experiment, can change some reduction parameters familiar to him (the description of the parameters is provided within the script) and run the script to processes the results of the experiment. User should not need to set up Mantid or modify the advanced reduction parameters.
# On the desktop of users account there are also links to sample Horace and Mantid scripts to do further analysis of the data.

=== Services, resources and Mantid settings necessary to allow users to process their data ===

This chapter contains the information about the data reduction service, implemented in ISIS.
User should familiarise himself with the information on [[Setting up Mantid, Matlab and auxiliary programs to process results of inelastic experiments.]]
if he is going to set up and process results of his experiments on a computers outside ISIS.

=== Autoreduction service ===
When Autoreduction service is enabled and set up properly, users data are also processed automatically, namely:
# Within RB folder a user finds folder named '''autoreduction'''. The folder should contain the reduced data processed automatically by [https://users.facilities.rl.ac.uk/auth/?service=http://reduce.isis.cclrc.ac.uk&redirecturl=http://reduce.isis.cclrc.ac.uk/ autoreduction service], which runs on a separate powerful machine and picks up the results of the experiments as they are completed and rapidly reduces them using a script similar to the one the users have in their RB folder.
#If necessary and convenient, the users should be able to change some parameters and reprocess some or all their runs using the web-interface and [https://users.facilities.rl.ac.uk/auth/?service=http://reduce.isis.cclrc.ac.uk&redirecturl=http://reduce.isis.cclrc.ac.uk/ autoreduction service].
#The common part of raw data processing (e.g. generation of the ''sqw'' files) can also be delegated to the [https://users.facilities.rl.ac.uk/auth/?service=http://reduce.isis.cclrc.ac.uk&redirecturl=http://reduce.isis.cclrc.ac.uk/ autoreduction service].

The autoreduction service is currently working in test mode, so its availability and reliability are not guaranteed.

=== Reduction scripts ===

[[ISIS direct reduction script description. ]] contains short description of the scripts, used to process results of the inelastic experiments in ISIS.

=== Chopper parameters ===

Reference material containing information about choppers, used by ISIS inelastic instruments, necessary for chopper '''flux&resolution''' calculations can be found in the [[Chopper Parameters]] chapter.

=== Help with editing this wiki ===

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]
* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Automatic user configuration

2017-03-22T18:15:25Z

Alex Buts: /* Automatic user configuration */

== Automatic user configuration ==

When an experiment is approved and scheduled by instrument scientist, [http://www.isis.stfc.ac.uk/user-office/ User office] assigns a user with his/her federal ID, which allows user to access ISIS network including [http://isiscompute.nd.rl.ac.uk/ isiscompute servers] and provides appropriate information to [Catalog service], giving the user rights to access the data through the Catalog.

At some stage, special [http://icatingest2.isis.cclrc.ac.uk/excitations.txt information file], which defines the participants of the experiments and the scientists who have access to the data of this experiment is placed on the [http://icatingest2.isis.cclrc.ac.uk/excitations.txt internal server] where it is accessed and automatically processed by [https://github.com/FITLinuxSupport/analysisscripts/blob/master/Build_Mantid_config.py issicompute configuration script]. The format of the file is .json and the sample contents is provided below.

The automatic isiscompute configuration script runs twice a day on [https://isiscompute.nd.rl.ac.uk/nxwebplayer isiscompue server] processing the information file, creating user accounts and appropriate RB folders and giving users permissions to access their RB folders on the basis of the information file. [mailto:alex.buts@stfc.ac.uk Alex Buts] and [https://www.facilities.rl.ac.uk/itsupport/default.aspx FBU IT support] [mailto:FITServiceDesk@stfc.ac.uk FBU IT support (Jeffs Warren)] have access and can modify this script. At the final stage this script runs [https://github.com/mantidproject/mantid/blob/master/scripts/Inelastic/Direct/ISISDirecInelasticConfig.py the Mantid configuration script for inelastic instruments], which configures each user's Mantid to use appropriate [[Experimental Data]], [[Instrument parameters and Instrument definitions]], necessary to reduce experimental data. It also provides users with initial version of [[Reduction scripts]], which users should modify to their requests according to the [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/ISIS_direct_reduction_script_description. description], using help of their instrument scientist if necessary. [mailto:alex.buts@stfc.ac.uk Alex Buts] is responsible for maintaining this script.

When initial [[Reduction scripts]] are copied to user's RB folder from the [https://github.com/mantidproject/scriptrepository/tree/master/direct_inelastic Mantid Script Repository] sample folder, script date is set to the experiment start date and the time -- to the exact midnight of the start date. Configuration script running twice a day checks this date and if it does not changed, replaces the existing script with the script, found in the Mantid Script Repository sample folder. If somebody touched the copied script, its date and time changes and [https://github.com/mantidproject/mantid/blob/master/scripts/Inelastic/Direct/ISISDirecInelasticConfig.py autoconfiguration script] stops replacing the file.

The diagram and the table below summarize the user creation workflow:

[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]

=== ISISOMPUTE User creation workflow ===

[[File:UserConfigWorkflow.png|1000px]]

Less visual but more detailed description of the workflow is provided in the table below. The number of the row in the table corresponds to the number of the block in the diagram above.

{| class="wikitable" style="color:blue;
! Stage Num
! Input
! Frequency, Actor & Event
! Result
! Person responsible
|-
| 1
| --
| Two weeks before Cycle, Crhis closes archive on the previous cycle. Sets up new cycle
| Cycle Folders appear in the archive.
| Chris Moreton-Smith
|-
| 2
| ---
| Somebody from the team sets up new cycle for cataloguing data.
| Data are searchable through iCat.
| Tom Griffin
|-
| 3
| Results of the review pannel
| On request, Instrument Scientists, schedules experiment in Scheduler application
| Input for scheduler script
| Instrument Scientist
|-
| 4
| e-mail to PI from 5.1, about experiment accepted
| Experimental team is added to visits system
| Input for scheduler scripts
| PI on accepted proposal
|-
| 5
| Results from 3 & 4
| Twice a day: '''Scheduler''' scripts runs processing inputs.
|
# Emails user office to create&renew FedID-s
# Generates [http://icatingest2.isis.cclrc.ac.uk/excitations.txt information file] containing RB number, Instrument, Cycle, visitors names & FedIDs
| Contact responsible from [mailto:isisdata@stfc.ac.uk data acquisition support group]
|-
| 6
| Output 5.1 from Scheduler script
| Create FedID & renew the old one
|
# User team as input for 5
# E-mails to users
| User's office
|-
| 7
| Output 5.2 from Scheduler script
| Twice a day: [https://github.com/FITLinuxSupport/analysisscripts/blob/master/Build_Mantid_config.py issicompute configuration script]
|
# Creates RB folders, User folders, access rights, autoreduction configuration.
# Users got rights to log into isiscompute and access their experiment folders.
# Starts Mantid Configuration script (8) and provides it with information about FedID&instrument&RB number
| Jeff Warren & Alex Buts
|-
| 8
| 7.3 and contents of MantidConfig folders
| [https://github.com/mantidproject/mantid/blob/master/scripts/Inelastic/Direct/ISISDirecInelasticConfig.py Mantid configuration script], started by [https://github.com/FITLinuxSupport/analysisscripts/blob/master/Build_Mantid_config.py issicompute configuration script]
|
# Mantid configured for first usage, [[Experimental Data]] and [[Instrument parameters and Instrument definitions]] data search path is set up and path to common inelastic script is configured.
# User scripts copied to users according their instruments and the contents of [https://github.com/mantidproject/scriptrepository/tree/master/direct_inelastic Mantid Users Script repository]
| Alex Buts
|-
| 9.
| Contents of [https://github.com/mantidproject/scriptrepository/tree/master/direct_inelastic Mantid Users Script repository] on the web. Contents of SVN servers on the web.
| Once an Hour: [https://github.com/FITLinuxSupport/analysisscripts/blob/master/UpdateSVN Inelastic data update script] pulls contents of the web repositories into files on isiscompute.
| [[Instrument parameters and Instrument definitions]] files in correspondent isiscompute folders are up-to date
| Jeff Warren & Alex Buts
|}

[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]

=== Extract from .json file defined users who has access to the data ===

{
"Last Updated": 20150413,
"experiments":
[
{
"RbNumber": "1234567",
"Instrument": "MERLIN",
"Cycle": "2013/4",
"StartDate": "20131203",
"Permissions": [
{
"email": "YYYY@stfc.ac.uk",
"fedid": "XXXXX"
},
{
"email": "warren.jeffs@stfc.ac.uk",
"fedid": "gpq43739"
},
{
"email": "alex.buts@stfc.ac.uk",
"fedid": "wkc26243"
}
]
},
{ "NEXT RB NUMBER""
}

[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]