Difference between revisions of "Instrument parameters and Instrument definitions"

From InelasticDataReduction
Jump to navigation Jump to search
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Instrument and experiment specific parameters ==  
+
== Experiment specific parameters ==  
  
To reduce experimental data, user needs access to [http://shadow.nd.rl.ac.uk/wiki/idr/index.php?title=Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data#Instrument_Definition_Files Instrument Definition Files] (IDF) defining main configuration of the instrument and parameters of the script, [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing which processes the data], and the range of auxiliary files such as '''detector calibration file''', defining actual neutronic positions of detectors in an experiment, '''map files''' (extension .map), defining the detector grouping, '''mask files''' (extension .msk) defining the lists of detectors which should not be processed and in less extend other files (.par, .phx, .nxs etc.)
+
To reduce experimental data described in the previous chapter, user needs access to [http://shadow.nd.rl.ac.uk/wiki/idr/index.php?title=Location_and_details_of_access_to_instrument_parameters,_necessary_to_reduce_data#Instrument_Definition_Files Instrument Definition Files] (IDF) defining main configuration of the instrument and parameters of the script, [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing which processes the data], and the range of auxiliary files such as '''detector calibration file''', defining actual neutronic positions of detectors in an experiment, '''map files''' (extension .map), defining the detector grouping, '''mask files''' (extension .msk) defining the lists of detectors which should not be processed and in less extend other files (.par, .phx, .nxs etc.)
  
On [http://www.isis.stfc.ac.uk/groups/excitations/data-analysis-computers/connecting-to-isiscomputendrlacuk-using-nomachine15120.html isiscompute servers] and instrument cabin computers these files can be accessed at:
+
The source of these files is stored in publicly available [https://github.com/pace-neutrons/InstrumentFiles GitHub archive] This archive is cloned to [https://isis.analysis.stfc.ac.uk/ IDAaaS workspaces] and instrument cabin computers at
  
 
   /usr/local/mprogs/InstrumentFiles/[INSTRUMENT_NAME]
 
   /usr/local/mprogs/InstrumentFiles/[INSTRUMENT_NAME]
Line 9: Line 9:
 
folder where the '''INSTRUMENT_NAME''' should be replaced by one out of '''[mari,maps,merlin,let]''' values list. Some instrument scientists create symbolic links (usually on Desktop) providing quick access to the files relevant for a particular instrument.  
 
folder where the '''INSTRUMENT_NAME''' should be replaced by one out of '''[mari,maps,merlin,let]''' values list. Some instrument scientists create symbolic links (usually on Desktop) providing quick access to the files relevant for a particular instrument.  
  
[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Automatic_user_configuration Automatic configuration scripts] running on [https://isiscompute.nd.rl.ac.uk/nxwebplayer isiscompue servers] automatically add appropriate folder, corresponding to the instrument to Mantid data search path (see picture below). This make the data, stored in the folder available to the [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing reduction scripts] and places the link to the instrument files to the Nomachine desktop.  
+
[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Automatic_user_configuration Automatic configuration scripts] running on IDAaaS automatically adds appropriate folder, corresponding to the instrument to the Mantid data search path (see picture below). This make the data, stored in the folder available to the [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing reduction scripts] and places the link to the instrument files the folder within IDAaaS machine menue.  
  
[[File:MantidDataSearchPath.png]]
+
[[File:instrumentFiles.png]]
  
The path to the requested parameters on this picture is set to '''mari''' instrument parameter's folder by the first row of the data search path folders list.
+
The path to the requested instrument parameters on the picture above is set to '''merlin''' instrument parameter's folder in the first row of the data search path folders list (blue arrow).
  
Currently instrument parameter are placed under [https://en.wikipedia.org/wiki/Apache_Subversion SVN source control] with server at:
 
  
[https://svn.isis.rl.ac.uk/InstrumentFiles/trunk https://svn.isis.rl.ac.uk/InstrumentFiles/trunk]
+
If user wants to run reduction on their own computer, he needs to checkout instrument files from the [https://github.com/pace-neutrons/InstrumentFiles GitHub archive]. Everybody can clone data from this server and some instrument scientists have write access to it. Contact [mailto:alex.buts@stfc.ac.uk Alex Buts] or [mailto:Duc.Le@stfc.ac.uk Duc Lee] if you believe you need write access to the server.
 +
On a Unix machine, an authorized person can add an instrument file to the source control and send it to the server by placing the file into appropriate instrument folder and issuing the following commands:
  
Everybody can read data from this server and some instrument scientists have write access to it. Contact [mailto:freddie.akeroyd@stfc.ac.uk Freddie Akeroyd] if you believe you  need write access to the server.
+
  git add "file_name"
On a Unix machine, an authorized person can added file to the source control and send it to the server by placing the file into appropriate folder and issuing the following commands:
+
  git commit -m" message describing purpose of the file"
 +
  git push
  
  svn add "file_name"
 
  svn commit -m" message describing purpose of the file"
 
  
[http://tortoisesvn.net/ Tortoise SVN] can be installed on a Windows machine to provide simple and straightforward GUI to control svn operations.
+
[https://github.com/FITLinuxSupport/analysisscripts/blob/master/UpdateSVN Simple scripts] are running each hour on instrument cabin machines and IDAaaS servers pulling data from [https://github.com/pace-neutrons/InstrumentFiles GitHub], so the file added to the server on any machine is ready to be used everywhere within an hour. The result of script operations is currently mailed to main instrument machine user (''mari'', ''let'',''maps'' or ''merlin'') so you can see results of these scripts work logging in as this user and typing
 
 
Simple scripts are running each hour on instrument cabin machines and [https://isiscompute.nd.rl.ac.uk/nxwebplayer isiscompue servers] pulling data from SVN server, so the file added to svn server on any machine is ready to be used everywhere within an hour. The result of their operations is currently mailed to main instrument machine user (''mari'', ''let'',''maps'' or ''merlin'') so you can see results of these scripts work logging in as this user and typing
 
 
   
 
   
 
  >>mail  
 
  >>mail  
  
command. The conservative logs are currently reviewed and regularly cleared up manually.   On [https://isiscompute.nd.rl.ac.uk/nxwebplayer isiscompue servers] only the last results are logged to:
+
command. The logs are regularly reviewed and cleared up manually.  
 
 
  /var/log/UpdateSVN.log.
 
 
 
available on '''ndmlnx01''' node only.
 
 
 
  
The SVN command:
+
The GIT command:
 
   
 
   
   svn update
+
   git pull
  
can be issued from svn folder of a particular machine (always '''/usr/local/mprogs/InstrumentFiles/''' but may have additional references on particular machines) to get immediate access to a new data, but on [https://isiscompute.nd.rl.ac.uk/nxwebplayer isiscompue servers] this command needs administrative access.   
+
can be issued from the instruments folder of a particular machine (always '''/usr/local/mprogs/InstrumentFiles/''' but may have additional references on particular machines) to get immediate access to a new data, but on [https://isis.analysis.stfc.ac.uk/ IDAaaS virtual machines] this command is restricted to the members of the excitation group staff.   
  
<span style="color:#0000ff; background:#FF69B4"> Five years ago it was the most convenient way to provide data synchronization between various computers, but other methods are available now.  
+
<span style="color:#0000ff; background:#FF69B4"> In 2015 git was the most convenient way to provide data synchronization between various computers, but other methods are available now.  
 
Group needs to discuss if this is still appropriate way of sharing data these days. </span>
 
Group needs to discuss if this is still appropriate way of sharing data these days. </span>
  
== Instrument Definition Files ==
+
[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]
 +
 
 +
== Instrument Specific Files (Instrument Definitions) ==
  
 
Instrument definition files, which describe the the instrument configuration and main parameters of the reduction are part of [http://docs.mantidproject.org/nightly/concepts/InstrumentDefinitionFile.html Mantid] and are usually located in ''/opt/Mantid/instrument'' folder on Unix or ''MantidInstallationFolder\instrument''  folder (''c:\MantidInstall\instrument'' by default) on Windows.
 
Instrument definition files, which describe the the instrument configuration and main parameters of the reduction are part of [http://docs.mantidproject.org/nightly/concepts/InstrumentDefinitionFile.html Mantid] and are usually located in ''/opt/Mantid/instrument'' folder on Unix or ''MantidInstallationFolder\instrument''  folder (''c:\MantidInstall\instrument'' by default) on Windows.
Line 55: Line 49:
 
   INSTRUMENT_NAME_Parameters.xml
 
   INSTRUMENT_NAME_Parameters.xml
  
file provide comprehensive list and description of almost all parameters used by [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing the data reduction script] and file  
+
file provide comprehensive list and description of almost all parameters used by [http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Location_and_details_of_access_to_sample_scripts_used_in_data_processing the data reduction script] and the file:
  
 
   INSTRUMENT_NAME_Definition.xml
 
   INSTRUMENT_NAME_Definition.xml
Line 67: Line 61:
 
*[https://github.com/mantidproject/mantid/blob/master/instrument/MERLIN_Parameters_after2013_4.xml MERLIN Reduction parameters]
 
*[https://github.com/mantidproject/mantid/blob/master/instrument/MERLIN_Parameters_after2013_4.xml MERLIN Reduction parameters]
 
*[https://github.com/mantidproject/mantid/blob/master/instrument/LET_Parameters_dr1to12.xml LET Reduction parameters]
 
*[https://github.com/mantidproject/mantid/blob/master/instrument/LET_Parameters_dr1to12.xml LET Reduction parameters]
 +
*[https://github.com/mantidproject/mantid/blob/master/instrument/HET_Parameters.xml HET Reduction parameters]
 +
 +
[http://shadow.nd.rl.ac.uk/wiki/idr/index.php/Setting_up_Mantid,_Matlab_and_auxiliary_programs_to_process_results_of_inelastic_experiments. Back to setting up Mantid]

Latest revision as of 16:07, 9 January 2023

Experiment specific parameters

To reduce experimental data described in the previous chapter, user needs access to Instrument Definition Files (IDF) defining main configuration of the instrument and parameters of the script, which processes the data, and the range of auxiliary files such as detector calibration file, defining actual neutronic positions of detectors in an experiment, map files (extension .map), defining the detector grouping, mask files (extension .msk) defining the lists of detectors which should not be processed and in less extend other files (.par, .phx, .nxs etc.)

The source of these files is stored in publicly available GitHub archive This archive is cloned to IDAaaS workspaces and instrument cabin computers at

 /usr/local/mprogs/InstrumentFiles/[INSTRUMENT_NAME]

folder where the INSTRUMENT_NAME should be replaced by one out of [mari,maps,merlin,let] values list. Some instrument scientists create symbolic links (usually on Desktop) providing quick access to the files relevant for a particular instrument.

Automatic configuration scripts running on IDAaaS automatically adds appropriate folder, corresponding to the instrument to the Mantid data search path (see picture below). This make the data, stored in the folder available to the reduction scripts and places the link to the instrument files the folder within IDAaaS machine menue.

InstrumentFiles.png

The path to the requested instrument parameters on the picture above is set to merlin instrument parameter's folder in the first row of the data search path folders list (blue arrow).


If user wants to run reduction on their own computer, he needs to checkout instrument files from the GitHub archive. Everybody can clone data from this server and some instrument scientists have write access to it. Contact Alex Buts or Duc Lee if you believe you need write access to the server. On a Unix machine, an authorized person can add an instrument file to the source control and send it to the server by placing the file into appropriate instrument folder and issuing the following commands:

 git add "file_name"
 git commit -m" message describing purpose of the file"
 git push


Simple scripts are running each hour on instrument cabin machines and IDAaaS servers pulling data from GitHub, so the file added to the server on any machine is ready to be used everywhere within an hour. The result of script operations is currently mailed to main instrument machine user (mari, let,maps or merlin) so you can see results of these scripts work logging in as this user and typing

>>mail 

command. The logs are regularly reviewed and cleared up manually.

The GIT command:

 git pull

can be issued from the instruments folder of a particular machine (always /usr/local/mprogs/InstrumentFiles/ but may have additional references on particular machines) to get immediate access to a new data, but on IDAaaS virtual machines this command is restricted to the members of the excitation group staff.

In 2015 git was the most convenient way to provide data synchronization between various computers, but other methods are available now. Group needs to discuss if this is still appropriate way of sharing data these days.

Back to setting up Mantid

Instrument Specific Files (Instrument Definitions)

Instrument definition files, which describe the the instrument configuration and main parameters of the reduction are part of Mantid and are usually located in /opt/Mantid/instrument folder on Unix or MantidInstallationFolder\instrument folder (c:\MantidInstall\instrument by default) on Windows.

Newer version of these files produced after Mantid release are automatically retrieved by Matnid at startup into ~/.mantid/instrument on Unix or c:\Users\USERACCOUNT_FOLDER\AppData\Roaming\mantidproject\mantid\instrument folder on Windows. Such files, if present for a particular instrument override files in Mantid installation directory.

 INSTRUMENT_NAME_Parameters.xml

file provide comprehensive list and description of almost all parameters used by the data reduction script and the file:

 INSTRUMENT_NAME_Definition.xml

describes all details of the instrument geometry. For further details of the file format and their contents user should look at Mantid IDF description pages.

Actual default parameters for inelastic data reduction scripts, which are used by Mantid reduction script can also be accessed on the web from Mantid Git repository:

Back to setting up Mantid