As you are beginning to appreciate, the exercise of using EISCAT data is a multi-stage process (e.g. data access, data integration, data analysis), so a natural thing to do is to chain some of these processes together to create a process pipeline.

Unfortunately because the Matlab-format data are contained in one file per dump, the data processing does not really lend itself to pipelining. However, because the old LDR-format mainland data can be contained in a single file for each data set, it is quite possible to chain consecutive processing commands together.

As we said in Section 6, simply staging LDR format data to disk is not actually a very useful thing to do, as we have very few software utilities to work with LDR files directly. Most of our subsequent software modules (e.g. raw data visualisation, analysis programs etc) work with either integrated data format or matlab-format data. The good news is that it is very easy to convert the data from LDR format into either of these formats, and it can be done without ever needing to write the LDR format data to disk, by the use of unix pipes.

The utility for converting from LDR format to integrated data format is called LDR2int. Once the data exist in integrated data format, they can be converted to matlab format using the utilities intpipe or int2GUP.

For instance, to go from a catalogue file containing a listing of LDR format data to a file of data on disk in integrated data format, one could run the following command:

getLR < {catalogue file} | LDR2int > {integrated data file}

Note that the integrated data file is a single file for the requested data set. Because it may be very large, integrated data files should also only be created on the /stager areas.