EISCAT UK support group logo EISCAT HQ logo
How to turn raw data into plasma parameters.

The process of turning raw data into analysed plasma parameters needs an analysis program that is capable of generating theoretical autocorrelation functions and applying system-dependent corrections to them so that they can be iteratively compared with the measured data. RAL supports two such analysis programs:

GUISDAP

The Grand Unified Incoherent Scatter Design and Analysis Program (GUISDAP) is the analysis program of choice for all modern EISCAT data (i.e. all of the data recorded in Matlab format) and is supported on all of the RAL machines. It is the only available analysis program for ESR or post-renovation mainland data. Pre-renovation mainland data can also be analysed by GUISDAP if it has been converted to Matlab format (e.g. using intpipe).

GUISDAP was originally developed in Finland in the mid 1990s as a collaboration between the Sodankyla Geophysical Observatory, the University of Oulu and the Finnish Meteorological Institute. Since then it has been through a number of versions, the latest of which is guisdap 8.5.

Unfortunately, version 8.5 does not run under Tru64 unix, so the version currently (March 2007) available at RAL is the previous release (version 8.4). In practice this does not matter too much, since most current experiments are analysable using v8.4. For advice about things which can only be done with guisdap 8.5, and some notes on how to access the latest version on EISCAT's own computers, see Ian's GUISDAP page.

Throughout its development, successive versions of GUISDAP have attempted to remain backwards-compatible, that is to say that new versions have added capability while continuing to support the facilities offered by previous versions. There is, however, one important exception to this rule. The most recent versions of GUISDAP cannot analyse the old, pre-renovation, mainland data. For that reason, we offer two ways of running GUISDAP, either via the command line (which can deal with all the newest data) or via a Matlab script (which does not analyse some of the newer experiments, but can cope with the older data).

As inputs, GUISDAP requires raw or integrated EISCAT data in Matlab format, together with various information such as the number of parameters to be fitted, the accuracy of the required fit, the number of allowed iteration steps and the model values being assigned to the non-fitted parameters. Several of these inputs are hidden from the user by default, however a user who wants to be a little more advanced has the possibility to go in and change almost any of the settings in the program.

The outputs of GUISDAP are also Matlab files, one per integration period, containing the fitted plasma parameters, the associated error bounds derived from the input data variance, the apriori values and the fit residuals, among others.

Running GUISDAP from the command line.

GUISDAP can be run from the command line, by simply typing:

guisdap

However, a number of command line arguments are needed in order to specify the various user-settable options. There are several command line arguments available, some of which are optional, but some of which are compulsory. Typing

guisdap -h

shows you the full list. The most important of the command line arguments are:

-C {system constant}

-e {experiment name}

-f {filelist containing the names of the input “mat” files}

-g {version number}

-h {help}

-I {input data directory}

-M {add extra directories into the matlab path}

-R {output data directory}

-s {site code}

-V {set global variable}

Of these arguments, only -e, -R, and -s are compulsory.

The experiment name would normally be the pulse code employed by the experiment, for example: tau0, tau2, steffe. This argument is included in order to set up the analysis for that pulse code.

The filelist is the name of the file containing the list of input file names, usually this file is created automatically by the integration program, and is called either filelist.dat or filelist.mat.

The version number is g84 by default (i.e. running v8.4) but if users for any reason want to use a previous version (e.g. to check consistency between different versions of the program) one could also specify -g g83 for example, to use version 8.3.

The -M option can be used to add a new directory into the Matlab search path such that it will be picked up at higher priority than the standard guisdap software directories. For example, if you want to use a modified version of a GUISDAP routine, you could put it in directory /home/eiscat/fred/myguisdap and then use

-M /home/eiscat/fred/myguisdap

to ensure that it was used in preference to the standard GUISDAP routines.

The site code indicates the EISCAT site at which the data was taken. It may be one of the following:

K (for Kiruna)

L (for Longyearbyen i.e. the ESR)

S (for Sodankylä)

T (for Tromsø UHF)

V (for Tromsø VHF)

The appropriate code will set up the analysis for that site, so that -s L will set up the analysis for ESR data.

The -V option will allow the user to set one or more Matlab global variables, which will be carried through the program, in addition to the global variables defined as part of the standard GUISDAP configuration.

Running GUISDAP from a script

The scripting version of GUISDAP utilises a slightly older version of the code, but retains the ability to analyse the pre-renovation (pre-2000) mainland data, which cannot be analysed by the command-line version of GUISDAP. Also, because a number of GUISDAP variables are set explicitly in the scripts, the scripted version of the program offers the user the possibility to play with some of the settings that are more deeply hidden in the command line version of the program.

To set up the script-based version of GUISDAP, you need to do two things:

  • Specify the matlab path to be used
  • Produce a script that specifies the analysis strategy

    • Path-Setting Scripts

    The matlab path which should be used, depends on which radar system your raw data comes from. While most of the GUISDAP software is common to all the EISCAT radars, some of the routines (e.g. those concerned with reading system parameter blocks and data files) are system-dependent. This system-dependence is handled by having multiple versions of the same routine (e.g. for ESR, pre-renovation mainland and post-renovation mainland) in different directories, which can be either included or excluded from the Matlab search path.

    Some EISCAT experiments may have the same name, but exist in multiple versions (e.g. the versions of the ESR tau0 experiment with pre-integration times of 3.2s, 6.4s and 12.8s). Once again, choosing any one of these needs one of multiple versions of the same routine to be specified.

    The set-up of the matlab path then determines which of these routines is picked up for use.

    In the directory /home/eiscat/ian/GUISDAP/setups/, one can find a number of scripts which set up the matlab path in the manner appropriate to the data being analysed. There are separate sub-directories for the clustered nodes and for some of our older machines (the latter are not really relevant from a users point-of-view). These scripts can be run under unix to enter matlab and start GUISDAP with the appropriate matlab path in place.

    For example, if the command /home/eiscat/ian/GUISDAP/setups/cluster/KST_new is given, matlab is entered with the correct paths set up for analysing post-renovation mainland data. Other options are:

    KST_old (for pre-2000 mainland data)

    ESR_tau0_preJune02 (for ESR 6.4s tau0 experiments before 01/06/2002)

    ESR_tau0_postJune02 (for ESR 6.4s tau0 experiments after 01/06/2002)

    ESR_gup (for other, usually earlier, ESR experiments).

    tau0_12.8 (for the 12.8s pre-integration version of tau0)

    tau0_3.2 (for the 3.2s pre-integration version of tau0)

    Data Analysis Scripts

    In order to kick off the analysis, GUISDAP needs to execute an experiment-dependent matlab script which specifies in detail how the analysis is to be done. The directory /home/eiscat/ian/GUISDAP/scripts contains examples of such scripts for all the experiments whose analysis is currently supported by the scripted version of GUISDAP. The directory has three sub-directories KST_new, KST_old and ESR, whose purpose is self-explanatory. The user can copy the chosen analysis script to his/her own directory and edit the things that might need changing (e.g. data path, result path, gating strategy etc.)

    For a user with the directory /home/eiscat/ian/GUISDAP/setups in their unix path, and an appropriately edited copy of the GUISDAP analysis script Analyse_cp1lt_lp.m in their current working directory, giving the commands:

    KST_new
    Analyse_cp1lt_lp

    should be all that is needed to start off the analysis.

    To analyse remote-site cp1l, you need the script:

    /home/eiscat/ian/GUISDAP/updates/KST_new

    Some documentation of the script-based version of GUISDAP can be found in these web pages. While some of this documentation is now rather out-of-date, there are good descriptions of these GUISDAP analysis scripts and what they contain, which should provide users with sufficient information to change the scripts to perform the type of analysis required.

    Note that GUISDAP also has a basic integration capability within the program. Because this integration capability is much less evolved than our other integration utilities, discussed above, it should only be used for simple time-bound integration. For any other integration strategy, integrate2 or intpipe should be used.

    It is also worth mentioning that GUISDAP has a graphics capability. If the four graphics flags in the analysis script are enabled (set to 1) the program displays the integrated data dump being analysed, the raw density profile, the best fit lag profile matrix for each range and the profiles of the fitted plasma parameters from each integration. This graphics capability might be very useful for getting an idea of how the program is working, but it produces a very substantial slow-down in the speed at which the program runs. Once a user has run the program once or twice with graphics enabled in order to see how they work, it is likely that they will wish to turn off the graphics (in the analysis script) for all future use of the program!

    Provided that everything has been set up correctly, simply executing the GUISDAP analysis script should be all that is needed for the program to begin working, and for result files (in matlab format) to begin accumulating in the directory specified by the result path in the analysis script. When the analysis of the specified data set is complete, the program should terminate and exit from matlab. It is, however, sometimes possible for GUISDAP to crash, leaving the matlab session hanging. Such cases usually correspond to problems, either with incorrectly staged data or with data, which has been correctly staged but was corrupted in some way. Sometimes the problem can simply be resolved by re-staging the data, or by advancing the start time in the analysis script just beyond the time in the data at which the failure occurred, and re-starting the analysis. If neither of these approaches works, then something more fundamental has gone wrong, and the user should seek help.

    GUISDAP Output Files

    On completion of the analysis, the result directory will be filled with matlab files holding the processed data. To obtain a RAL NCAR-format result file, one needs to use the g2n program. One can find information on g2n in Section 13.

    The RAL Analysis Program

    The RAL Analysis Program is used to analyse LDR format data. Hence it is normally used for the analysis of pre-renovation mainland data. The RAL Analysis Program is supported on all of the group's machines, and can be accessed by typing:

    Analysis

    An X-windows based GUI will then appear. By using this GUI to define the analysis process, one can in fact specify all stages from data staging to data analysis. This is because the GUI requests the catalogue and idef filenames in addition to the driver and model files needed for the analysis. The full command underlying this process is in fact

    getLR < {catalogue file} | LDR2int | intpipe -id {idef file} | analyse -m {model file} -d {driver file} -a {ascii output} -e {error output file} -n {ncar output file}

    This software chain is, however, not directly visible to the user. The RAL Analysis Program takes in two input files, namely an ionospheric model and an experiment-dependent driver file, which defines experiment parameters such as the gating and also defines the fit strategy.

    The default ionospheric model file is located in ,code>/soft/eiscat/etc/analysis/model/standard.modl. Software utilities also exist for creating user-specified models, but are beyond the scope of this document.

    Driver files for all the experiments supported by the RAL Analysis Program can be found in /soft/eiscat/etc/analysis/driver.

    The RAL Analysis Program produces three output files - an ASCII file containing the fitted plasma parameters and their associated uncertainties, an ASCII file containing human-readable (but rather cryptic) error information and a binary file of analysed data in "RAL-NCAR" format (also called rslt format) which is a local version of the international standard format for analysed incoherent scatter radar data.

    The RAL Analysis Program is fully documented on the RAL Analysis Program Pages.

    Provided that the analysis has been set up correctly (e.g. by completing a sufficient number of fields in the GUI to define the ionospheric model, driver file and the data input and output files), then pressing the "Go" button on the GUI should be all that is needed for the program to start running. As with GUISDAP, it is possible for the program to crash midway through. If this occurs, either advancing the start time (in the idef file) to a time beyond which the failure occurred, or re-staging the raw data, may fix the problem. If not, then the user should seek help.

    Powerprofile

    The Powerprofile program is a reduced version of the RAL Analysis Program used to produce "power profiles", namely estimates of raw electron density as a function of height derived from zero lag measurements. The Powerprofile program has no fitting algorithm. Instead, it splits the zero lag profiles out from the rest of the data dump, subtracts the background, range corrects them and scales them into approximate values of electron density. No corrections for temperature ratio or Debye length are performed, hence the name "Raw Electron Density".

    As with the RAL Analysis Program, the input is in integrated data format, and the output is in binary "RAL-NCAR" (rslt) format, with a human-readable error message file. Further software modules exist to produce ascii output from the binary output data files. A driver file is still needed to define the data layout and gating information, and can be found in /soft/eiscat/etc/analysis/driver. No ionospheric model is needed in this application. Once again, the program is driven by a GUI, obtained by typing

    Powerprofile

    Filling in the fields of the GUI and pressing the "Go" button should start the program running. Because there is very little complexity in the data manipulation, the program generally runs very fast!

    The Powerprofile program was written to derive power profiles from pre-renovation mainland data. However, it has since been extended to produce power profiles from ESR data and post-renovation mainland data, provided that this data has first been converted to LDR format using the genLR program (see Section 12).

    D-region Analysis There are two d-region analysis programs:

    Both d-region analysis programs are written in FORTRAN and work in a similar way. The program reads the LDR integrated-format data and fits the Auto-Correlation Function (ACF) to and exponential curve. A Fourier Tranfsorm is then performed on the ACF to get spectra, to which a Lorentzian curve is fitted.

    The area under the Lorentzian-shaped spectrum gives a value of the returned power and therefore the Electron Density. The Doppler-shift of the spectrum gives a measure of the plasma velocity and the Spectral Half-width gives an indication of the energy, but given the low heights that this experiment operates, the complicated chemistry makes it difficult (if not impossible) to get any temperature information.

    CP6 Analysis

    The CP6 Analysis program reads in the CP-6 integrated data files, which do not hold ACFs. Therefore, before fitting exponential curves to the ACFs, the integrated data are converted to ACFs. This is done within the CP6 Analysis program so the user does not need to worry about this, but if you are going to look at the integrated data, you will not see any ACFs.

    To run the CP6 Analysis program, type:

    CP6_Analysis

    The same X-windows GUI that is used for the RAL Analysis program will appear and you simply fill in the relevant boxes. The CP6 Analysis program takes in two files: a Correction Matrix file that gives the correct correlator weightings to the various lags in the ACF, and a driver file, which defines parameters such as gating, height ranges etc. The CP6 driver file is much less complicated than the driver files used in the RAL Analysis Program. The Correction Matrix file can be found in:

    ~vikki/CP6_ANALYSIS/cp6_cormat.txt

    The CP6 driver file can be found in:

    /soft/eiscat/etc/analysis/driver/ (on the cluster) or /home/common/drivers/ (on eiscatd)

    As with the RAL Analysis Program the output files created are: an ASCII results file, an error file (in ASCII) and a RAL-NCAR file (rslt).

    As with the RAL Analysis Program and Powerprofile, you have a choice of using: * a TAPECAT-catalogue file and an idef file,

    Fill in the relevant boxes in the X-windows display and press the "Go" button to start the program running.

    A command-line version of the CP6 Analysis program is also available and is called cp6_anal. It has the following format:

    cp6_anal -m ~vikki/CP6_ANALYSIS/cp6_cormat.txt -a {ACSII output file} -i {input integrated data file} -d {driver file} -e {error output file} -n {ncar output file} -s {system} [-vaz {azimuth}] [-vel {elevation}]

    For example, if you wanted to analyse some CP6 data taken on the VHF radar, the dish was point vertically, you had an integrated data file called cp6_120s.int and your output files were going to be called 1990_cp6.{end}:

    cp6_anal -m ~vikki/CP6_ANALYSIS/cp6_cormat.txt -a 1990_cp6.asci -i cp6_120s.int -d /soft/eiscat/etc/analysis/driver/cp6bt_pls2p.driv -e 1990_cp6.err -n 1990_cp6.rslt -s VHF -vaz 0.0 -vel 90.0

    dlayer_anal

    The dlayer_anal program works in a very similar way to the CP6 Analysis program. The dlayer raw data file contains three ACFs and two power profiles:

    Only one set of (ACFs) can be analysed at a time and the dlayer_anal program will only work with LDR integrated data. To convert from the raw matlab files to the .int files needed for the program, you must run the program genLR2 (See Section 12.4.2 for a description). genLR2 will pull out of the matlab raw data file only one of the lag-profile matrices and one power profile and create an ACF from that, which is written to an integrated data file.

    Currently, there is no X-windows interface, so it must be run by using the command line:

    dlayer_anal -a {ACSII output file} -i {input integrated data file} -d {driver file} -e {error output file} -n {ncar output file} -s {system} [-vaz {azimuth}] [-vel {elevation}]

    The driver files used with dlayer_anal are not the same as the CP6_Analysis program. The dlayer_anal driver files can be found in the directory:

    /soft/eiscat/etc/analysis/d-layer/drivers

    There are driver files for each combination of ACF and power profile for the dlayer data. It is also possible to output the dlayer spectra for each integration period to a separate file. Each file would contain the spectra for all 111 height-gates in every integration period. There is a variable in the driver file that turns on or off this option, but please note that if you choose to create files of spectra, there a be a spectra-file for each integration interval in the data you are analysing. This will slow down the speed of the program and fill up your disk space quickly, so use this option wisely.