9: Visualising raw data
As we have seen previously, raw (or integrated raw) data can come into two formats. From the post-renovation KST mainland radars and the ESR radar, the data appear as matlab files. These matlab files can contain raw data, written at the radar at the pre-integration period, or integrated data subsequently integrated to a longer post-integration period using the integrate package. The raw data from the pre-renovation KST radars appear in LDR format, but for visualisation purposes we would first change them into integrated data format, using intpipe.
Visualising Raw (or Integrated) Data from Matlab Files
Of course, the simplest way of visually inspecting raw (or integrated) data dumps in matlab format is simply to load the file into matlab. Doing this, one will see arrays called d_ExpInfo, d_parbl and d_data. In the case of integrated matlab-format data, prepared using the integrate2 program, one would also see additional arrays called i_var1 and i_var2, containing estimates of the measured variance and variables called n_preint and pre_integrated. The value of pre_integrated would be 1, and the value of n_preint would be some value greater than 1, corresponding to the number of pre-integrations that had been added.
The d_ExpInfo array is a human-readable (but cryptic!) array of ASCII characters written to the data file, containing information on the experiment name etc. The d_parbl array is an array of some tens of integer numbers corresponding to the system parameter block. It contains information about time, antenna pointing direction, transmitter power, system status etc.
The arrays d_data, i_var1 and i_var2 (if present) will all be complex, as the data has both real and imaginary parts. To see the real and imaginary parts of the data dump (i.e. power profiles and lag profile matrices) one could give the command
plot(real(d_data));
plot(imag(d_data));
and so on.
This method of visualisation, of course, shows only the basic contents of the dump, such as lag profile matrices for signal, background and calibration. For most users, the data dump would be more intelligible if sorted into autocorrelation functions, or if these were Fourier transformed into spectra. For this purpose, a matlab-based program called rtg (originally standing for real-time graphics) is used. This program comes in two flavours - a program created for post-renovation mainland data (hereafter referred to as rtg) and a program optimised for ESR data (hereafter called rtg3). Whatever happened to rtg2 is a mystery!
The rtg program for the post-renovation mainland system
The program can be found in the directory
/soft/eiscat/share/rtg
If this command is run, matlab is entered and a window with the title "EISCAT - rtg" appears. In order to get some data into the program, choose the widget third from the left on the bottom raw, which probably has a name like "RT Kir". This will give you access to a pull down menu, in which one of the options is "Disk". Choosing this option gives you a window entitled "Pick a dummy in directory", and you can use this by altering the "Filter" to choose the files you want to look at, i.e. /stager/b/joebloggs/raw_data/*.mat.
When you have chosen the directory, a list of files with names matching your filter will appear. The program can be started by clicking on one of these and pressing the "Go" button.
At this point, the program will check the experiment name. Provided that the name corresponds to one of the experiments supported by the software, the program will begin to run. Since this program was designed as a tool for real-time data display, its standard operation is to play through the data (one dump at a time) at a speed dictated by the pre-integration period. Thus, one five-second dump is displayed every five seconds. This is intensely annoying for off-line work (and one day we will get around to fixing it!).
There are various other functions buried in the widgets (like the ability to plot colour-coded time series of the raw data, and the ability to convert from visualising spectra to acfs. The user is encouraged to play with the buttons to get an idea of what the program can do.
The rtg3 program for the EISCAT Svalbard Radar As above, this is a matlab-based program, which can be found in /home/eiscat/ian/matlab/rtg3. After entering matlab and starting the program, a window entitled "rtg" will appear. To get some data into the program, pull down the "Data" menu, and choose the "Filelist" option, together with "Open Directory" in the box that will appear to the right of the menu. A window titled "Select a Directory" will appear, which can be used to choose the directory containing the raw data. The "Select a Directory" window will now show the number of data files found and the start and end times of the raw data contained in the selected directory. If the "OK" option is selected, the data will begin to be loaded into the program.
The first question to be established is whether these data come from an experiment (gup0, gup3 or tau0) recognised by the program. If this is so, a second window will then appear, initially displaying the data in "s0,g0" form (a zero lag profile and the associated gain profile).
To change the display, pick up the "s0,g0" widget, and change it to something more informative, such as "SP r^2" which produces a wire plot and waterfall plot of range-corrected spectra. The other widget in the right-hand window (labelled "AC1" if you have selected tau0 data) can be used to control the modulation from which the data is displayed. Since tau0 consists of two alternating code schemes covering different height ranges, settings of "AC1" and "AC2" are possible when plotting raw data from the tau0 experiment. The contents of any of these windows can be printed as hardcopy by selecting an appropriate option from the "File" menu depending on the desired plot file format.
A very useful feature of rtg3 is the ability to "play" one's way through the contents of the data directory, displaying sequential dumps like a movie. To achieve this, choose the "Data" menu and take the "Player" option. Using the "Define" option allows one to define the extent and duration of the data period (defaults are forward direction and infinite length) as well as the time to pause between adjacent displays (Note: this last feature may not always work correctly). Choosing the "Play" or "Re-Play" options will start the play-through functionality. Note that you have to "Stop" this before any other property of the display can be changed.
The "PI" menu can be used to specify an integration period that can be applied (dump wise or time wise) to the data. The "Integrate" option in the "Data" menu causes this integration to be implemented.
The rtg3 program has some other nice functionality, although not all of it is currently supported at RAL. The user is invited to try it out for himself or herself to see what is available.
Raw Data Visualisation for Pre-Renovation Mainland (LDR format) Data
RAL has no visualisation software for raw data in LDR format. In order to visualise these data, one has first to convert them to integrated data format. Since this conversion is also needed in order to do any other work with these data (such as analysis) this should not be too much of a hardship.
The raw data dump itself from any experiment (power profiles, acfs and/or lag profile matrices) can be viewed in idl, using the software modules int_hdr and int_record. These are used as follows:
Suppose you wrote your summed format integrated data file (using intpipe) to a file called data.int. To look at this file in idl, you need to do the following:
idl
openr,1,' data.int'
hdr = int_hdr(1)
data = int_record(1,hdr.format)
plot, data.ldrs(*)
As above, this method of looking at the data may not go very far. In order to see the acfs, spectra etc, one should then read these data into one of a number of experiment-specific idl routines which can be found on the directory /home/common/idl (on eiscatd) or /soft/eiscat/shlib/idl/integrate (on the cluster).
For instance, suppose that the above data came from the VHF CP6 experiment. One could use the program cp_6_b_v.pro to create a data structure containing the acfs and power profiles and plot them as follows:
a=cp_6_b_v(data.ldrs)
plot,a.ACFS
plot,a.PP
To produce plots of spectra, one could Fourier Transform these acfs using the idl routine acf_to_spec.pro
spectra=acf_to_spec(a.ACFS)
plot,spectra
This software lacks the ability to play through extended data sets like rtg3, and the integration period can only be changed by varying the integration of the input data. However, these routines provide the basic functionality that users should need to visualise pre-renovation mainland data. The following modulation schemes are supported: cp1ht, cp1jt, cp1kt, cp4bv, cp6bv, cp7dv, cp7fv, which together represent a large proportion of the pre-renovation data set. It is not too difficult to develop software for most other similar pre-renovation experiments.