WAPP, Wide Band Arecibo Pulsar Processor

This page is Jeffs notes on the status of the wapp. The wideband arecibo pulsar processor. The WAPP Home Page is too difficult to change.

This page last updated on 6 Oct 2004.

Latest Fixes and changes 6 Oct 04

A new keyword to the pulsar catalog observing has been added:

OFFSETS ddmmss ddmmss units
Units can be either "radec" or "azza". If its radec, then its hhmmss and ddmmss, otherwize its azza in degrees. These offsets will be applied to all subsequent sources specified in the list.

Latest Fixes and changes 30 Sept 04

Latest Fixes and changes 17 Sept 04

The WAPP header was updated today to include the position of the antenna at every alfa beam, the alfa rotation angle, the alfa az/za offsets from center,the paralactic angle at the zero beam and receiver name. The added variables are:
    double rfeed_offaz[7];  /* deg az rotated offset all alfa beam to center */
    double rfeed_offza[7];  /* deg za rotated offset all alfa beam to center */
    double prfeed_offaz;  /* deg az offset to center of alfa beam */
    double prfeed_offza;  /* deg za offset to center of alfa beam */
    double alfa_raj[7];   /* hr starting actual ra position of alfa beams */
    double alfa_decj[7];  /* deg starting actual dec position of alfa beams */
    double alfa_az[7];    /* deg starting actual az position of alfa beams */
    double alfa_za[7];    /* deg starting actual za postion of alfa beams */
    double alfa_ang;      /* deg alfa rotation angle  */
    double para_ang;      /* deg paralactic angle of center beam */
    char frontend[24];    /* receiver abbrev name */

    unsigned char prfeed;  /* centered alfa beam */

Also, there is a new observing screen under Pulsar Observing => More => Pulsar Mapping Modes

Included here are the familiar modes as well as two new ones for the PALFA team. Alfa Beam Check and ALFA calibration check.

Latest Fixes and changes 2 Apr 04

Today I changed the folding and spectral line FFT algorithms to use the lags+1 points with data[lags] = 0.0. This will give the frequency channel spacing of bw/n points which was assumed by the old pre-september algorithm. I changed the foldfft routine to do this as well.

Just as I did this today Jon Hagen produced a document describing a rigorous fft for this problem. Let me take the time to understand it - so the algorithm may change again...

Latest Fixes and changes 29 Mar 04

Latest Fixes and changes 10 Dec 03

Small change to WAPP start and stop controls

11 Nov. I changed the start and stop controls on the WAPP pulsar observing. Now query observation will return a reasonable message. Also power level setting controls get through so IF system levels can be set when the monitor is running. Careful - this also means that levels can be set while observing. I will leave it for now. I can limit this if it is deemed a problem. Please tell me if these fixes have broken something that I havent tested.

Small change to NON-WAPP pulsar iflo setup

Selecting "non wapp iflo setup" now always adjusts power, if1 and if2 after setting up the iflo. This matches what the old GUI did. Capable observers should be checking power levels under IF/LO Control anyway. Observers who dont check the power will now be saved, usually, depending on where the telescope is at the time of selection.

WAPP Pulsar Header Changed

17 September 2003. The WAPP Pulsar header changed. A complete state of the IF/LO system is now included in the the header. Most software should be unaffected as it only adds a number of values to the end of the header. The version number was increased to 6. (The comments are pretty good, but I will document the details later.)

Data reduction programs that hard-code the header size will break. See WAPP Data File Support below for a description of the Linux library routines that I offer to make it easy to generally find header values and data. I'd be happy to help Solaris users or others having trouble.

And another thing to mention. The folding and spectral line modes now use fftw3 (rather than fftw2) to do the lag to spectra signal processing. (You shouldn't notice except its faster.)

Gigabit Ethernet installed on WAPPs

Gigabit ethernet is now installed on all wapps. A 2 Gigabyte file transfered at 17 Mb/sec using the nfs crossmount. It transfered at 32 Mb/sec using ftp. Like I thought, use ftp and not nfs. August 2003

WAPP errors discovered

17 July 2003. We discovered a serious bug. The WAPPs have not been triggering properly as some observers have guessed. We believe this has been the case for the period:
Tuesday 8 July 9:00 am till 16 July at 6:00 PM.

A series of failures occurred.

The start times for the data taken are not precisely known but in the absence of network overhead, the starting time should be about 0.5 seconds sooner than reported.

I apologize to those who lost data. I'm sure makeup time can be found.

I discovered another bug related to the OS upgrade. There was a timing problem in the edt driver. When this occurs part of the correlator hardware configuration would have waited for an entire driver timeout of 10 seconds before the configuration completed. This was intermittent and never seen in the offline diagnostic software. This bug would have caused the delayed start that we saw. Fixed now. If the wapps were triggering correctly then the start times, although delayed, will reflect the correct starting second.

This may be the elusive "network" problem we have been hunting.

WAPP pulsar setup screen changed

14 July 03. The Apply WAPP Setup button was replaced with two buttons. The Sanity Test and Setup IFLO. The Setup IFLO button should be pressed after making changes to the IFLO setup especially the desired frequencies.

Here is what the buttons do:

New features added to Catalog Observing

10 July 03 Two new keywords were added to the third field in catalog observing. "toward" and "cal_toward". These allow you to start the wapp (with or without cal) while slewing to the next source. Also the caltype (which noise tube) and whether you want the 25Hz cal or the continous noise is now an option on the catalog observing GUI. Details below.

WAPP upgrade complete

7July 03, we completed the upgrade of the wapp os to Red Hat 8.0, 2.4.18-14 kernel. The wapps are up and running. Some details for those interested:

WAPP demultiplexiing bug found

For 3 level 50 MHz polarization modes (both 16 and 32 bit) there was a bug discovered in the demultiplexing routines. All previous data taken in this mode is suspect. The data had the correct polarization but not from the same correlator dump. fixed 18 June 2003.

WAPP Folding Mode working

Late June 2003 David Nice, Ingrid Stair and others helped me debug the WAPP folding mode. Should be working. Thanks for the help.

WAPP File Name Convention Changed

On 17 June 2003 we changed the wapp file naming convention. The wapp file names are now, /share/wapp11/project_id.source_name.wapp[234].xxmjd.xxxx The file extension is now 4 digits and is global to all wapps and wapp disks. Now all multi-wapp observations will now start with the same file extension number.

Four WAPPS Operational

Four WAPPs are now available for pulsar observing. Four freqencies over a range of bandwith up to 100MHz (each) are available to observers. The four WAPPs are accessed from the CIMA User's interface. The machines are named wapp wapp2 wapp3 and wapp4. The WAPP capabilities are spelled out here.

All four WAPPS now have reasonable disk space and are usable from the CIMA User's interface. Four WAPPs can be used for observing although there probably remain bugs related to things we havent thought to try. Most people who have tried have had some success. It seems that most trouble comes with configuring the IF system.

The old gui had an Auto-Observe feature and two scripts were used to take some calibration data. These are /share/obs4/usr/pulsar/WAPP.CAL and WAPP.PSR. These are now written into the CIMA executive and are callable from the WAPP Pulsar screen->More

These are in. Ramesh has tested them to his satisfaction. The source names are appended with _north _south etc... If you have other auto-observe features/scripts/parameters that you want to include, tell me I will add them for all to use.

For a while they didnt abort properly. Should be fixed now.

Pulsar Calibration Script -

Pulsar Pointing Calibration Script-

Here is a observing script that I wrote for Fernando Camillo. Please use it if its suits your needs. This script will take WAPP observations from a list in a command file. It moves the telescope to the given source and then takes wapp data for the given number of seconds. The observing mode looks at the current source name, trys to find it in the command file and starts the observation there. In this way a command file can be started in the middle.

WAPP Pulsar Catalog Observing-

Late January 2003 the wapp11 raid drive failed completely. We reconfigured it with 4 drives. Its smaller but it is working.

There is a bug in the LO configuration that assumes wapps enabled sequentially. If you wanted to enable wapp four only, for example, do the following workaround:

Trouble Shooting Suggestions

In Pulsar mode, if you press "Apply Current Setup", and you dont see "sanity check passed" for each selected WAPP, there is a problem. When other obvious things dont work, try restarting the wapp software. There is a button under Pulsar=>WAPP=>More called "Restart all WAPPs". Another thing to try is to login to the afflicted wapp with "rlogin wapp -l wapp". After login, you can restart the wapp software with "start_wapp".

Its almost never the case that having the operator reboot the wapp will help. But there is one case where I know how to hang the driver so that only a reboot will fix it. Really, I'd like to check it first. With the requsite terabye file-system-check that will result, a cold reboot will take as long as calling me.

Improvements to SNAP

I made a few changes to SNAP. Snap now runs on most any linux machine. Here is how to run it on WAPP4.

xhost wapp4
rlogin wapp4 -lwapp
DISPLAY=observer2:0.1
export DISPLAY
cd /share/wappsrc/snap
snap
I changed the pulldown on the upper left so that you can choose the wapp you want to view. Remember to choose a wapp thats is active - either running monitor or daking data and it should switch just fine. Otherwise it wont do anything, of course. If you arent using all 4 WAPPS at once you should run snap on one of the WAPPs that you arent using. Its CPU will be all yours.

Snap was slow because we "upgraded" the van vleck correction routines in the lib area so that it used a more rigorous algorithm. I restored the old Andy Dowd algorithm to snap. Dont believe what snap does to the 9-level data. It will give you an overall reasonable filter shape, just dont believe the fine details. Snap still doesnt display properly in folding mode.

Direct Sampling Available

The new WAPP Direct Sampling mode is available but has yet to be independently tested. For the Mode choose Sampler instead of Search or Folding. The Bandwidth, 50 MHz or less is the only respected parameter. It takes data until you tell it to stop. It does this quietly. ( I need to improve this. Maybe this mode needs its own screen. I dunno.) The data will be saved as search data bit-packed depending on the bw:

2 channels,continuous sampling

Bandwidth(MHz) Data Rate Mbyte/s Bit Packing
50 25 1
25 25 2
12.5 25 4
6.25 12.5 4
3.125 6.25 4
1.56 3.12 4
0.78 1.56 4
0.390 0.78 4
0.195 0.39 4

Direct Sampled Data Format

The direct sampled data is similar to WAPP search data format. It writes multiple files in 2GB chunks. The binary data follows the header. I'm sure improvements to the header will be necessary. Please suggest some. The data structure from Bill Sisk...

Notes for Pulsar Observing with WAPP

Here are a few notes for observing with the CIMA and pulsar WAPPs.

I must admit I'm not that thrilled with the human interface of this screen. Please be patient. I will be making changes as they become obvious.

All WAPPs now have digital filters

All four WAPPs now have digital filters installed. These are available for pulsar observing. Bandwidths less than 50Mhz are now supported.

Disk Status/Tape Status

wapp disk drives:

wapp currently has no tape drives. wapp2 disk drives: wapp3 disk drives: wapp4 disk drives:

All drives are now exported with the same name as on the machine. The 700GB Raid drives were measured at 0-10% full, 34 Mb/sec. Over 10% full 38 Mb/sec.

WAPP data file support

Here are a few notes on the wapp data file structure and the support routines. The wapp data files for pulsar observing have the following structure:

There are a number of procedures provided that allow the user to look at this data.

These routines all use an underlying library that makes reading and parsing the header easy. For example,


  #include "/share/wapp/include/key.h"

  struct HEADERP *h;
  struct HEADERKEY *key;
  char *file = "/share/wapp1/wappfile"; 

  h = head_parse( file );

  find_hdrval( h, "header_size", &header_val );
  header_size = *(int *)header_val.value;

  find_hdrval( h, "num_lags", &header_val );
  num_lags = *(int *)header_val.value;

  find_hdrval( h, "nifs", &header_val );     /*-- Number ifs? --*/
  nifs = *(int *)header_val.value;

  find_hdrval( h, "lagformat", &header_val );  /*-- 32 bit mode? --*/
  lagformat = *(int *)header_val.value;

  close_parse(h);

find_hdrval returns 0 on success. Use head_parse to open and parse out the file. Then use find_hdrval to read the data into your variable. If you want to generically loop through all the header variables try this:

  key = h->head;
  while( key ) {
    dosomething
    key = key->next;
  }

To link these routines link use /share/wapp/lib/libwapp.a. The source is there as well. The relevant files are:
     y.tab.c: mkheader.y
             yacc -vdt mkheader.y

     test_parse: test_parse.o head_parse.o y.tab.o
             gcc -g -o test_parse test_parse.o y.tab.o head_parse.o
Look at the Makefile for more details. I encourage users that process their own data to use some variant of these routines as they will smoothly handle any future changes to the wapp header. Look at /share/wapp/utility/asciidump.c for an easy practical example.

New WAPP utility

There is a new utility in /share/wapp/utility called:
ndumps file
This procedure returns the number of dumps - integrations in time - contained in the file. I should also remind observers that there is a header parameter called "timeoff" that shows the number of dumps recorded in previous files. Its should be noted that the file size is a poor statistic as the header file and header can change at any time.

To recover timeoff from a wapp data file
/share/wapp/utility/asciidump file | grep timeoff

WAPP Software Bug discovered

A subtle bug was discovered (and cured) on 24 September 2002. It only effects 1 If 3-Level 100 Mhz mode. The demultiplexing was being done incorrectly, but in a subtle way so that normal 100 Mhz filter shapes came through. When we happened to send a 50 Mhz signal though the wapp we saw a bi-modal filter shape which is an artifact of the bug. I doubt many users have used this 1 IF mode. Unfortunate victims should contact me for the details. It will not effect summed modes or 2 IF modes.

Data Archiving Policy

It is the responsibilty of the observer to backup and remove the data. The telecope operator will be happy to put tapes in the machine if you are remote. If there is no space for your experiment, that is your problem. There is a utility called wappstats that will tally project disk usage on the wapp.

Release Notes

Thu Jan 9 2003

There was trouble reported with multiwapp timing. We took some data with an artificial pulsar piped into all wapps, 64 lags 64 usecs, with a period 0.8 seconds, and a 20% duty cycle.

/share/wapp11/timingtest2.wapp.52648.002
/share/wapp23/timingtest2.wapp2.52648.002
/share/wapp31/timingtest2.wapp3.52648.001
/share/wapp41/timingtest2.wapp4.52648.001

When I fold it into 100 bins I get one changeover bin at bin 19 from all sets of data. This tells me its working. Maybe others can see something wrong with this data or suggest a more robust timing test.

wapp4 now supports narrow band observing. wapp4 with cima now supports all designed narrow bandwidths. Filters for the other three are being built and tested.

The wapp Mammoth tape drive has been removed for repair. The DLT drive is now /dev/nst0.

The timing bug is still there as of Jan 7, 2003. We are looking. Also please note the disk drive changes. wapp12 => wapp24 ...

There was a timing bug reported by Scott Ransom in WAPP4 where WAPP4 was not triggered to start correctly. It was a cable mix-up. This was fixed on Monday 30 Dec 2002. Before this WAPP4 was not reliable on its start trigger.

With the advent of multi wapp and the CIMA gui, wapp filenames are changed to include the hostname of the wapp that took the data. The the wapp header version changed too to include the hostname in the header. So expect, wapp wapp2, wapp3 wapp4.

The CIMA wapp control was incorectly setting src_ra and src_dec to hours degrees instead of hhmmss.s and ddmmss.s. Should be fixed after 23 Dec 2002.

People are always asking me about the maximum performance of the WAPP. The limit is 45 Mb per second out of the correlator and 24 mb/sec to the hard drives. The correlator can trade lags for speed and the formula is:


lags * bpl * chans * interleave * ninelev = msecs * 45

  lags is the number of lags desired
  msecs is the wapp dump time in microseconds
  bpl is bytes per lags ( 2 for 16 bit mode, 4 for 32-bit mode )
           8-bit truncation mode is the same as 32-bit 

  chans the number of channels, summed mode is still 2 channels.

  interleave is 2 for 100mhz or 1 for the other bws.
  ninelev is 4 when nine level mode is selected and 1 if 3 level.

The wapp now supports all of the 32 bit modes, and all 16 bit modes. It supports the bit selection mode which allows the user to select 16 of 32 bits for longer integration times. This is the equivalent of averaging. Duncan Lorimer looked into this with a few tests: 32-bit tests The 8-bit trunctation mode is installed as of April 2002. The observer must be careful to choose the correct truncation bits to make sure the data is valid. This is well indicated by the power level returned. If you run monitor in 32-bit mode, a bit selection recommendation is given to help you choose to truncate your data. Also a 4-bit truncation is planned. Take care with this as the correct trunctation is power level dependent. There must be a better way to select this. I welcome comments. jeffh@naic.edu

The "WARN low power could indicate 16 bit overflow" message was removed in favor of changing the power level color.

On 25 February 2002 the wapp computer was upgraded to a dual 1Ghz pentium intel server running redhat 7.1. Now all four wapps are this model.

The wapp now supports folding. The header can accomodate up to 16 entries of up to 12 polynomial coefficients each. Only one polyco file is necessary for multiple days of observing and multiple sources.

There was a bug in the GUI where the project ID was not being written to the wapp header. This was from 20 Feb 2002 till 28 Feb 2002.

Known bugs:

I had some questions about the accuracy of some of the times in the wapp header for folding mode. I think they are accurate: