How to start and monitor the online PALFA data reduction on the ASP. Notes by DRL/DJC, 12 August 2004. (small update for viewer filter by Joeri, 7 April 2005) (modified by DRL, 31 July 2005) Overview -------- There are two main processing schemes currently implemented -- the so-called "quick" option which ftps, demultiplexes and searches the resulting individual pixels at low resolution (32 x 3.125 MHz channels and 1.024 ms sampling); and the "long" option which searches the data at full resolution (256 x 0.39 MHz channels and 64 us sampling). The "quick" processing must be run first and is designed to work while the data are being collected on the WAPPs. The "long" processing can only be run once the "quick" processing is complete and assumes that the demultiplexed data are on the WAPP. Currently a standard FFT (seek) and single-pulse searches (Cornell code) are run on the quick pipeline, while the long processing has just an FFT search performed. The results are written to /home/alfa/realtime/results on the ASP but are best viewed through the php web server (see below). NodeStatus ---------- SSH into aspmaster as user alfa. Paulo or any of the ASPers will tell you the password. Type "aspmon" and a monitor/control GUI will pop up. Check that the current state of all the processors is "stopped on request" or ends in "done". It is worth checking that the processor usage is close to 0%, this is shown for each node on the right side of the GUI. If this is not the case, there is probably something still running and you should hit the "Stop" button. Before starting the processing, make sure the local disks on the nodes are clean by hitting the "Clear Files" button. Check the disk space on each node. If the number is highlighted in yellow then there is not enough space and this node will not be used. If it is highlighted in green then there may not be enough space for the whole run and may be not be used later. You are now ready to start processing. Hit the "Start" button and you should see the four ftp processes shown at the top change to blue and say "waiting for data". The colours of all the nodes should turn blue and end with "done". Once your first scan is finished, they will start to copy data onto the local disks. Processing will begin once files are completely ftp'd over (it takes of order the integration time to ftp the files). Below the disk space and cpu monitor of each node there is a summary of the files waiting to be processed. It is in the format: files in pool - split files (incoming files). If the files in the pool becomes > 5 it will highlight yellow. This may occur when a process is unable to finish properly, check that the processor usage is at 0.0%. It can usually be fixed by 'echo "none done" > /palfa/node?/status/status.proc?'. For example, to manually clear the two processors "a" and "b" on node 5, you would do the following: echo "none done" > /palfa/node5/status/status.proca echo "none done" > /palfa/node5/status/status.procb As each file is successfully copied onto each node it will automatically begin processing. You can look at the results by opening another window on aspmaster and running the php web server by hitting the "View results" button (see below for further details about viewing the output). After the observing has finished the processing will need time to catch up. When all the ftps and nodes turn blue the files should read 0 - 0 (0) and the processor usage should drop to 0%. At this point you can click "Stop" and in ~1 min all processes should turn yellow and report "stopped on request". If you wish to do the long processing you can click "Long processing" once you have finished the quick processing. The split files are copied to the pool and processing occurs as before. Processing is ended in the same way as with the quick processing. If you wish to kill all processes immediately use the "Kill" button. This may take several 10s of seconds to run since it has to ssh into each node and kill any relevant processes. Once this is complete, you should click on "stop" and then either "start" or "reprocess". Notes: The CPU usage lags the GUI by ~40 secs If you quit the GUI processing continues After "stop" has been clicked some processes may continue for several mins (up to 10 mins for the reprocess) so the CPU monitors provide a good check. When these processes finish they will report "done" overwriting the "stopped on request" Candidate Viewer ---------------- The candidate viewer can be run from any browser on any computer on the AO network, not just ASP. Enter the URL http://aspmaster.naic.edu:8080/alfa/viewer.php and it will show you the results for today. At the top you can choose to see results for a specific MJD and scan number range. This is useful as the more pointings that are displayed the longer the page takes to build. You can also choose to view the quick or long processing. There is a box where you can enter your name or initials. This will allow you to keep track of the beams which you have viewed. NOTE: You need to click "update" after you make a change in the Min/Max scan number or name boxes. The pointings are listed below. You can view the candidates of each beam by following the link. You can jump to the ps files of the top 3 candidates and single pulse search by following by the links below. Any beam which you have viewed will be subsequently highlighted in green. In the 'view candidates' page the top 10 candidates and the result of the single pulse search are listed. Clicking on the figure will link you to the full ps file. If there is a nearby pulsar (within 10 arcmin of the central beam) in the pulsar catalogue it will appear at the top of the page. Plots for candidates that fall in one of the Pmin Pmax SNRmin SNRmax DMmin DMmax ranges defined in the ~alfa/.rfi file are replaced by a "filtered signal" link to the plot to allow some very basic customizable filtering. If you've entered in the top-right corner box, the filter will use file ~alfa/.rfi in stead. At the bottom of the page there are several links. "list" returns you to the pointings list. "Next / Prev Beam" will take you to the next / prev beam and will wrap to the next / prev pointing if necessary. "Next / Prev unviewed beam" will take you to next / prev beam which your username hasn't viewed, i.e. does not have a green box. NOTE: If you go "back" rather than using the "list" link the green boxes may not appear to be updated. Click reload/refresh to update. If you use the same username from a different browser or at a different time the beams which you have viewed will be retained. NOTE: Wapp2 is the slowest to transfer so beams 2 & 3 are often the last to arrive. NOTE: creation of the PNG files is more robust than creation of the PS files so occasionally links to the ps files may be missing,