##########################################################
# Python scripts for interfacing with the PALFA database #
##########################################################

by: Jason Hessels (hessels@physics.mcgill.ca)
last updated: August 5th, 2004

(updated on August 9th, 2004 by Beth Reid breid@princeton.edu)

Here is a quick description of some of the scripts Beth, Ramach and I
have been developing to interface with the database:

NB: To run a python script on fusion00, use '/usr/bin/python' instead
of just 'python'.  Otherwise, you won't be using the right version of
python.  If you still get errors, you may need to add some
library paths to your .cshrc/.bashrc file (see ~jhessels/.bashrc).
If you really can't get it to work, you can try going into bash and
then sourcing ~jhessels/.bashrc.  For each of the scripts, if you
type /usr/bin/python 'scriptname' --help you'll get the list of 
command line flags you can use with that particular script.
All the scripts are in /share/p1739.2/PALFA.  The scripts are under
heavy development and will get fancier and more robust with time.
We also need to comment the code (and clean it up in some areas!).

1) "fill_observed.py": This parses /share/obs4/usr/pulsar/p1944/p1944.cimalog
to get all the information about the observations and then reads the information into the
observations table of the database.  Usually I copy p1944.cimalog AND
P1944.cimalog (people have been writing to both) from
the /share/obs4/... directory to /share/p1739.2/PALFA after each observing
session and then run:

fusion00> /usr/bin/python fill_observed.py --passwd 305m_adm

This has been updated to accomodate (possibly unintentional) pointing offsets
 using David Nice's code alfawhere.  We'll still need to update this file when
 the beams are able to rotate.      

I've updated so you have the option of checking schedule for known pulsars at
the same time (and I run from my home directory where there are the latest
copies of BOTH p1944.cimalog and P1944.cimalog -- people have been writing to
both).  You can also run that check on its own (see below for options)

fusion00> /usr/bin/python /share/p1739.2/PALFA/fill_observed.py --passwd 305m_adm --check G 


2) "fill_prelimproc.py": This is now run automatically as part of Dunc and
David Champion's preliminary searching of decimated data on the ASP.
In principal, the tables 'prelimproc' and 'longerproc' are updated automatically
every time there is 'realtime' processing of a pointing.  If the realtime
processing is processing a pointing and beam that is already in the table then
this is deleted from the table and overwritten with the new results.
We may or may not want this to work this way.  Dunc's code is currently 
calling fill_prelimproc.py from a copy on 'asp' in ~alfa/bin.

Regarding fill_prelimprocplus.py realtime checking with known candidates is not possible, since we
update the observations table from the cimalog file after the fact (this
contains all the telescope pointings needed to compare with known pulsars).

I also made scripts man_fill_prelimproc.py and man_fill_prelimproclong.py that
input a file containing a list of the Dunc/David text files to enter into the
database, so that the database can be updated after the fact if necessary.
That file was generated with the find command: 
find . -name "*.txt" (where . indicates starting directory -- works
recursively)

This has to be run on ASP, because that's where the data is.  

3) "plot_prelimcands.py": This does plotting of the cands in the prelimproc table.
One can plot period, SNR, time, DM versus any of these quantities.
For instance, one can plot the candidates in period vs time and have a marker for
each candidate whose size is proportional to the SNR of that candidate.
To do that, type:

fusion00> /usr/bin/python plot_prelimcands.py --passwd alfa_305m --zaxis SNR --mjd 53221

You can also plot candidates from individual beams using --pixel and the beam number.

3b) "plot_prelimcandsint.py": int stands for interactive.  It cues you for
things you can do/plot.  There are lots of options and it's rather messy, but
once we figure out what's nice to plot we can go from there.  Here are a few
examples:

To start running:

fusion00.naic.edu>/usr/bin/python plot_prelimcandsint.py --passwd alfa_305m

The first option is to use mysql commands directly to generate columns you'd
like to plot. Here's an example.  This will find how many occurrences for a
particular scan/beam_num (goes into the field count(*)) where the pointing_ID
begins with G (so it's not pointing at a known pulsar), and where the period
is not in the now-famous 118 - 122 ms region where we've been getting lots of
hits.  the 'having count(*) < 20' returns only entries with less than 20
counts.  

Enter a mysql query if you want (default none): select prelimproc.obs_ID,
beam_num, count(*) from prelimproc, observations where prelimproc.obs_ID =
observations.obs_ID and observations.pointing_ID like "G%" and
(prelimproc.period < 118 or prelimproc.period > 122) group by obs_ID, beam_num
having count(*) < 20;

Enter x axis column: 2 (specifies x values are in column 2 -- they're the
counts)

Enter xmin (default to min(x): 
Enter xmax (default to max(x): 

Enter y axis column (-1 for histogram, -2 for lnhist): -1 (we want a
histogram; otherwise you can specify another column, or -2 gives a histogram
with logarithmic axes)
 
Enter # of hist bins (default 20): 

Enter z axis column (0 for none): 0

And there you have it.

Another option is to scan through a day scan by scan looking at a particular
plot.

Would you like to scan through an entire day? (y or n) y
Enter mjd to scan: 53225
Lower limit on scan # (default 0000): 0000
Upper limit on scan # (default 9999): 0030
Enter pixel (return for all): 
Enter a minimum DM to extract from database (default 0): 5
Enter a maximum DM to extract from database (default 1000): 
Enter a minimum SNR to extract from database (default 0): 
Enter a maximum SNR to extract from database (default 100000): 
Enter a minimum period to extract from database (default 0): 1
Enter a maximum period to extract from database (default 100000): 1000
Enter x axis (period, frequency, SNR, DM, time): period
Enter xmin (default to min(x): 
Enter xmax (default to max(x): 
Enter y axis (period, frequency, SNR, DM, time, histogram, lnhist): DM
Enter ymin (default to min(y): 
Enter ymax (default to max(y): 
Enter z axis (period, frequency, SNR, DM, time, default none): SNR

And finally let's make some regular plots:

Enter x axis (period, frequency, SNR, DM, time): time
Enter xmin (default to min(x): 
Enter xmax (default to max(x): 
Enter y axis (period, frequency, SNR, DM, time, histogram, lnhist): frequency
Enter ymin (default to min(y): 
Enter ymax (default to max(y): 
Enter z axis (period, frequency, SNR, DM, time, default none): SNR
Enter mjd (return for all): 53225
Enter pixel (return for all): 3
Enter 4 digit scan number (return for all): 

4) "plot_pointings.py": This plots the centres of the pointing grid positions (which
are in the table 'pointings') along with the pointings that have been observed (which
are marked in red for the central beam and green for the six outer beams) and the known
pulsars (plotted as yellow stars).  One can change the size of the viewing area to
zoom in on a particular part of the sky, and you can plot the pointings in either
galactic or equatorial coordinates.  Eventually we will write a utility to do zooming
by selecting an area with the mouse... it isn't quite that fancy yet.
To get a nice plot, type:

fusion00> /usr/bin/python plot_pointings.py --passwd 305m_adm --ymin -2 --ymax 2 --xmax 55 --xmin 51

...can you see where we've observed the first PALFA pulsar numerous times?

5) "makeobsscript.py": This isn't as sophisticated as it needs to be yet (especially
since we can't rotate the feed!), but the idea is that this script receives an
MJD and session length and using the database determines what is in the sky to observe 
(that hasn't already been observed) and then prints out an observing schedule that the
observer can use in CIMA.

6) "check_schedule.py": This looks through the 'observations' table and determines
what known pulsars were close to where we have pointed.  The current limit for
'close' is 6 arcminutes away (you can change this by adjusting the variable at
the top of the file, beam_cov).
There are several useful options:
--scan: select a particular scan you'd like to check.
--mjd: only check a particular mjd
--startmjd: include everything after and including this mjd
--Letter: 'G' means look only at observations with pointing ID beginning with
G, meaning it's not a known pulsar already.  'BJ' looks at pointing ID's
beginning with letters B or J.

fusion00.naic.edu>/usr/bin/python check_schedule.py --passwd 305m_adm --Letter G

This can also be run from fill_observed.py -- see above.

7) "check_known.py": Checks through either prelimproc or longerproc in the
database (choose which one with flag proctype), and looks for entries with
similar ra, decl, and periods as known pulsars.
Options:
--scan: select a particular scan you'd like to check. 
--mjd: only check a particular mjd 
--startmjd: include everything after and including this mjd 
--pixel: only look at a certain beam_num
--N NULL will delete current entries in the known column (in case you've got
corrected ra/decl's in the observations), but only for the scans you're updating. 

fusion00.naic.edu>/usr/bin/python check_known.py --passwd 305m_adm --startmjd
53225

Script development has just begun, but I think we have some of the fundamental code
we need.  We'll have to discuss what other scripts are needed soonish...