File contents

Wednesday 21st October 2009   APM Meeting tables    11:30-13:00

Present:        MJI, RWA, EGS, JRL, JPE, MR, SC, STH 
Apologies:      NAW, RGM

Agenda
------
1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, VST .....
6.  WFCAM update
7.  VISTA update
8.  AOB - grant status etc..

D'oh!
-----

MJI forgot to officially welcome JPE to the CASU meetings and also to 
congratulate him on his choice of the IoA for his Sabbatical.

Minutes
-------

1. Actions from the last meeting

MJI  stage I software release on hold due to other pressures - ongoing     <<<<
EGS

MJI  ditto technical paper - ongoing                                       <<<<

MJI  ditto utility to convert binary tables plus corrections - ongoing     <<<<

JRL  duly wrote his ADASS talk and delivered it in multiple ways

STH  briefly described the Palomar Transient factory. Managed by Nicholas Law, 
     it uses the 48-inch Oschin Schmidt to do an optical survey of the sky in 
     8 square degree snapshots and has so far produced two papers.

MJI  still comparing WFCAM dome flats with twilight sky flats. Low light   <<<<
     level linearity sequences have also been taken for both WFCAM and VISTA 
     and it will be interesting to see how the linearity properties compare 
     to WFCAM

EGS  has made the Plone system entry to the top level pages for VISTA similar 
     to that of WFCAM

EGS  checked out and updated the WFCAM survey progress web pages

RWA  has inserted the VISTA performance specs emailed out by Magda on the 
     VISTA technical pages

SC   said that the Chubb engineer has not yet visited the Kavli so no 
     progress on the high temperature monitor. Another discussion with 
     Paul Aslin is required.                                               <<<<
 

2. Comments on the WFAU minutes

DR6 happened about a week ago.  There are still some holdups with the DR6 
version of GPS and UDS, partly due to us and delayed reprocessing and partly 
due to one of life's eternal mysteries.  WFAU are getting a bit itchy and
scratchy about up to date examples of processed VISTA FITS files and after 
a brief discussion EGS was volunteered to make sure the night of September <<<<
27th (all calibration pawprint OBs as it happens) is available for slurping
after it has been checked.

After much CASU itching and scratching the format of the FITS tile images 
has finally been decided upon and MJI will ensure an example of this wends
its way down the UKlight fibre shortly.                                    <<<<

It was noted that all the VISTA FITS files magically adhere to the FITS 
standard, even the catalogues.


3. Recent and upcoming meetings

There was a VISTA IOT on October 6th in which among other things science 
verification and scheduling was discussed. SV started on October 15th and 
runs until the end of the month. ESO SV team members are attempting to 
produce science quality data at the telescope but JRL thought this was not 
going to happen without a lot of extras.  We expect the first batch of SV 
data at the end of October and will process the data here from scratch as 
per usual.  The plan is then to visit ESO late November to discuss the data 
processing science outputs directly with the ESO SV team.

JRL reported that his ADASS trip was very successful, particularly in 
discussions after his talk when he was subjected to many questions about 
VISTA and the CASU pipelines.  He also had many useful conversations with 
ESO attendees and even better, his mysterious back injury dissipated through
a combination of immobility and Sake.

MJI and NAW went to a EUCLID meeting in Durham. Although this is notionally
a Dark Energy mission (weak lensing, BAOs etc..) its proposed three cameras 
are of interest to a range of auxilliary science, particularly a near-IR
imager which will scan 20000 square degrees of high latitude sky down to 
24th mag in Y,J,H (and possibly Ks) in 5 minutes per shot (JPE held head in 
hands here, but cheered up when he heard that the earliest possible launch is 
2018). MJI, NAW and STH are investing a modicum of effort to keep an eye on 
this and PLATO, the other M-class space mission of interest.

STH noted there was a PLATO meeting in early December in preparation for a 
funding bid to STFC for a Phase B study.  The same is happening for EUCLID.


4. Data archives

AAT, ING and WFCAM are up-to-date as usual.  The Cassegrain UKIRT archive
still awaits its final putsch involving (re)acquiring 2 to 3 dozen nights 
to complete the data archive.  JRL promised to finish this by Christmas
[which is odd since MJI recalled the same IoU in last year's minutes].


5, Optical /NIR processing

There is a lot of optical processing happening all directed at science 
research projects of various CASU individuals.  Currently under the
kosh are data from : Subaru SuprimeCam; KPNO 4m; CTIO 4m; and about to
hit the road more PAndAS MegaCam survey data from the 2009 Autumn season.

Ironically, VST rumoured to be aiming to begin surveying November 2010,
will produce less data for CASU to process than the current crop of
optical imagers already being dealt with. 


6. WFCAM update

WFCAM is back taking data after its two week vacation (much appreciated by
all) and as usual, after all the loving care and attention it received, 
came back with a few extra bonus features.  MJI was asked to check the
effects of an increased reset anomaly on #4 seen in the darks (~10% bigger
effect near edges) on the science products.  Although no noticeable impact
on processed science images, while checking this he noticed extra pickup 
noise on #3, but not the other detectors, readily visible in the science 
images. This visibly does not have the usual quadrant symmetry seen in the 
curtaining effect, otherwise it would have been removed in that part of
the correction process. 

This took a while to intercept since a full set of broadband flats were not
available until 20091015 due to uncooperative weather at UKIRT.  Processing
recommenced last week and should soon catch up to the incoming data flow.

[Note added post meeting: after informing JAC of this new feature JAC staff 
tracked this down to pickup from a cooling fan in an electronics rack and a 
workaround was implemented to remove the problem as of 20091021.]

MJI said that as of 20091001 processed data he had finally implemented the 
udpated CFITSIO Rice compression software for 16-bit integer data (using 
CFITSIO v3.14) now that most recent versions of DS9 correctly uncompress 
these on-the-fly. This is only relevant for the confidence maps.  The most 
recent versions of CFITSIO and DS9 also seamlessly uncompress the earlier Rice 
compressed 16-bit data (which was actually converted to 32-bit integer during 
the compression process). The 16 bit-specific compression is slightly more 
efficient and produces slightly smaller (by a few %) compressed files.

MJI noted the upcoming "six-monthly" UKIRT Board Meeting in early December 
and needs to compile a report on these and other matters as usual.         <<<<

Reprocessing is on hold while MR takes a well-earned pipeline break and also
because of VISTA PR colour image massaging.  The reprocessing queue is down 
to some nebulous GPS data and the the UDS J,H from 05B-07B.  From 09A, UDS 
data gets its object masking pawsky fix automatically during pipeline 
processing.

EGS produced at short notice some nice summary statistics on the various 
UKIDSS surveys for Steve Warren.  By some miracle this agreed exactly
with the WFAU stats confirming that transfer and ingestion does not suffer
from a Black Hole in the UKlight network in the Leeds area.


7. VISTA update

JPE hand delivered 5 USB disks of VISTA PR data containing nights from 27th
September to 7th October to IoA on Sunday 11th October, where after some 
interesting shenanigans involving a fridge, some blocks of metal and much
patience, it was read on for immediate processing. In parallel a speeded-up 
offcial copy of this arrived via the usual ESO delivery five days later. 
Virtually all of this is PR data and is, as expected, stress testing the 
pipeline and its occupants to destruction.  As acceptable PR images sputter
out the farside they are being made available to Richard Hook and the ESO PR 
team for suitable mastication.  We always expected processing this data would
be far worse than doing science with VISTA and indeed it has surpassed our 
expectations.  JPE noted that as a breather prior to SV data arriving, the
next ESO disk would be mostly air guitar data.

The PR processing has yielded some nice images of the Fornax Cluster and 
the Flame Nebula in addition to the GC and M55 pictures already in the can.   
Various other PR images are still being worked on and assessed with the
deadline for finishing off these the end of October in good time for 
handover.  JPE described the ESOcasts which are propagated around the site
(see  http://www.eso.org/public/archives/esocast.html) and showed some of
the publicity images. MJI said he was keen to get the PR exercise out of the 
way before the SV data comes in.

JRL pointed out some problems with flat fields due to the close proximity 
of a 5th mag star whose diffraction spike was occasionally in the field of 
view when the telescope was jittered, so care needs to be taken when 
co-adding flats. Another problem arises when taking short exposures. Steven 
Beard produced a routine for calculating exposure times for given count rates 
but it does not allow for readout time (1 second) so at short exposures 
it is easy to overexpose the frame (particularly for #5).  A workable 
rule-of-thumb is that if the flat field exposure time is less than 5 seconds 
to give ~10K ADUs/pxiel then the sky is too bright, so do not use 
short exposures or those already obtained which are affected. It was also
noted that we would like flats taken in the half hour before sunrise to
compare with the sunset twilight flats. JPE confirmed this was not happening 
at present. WFCAM suffers from serious persistence problems hence the
pre-dawn twilight flats.  Athough we expect VISTA to not suffer as badly from
persistence effects as WFCAM, testing the characteristics on sky is still 
necessary.

Quite a few nights of data had split OBs i.e. where an OB was interrupted
and then restarted from where it left off.  This currently happens when the
AG drops out and JPE said that it was possible to restart a pawprint if this 
happens in the middle of the procedure.  Unfortunately the headers do not 
reflect the connectivity between partial OBs in cases like this and JRL
will have to implement his usual WFCAM-like association process as a
workaround.                   

The dome lights have been upgraded but not used in anger.  There is some
confusion about their readiness for use but since the last usable linearity
sequence was taken way back in June and many things have been adjusted since
then a new set of measures is highly desirable. JRL said he had already 
enquired about this and this topic was also raised at the recent IOT 
telecon and is high on our wishlist.  (It turns out that the linearity
sequences taken in August that were not quite deep enough for purpose were 
also meant to be used as a test of the stability of a trial light source.
JRL will investigate.)                                                     <<<<

The recent successful attempts to speed up delivery of VISTA data via ESO 
to CASU prompted suggestions that this should be repeated for SV data if 
possible.

JRL noted that offset sky observations are still causing problems in 
associating desire with pipeline reality. Steven Beard's offset sky 
template, which is designed to circumvent these problems, cannot be tried 
until the camera software is rebuilt to include it. This is unlikely to 
occur until after SV is completed.

MJI provided a brief mosaicing update starting with the different mosaicing/ 
tiling software. Up to now we have been using Montage for trialling.  This
generally works well but is a tad minimalist (and non-standard WCS) in its 
headers and also produces 64-bit floating point data.  In parallel we have
been developing (upgrading) our own mosaicing software to cope with
VIRCAM tiles (96 correlated offsets to compute) and produce valid headers 
and 32-bit output.  This is being stress tested by using it to produce the
tiles for the colour images.

JPE said that the Verifications Status review panel was worried about 
possible time-dependent astrometric distortion in the focal plane and asked 
for it to be monitored by the QC pipeline. MJI commented that during normal 
processing we continually monitor the astrometric performance and where 
required generate residual distortion patterns by stacking lots of pointings. 
He is generating some for the various filters anyway to assess wavelength-
dependence of the distortion pattern.                                      <<<<

STH is checking colour equations and throughput and re-affirms his agreement 
with the numbers on the webpage. He confirms that the low value for WFCAM 
(16%) in the bluer passbands is correct and that this is mainly due to the 
much lower QE of the detectors. 

Photometric illumination corrections for Z,Y,J,H,Ks will be handled in a
similar way to WFCAM i.e. computed from stacked 2MASS photometric solution 
residuals.  However there are two sets of narrow band filters in the camera,
that in addition to standard illumination corrections also have variable
bandpass as a function of position in the field: the NB118 - that we knew
about; and a hybrid NB980/985 split half and half over the detectors - that 
we didn't.  The latter have never been exposed to on-sky light and Tim de 
Zeeuw has allocated 3 nights for them to be assessed.  MJI asked about
acquiring the filter transmission curves for the NB sets for feeding into
the CASU ETC. JPE will send MJI this filter data.                          <<<<

Whilst on the topic MJI reiterated that the CASU ETC (v1.4) gives plausible 
nos. for the broadband filters and that apparently the largest differences
with the ESO ETC for VISTA are due to the way the expected sky background 
level is computed in the ESO version.  Whilst rooting around with ETCs
MJI noticed that the previous version (v1.3) was still directly accessible
and had the system folk redirect the link to the latest version.

MJI said that the stability of the filter wheel positioning was still being 
monitored.  EGS is examining the generated sky correction frames for the
tell-tale signature.  The last time a problem was noticed was September 29/30.
JPE commented he meant to re-datum the filter wheel each night to monitor 
accumulated drift but did not always do so.                                <<<<

MJI reported on a slight panic about micro-stepping after a query from the
VIDEO PI.  Apparently the VISTA user manual is somewhat confusing on this
point.  He reiterated the CASU recommendation to not use micro-stepping
and instead "drizzle" carefully chosen dither sequences onto a finer grid
under-sampling is a concern.  The experience with micro-stepping with WFCAM 
has been mixed.  The basic problem is that the PSF is not stable on short 
timescales so you don't correctly sample the PSF by microstepping. One driver 
with WFCAM was to avoid pixel interpolation schemes in the processing so
in this case microstepping is a viable option.  With VISTA there is no choice, 
due to the larger field distortion you have to interpolate when combining 
deep stacks anyway so you may as well interpolate the stacks onto a finer 
grid if you so choose.

STH said that he would look at image quality again when more recent good 
quality processed data is available. JPE expected that some of the SMC data 
taken October 5-7 would be suitable and JRL noted that this would soon be 
available. JPE said that VISTA's EED50 had been formally accepted as within 
specification, but EED80 was not and required a waiver. VPO were required to 
provide a report on EED over range of filter, alt, az, reversal positions and 
wind conditions in good seeing (without M2 micro-oscillations). There 
followed a discussion on the SIQ targets and how they depend on assumptions 
about the PSF profile.                                                     <<<<

STH was keen to see trial survey OBs before they are used in anger and 
suggested that we should have access to them at the same time as ESO 
otherwise it is too late. JPE commented that ESO would be unlikely do this 
but that the PIs could anyway, however he would ask ESO.                   <<<<

MJI and JRL are still chewing over Rice compression options for VISTA 
processed data - a recent poster paper presented at ADASS by Bill Pence 
is relevant - but almost certainly will plump for the WFCAM style.  This 
means that all the current VISTA processed images are still uncompressed.  


8. Any other business

All of the GAIA hardware is now running and MJI said that 2 or 3 out of 
the 4 A/C units were running at any one time giving some welcome redundancy
in capacity.

Both JRL and EGS have had problems with mains spikes on the power supply to 
their PCs recently. A spare anti-surge unit is available and could be 
installed in their office upon receipt of suitable quantities of beer.     <<<<

MJI asked SC to investigate alternative backup strategies on non-cluster 
machines (i.e. apm 05, 25, 26, 28 etc..) for the next meeting.             <<<<

After much hassle with the rolling grant MJI noted that we are finally in a 
position to advertise for the vacancy required to properly cover VISTA 
pipeline processing.  The advert is expected to be placed shortly on the
AAS job register with closing date the end of November, interviews in early 
January, aiming for a start 1st April.


Continuing actions
------------------

EGS  add catalogue software and binary table conversion software to 
MJI  software release pages  

MJI  finish off technical paper updates

MJI  write utility to convert binary tables with all corrections

MJI  compare WFCAM dome flats with twilight sky flats

SC   progress the high temperature monitor installation


New actions
-----------

EGS  email WFAU about VISTA processed data for night of September 27th 

MJI  do the same with an example of a tiled image

MJI  compile and write the usual UKIRT Board report

JRL  investigate stability of trial light source from August data

MJI  assess wavelength-dependence of VISTA astrometric distortion 

JPE  send MJI the NB filter transmission curves

EGS  monitor sky frames for signs of filter wheel problems

STH  reassess SIQ on the SMC data taken in good seeing

JRL  investigate alternatives for Rice compressing processed VISTA images 
MJI  a la Bill Pence

JRL  supply some beer if they want the mains filter installing
EGS

SC   investigate some strategies for backing up non-cluster machines