File contents

11th March 2009   APM meeting tables  11:30-12:52

Present:          MJI, JRL, STH, MR, EGS, RGM, RWA, NAW, SC, DWE
Apologies:        

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru
6.  WFCAM update
7.  VISTA update
8.  AOB - rolling grant status etc..

Minutes
-------

1.  Actions from last meeting

STH  the technical pages are still not up to date - ongoing                <<<<
MJI

JRL  has made some progress on producing suitable images for the web page 
     of HawkI publicity for CASU - ongoing                                 <<<<

MJI  have not finished off the stage-I software release pages - ongoing    <<<<
EGS  RGM reminded us about the request to write an example utility to 
     convert catalogue binary tables to ASCII with all the assorted 
     photometric corrections applied.                                      <<<<

STH  has put the photometric calibration paper on the Plone and VDFS sites

MJI  has not finished off the technical paper - ongoing                    <<<<

MJI  contacted Mark Casali about the HawkI radial distortion and sent him 
     some example plots illustrating the improvement using correct ZPN model

RGM  liaise with JRL about processing HAWKI PI data - still ongoing since
     awaiting previous SV HAWKI data completion                            <<<<

JRL  has produced a definitive solution to the pipeline versioning issue 
     and implemented it for 09A processing
     
EGS  backed-up the online 2008 ING archive data
     
JRL  has not finished the HAWKI NB image data processing to improve sky 
     subtraction - still ongoing                                           <<<<
     
JRL  contacted JAC about upgrading to a more recent CFITSIO version in 
     the conversion software 
     
JRL  have made good progress in sorting out the VISTA documentation set
STH  and ensuring that the latest copies are available on the internal 
RWA  web pages. JRL has produced a spreadsheet documenting all this. A
     few documents still remain to be indexed and cross-checked - ongoing  <<<<
     
STH  has contacted the VISTA PSPIs regarding the need to get permission 
     through JPE for access to reduced VISTA test frames


2.  Comments on WFAU minutes

The various recent hardware and software (32/64-bit) foibles were noted.
JRL commented that the pipeline and particular the current version of the 
cataloguing software, works fine on both 32-bit and 64-bit operating systems.
He was reminded to send the latest version of the cataloguing software to 
WFAU together with versioning information on library dependencies.         <<<<

The rediscovery of the CFITSIO 16-bit compression feature was noted. This
is one of the reasons we still use an earlier version of CFITSIO to compress
the confidence maps.

JRL appreciated the SVN suggestion but commented that despite it having
several advantages over CVS there was a snag. The problem is that the 
VIRCAM pipeline software including ESO software deliverables, which shares 
much of the WFCAM code, sits in a CVS housed at ESO. He felt that running
two similar but different versioning systems in parallel could nuke his brain. 

MJI reported on a brief discussion with NJC regarding deep stacks of ~100s
of frames and concommittent memory issues -v- use of intermediate stacks and
apparent degradation of image quality.  MJI had seen similar effects with
other data and thought the problem was most likely caused by seeing 
degradation due to "astrometric" blurring.  The solution we usually adopt 
is to always update the astrometry of intermediate stacks via the usual 
catalogue method.


3.  Recent and upcoming meetings

MJI and JRL commented that they had recently learnt of regular VISTA 
(Instrument Operation Team) telecons to discuss various issues including 
data quality and properties.  These are held the last week of every month 
and are organised by Thomas Szeifert at ESO.  We thought it was now timely 
for CASU to be directly represented at these given that we have a.) measured 
many of the parameters of interest and b.) know at least as much about the 
properties of VIRCAM as anybody else.  The next telecon is apparently 
scheduled for March 26th and MJI will inquire if someone from CASU should 
take part.                                                                 <<<<

The upcoming JENAM fete was mentioned and a whip-round for volunteers for 
talks, though painful, was relatively unyielding.  However, NAW raised a
paw and will give a CASU-related talk at JENAM (and in the time it takes 
to write these minutes has already submitted an abstract).                 <<<<

MJI was part of the WFMOS roadshow entertaining the Gemini Panel in Hilo
February 22,23 in a supposed shoot-out of the two conceptual design 
studies.  He noted that this was the first ever ppt talk he had written
in advance (3 days) of a presentation and was still in shock and awe.

Regular Plato meetings continue to occupy NAW and STH but at least STFC
have funded, albeit at modest level, our contribution to the design study,
unlike everything else we do.

Other upcoming events: NAW noted that Jonathan Fay, lead developer of 
Microsoft's World Wide Telescope will be visiting the IoA after the JENAM,
viz 23-24th April 2009. He will be talking at the VO session (t) of the 
JENAM - 22nd April 2009.


4.  Data archives update     - AAT, ING, WFCAM, UKIRT

AAT:   data dribbles in - up to date

ING:   data trickles in - up to date

WFCAM: data sputters in - up to date

UKIRT: RWA is reading on all the old DLT tapes prior to backing them all up
on LTO4 media as a precaution against degradation/ancient tape readers.
(MJI - do we have the 08B CASS data ?)


5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru

There has been a lot of action on the IPHAS/UVEX front with EGS reducing all
the 2008 season data taken on the INT WFC.  The IPHAS data has been ingested 
into the DQC DB and also released to the consortium via the usual web pages.
UVEX is still being footled with due to assorted discussions regarding 
band-merging with IPHAS and the variable no. of passbands observed.

Cross-matching of these two datasets and band-merging of UVEX data is more 
of an open issue. The UVEX team seem unable to come to terms with proper use 
of the band-merged FITS binary tables - which would solve all of their 
problems.  To save time and effort at this end, EGS will find out what coding 
language they are using, and within reason will produce an example piece of 
code to illustrate how to go about it.                                     <<<<

There was a brief discussion about IPHAS data access through VIZIER.  CASU
could not see the point of doing this given the existing public access and
the problems and extra effort involved in releasing it through VIZIER as well.
NAW was volunteered to compose a polite but firm email to the IPHAS 
consortium regarding not duplicating the data release through VIZIER.      <<<<
Of course if they want to provide the effort themselves - fine.

MegaCam is still awesome but some recent Subaru data on NGC891 that Rodrigo 
Ibata sent over for diagnosis tests is awesomer^2.  In ~10 hour integrations
in V,i in 0.5 arcsec seeing last Autumn using the new Hamamatsu detectors 
this reaches unprecedented ground-based 5-sigma imaging depths of 28th 
magnitude in i and 29th magnitude in V.  The same depth as equivalent
HST ACS observations, but over a 30 arcmin field !


6.  WFCAM update

09A processing was held up for about a month while waiting for a full set of 
sky flats to be taken.  The long wait was mainly caused by uncooperative 
weather which has the side effect of not that much data to process either.
This has all now been remedied and as a glance at the processing status
page for 09A (http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/
report_night_reduction_status_casu?semester=09A&SUBMIT=Submit+Query) will 
reveal, with the help of the weather we are now back on track with processing.

MJI asked about the status of duplicating PSB raw data access scripts so we 
have a completely independent backup when it inevitably goes wrong at the
next computer hiccup.  MR will chase this up though so far transfer and ingest
of 09A data have gone seamlessly.                                          <<<<

MJI recalled a request from Derek Ives via AJA for information about possible 
non-linearity at low light levels for Rockwell detectors.  He pointed them to 
the web pages containing the best direct measures on non-linearity we have, 
but noted that those measures are long in the tooth and don't really 
adequately sample the relevant ADU range - though in mitigation there is 
currently no compelling evidence of any signficant non-lineary effects in
WFCAM photometry.  As Version 3 of the WFCAM flat field screen should just 
about be in place by now it would be timely to repeat a set of linearity 
measures as noted previously.                                              <<<<

JRL and MJI reported on a telecon with Seb and Omar about UDS 05B reprocessing 
results.  Unfortunately the reprocessed 05B K-band data, using the pawprint
sky method (developed for UltraVISTA), enthused them so much (0.3 mags deeper)
they now want to reprocess ALL the UDS data the same way. There are still a 
few potential gotchas with unfettered use of this method, so Seb and Omar 
were asked to do a few further tests to make sure this was what they REALLY 
REALLY want.  (MJI noted that the results of these came in just before this 
meeting. He will forward the UDS test results to JRL.)                     <<<<

This lead to a short discussion of the various sky estimation strategies that
the pipeline now has at its disposal: 

none      - not enough data to form a sky so don't bother;
leftovers - odds and sods throughout the night combined to make sky frame;
pawprint  - sweaty iterative stacking with ever deeper object masking;
tile      - default, usually the best option but needs tile of pawprints

How these work and more about them needs to go on the technical plone pages 
- already noted as an ongoing action. (Note that current offset sky strategy
makes use of the pawprint method by default, though MJI had already mused 
why not tile as alternative since can give better skies in many cases.)

Although the information about sky strategy adopted is in the processing 
SUMMARY file in each date subdir, RGM asked if we could place a keyword in 
the sky FITS file annotating which method has been used.  Since this is 
easy to do, it is also worth adding to already processed 09A data as it    <<<< 
hasn't been flagged for copying yet. No doubt there will be a request to 
have it propogated through the WSA shortly hereafter.

MJI noted that WFAU transfers of recatalogued 08A data were now complete
but that 08B were still awaiting their turn.  This presumably means we
can now fix the Petrosian etc... fluxes and radii for blended objects for
05A -> 08A inclusive ??

For MAR eyes only - Jim admitted he hadn't put in place the code to compute
dither offsets en masse for WFCAM.  He is beavering away on it now since
we notice similar occassional problems for HAWKI and VIRCAM. RGM asked
for a keyword to flag when there was a problem with different computed offsets
for the detectors, but subsequent to the meeting we decided to compute the 
dither offsets in a single robust LS fit in xi,xn space to circumvent
too many iff's.

[This is a known (admittedly rare) problem. The pipeline computes the observed 
dither offsets for each detector separately.  It is therefore possible for it 
to get the offsets in a dither sequence correct for say detectors 1,2,3 but 
only partially correct for detector 4.  When they are stacked this means
that some of the subcomponent images effectively get rejected causing
an effective shorter total exposure and hence the problem with perceived
magnitudes on that detector.]

A coffee time conversation regarding edge effects on photometry and how
to deal with them in a transparent way then sprang to mind. The suggestion
of using some of the surplus bits in parameter #55 to encode the 
average confidence level within aper3 was mooted.  This seems like a good
idea since it covers, partial exposure in dithers, surplus not so good
pixels, edges etc....  The current use is #bad (i.e. 0 confidence pixels)
but there is plenty of room to include the average conf too.  What do
WFAU think ??


7.  VISTA update

JRL and STH summarised the current state-of-play.

Filter wheel: A lot of the problems with the data have been linked to a 
problem with the filter wheel which seemed to have a great deal of trouble 
moving to specified locations. This is crucial if the flat fielding is to 
work properly. After the camera was warmed and opened up it transpired that 
a screw in the filter wheel had worked itself loose and was digging into the 
side of the container causing flakes of paint, undercoat and finally metal 
to rain down on the filter surfaces.  The resulting particles cast shadows 
and hence caused some funny flat fielding properties, which we now hope will 
be a thing of the past.  During the time VIRCAM was off the telescope the Z 
filter has now been installed.

Data reduction: Data from 1st December through to end of January have all 
been reduced in a 'standard' way. The tiles were all done using a tilesky
algorithm very similar to WFCAM. Those that weren't part of tiles were done 
using the masked pawsky algorithm (also now used in WFCAM processing). By 
and large the results are very encouraging although there are still a 
couple of outstanding issues:

 -- detector 16 has a worrying 'feature' in its top half which gets very
    noisy and has a strange cross hatched pattern with blotches that don't
    subtract out well in most cases. The source of this is being investigated.

 -- observations in a region with some very bright stars have shown some
    interesting reflections that can occur when one of these very bright 
    stars is located between two detectors. This is also currently being
    investigated further.

 -- preliminary analysis of standard star data shows a problem with the
    linearity of the results. JRL will return to the linearity sequences
    that have been observed and hopes that some new ones will come along
    very soon.

The full results of the reduction will be collated in the next week or so 
and MR will post them in a manner similar to the WFCAM DR pages, although 
this will not be a public page yet.

Offset skies: After being told ad-nauseam that it is not possible to schedule
offset sky observations linked to an object observation using the ESO P2PP
infrastructure, recent upgrades announced at the September PI workshop beg
to differ.  We have now been asked to draft a proposal to define how these 
should be done with VISTA.

Schedule: VIRCAM is to be reinstalled on the telescope on 17th March, with 
observing commencing at the end of the month to revisit the technical 
verification of the system. 

STH then showed some preliminary results from the VISTA processing of data  
taken in Sep, Oct and Dec 2008.  In brief:

1. As noted earlier there is a significant residual non-linearity in the 
photometry (shown in a 2MASS comparison plot). JRL will look at the 
application of the linearity model to try and understand where this comes
about.  In particular STH and JRL will test not applying the non-linearity 
correction to a suitable night of data to see how this affects the 
observed non-linearity.                                                    <<<<

2. The colour terms with respect to 2MASS are fairly small for JHKs, as  
expected.  Further progress here awaits fixing the non-linearity issue.

3. The sky brightness measurements have been compared with WFCAM, with  
the proviso that: (1) no data have yet been omitted for cirrus and  
lunar proximity. Early indications are that the sky/background looks similar 
brightness in YJH and 0.2 mags brighter in Ks.


8.  AOB

MJI gave an update on the rolling grant status from the previous round
(i.e. due to start 1st April 2008) which was stalled yet again despite
a huge amount of effort on our part.  This is particularly galling given 
that PDRA support has already been awarded this current grant round for
VISTA and VST science exploitation.

SC reported on recent hardware additions to supply sufficient disk storage
for VISTA and WFCAM to last this current year.  MJI noted that it had better 
be since that is the last of the equipment dosh.


Continuing actions
------------------

STH  check that Plone technical description pages are up to date 
MJI

JRL  produce web page of HawkI publicity for CASU

MJI  finish off stage-I software release pages
EGS  

MJI  finish off technical paper

RGM  liaise with JRL about processing HAWKI PI data

JRL  redo the HAWKI NB image data processing to improve sky subtraction

JRL  sort out remaining VISTA documentation set and make sure latest copies
STH  are available on internal web pages
RWA


New actions
-----------

MJI  write utility to convert binary tables with all corrections applied 
     to ASCII

JRL  send latest version of cataloguing software to WFAU

MJI  contact JPE about one of group partaking in VISTA telecon 24th March

NAW  submit abstract for CASU talk at JENAM

JRL  place keyword in sky FITS file describing which method has been used

EGS  inquire what language UVEX team are using for coding and produce example

NAW  send polite but firm email to IPHAS about VIZIER data release

MR   chase up wherewithall to produce duplicate WFCAM transfer script

MJI  inquire about status of new WFCAM linearity measures

MJI  forward UDS test results to JRL

STH  test not applying non-linearity correction to see how this might affect 
JRL  observed non-linearity -v- 2mass