File contents

Monday 18th May 2009   APM Meeting tables    11:30-13:00

Present:        MJI, JRL, STH, SC, RWA
Apologies:      NAW, MR, EGS, RGM, DWE

Agenda
------
1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT, VISTA
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, Gemini, VIMOS
6.  WFCAM update
7.  VISTA update
8.  AOB 

Minutes
-------

1. Actions from the last meeting

STH   PLONE pages. Sky subtraction and dribbling text updates have been added 
RWA   (it later transpired that the sky subtraction text should also be on 
      the WFCAM as well as the VISTA technical pages.)  RWA will continue 
      checking the technical pages - instances of poor alignment noted.    <<<<

STH   HAWK-I colour image of Carina. STH has put this on a private web page 
      with password and demo'd zoomifying at the meeting.  JRL still has
      to write some text to go with it.                                    <<<<
      
MJI   says (re)photometry for HAWK-I still to be done                      <<<<

MJI   Stage 1 release software - still ongoing                             <<<<
EGS

MJI   updates to technical paper started, but not yet finished             <<<<

RGM   liaised with JRL about HAWKI  - data processed

JRL   has sorted out the VISTA docs and pdfs so that this is essentially 
      complete and they are all now on the VDFS internal web pages.

MJI   write utility to convert binary tables with all corrections applied
      - in the queue but not high priority                                 <<<<

JRL   has sent the latest catalogue software to WFAU

JRL   implemented keyword in sky FITS files and back copied it to all 09A
      sky files

EGS   has not yet sent off example code to UVEX team                       <<<<

NAW   polite but firm e-mail to VIZIER about IPHAS - ongoing               <<<<

MR    sorted out WFCAM transfer script with result that MJI now gets two 
      e-mails per day from MR rather than PSB

STH   inquired about linearity sequence status and noted nice new screen 
      but no lamps yet    

MJI   wrote and sent the two reports to JPE for distribution
JRL   

MJI   has put the minutes he has of the IOT telecons on the internal web 
      pages, though he noted that the minutes of the previous April meeting 
      have not appeared yet

STH   is liaising with ASTEP folk and has started to construct a webpage   <<<<

MR    status of comparison of FITS file header problems on 20090312 with 
      original sdf files unknown - ongoing                                 <<<<

MJI   compiled and sent off report to AJA for the upcoming UKIRT Board meeting
      and put a copy of it and the UKIDSS submission on the internal web page

JRL   has been in communication with JAC about cfitsio version in use there

JRL   VISTA processing status webpage. A lot of action on this and it is now, 
MJI   thanks to MR, doing live updates. JRL has also has made access to 
      reduced data possible by our usual password-protected web page access.
      MJI is steadily verifying, correcting astrometry and producing
      summary QC plots for the live web page. 1st pass photometric ce's and 
      ZPs have been generated based on STH touchstone field observations.  <<<<

MJI   sorting out the VISTA i/o disks is on hold until the GAIA team finish
      assorted i/o tests on the new RAID system.                           <<<<


2. Comments on WFAU minutes

MJI noted that as a result of the recent computer attack on the IoA cluster,  
all private authorized keys on the APM cluster were disabled as a security
measure. It wasn't clear at the time who was using this method but it turned 
out to be the ING incoming transfers and the WFAU outgoing transfers. The 
relevant keys have now been regenerated.

There have been several comments about processing and reprocessing status of 
various things - see the WFCAM section later for the current state-of-play.


3. Recent and upcoming meetings

There will be a UKIRT Board Meeting in Nottingham on Thursday, and SJW, NCH 
and MJI will attend for the pm session.

EGS later reported via email that he gave a CASU presentation in the VO and 
Distributed Computing session at NAM 2009. He noted that it looks like 
Europe and the US are taking the VO seriously.

VISTA IOT telecon on April 28.  JRL and MJI attended from Cambridge and STH
from Mauna Kea (albeit briefly due to inclement weather).  JRL and MJI 
submitted reports on sky subtraction strategies and science verification 
results and they can be found under the VISTA commissioning internal web 
pages.

The next telecon is due in about a week. There is a definite need for a sky 
offset template (HAWKI uses 4 different flavours, but 1 would do for VISTA)
as noted during the previous telecon. After a brief discussion on possible
practical ways to implement this it was felt that this should be taken 
offline and the constraints discussed with JPE and Steven Beard before 
proceeding further.  STH noted that as he was imminently about to fly out 
to Paranal he would discuss this further with JPE when he arrives.

MJI noted that the PLATO Science consortium will meet in Belfast on June 11. 
STH can't go but NAW will attend and further impart some wisdom.

JRL commented that ADASS 19 is now open for registration (Sapporo, Japan) and
he thinks we ought to contribute something on VISTA processing.  MJI agreed
but pointed out that the travel budget is currently zero and still awaiting
a delivery from STFC.

At the JENAM meeting Tim de Zeeuw said that first light for VST would be
this September - watch this space, but don't hold your breath for the
first survey data.


4. Data archives

AAT - Jim says that the AAO now write to DVD in real time. He has written a 
script to download the data as each DVD is completed and we are now running 
~1 night behind. MJI noted it might not be a bad idea to use a semaphore 
system of some sort rather than dead reckoning to ensure synchronicity.    <<<<

The RAID array that this is being stored on seems to be having a hard time
of it recently and has had two different disk drive units going awol within
the space of a week.  No obvious reason other than stochasticity but at
the time of writing another (different again) channel is going awol - d'er!

ING - data keeps on rolling in and being verified, ingested and backed up
to tape.

WFCAM - MR has more fully automated the MEFing and verification step, which
together with the automatic data transfer stage from JAC to CASU has further
minimised overheads for this aspect (good job too!), but still needs manual
tape archiving further downstream.

UKIRT - remaining few DLTs and LTOs still to read

VISTA - MJI noted a report in the latest ESO news letter that from mid-2009 
bulk ESO archive users need to be ready to receive Blu-Ray disks (sort of 
souped up DVD) and it warns users to have suitable hardware in place. 
MJI asked JRL to double check that this is not going to affect us and that
the disk bricks will still continue to be the VISTA transport medium for 
the near future.                                                           <<<<


5. Optical/NIR processing 

HAWKI - as noted previously, JRL has processed the latest HawkI data for RGM
and MJI still hasn't finished off regenerating the Carina catalogues.  Yet
another excursion for the nebulosity filter.

INT WFC - MJI had a pleasant observing run on the INT WFC towards the end of 
April, generated and analysed some new non-linearity observations and 
finished the processing of data from his observing run before he got back.

MegaCam - nothing to report

Subaru -  EGS says he has finished processing deep optical data from Subaru,
pipeline runs smoothly and produced beautiful stacks.

Gemini - MJI processed some SLUGS imaging g,r,i data for an IoA-led lensing 
consortium

VIMOS  - a pile of venerable P74 and P75 deep VIMOS imaging data on nearby
galaxies has also been processed to final stack stage.  All that remains is
to trim the crud, remove the assorted reflections and generate some 
catalogues.  


6. WFCAM update.  

09A processing is going smoothly after a slight pause to generate a full
set of May flats before starting the May processing.  MJI is doing the honours
while MR takes a well-earned break observing with WFCAM, or maybe the break
is the holiday bit after.  As usual when at full steam the pipeline processing
to completion runs about 5-7 days behind the date of observation (but recall
the observation "date" is UT-based and it takes ~24 hours from observing
finishing for the data to be transferred, verified and ingested here).

At least one logjam could finally be cleared now that sufficient 09A data has 
been accrued to compute the photometry corrections.  The final checks and 
photometry udpates were made to January-April 09A data over the w/e and this 
has now been flagged as ready for transfer to WFAU.

A disk hardware failure on a RAID unit on one of the processing engines 
required a disk drive to be replaced.  The bad news is that this flavour
is now out of warranty, the good news is that we have around 8 spares 
from a now defunct older RAID array.

Reprocessing is currently stymied somewhat by insufficient hours in the week
but moaning aside the current status is:

the UDS K-band reprocessed data awaits the final thumbs up from the UDS team
but is looking good.  MJI was asked to help make a quick assessment of the
depth (expected -v- reality) and predicted from purely photon noise 
limitations due to sky, a 5-sigma depth of K=22.3 within a 2 arcsec diameter
aperture from the no. of components in the latest stack, or ~22.1 for an 
average point source - i.e. accounting for aperture loss.  His analysis of 
the random noise component (allowing for interpolation correlated noise) of 
a 10kx10k chunk of the proposed deep stack suggested K=22.2, in reasonable
agreement. Larger scale correlations present in the deep stack will conspire 
to lower this limit by ~0.1 mags but this is now fairly close to what is
being measured by the UDS folk.  MJI will double check with Omar at the 
Board meeting before making this available for transfer and also discuss 
with NCH the simplest approach to this.

MJI finally started the GPS nebulosity reprocessing after writing a script
to do all the hard work. The filtering took a few days and the catalogue
regeneration several more.  It now needs assorted tests running in
conjunction with GPS folk.

Lack of available effort has meant that the Petrosian etc. reprocessing/
updates for 05A-08A still haven't been started and neither has supplying
WFAU with the relevant script/details.

MJI noted that there is an upcoming PPRP meeting in Glasgow at which, among 
other things, a proposal to build a planet-finder spectrograph for UKIRT will 
be discussed.  He noted that if successful we could be involved in the
archiving aspects.

There has also been a flurry of e-mails about an old hot potato, flushing the 
detectors after changing filters. In practice this amounts to a single 
"quick look" recipe exposure of similar parameters to the upcoming science 
observations after each filter change.  For some projects this adds 
negligible overheads (~1%) for others it can be up to ~5% or more.  The need 
for this is going to be reassessed.  It mainly (solely?) seems to affect 
detector#1 and was first spotted in SV data taken in April 2005 by SJW. 


7. VISTA update

CASU have received complementary remarks from Thomas Szeifert on the quality 
of the output from the summit pipeline.

JRL notes he has made a few more bug fixes (Shirley not) to VIRCAM software 
and is making good progress on the UK science pipeline. No new useful science
data has yet been received but we anticpate once the detectors are running
at the correct operating temperature and the camera is back on the telescope
that characterisation will begin in earnest once more.  STH has flown out to 
Paranal to help JPE and WJS with this process.

MJI asked about the progress with the tiling software, tests with MONTAGE, 
and in particular what the contents of the tile FITS headers will be, i.e. 
what gets inherited, what doesn't, what extra keywords etc...  JRL said he 
would produce an example tile FITS headers that could be used as the basis 
for discussion with interested UK parties.                                 <<<<

As noted earlier the VISTA processing status webpages are now live after
a lot of recent work. They are accessible remotely but deliberately are
hidden from casual view.  All of the useful processed VISTA data up to the
end of January is being checked astrometrically and photometrically and
the usual WFCAM style QC information is being generated.  The first tranche
of this is available through the web page and the first pass colour equations
and default zero-points have been produced.  New raw data is picked up
and logged on these pages as it comes in after being copied, verified and
ingested.  

MJI commented that JPE was planning to spend his upcoming sabbatical at the 
IoA and given the current STFC funding crisis we can put him to good work
helping to run the VISTA pipeline. 


8. AOB 

The TV/wall mounted monitor for the CASU meeting area has been in abeyance
for a while and as SC raised this topic MJI asked SC to investigate the 
possibility of a display similar to that on the wall outside the Sackler
lecture theatre -v- a large format LCD TV (which was the original plan).   <<<<


Continuing actions
------------------

RWA  continue checking through the technical pages looking for glitches et al.

JRL  produce web page blurb for HawkI 

MJI  finish off HAWKI reprocessed data by nebulosity filtering and 
     recataloguing

MJI  finish off stage-I software release pages
EGS  

MJI  finish off technical paper updates

MJI  write utility to convert binary tables with all corrections

EGS  send off example code to UVEX team                    

NAW  polite but firm e-mail to VIZIER about IPHAS reqired

STH  put ASTEP info on webpage

MR   check with JAC to see if SDF files have same detector#4 header problem

MJI  continue producing summaries a la WFCAM of reduced VISTA data

MJI  sort out VISTA i/o disks


New actions
-----------

JRL  set up semaphore system for AAO transfers

JRL  check any impact of ESO archive decision to go to Blu-Ray disks

JRL  produce draft FITS headers for tile products

SC   investigate TV/monitor similar to that in the Hoyle for CASU meeting area.