File contents

28th June 2006  Hoyle committee room 13.00 - 14:10

Present:    DWE, MJI, JRL, PSB, MR, EGS
Apologies:  WJS, RGM, STH

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Report on VISAS meeting
4.  Data archives update
5.  Optical processing update
6.  WFCAM news
7.  VDFS news
8.  AOB

Minutes
-------

1.  Actions from last meeting

MJI  has made no further progress investigating the merging of 
     classification statistics - ongoing                                   <<<<

WJS  has not contacted the VPO re: ETC-related digitized data.  This
     and the related action have been deprecated after an email from JPE.
     PSB commented that JPE has acquired some mirror reflectivity
     data (but has not passed this on to us yet).

JRL  has not implemented the fix to the problem with the UKIRT archive
     (missing header information problem) - ongoing                        <<<<

EGS  has not updated the INT WFC web pages with the latest flat-field 
     and fringe frames - ongoing                                           <<<<

STH  it is not known if the data from the 2MASS Touchstone fields has been
     acquired (STH not present).  This is needed for further linearity
     tests - ongoing                                                       <<<<

DWE  have updated the hardware whereabouts page (HTML and Plone)
MJI

PSB  had a short meeting to finalise updates to DRLD
JRL
MJI

DWE  are still finalising content of Plone site - ongoing                  <<<<
EGS
STH
MR


2.  Comments on WFAU minutes

The previous "Rant of the Week" concerning the UDS processing was noted. 
CASU agrees and extends its sympathies.

MJI noted that the latest minutes stated that the agreed single thread
network transfer test would be carried out after DR1 has been released.


3.  Report on VISTA PSP meeting

JRL reported on the VISTA PSP meeting he attended in Edinburgh on 20 June. The
main activity at the meeting was horse-trading between the original
proposals, which was very successful. The original set of proposals shows a
3 to 4 times oversubscription in the available observing time. By merging
some of these proposals, the amount of oversubscription was reduced.
From the 10 surviving proposals at the start of the meeting, there now
remain 6 (Magellanic Clouds, Galactic Bulge, VIDEO, VIKING, VHS, a UDS).

What JRL found worrying was the way many of these projects were intending to
take data. There is a need for us to give clear advice on observing recipes
particularly those proposals that do not want to dither. Many are concerned 
at the overheads reported by the ETC, believing they will maximise their 
science return by not dithering.  PSB pointed out that it was very important 
to dither given the bad pixel distributions and notes that astronomers may 
not be aware that the acceptance level for the proportion of bad pixels per 
chip is 5%. JRL said that we have been asked by the PSP (via Marc Casali) 
to produce a crib sheet of observing strategies.

There is clearly a need to re-examine the overheads used in the ETC due to 
the recent changes with respect to parallelization eg. writing data to disk 
while filter changing or dither offsetting.  PSB reported on a phone call 
where JPE said that he would write a specification for the overhead 
algorithm.  This will be forwarded to ESO for the official VISTA ETC
and to CASU for possible inclusion in the test ETC.  However, MJI also noted 
that in his experience overheads were rarely overestimated and erring the 
other way was even worse.

JRL also reported that there seemed to be a distinct lack of manpower 
considerations for data reductions beyond that provided by the CASU pipeline 
in these proposals.  ESO DMD will undoubtedly pick up on this.

The merged proposals will now be rewritten and resubmitted to the selection
panel including revised data management plans.


4.  Data archives update     - AAT, ING, WFCAM, UKIRT

AAT - no change

ING - EGS said that the data for 1996 had now been refactored into date 
subdirs and duplicate entries flagged.  The period 1996 to 2006 is now 
almost complete excepting some INT data from the latter half of 2004,
around the transition between DVD and Internet transfer of data from La
Palma. MJI and EGS will arrange for RWA to fill in this gap and stress 
test some more DVD readers.                                                <<<<

WFCAM - MR said that tapes have been received up to 15 June. All the data up
to the 7th of June have been MEF'd and are available through the raw data
archive. He is currently dealing with the last set of tapes, 8th-15th of
June.  Transfer to ESO of 06A UKIDSS and calibration data has commenced.

JRL said that he had a nephew coming for a couple of weeks for work
experience as part of school course requirements and that helping
with reading tapes and MEF'ing files might be a suitable task.  We will
see.

UKIRT - no new data tapes.


5.  Optical processing update

EGS said that he had been reducing some INT data for Dan Smith and Martin
Jarvis.


6.  WFCAM news

(Non)linearity saga - this one could run for a long time.  JRL says that 
his analysis of the photometric test set (same field but exposure times 
ranging from 1s to 60s) reveals no clear evidence for non-linearity until 
close to saturation.  MJI asked to be pointed to the dome sequences taken 
on the same day so he can carry out further checks.                        <<<<

MR reported on the progress of 06A reductions.  MR reminded us that most of
the May data has assorted randomly missing chips and also a bunch of files
with null values for detector #2 (as reported in previous minutes).  After
several modifications the pipeline now deals with these more robustly. A 
small number of files have also been found with incorrect offsets in the 
header. MR was asked to email Paul Hirst (CC Andy Adamson) concerning this 
and any other problems he has found.                                       <<<<

Since 2 June, all chips are present in the files, there are still some 
null fulls for chip#2.  This is still about 1% of files !

MR noted that some MSB observations have been split in an odd way and this 
has caused problems due to the interesting way they have been deconstructed 
and then partially started again and continued. These MSBs are effectively 
unreduceable without much header fiddling. MR said that these MSBs were all
PI data.

MR also said that he and MJI have been re-reducing some 05B data for UDS. It 
was noted that a recent paper on the UDS (astro-ph/0606386) claims that our 
pipeline components are not optimal for deep stacking and cataloguing.  
MR and EGS said that they are looking into this independently              <<<<

MJI said that the additional air conditioning has been installed successfully
and has been running for the last 2 weeks.  This provides an extra 7kW of 
cooling capacity and should keep the CPUs and disks happy in the soon-to-be
ex-store room.  Together with the previously installed kit we now have the
wherewithall to create an artificial indoor ski slope or ice rink.

Two existing computers/RAID arrays have now been rack mounted (rather than
desktop mounted), cass03 has been finally retired, MJI finally got his new 
desktop machine after months of waiting, and much old (solaris) disk 
rationalisation has occurred. It is intended that two other computers (apm18 
and apm23) will be moved from their current racks into the new racks in the 
store room to reduce the ambient noise level in the main APM area.  There are 
no immediate plans, or requirements, to purchase any new hardware.


7.  VDFS news

PSB reported that he had a productive visit to RAL and that he has returned 
with 270 files - all with full headers. Currently, the RAL team are in the 
middle of intensive spot projection tests of VIRCAM. The data PSB obtained 
included linearity sequences, noise/gain runs, darks and 1 persistence run. 
It is unlikely that we will get any more data before commissioning on Paranal
begins so we can deconstruct this data in earnest and see what we can learn
about VIRCAM.

JRL reported that the ESO end-to-end test had been very successful. We
supplied simulated data to test out our supplied V0.3 recipe modules. ESO 
also carried out some archive ingestion tests using "real" VIRCAM data. There 
were a few minor header problems reported, but only 1 (out of 6) was of CASU 
origin.

PSB said that it would be useful to configure the VIRCAM data-acquistion
system at RAL to be close to the production system, such that most of the
real overheads could be determined empirically to feed into the ETC. Such
tests could be carried out with a warm, but complete, camera.


8.  AOB

None - record finish time - the absence of RGM was purely a coincidence.


Continuing Actions
------------------

MJI  finish investigating merged classification statistics

JRL  implement fix to problem with UKIRT archive (missing header 
     information problem)

EGS  update INT WFC web pages with latest flat-field and fringe frames

STH  acquire data from the 2MASS Touchstone fields for linearity tests

DWE  finalize content of Plone site by end of month (which one ?)
EGS
STH
MR


New Actions
-----------

MJI  arrange with RWA to fill in the 2004 gap for the INT in the ING archive
EGS

JRL  email MJI on whereabouts of dome linearity sequences

MR   email Paul Hirst (CC Andy Adamson) concerning incorrect offsets in 
     file headers (and any other problems)

EGS  investigate UDS claims regarding pipeline components not being optimal
MR