File contents

31st August 2006  Hoyle Committee Room 10.00 - 13:00

Present:    DWE, MJI, JRL, PSB, MR, STH, EGS, RGM
Apologies:  WJS

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  News from IAU GA
4.  Upcoming VDMT meeting
5.  CASU help saga
6.  Data archives update
7.  Optical processing update
8.  WFCAM news
9.  VISTA news
10. AOB                       


Minutes
-------

1.  Actions from last meeting

JRL  has not implemented the fix to the problem with the UKIRT archive
     (missing header information problem) - ongoing                        <<<<

STH  have had more than one meeting to plan the update to the Calibration 
PSB  Plan document (VIS-SPE-IOA-20000-0002). 
MJI

DWE  have had a number of meetings and have a draft version of the Plone 
EGS  site ready for reviewing by the rest of the group. DWE will send out
STH  a reminder of the address. Please send comments to DWE and EGS.       <<<<
MR

MJI  emailed JAC to see if it would be possible to exclude daytime 
     engineering test data from the data sent to Cambridge. This is being 
     looked into by JAC.

PSB  has set up the casuhelp email account. This is aliased to the wfcam
     account and the .procmail diverts all emails to MJI.

DWE  has placed the casuhelp email address on the web/Plone pages

STH  has not yet acquired the tiled supernova remnant image. JMI did not
     have access to the data, so he will now try Albert Zijlstra - ongoing <<<<

MJI  decided not to draw up a response to the UDS EDR paper - withdrawn   
MR

STH  has not investigated the effects of stacking on the 2MASS/WFCAM 
     non-linearity comparison - ongoing                                    <<<<

PSB  have had many meetings to decide on what documents are necessary for the
STH  UK review
JRL
MJI

STH  attended a meeting regarding the Z filter (with JPE and PCH). There was
PSB  an agreement to have a filter that looks very much like the WFCAM one.
     It would be interesting to know if this has been ordered yet.
     
DWE  forwarded the Gaia meeting email to the rest of the group.

ALL  the ADASS 2006 deadline for abstracts has passed. EGS has submitted an
     abstract.


2.  Comments on WFAU minutes

PSB described some of the transfer tests that have been carried out and is
pleased that they have been done. He added that nothing much is happening on
the UKLight front, although the Cambridge end is ready to go.

DWE raised the issue of CASU web addresses and the need to inform people of
any changes when we swap over to the Plone system. There then followed a
discussion regarding the transfer period between the HTML and Plone sites.
For the first month or so, it was felt that the 2 systems should be running
in parallel. The speed of the Plone server was also discussed and it was
decided that a dedicated server should be bought. EGS, PSB and MJI will
write a specification for this machine.                                    <<<<
It was also decided that this machine would host the ING archive and 
possibly other web-based services.

DWE asked for more details regarding the flagging of persistent objects. 
MJI said that he, SJW and NCH were considering what needs to be done as
part of a general discussion on flagging "bad" objects.

MJI mentioned that there had been problems in sending a header updating
script to WFAU. The problem was caused by the attached script being 
embargoed for security reasons and then redirected to a black hole, without 
informing either recipient or sender that this had been done. The solution 
was to send the Perl script gzipped.


3.  News from IAU GA

DWE said that the IAU assembly had been very interesting including some 
entertaining planet-definition meetings. Another measure of flux for
extended images was mentioned by Konrad Kuijken during one of the talks 
(psf-corrected aperture-matched colours).  MJI said he was aware of this 
measure and had discussed it with Konrad at a VST public survey meeting 
last year.

The Commission 8 meeting was more interesting than usual since it was
dominated by new telescopes and space projects rather than results from
outdated equipment. An extensive document explaining the most recent IAU
resolutions was mentioned (USNO Circular 179 by Kaplan). Pat Wallace gave a
summary of this which will hopefully by available from the Commission 8
website (which we will be hosting for the next 6 years due to DWE being the
new vice-president).  The most relevant items here are dropping the redundant
EQUINOX keyword when using ICRS and the abolition of Sidereal Time (it is 
replaced by the more fundamental Earth Angle Time).


4.  Upcoming VDMT meeting

MJI said that there was a bit of uncertainty about the next VDMT meeting and
that it might be held on the 4th or 5th September (or postponed by at least
a week). No agenda has been distributed yet.


5.  CASU help saga

MJI reported on the latest emails received to give a flavour of the type of
questions asked. 12 relevant emails had been received in the period since 
casuhelp had been set up.  On average each took about 1 hour to deal with
to fruition (one email reply is still pending). A number of them covered 
missing data, or data not yet transfered to WFAU.  Others were giving 
feedback and asking questions about the VDFS and the VISTA public surveys

There was then a discussion regarding how visible the casuhelp email address
was since none of the above emails went directly to casuhelp.  DWE was tasked 
to add the casuhelp email address to the bottom of each Plone page.        <<<<
STH was asked to contact SJW to place this email address on the UKIDSS
website and advertize it.                                                  <<<<


6.  Data archives update

AAT: JRL reported that AAT had swapped over to use PostgreSQL and that the
automatic updating of the archive had thus broken. JRL will fix this.      <<<<

ING: JRL said that EGS had done an excellent job with the ING archive user
interface. This covers 1996+ data and should now be the default interface. 
If earlier data is required, users should be redirected to the old 
interface.  The new interface is almost ready to go live.                  <<<<

There are still a few things that need fixing in the headers of some of the
files with the 1996+ data such as some FILTER and OBS keywords. JRL will
write a script for RWA to interactively run on these files. 

WFCAM: MR reported that all tapes up to 31 July had been received (ie. all
of 06A). The raw archive was also up-to-date. ESO transfers have about one
week of data to go before they have completed 06A. All the data has also
been processed apart from one night (27 July), which is still going after 6
days of processing. This is because it is part of of the GPS survey and is a
very crowded field. RGM pointed out that the WFCAM summit pipeline sometimes
had backlog problems because of these fields. JRL said that this was due to
the requirement of processing the whole magnitude range. For the VISTA
summit pipeline this will not be a problem since the requirement is for DQC
only and the magnitude range to be processed can be limited.

UKIRT: JRL said that a couple of fixes had to be carried out after the 
Debianizing of the host computer. JRL was asked to email AJA that all  
should now be OK for routine inquiries.                                    <<<<


7.  Optical processing update

MJI said that the revised Software Management Plans for the VST surveys 
were sent in by 10 August.

He has also been finishing off processing the 1/2 Tbyte of UVEX data for 
Paul Groot (a proto-type Northern equivalent for part of the VST VPHAS+).

EGS will be re-reducing some data due to a RAID system meltdown (see later).


8.  WFCAM news

Progress with 06A: As mentioned previously, only one night from 06A is left
to process. This night, 27 July, would be a good one to stress test any
pipeline. MJI said that he was checking each night and then flagging them
as ready to copy afterwards. So far, 23 nights have been checked.

MR reported that he was reingesting all the 05A, 05B and 06A data into 
the DQC database.  This will take of order a week to complete. 

Calibration update: STH said that updated (slightly) colour equations have 
been derived from all of 05A and 05B and that these have been applied to the 
calculation of the photometric zeropoints of all data processed so far (05A, 
05B and 06A). DWE and JRL asked STH/MJI to email these colour equations.   <<<<

STH reported that SJW is writing a UKIDSS paper and that from changes in 
the median colours he is finding an RMS colour variation of 2.5%.  MJI 
suggested matching locii from colour-colour diagrams as a more robust method.

ZY zeropoint: STH noted that as there were no standards for these
passbands, extrapolation from 2MASS JHK values using linear colour equations
was being used.  However, the relationship between Y and the JHK bands is 
both non-linear in colour and luminosity class-dependent.  As a result,
A0 stars (for example) were offset by 0.1 mag in Y.  Thus not all star 
types are on the Vega system for these passbands.

PSF fitting: DWE reported that the tests with real data have gone reasonably
successfully. More checking is required with the M17 case although Prague
and Dubrovnik have got in the way of this. A new modification has been
carried out to the standard pipeline astrometry making it as good as PSF
fitting across the entire magnitude range for most circumstances.

After some discussion it was felt that there was a need to benchmark the
PSF software with respect to other PSF fitting software. STH suggested Phil 
Lucas as a possible person to contact on this point.

It was noted that both Paul Hirst and Sandy Leggett have moved from UKIRT to
Gemini.


9.  VISTA news

VIRCAM status: PSB noted that a VIRCAM shipping readiness meeting was being
held at RAL.  The dubious chip has now been replaced and there was a 
possibility that we would get a digital QE response curve for the replacement
chip. He again stressed that if there was another opportunity to acquire
some test data, it would be very helpful.  The possibility of doing this 
later on Paranal was discussed.

Recipe keywords: the VISTA VIRCAM data-acquistion software provides a 
RECIPE keyword in the FITS header to drive the UK VISTA Science pipeline. 
This keyword is explicitly ignored by the ESO Quality-Control pipelines
but is required for options within the UK pipeline processing.  PSB,
JRL and MJI need to come up with a proposal for these and discuss
with SMB.                                                                  <<<<

VDFS docs status: The ETC has now been updated to have the new overhead
algorithms. This results in a ~10% improvement in observing efficiency.

JRL said that he wanted to update the data flow figures in the DRLD. STH 
will provide these.                                                        <<<<

The VISTA data transfer from ESO is currenty planned to be via disks.
These are currently standardized at 250 Gbyte Linux file systems (ext3).
The benefits (factor of 10 less disk traffic) of using the latest 750 Gbyte 
disks with Rice compression were discussed. PSB thought that an ICD on
data transfer from Garching to Cambridge was needed.  He will raise
this issue with JPE.                                                       <<<<

Upcoming UK review: MJI outlined the documents and web access needed 

Overview of pipeline processing philosophy                                MJI

Document summarising the documents and discussing the UK URD 
requirements with comments and pointers to how/where these will be        MJI
met/discussed or otherwise (may include addendum of VISTA public          STH
survey requirements)                                                   

Detail of the pipeline processing (aka latest Calibration plan and DRLD)  JRL
which covers calibration and pawprint processing                          PSB
                                                                          
Extra document in same vein describing additional processing operations for 
UK processing eg. mainly tiling/mosaicing and interpolation options       JRL
                                                                            
Hardware design document including proposed data transfer mechanisms, 
hardware for processing and storage system                                PSB

DQC and pipeline progress database and web pages                          MR
                                                                          EGS

More detailed discussion document for photometric calibration based on    STH
WFCAM strategy.

Draft of WFCAM processing technical paper (as example)                    MJI

Plone website including technical description of WFCAM processing etc..   DWE
examples of progress web pages etc... as examples                         EGS
                                                                          MR 

10. AOB

ADASS 2007: JRL said that he wanted a meeting next Monday to discuss
organisational details for ADASS 2007.

Hardware update: MJI reported that the equipment grant (April 2005) has 
finally made it, so we can now sort out terminating the old grant and
assessing how we stand with equipment procurement.

The solaris UFS file system /data/cass55c was totally fsck'd and all data 
lost.  This RAID array has now been moved to the linux machines and rebadged
as the XFS filesystem /data/apm23_c (7 Tbyte RAID5). It is unclear what 
caused the complete meltdown but there appears to be nothing wrong with
the disks.

Another 10 Tbyte RAID6 array has been acquired and mounted as /data/apm23_d.

Power Cut: To prevent any problems from the mega power outage scheduled for 
the weekend of 16/17 September, all CASU machines will be switched off late
on Friday 15th and isolated from the mains for the duration (a generator will 
be supplying temporary power over some of the w/e and we don't want that
do we ?).  They will be re-connected on Monday 18th, or thereabouts, depending 
on whether or not we have a stable power supply again by then.  However,
this will not be the end of the saga and sometime after this, but in the
very near future, the power will again effectively go off for a whole w/e.


Continuing Actions
------------------

JRL  finish the fix to problem with UKIRT archive (missing header information)

STH  acquire tiled supernova remnant image

STH  investigate effects of stacking on 2MASS/WFCAM non-linearity comparison


New Actions
-----------

DWE  remind ALL about location of Plone site etc.

ALL  review Plone site and send comments to DWE and EGS

EGS  decide on a specification for the CASU web server
PSB
MJI

DWE  add the casuhelp email address to the bottom of each Plone page

STH  contact SJW to place this email address on the UKIDSS website and 
     advertize it

JRL  repair AAT archive (PostgreSQL problems)

EGS  link new ING archive interface to current ING archive page and
JRL  make it the default with option to use previous interface for
     pre-1996 data

JRL  email AJA about fixes to UKIRT archive

MJI  email new WFCAM colour equations to DWE and JRL
STH

PSB  agree on a proposal for RECIPE keywords and discuss with SMB
JRL
MJI

STH  provide JRL with new figures for DRLD

PSB  email JPE about ICD for Garching-Cambridge interface