File contents

26th August 2004  Hoyle meeting room 10.00 - 12:30

Present:   JRL, PSB, MJI, DWE, WJS, STH, JMI, EGS & RWA (after coffee)
Apologies: RGM

MJI welcomed Eduardo (EGS) to the group meetings and pointed out that 
Eduardo will be working part-time on CASU projects from the beginning of
October.

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent VDMT and JAC telecons
4.  Upgrades to ING archive
5.  VEGA meeting report
6.  Pipeline processing IPHAS data
7.  Update on CPL + QFITS document
8.  APM building works
9.  AOB

Minutes
-------

1.  Actions from last meeting

MJI  has copied the correct version of the 040706 minutes to the online
     location.

STH  has supplied MJI with the location of the FIRES catalogues

MJI  has completed generating the catalogues from the FIRES data, but there
     has been no progress in the comparison, factoring in seeing-weighted
     stacking is needed first, ongoing                                     <<<<

JRL  have distributed the QFITS/CPL document (see later)
PSB

STH  has received a response from Simon Dye regarding the generated MSBs
     from the survey heads. Only Phil Lucas (GPS) has so far done this. 
     STH has a copy of these and will investigate their suitability.       <<<<

ALL  some comments have come in to WJS regarding the URD (v0.7).  Is this
     all ?  The release date is 31 August, so if there are any further 
     comments, they should be forwarded by the end of the week.

MJI  has managed to find storage for most of the homeless APM office 
     equipment during the building work. This is in the link between 
     Greenwich House and the Observatory building. MJI added that RWA has 
     already moved into JRL's office and that MTB is in the process of 
     moving out.  MJI's new pad, where most of the computer equipment will 
     go, will get a portable cooling unit.
     
PSB  is now happy with his bit of the Q3 plan.

JRL  has acquired the WFCAM acceptance test data from Paul Hirst.  There 
     are still holdups acquiring science detector test data.  This awaits
     unpacking the WFCAM kit in Hawaii.  Meanwhile, JRL will investigate
     the WFCAM acceptance test data from the engineering array             <<<<

MJI  has emailed his URD (V0.7) comments to WJS.

PSB  has forwarded the DIC to JPE who has passed it on to Michelle Peron.

STH  was unsure about the status of the RIX responses.  MJI will contact
     JPE to see if he has collated them and passed them on to Michelle
     Peron.                                                                <<<<

MJI  raised the issue of a RIX response meeting at the VDMT meeting, but 
     this is contigent on getting the RIX response document finalised.
     He will raise the issue again, but the date is unlikely to be 
     before mid-September.                                                 <<<<

JRL  has started updating the pipeline to more gracefully deal with lack of 
     fringe (or suitable flat) frames, ongoing.                            <<<<

     There was a general discussion about the need for new default fringe
     frames for the INT WFC. The current ones are well past their sell-by 
     date. JMI volunteered to look through recent observers logs to see 
     if any suitable data is available.                                    <<<<

JRL  is in the process of building in an option of using local versions of 
     the astrometric and photometric calibration catalogues into the 
     pipeline based on JMI's local catalogue access code, ongoing.         <<<<
     (Moral: as expected, external pipeline dependency not good.)

JMI  have rack mounted the apm2 and apm3 general purpose data processing
MJI  systems.


2.  Comments on WFAU minutes

From the last 3 WFAU meetings, MJI was interested in the discussion
regarding access to the WFCAM science archive.  There was some concern
about who would police the registration, authentication and data access,
particularly with regard to third-party collaborations.  It was felt that
the latter in particular was really an issue at the UKIDSS consortium
and UKIRT Board level.


3.  Recent VDMT and JAC telecons

At the last VDMT meeting, in lieu of the delays with the RIX response 
document, it was suggested that PSB should begin the Exposure Time 
Calculator (ETC) specification using ISAAC as a starting point.            <<<<

The 2nd August JAC telecon (sans MJI) consisted mainly of a discussion
about details of the pipeline between JRL and Paul Hirst.

The following telecon was on 23 August where there was further discussion 
regarding the slow reading speed of the LTO-I tapes recently received from 
the JAC (with UKIRT archive data on them).  At the moment our HP LTO-II 
drive (in the Overland loader) is reading them at 1-2 Mbyte/s ie. one night 
of WFCAM data would take more than one day to read in.  This is undoubtedly 
related to a tape record blocksize=512 bytes problem, which is what gtar 
reports the blocksize to be (instead of the usual default of 20x512 bytes).  
Tests using the X-ray group's HP LTO-I drive give the same problem and 
tests on both Linux and Solaris, using gtar and dd also give the same
problem.  Rewriting the same data to the JAC LTO-I tape and to one of
our own LTO-II tapes using our LTO-II drive gives read/write speeds on apm6
of between 50-100 Gbytes/hour.  The rewritten LTO-I tape will be sent 
back to Hawaii for further (compatibility) tests, after verifying that
the X-ray group's LTO-I drive can read it at flank speed.                  <<<<

It was noted in passing that since the LTO-II tape drive system has been 
swapped to a Linux box (apm6) the I/O speed is about a factor of two 
faster.

[Note added after the meeting: it transpires that the WFCAM LTO-I drives 
are manufactured by Seagate, leading to further avenues to explore.]

The news on the corrector front is guardedly optimistic, but even if it 
is delivered before the end of the first commissioning period in November,
the current schedule will still hold.  After engineering commissioning,
and autoguider commissioning, before WFCAM comes off the telescope, there 
will be a limited series of on-sky tests, the details of which are still 
under discussion.  

The current plan is then for WFCAM to come off for December/January and go
back on UKIRT in February to complete the commissioning in February/March,
followed by ~1 month of science observing.  WFCAM will then be next on in
~late summer ready for the beginning of the UDS observing season. This 
format is similar to the original plan and will enable CASU and WFAU to 
thoroughly shake down the end-to-end system based on the first month of 
survey science data.


4.  Upgrades to ING archive

RWA was invited to the meeting to report on various recent upgrades that 
have been carried out to the ING archive. Filling user requests under the 
old system was becoming too labour intensive based as it was on manually
reading from a mix of DAT tapes, CDs and DVDs.  Since the total raw size 
of the ING archive is (only) 14.6 Tbytes, the availability of efficient
lossless Rice tile compression and the advent of relatively inexpensive 
commodity mass storage disk systems have made placing the entire archive
online a feasible option.

This is a necessary prerequisite to the desired objective of automating 
the data retrieval and distribution, and indeed is one of several urgent 
tasks that the CASU ING archive working group had previously identified.

Over the last year or two this has involved reading ALL of the collection 
of 6600 DAT tapes onto RAID5 disk arrays. These DATs were themselves copies 
of copies of ...... the original tapes and there was some concern that the 
more popular requested tapes were reaching the end of both their 
sell-by-date and their working life.

These 6600 tapes (the original no. of tapes was several times that), cover 
the period 1982-1995, and have been checked.  They now take up (only sic!) 
730 Gbytes and are backed up on 3 LTO-II tapes !!  Mind you a battle 
royale is still ongoing to fix their many assorted FITS header foibles 
before they are Rice tile compressed.

The data since 1995 has been stored on 11,300 CDs and 1800 DVDs and a 
similar exercise is now well underway for the CD-rom collection, assorted 
problems notwithstanding.

In parallel with this JMI has been doing some urgent upgrades to the 
archive manager software collection both to make it more efficient and
to incorporate the new disk-based data scheme.  He has also been 
investigating automating calibration frame selection (it previously was
done manually) and putting the onus back on the user to select the 
relevant subset from the suggested calibration available.

All of this activity is also a necessary prerequisite for the next stage
of the planned upgrades which will include developing the schema and
infrastructure to support the option of on-the-fly data processing.


5.  VEGA meeting report

The VEGA meeting was attended by representatives from Cambridge, Edinburgh,
VISTA, Leicester, and MSSL.  Much discussion of management strategy ensued
with JMS giving a presentation on Earned Value Analysis (EVA) which 
highlighted both the advantages (simplicity) and disadvantages (simplistic).  
There was some concern that if external factors are not taken account of 
(eg. the delay of WFCAM and the consequent redeployment of some staff 
effort), the bald graph of weighted %completeness -v- time could be open 
to misinterpretation.  

There was also some discussion of future project oversight.  VDFS will 
press for oversight via the VISTA Board by augmenting the board membership
with a VDUC-nominated representative for VDFS matters.  Oversight for
the GAIA part is expected to be deferred until the larger bid is completed.

JeS forms for the VDFS part of the bid have been submitted to PPARC.  The
plan for the GAIA part is that 1.5 FTEs will be based in Cambridge and 0.5
FTE in Leicester.  Among other things their main task will be to work on 
preparing a bid to PPARC (mainly for the development of photometric 
techniques, data processing operations and a real time photometry database) 
in time for the ESA GAIA AO early next year.

It was also noted at this meeting that CASU and WFAU have expressed an 
interest in potential involvement with data processing and archiving any 
VST public survey data that UK groups successfully bid for (a requirement 
on any succesful bid).  To coordinate UK bids WJS and SJW (on behalf of the 
VISTA and UKIDSS consortia) are organising a meeting of interested parties 
in Cambridge for 16th September.  The VST will provide a timely 3-4 year 
window of opportunity for doing optical surveys in the Southern hemisphere, 
before the likely advent of one or other Southern Dark (Energy) Cameras.


6.  Pipeline processing IPHAS data

JMI reported that he had been successfully running the WFCAM prototype
pipeline on the IPHAS data (a Galactic plane survey in H_alpha, r' and i' 
+/-5 degrees either side of the GP and ~220 degress in longitude).  Each 
~one week run of data (3 received so far for this season begining in June) 
of about 250 Gbytes comes on a Firewire disc for ingestion into the
processing system.  

Over the year the IPHAS project has been running (9 lots of ~one-week runs), 
over 2 Tbytes of data have been imported this way, and this transport 
mechanism has proven a very efficient and cost-effective solution to the 
bulk shipping of data (with Rice tile compression you could of course get 
~750 Gbytes of raw data on each disk !!)  

Each ~1 week run of ~250 Gbytes takes about 2 days to process using the 
WFCAM hardware architecture, of which ~1 day is raw computer processing.
Each MEF can be processed in about 2 minutes per CPU (there are 8x this
available for parallel use). This has proven to be an excellent test  
and shakedown, of both the pipeline and the monitoring processes.


7.  Update on CPL + QFITS document

Nuria McKay (ROE) and JMS have fed back useful comments on the document
and Nuria has obtained a copy of the latest CPL guide for PSB and JRL to 
browse.  MJI and JMI borrowed PSB's copy to get up to speed on the current
state-of-play with CPL.


8.  APM building works

On Friday 27th August, Paul Aslin will have a meeting with Estate Management
and the architects in order to finalise timescales. It is anticipated that
building work will commence toward the end of September and be completed
by the New Year (see also item in 1.)


9.  AOB

MJI reminded us that CASU have 0.8 of allocated secretarial effort and that
we have been encouraged to make use of Jeannette Gilbert for this now that
JPO has gone.

STH and MJI discussed planning extra tasks for the next quarter, in lieu
of some of the previously planned commissioning work, to sort out the
documents needed for the UK VDFS review scheduled for the New Year. 
It was suggested that all review documents be sent out as pdf files, but 
to minimise work that the current diversity of document types be retained,
providing a minimum set of style requirements are met.                     <<<<

Due to incorruptibility JRL will be giving a poster rather than an oral 
presentation at the ADASS conference (24 October). JRL will sort out the 
poster material soon so that Richard Sword has enough time to work it up
into a more professional looking piece.                                    <<<<

STH pointed out that the next VDMT meeting will be on 3rd September and 
asked for the monthly reports to be sent in by Tuesday (31st August).      <<<<

There will be a meeting to discuss proposals with the Herschel Space
Observatory (far infrared and submillimetre photometry and spectroscopy) 
on 21-23 September.  WJS will be going and will report back.               <<<<

WJS noted that the next VISTA Project Board meeting will be on 2nd 
September, the day before the next VDMT.


Continuing Actions
------------------

MJI  compare FIRES project reduced data catalogues with ours

MJI  at next VDMT raise the issue of a RIX response meeting/telecon for 
     mid-September

JRL  finish updating pipeline to more gracefully deal with lack of fringe 
     (or suitable flat) frames

JRL  finish building in option of using local version of calibration 
     catalogues in pipeline

     
New Actions
-----------

JRL  investigate the WFCAM acceptance test data (that was obtained from Paul
     Hirst) via the slow tape

STH  investigate GPS MSBs (and any others obtained via Simon Dye)

MJI  ask JPE if he has collated the RIX responses and passed them on to 
     Michelle Peron

JMI  look through the INT WFC observering logs to see if any suitable data 
     is available to form updated master fringe frames

PSB  look at the Exposure Time Calculator (ETC) documentation for ISAAC data 
     and write a draft specification for the VISTA ETC

MJI  send an LTO-I tape written at Cambridge to Hawaii to pursue slow tape
     reading problem

STH  allocate a WP subtask for the next quarter to sort out the documents 
     needed for the UK VDFS review and review the style requirements

JRL  sort out the poster material needed for the ADASS conference

ALL  send in monthly reports to STH by Tuesday (31st August)

WJS  attend Herschel meeting and report back at subsequent CASU meeting