File contents

3 June 2003 

Present  : MJI, STH, JRL, PSB, DWE

Agenda
------

1. Actions from previous minutes
2. Comments on recent WFAU minutes
3. Highlights/lowlights for monthly report
4. Preparation for June 12/13 WFCAM meeting
5. VISTA camera testing issues pre-delivery (Q2 2005)
6. VEGA e-science bid 
7. Comments on Perry's reprocessing notes for VEGA bid 
8. AOB


Minutes
-------

1. Actions from previous minutes

DWE  obtained Ian Bond's difference imaging software and has carried out
     initial tests on easy data. Single 2kx4k images took 70-80s to process
     on a 1.9 GHz PC.  The next step is to trial using data with different 
     seeing and offsets eg. WFC data for SN searches.                      <<<<
     The current version doesn't work on MEFs so DWE will ask IAB to
     modify his software.                                                  <<<<
MJI  acquired the 2MASS catalogue and placed it on apm3 
     (/local/raid/mike/2mass). This is in gzipped ASCII format. There was 
     some discussion about what format we should convert the data to, and
     after the meeting we decided to use sorted FITS binary tables as we 
     did for the EDRs.
MJI  has reminded JRE/WJS about the need to update science/user 
     requirements documents for VISTA
JRL  has reminded AA about need to acquire H and K twilight flats.
STH  still needs to contact AA about use of service night, possibly for    <<<<
     standard observations though not clear how much use this will be.
STH  has finished off the first draft of the photometry document.

MJI  will discuss the new plan for a survey manager with AA at the Edinburgh
     meeting. STH confirmed what DWE reported at the last meeting and that
     the intention is for 2 x 0.5 posts one based at JAC the other in UK   <<<<
JRL  booked the hotel for us for Edinburgh meeting
DWE  booked flights for us for Edinburgh meeting
STH  was not able to obtain more useful versions of the Astrowise documents.


2. Comments on recent WFAU minutes

DWE was impressed at the turnaround speed for producing the minutes of WFAU
meetings.

It was generally considered that a data transfer rate of 4 Mbytes/s was
readily achievable and good enough for general data transfer. Tests to 
Leicester from Cambridge achieved this. Similarly, pushing data from apm3 
up to Edinburgh also achieved this rate. PSB has liasesd with ETH on 
network transfers IoA -> ROE.  Get same basic speed ~4MB/s even from IoA 
Ultra 1/140 ftp server (when using normal ftp, ie. unencrypted). Same test 
to an ROE Sun gave <1MB/s, conclude some machines 'see' a real bottleneck. 
Sorted out firewall issues with ROE not passing some of IoA IPv4 space. 


3. Highlights/lowlights for monthly report

The monthly report is being collated by STH.  A brief discussion of the 
salient points ensued and the relevant extracts have been emailed to
Simon to fold into the workpackage summary and gantt charts.

There some concern over the lack of progress of the simulations 
workpackages and its impact on other packages.  JRL said that a better 
source of data for throughput simulations would be CIRSI data.  However,   <<<<
there is also a need for suitable data for interpolation tests and
DWE thought that the WFC option be better for this.                        <<<<
There was also some discussion regarding the need for totally synthetic 
data (eg. SDSS) to test catalogue generation. STH will investigate the 
the SDSS software for this purpose.                                        <<<<


4. Preparation for June 12/13 WFCAM meeting

STH reported that he was working on the photometric standards presentation.
The pre-meeting document will be sent to Mark tomorrow after we have all
had a chance to comment on it.  This will also feed into meeting on 6th
here with PCH and SJW to discuss calibration and other issues.            <<<<

MJI and JRL will produce their documents for items 6,7 "QA pipeline 
processing at the summit" and "Unresolved issues ...." by Friday.          <<<<

PSB raised the issue of feedback of the DQC and differences between the
summit and the Cambridge pipeline. DWE gave a summary of AA's viewpoint
regarding this which he gave at the 14th May meeting at Imperial. PSB
wondered if there were implications for VISTA design. There was then 
some discussion regarding current ESO policy on this topic.

We should all read the documents sent out by Mark Casali before the 
Edinburgh meeting for discussion en route.                                 <<<<


5. VISTA camera testing issues pre-delivery (Q2 2005)

Steven Beard raised the question with PSB about whether the pipeline 
DQC modules will be ready in time for the VISTA in-lab commissioning 
(Q2 2005)?  The basic answer is yes, provided the FITS headers and data 
container structure are MEFs - which appears to be the case.

However, it is unclear exactly what quality control measurements would be
required. If something non-standard is required, we need to know about it. 
PSB will investigate and report back.                                      <<<<


6. VEGA e-science bid

There was a general discussion regarding the AstroGrid2 and VEGA 
e-science bids and how it might impact our activities.  General data 
centre activities are folded in the AstroGrid2 bid; further data 
processing development in the VEGA bid.

MJI reported that the new CASU grant will be a revisable grant rather 
than a rolling grant.  This reflects the project-based nature of
most of the likely funding.  An almost correct letter announcing the grant
was received by Paul Aslin on 20th May.  Minor revisions are being
incorporated in Swindon and we expect (sic!) the final? version to be 
sent any time now !

There was some concern expressed about how much/little secretarial support 
we get.  Some of the problem has been in identifying which tasks to 
delegate and a reluctance on our part to impose on the general 
secretarial pool.  MJI pointed out that the new grant (and the old)
has formula funding for secretarial effort at the level of 0.1 FTE
per PDRA position.  He will raise this issue with Paul Aslin and suggest
we get assigned use of one or two named individuals rather than the
current unsatisfactory arrangement.


7. Comments on Perry's reprocessing notes for VEGA bid

The number of versions of the pipeline has been increased from 3 to 6 .
In the VEGA bid Perry wrote a section about the possible need for 
reprocessing of ALL the data each time a new version is released and
the concommittant issues of storing ALL the old versions of the data
products.

To summarise the discussion:

Versions 1 - 6 of the pipeline and archive are major releases.  In 
addition there will be minor enhancements, bug fixes and so on eg. 1.1
1.2 .....  Clearly, it will be impossible and unnecessary to reprocess 
ALL the data everytime a minor change to some package is made, but this
raises the issue of when reprocessing will be necessary and how
end-to-end it has to be.

Clearly, major bug fixes will necessitate reprocessing; tuning of the
pipeline, particularly during commissioning and the first few months of 
science data, will require several iterations to get the best results.   
For WFCAM we propose to not release any of the data to the general community
during this version 1 shakedown phase (Feb-Jul 2004) other than to WFAU for 
trial ingestion and shakedown at their end and to the heads of the survey 
teams for assessment.

Further downstream, as it stands the VEGA proposal is to reprocess ALL 
the data with each major version release of the pipeline.  However, this 
is not sensible if only some later part of the pipeline has changed 
eg. the catalogue generation.   The pipeline is modular not only in its 
operations but also in its underpinning structure.  This gives us the 
flexiblity to only have to rerun the parts of the pipeline that have 
changed and to incrementally archive only those parts of the data products 
that have changed.


8. AOB

a. Data Processing and Storage Hardware

The plan for the data processing and storage hardware is to rack mount 
it all in the APM building.  This will have to mesh in with the 
refurbishment of the APM building to create more office space, being 
undertaken by the University.  MJI will contact Paul Aslin and try and
get an updated schedule for this work since some of it will be 
disruptive.                                                                >>>>

On a similar note JRL's office and several adjacent offices in the SPO
are being de-asbestoss'd in the near future.  We should take this 
opportunity to rack mount the SUN hardware with the Data Centre 
archives and servers stuff and move it to the APM building for the 
duration and possibly longer.                                              >>>>

b. UKIDSS meeting

STH reported that he had attended a UKIDSS meeting (GPS, GCS and LAS). 
Two points of interest:
1) GPS only plan to observe offset skies every two hours which is not 
sufficient.  They will need to revise their plan.  Some science progs 
have a requirement to measure JHK in sequence. If we have a data reduction 
requirement which conflicts with this then we need to sort this out. 
A possible compromise is a minimum duration single filter block.

2) A significant fraction of the time, in otherwise good conditions,
will be non-photometric due to thin cirrus etc.  An additional problem 
in such non-photometric conditions is possible differential transparency 
variations across the wide WFCAM field, making post facto bootstrapping, 
using adjacent fields, of somewhat limited utility.  Accurately 
calibrating science data taken in non-photometric conditions will be a
non-trival problem and also needs addressing at the survey strategy 
design phase.


Continuing actions
------------------

MJI  carry on with Rice compression tests: randomness of the noise 
JRL  and stacking tests

STH  still needs to contact AA about use of service night

JRL  compare dark sky flats with the twilight ones

MJI  contact AA regarding new plan for survey manager (2 x 0.5 posts)

New actions
------------

DWE  test Ian Bond's software on data with different seeing and offsets.

DWE  encourage IAB to modify his software to work on MEFs.

JRL  produce simulated data for end-to-end tests of pipeline using CIRSI
     data but with WFCAM-style FITS headers

STH  investigate SDSS software for generating totally synthetic data

DWE  use WFC data to simulate undersampled offset data for interpolation
     and difference imaging tests

STH  prepare pre-meeting documents for the Edinburgh WFCAM meeting
MJI
JRL

All  read pre-meeting documents for the Edinburgh meeting

PSB  find out what quality control measurements will be required during the
     in-lab commissioning for VISTA

MJI  find out timescale for APM building refurbishement and organise skip
     for junk removal in advance of

PSB  rack mount JRL's SUN equipement and move to APM building