You are here: Home / Documents / Minutes of CASU Meetings / minutes_031216.txt

Plain Text icon — Plain Text, 15 KB (15517 bytes)

File contents

16 December 2003 

Present:   MJI, JRL, PSB, DWE, STH, RGM

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Report from Q4M2 VDMT (2nd December)
4.  Report on JAC telecon
5.  Report on Astrowise visit
6.  Report on ESO visit
7.  Progress with ISAAC reductions
8.  APM building revamp - update
9.  AOB

Minutes
-------

1.  Actions from last meeting

STH  has continued checking the photometric standard fields for bright star
     contamination.  DSS images of the relevant regions have been
     obtained but not fully checked yet.  MJI suggested using a catalogue 
     (eg Hipparcos) to help with the checking.                             <<<<

STH  the numbering of documents is now 85% complete. STH promised that it
     would be completed before he goes away.                               <<<<

MJI  the information regarding the QE variations of the 4 science-grade 
     detectors has been distributed by email. The current information is not
     that detailed. All are different, so significantly different colour 
     equations may be needed for all chips if these variations persist.
     A list of lab tests has been produced in conjunction with MMC, JRL and
     SJW to be carried out in January.  MJI said that he would place the 
     list on the internal web pages.                                       <<<<

JRL  has not finished the reduction of the UFTI service data - ongoing     <<<<

STH  is up-to-date on acquiring the UFTI service data - ongoing            <<<<

STH  has checked with SJW about the issue of manual CIRSI and FIRES 
     comparisons.  SJW is not interested in the CIRSI data and although 
     interested, he has no time to process the FIRES data.

STH  checking the spatial systematics in the UFTI standards data and new 
     test data is an ongoing task.                                         <<<<

MJI  supplied WFAU with a night of processed WFCAM-CIRSI test data with 
     updated headers and Rice tile compressed versions to test WSA data 
     ingestion.  WFAU have said that the data was fine and that it does
     conform to their schema.

STH  has updated the risk registers (long-term and short-term illness risk) 
     and placed them on the web

MJI  are providing JPE with budget requirements for the next VDFS bid (Jan).
STH  This will be iterated upon with JPE and RGM.  An estimate of the current
     VDFS costs is also required with the costs partitioned into 6 month 
     intervals starting 1 April 2002 and broken down into Travel, Consumables,
     Equipment and Staff. This also needs to be extrapolated to end of 
     September 2004 - ongoing                                              <<<<

PSB  finalised agenda and goals for their ESO visit. See later report on 
JRL  visit which has been placed on the internal web pages                 

ALL  sent in brief reports on progress for Q4M2 VDMT

PSB  found out about date for PDR at the ESO meeting (see later item)

DWE  asked Paul Aslin regarding the DAC election. Apparently the information
     regarding the nominations was placed on the notice boards and in the
     pigeonholes of academic staff "on the assumption that it would reach 
     those interested". MJI said that it looked like all Hoyle pigeonholes
     had got one.

STH  sent in the written nomination for the DAC.  STH duly got elected.
JRL  He said "Things will change".
MJI

MJI  Attended APM building refurbishment meeting (see later item)
PSB  


2.  Comments on WFAU minutes

The only comment was regarding the data transfer rate, which raised the 
point that a data transfer test to ESO has been done which achieved 
1 MByte/s, for several Gbytes of recalcitrant FIRES data, a minor miracle 
given the current 34 Mbit/s bottleneck at Garching.


3.  Report from Q4M2 VDMT (2nd December)

STH reported on the last VDMT meeting, noting that CASU delivered their 
usual WP progress spreadsheets and summary documents, and then summarised
the rest of the meeting via going through the action list:

* JPE wants to work more with PSB to progress the ESO documents. This is 
ongoing.  The issue of the VISTA Science (and User) Requirements document 
was raised again.  WJS needs to rewrite this document before the VDFS PDR. 
This will be raised at the next management meeting and is also tied in with
the formation of the VDUC.                                                 >>>>
  
* JPE needs to add more management documentation to his VDFS web site. RGM
said that he had visited this web site and that 50% of the links were broken.

* JPE will generate an updated master list of dates needed for the CASU
GANTT charts.

* MJI to supply WFAU with WFCAM-like CIRSI data for ingest testing (done).

* MJI/STH to supply budget information to JPE (in progress).

* STH/NCH to discuss calibration plan - in progress, awaiting feedback(?)

* CASU to have regular telecons with JAC.  Rather than be prescriptive over
timetabling these it was decided that these would be organised as and when
they were necessary and at convenient times (since the VDMT there have been
2 telecons with JAC)

The next VDMT meeting will be on 9 January in Cambridge.

After the main VDMT meeting MJI, JPE and PMW discussed VDFS budget issues.


4.  Report on JAC telecon

JRL reported on the last JAC telecon which was held two weeks ago. This was
a short (30 minutes) meeting which mainly dealt with the state of the
pipeline and what the current plan is.

The big issue was how to do pipeline acceptance. How can this be done with
no real data?  What has been proposed was that the CASU pipelines process 
UFTI data for a comparison with the current JAC pipeline.  Since this test 
has already been carried out at JAC, JRL will repeat in Cambridge using the
standard pipeline here and then write a short report on both sets of 
comparisons in time to present at the next VDMT.                           <<<<

The next meeting will be 9pm tonight (the main outcome of which MJI has 
paraphrased in the above).


5.  Report on Astrowise visit

MJI reported on his visit to Groningen to find out about Astrowise.  
Representatives from WFAU - NCH, IAB and ETWS - were also there with a 
particular interest in the Astrowise database operations.  Ed Valentijn and 
colleagues presented talks explaining the Astrowise concept and summarised 
progress to date.  

The following is a brief precis of MJI's notes of the visit:

The Astrowise mission is a Virtual Survey System which seems to be attempting
to factor in most of the elements of AstroGrid, the pipeline processing 
operations at CASU, and the science archive activities at WFAU.   

It is a highly abstracted layered concept, all Object-Oriented, and built 
around an Oracle 9i databse and Python scripting language.  Everything is 
controlled through and by the database.

One of the main uses for Astrowise will be to deal with OmegaCam VST data. 

The user interface is via Python I/F to SQL and the database provides both
an environment (for data processing) and a dynamic archive - all the way
through from survey design to end products.

The pipeline processing components are virtually all imported:   
  image processing is via the Eclipse C library written by N. Devillard (ESO)
  wrapped with SWIG;
  coaddition uses Bertin's SWARP package via system level calls;
  source extraction is via Sextractor;
  uses LDAC for astrometry and photometry;

Astrowise views pipelines as data administration rather than processing 
since everything is DB driven.

CPU is via servers that through the DB schedule CPU jobs by polling known 
systems to see if they have spare capacity.  Gbit ethernet is used for
some of the interprocess communication.  Much of the processing is done 
using 32-node Beowulf clusters (eg. 32 CCDs in OmegaCam, philosophy is 1 
node per CCD).


6.  Report on ESO visit

PSB reported on the visit of himself, JRL, JPE and JMS to ESO, to talk
with representatives of DMD (Peter Quinn, Adam D. Benoit P., Michelle Peron)
about VDFS issues.

The VDFS delivery schedule was discussed.  Working backwards from 
commissioning: PAE 3 months before 1st system integration; FDR ~1 year 
before PAE (ie. around October 2004); PDR ~6 months before FDR (ie. around
April 2004).  

Peter Quinn emphasised that ESO DMD produces calibration for their raw 
data archive, but do not currently engage in surveys, nor produce science 
products of the type generated by CASU.  This might change in the future
but would require changes to the OB preparation tools and may require
the concept of linked OBs.

The issue was raised if CASU personnel would be needed for the first VDFS
commissioning. If this was the case then budgeting for this will be needed.
VDFS should have physical presence, during `commissioning 1', manual 
operations, `commissioning 2' has a working pipeline (including calibration 
library), with 3-month separation typically between C1 + C2.  The VDFS 
commissioning plan should be integrated with the Camera plan.  Design of
allowed observing templates are frozen 4 months before phase 2 proposal
(eg. a dither sequence should be one template to trigger reduction correctly).

Operations/Storage/disk arrays/connectivity: it is possible that in future 
there will be a direct fibre connect (EC `remote') in Chile.  EC have a 45 
Mbit/s ring in Chile/Argentina and might get 155Mbit/s -> JANET,  ESO are
currently using IDE disks for data transfer, there could be ~2 week
lag if problems, and so need a buffer at Paranal.  There is a "small" failure 
rate with discs, but no one present was able to quantify this.

The ESO (Garching) internet connection is quite slow.  They currently have 
a 34 Mbit/s link to the internet, but will upgrade this to 155 Mbit/s at 
some point in the future.  It was pointed out that there is 1 Gbit/s 
connectivity from CASU to the SuperJanet backbone and that the 1 MByte/s 
data transfer from ESO to Cambridge (mentioned earlier) amounts to 25% of 
the 34 Mbit/s capacity.

ESO are very keen for CASU to use the ESO in-house CPL (C modules and C 
'glue') and QFITS libraries, although delivering complete pipeline modules 
based around our own software modules and CFITSIO is also not ruled out.  
The use of CPL was not seen as a problem, but using QFITS would require a 
lot of extra work since the CASU pipeline is heavily dependent on CFITSIO.  
Furthermore, QFITS does not have Rice tile compression, nor WCS support - 
though the latter is not much of a problem since the pipeline uses WCSTOOLS 
for most WCS calculations.  CASU will request Rice tile compression as a 
native (transparent) format for QFITS.

The PDR for VISTA is going to be April+. The documentation needed for this
probably needs to be ready at least a month before this eg. mid-March.

The status of other documents has implications for the VDFS ESO document
deliverables.  The Data Reduction document cannot be done until the CPL 
document is released. The User Requirement document needs the Science 
Requirement document (see earlier comment). This probably needs VDUC to 
help push things along. 

WFCAM MEFs: it was agreed to add ESO-archive compliant header items 
on-the-fly; ESO would rather NOT have checksum in the MEF (since ESO 
generate their own).  Benoit and Adam wanted a clearer idea of what they
would be getting.  MJI suggested pointing them to the online example WFCAM
FITS header and possibly a WFCAM dummy CIRSI example.  It was agreed CASU 
will only present non-proprietry (ie. UKIDSS) WFCAM data to the ESO-supplied
transfer tool                                                              <<<<

Doxygen and Qfits have now been installed in Cambridge for evaluation.

Overall, the ESO trip was regarded as very positive. PSB said "all the
problems were little ones".


7.  Progress with ISAAC reductions

JRL reported that he was "nearly there". All the data had been checked and 
the missing frames recently acquired.  This now amounts to some 4000 science 
frames (~1000 J, 1000 H, and 2000 Ks), ~25 GB of data.  Different 
flatfielding and sky subtraction strategies are being investigated and
the results will be presented early in January for CASU group discussion.


8.  APM building revamp - update

On Monday 1st December, MJI, PSB and Paul Aslin met up with two architects 
and Estate Management representatives to go over possible choices for the 
refurbishment of the APM building.

A number of options were presented, considered and discussed, taking on board
the wish of the CASU group members not to have a large open plan office area. 
The end result was that the architects withdrew to draw up some plans for 
futher discussion in January.

The timescale of the refurbishment will be summer++, stunningly well matched
to the current commissioning schedule for WFCAM.

It was apparent that no serious thought had been given to all the
computer equipment that is currently in the APM building while the
refurbishment is being carried out (let alone the huge plate and film
and ING archive stores).  Portacabins were suggested, but the security and 
network access problems with this solution caused it to subsequently be 
ruled out.


9.  AOB

STH raised the issue of the reports that have to be in by the next VDMT
on 9th January.  STH will be between two observing trips during that week,
so that these reports must be in by the 5th January.  STH also said that 
he would much prefer it even more if they were in before Christmas. These 
reports are the Q4m3 monthly report and a first pass at 04Q1 deliverables. <<<<

The rest of us will have a group meeting early during that week.

RGM reported that Steve Warren had submitted a memorandum to the UKIRT board
which had not been circulated for comments beforehand.  A number of issues
were raised, mainly relating to responsibility for the the design of the MSBs 
and lack of oversight by the UKIRT board of the VDFS work for WFCAM.

It was felt that much ado about MSBs was caused by earlier misunderstandings
regarding responsibilities and that most of this was cleared up at the recent
UKIDSS meeting in Oxford.

In future the oversight issue will be remedied by VDUC acting as a conduit 
between VDFS and UKIRT and that this forum will also help mitigate against
further communication problems between UKIDSS and VDFS.  


Continuing Actions
------------------

STH  finish the southern extension of the photometric standards (eg. bright 
     star checking)

STH  finish off numbering of documents and email authors with the appropriate 
     information
     
MJI  place list of lab tests to be carried out in January on the web

JRL  finish reduction of UFTI service data

STH  acquire UFTI service data as it becomes available and oversee UFTI
     reduction and comparison 

STH  look at the spatial systematics in the UFTI standards data and new test
     data

MJI  iterate with JPE regarding budget requirements for next VDFS bid (Jan) 
STH  and the spend since April 2002


New Actions
-----------

MJI  raise issue of WJS writing the VISTA Science Requirements document at
STH  the next management meeting, stressing how important it is

JRL  write a report on the results of comparisons between UFTI data processed
     using Cambridge pipelines and using normal JAC pipeline and present
     at next CASU meeting

JRL  send example WFCAM FITS header and dummy data to Benoit and Adam
MJI

ALL  send Q4m3 monthly report and 04Q1 deliverables to STH before Christmas
     (or by 5th January at latest)