You are here: Home / Documents / Minutes of CASU Meetings / minutes_040210.txt

Plain Text icon — Plain Text, 15 KB (15503 bytes)

File contents

10 February 2004  Observatory meeting room 10.00 - 12.30

Present:   MJI, JRL, PSB, DWE, STH
Apologies: WJS, RGM


Agenda
------

1.   Actions from last meeting
2.   Comments on WFAU minutes
3.   Report on JAC telecon of 28th January
4.   Report from PPRP meeting and VDMT of the 4th February
5.   Status of ESO PDR deliverables
6.   Data transfers to WFAU and Garching
7.   Status of ISAAC reductions
8.   AOB


Minutes
-------

1.   Actions from last meeting

STH  met with Paul Hewett at the end of last week to discuss various
     photometric calibration issues that concern WFCAM and VISTA. The work
     has been divided up between them and should be done by the end of next
     week - ongoing                                                        <<<<

STH  checking the spatial systematics in the UFTI data is an ongoing task.
     No more UFTI data on this issue is foreseen, so this should be 
     completed by the next meeting.  STH also added that he has been in 
     touch with people at Caltech regarding CSO Tau data to see what (if 
     any) correlation exists between this and extinction, as part of the 
     photometry work package                                               <<<<
     
JRL  has sent some dummy WFCAM FITS headers to Benoit and Adam. This 
     probably will need one more iteration. The clarifications needed 
     could not be obtained from the example SDF header official WFCAM 
     document.                                                             <<<<

MJI  has been in touch with Andy Adamson and he (AA) will ask the ATC for the
     requested example SDF file(s) with all header packets present.  This
     is required for a full end-to-end test.

JRL  have looked at the issues involved in the CPL/QFITS route (see item 5)
PSB

MJI  asked AA at the JAC telecon about the progress of the engineering tests. 
     Apparently coplanarity tests have just been completed, optical alignment
     comes next.  A further email from Mark Casali said that they had cooled 
     2 science grade chips down and planned to cool the other two down shortly
     before commencing more coplanarity checks followed by device testing.

MJI  has been in touch with MMC and confirmed that the data from the lab tests
     do not need to have the full header packets in them and that we would
     prefer them in FITS format.  Transfer mechanisms to Cambridge are still
     under disucssion.
     
STH  finished the various GANTT charts and other documents needed for the
     PPRP and VDMT meetings.
     
JRL  produced copies of his UFTI reports for the PPRP meeting

JRL  JRL passed on the details of the missing ISAAC frames.  Although the 
STH  files were present, they only contained the headers. STH is still in 
     the process of tracking these ~100 or so frames down - ongoing        <<<<

MJI  has carried out the stacking of the ISAAC data (see item 7)

STH  has acquired the FIRES reduced data to compare with ours.  MJI said 
     that this example of the FIRES data had probably been stacked to 
     maximise faint image detection (ie. using some sort of matched detection 
     filter) hence the noise is smoothed (ie. correlated) over several pixels
     making direct comparison not so straightforward.  Visually the two
     sets are comparable.  He will do some further catalogue-based tests   <<<<

STH  calibrating the FIRES data is an ongoing task                         <<<<

JRL  has written an outline of a report on the ISAAC reductions. This will
     be iterated with MJI - ongoing                                        <<<<

JRL  has done tests on the ISAAC data doing flatfielding first, followed by 
     sky estimation and then sky subtraction.  The result was very little 
     difference, a 0.3% to 0.5% improvement in sky noise.  The motivation
     for doing the processing this way is to improve the rejection of faint
     images (and faint extended halos) in the sky estimation phase.

MJI  WJS has been invited to these meetings (but is in Edinburgh this week).

ALL  sent in end-of-month reports to STH

STH  emailed our VDUC suggestions to JPE. Tim Naylor is to be the chairman,
     but the rest of the committee hasn't been finalized (still waiting for
     some nominations to come in). It was thought unlikely that a VDUC 
     meeting would be held before March.  One of the first issues CASU would
     like VDUC to consider is the pipeline (and archive) User Requirements 
     for VDFS.  WJS is drafting a first version of this.
    
MJI  have discussed dome flats for VISTA with WJS and confirmed that 10-20%
PSB  variation in illumination across the array was fine.  The main 
     requirement (for non-linearity measures) is to have stable illumination.

    
2.   Comments on WFAU minutes

MJI noted that WFAU had asked us to consider checksum tests for the data 
transfer, together with a number of related issues in an email - see item 6.

Nobody could figure out what "mouse-bat-folicle-goose-creature-ampersand"
referred to.


3.   Report on JAC telecon of 28th January

A number of questions were put to AA:

1. engineering chip tests: no news at the time but see earlier item
2. getting SDF file for end-to-end test: AA will see to this
3. dome flats: confirmed that we don't expect super uniformity
4. more UFTI service data: confirmed none is planned and agreed none wanted
5. commissioning personnel: agreed to outline CASU plan but noted CASU 
   team probably not required on-island before mid-August
6. CSV AA & AL: found out about progress (confidential)

MJI reported that AA wanted to know more about the various reviews coming
up this year and was concerned (as is MJI) about the number of them and 
their possible interference with WFCAM commissioning.

The next telecon is scheduled for 18th Feb. This is delayed by about a week
due to JAC being busy with Michelle recommissioning.


4.   Report from PPRP meeting and VDMT on the 4th February

MJI reported that the PPRP meeting, a full house with standing room only,
seemed to go well and that we should hear officially shortly.  There was a 
lot of detailed questioning on several aspects of the programme.  MJI noted 
that having a uniform presentation style for the 4 talks (JPE,MJI,NCH,GIL) 
helped to create a better impression, thanks to ???? for suggesting it and 
to NAW for organising it.

WFCAM and VISTA operations was deliberately not included in this bid and 
will have to be bid for later.  STH suggested that we revisit the notes 
taken from DWE's and his meeting in Leicester in order to get some further 
ideas regarding the running of operations.

The VDMT meeting was an anticlimax by comparison. The main actions of 
relevance are that: someone from CASU should attend the VoProc meeting in 
Leicester on 25th February (JRL said that he would go since the topics were
more in his domain of expertise);  the EIS MVM pipeline is available and
MJI was asked to report on its relevance, or otherwise, to VDFS.           <<<<


5.   Status of ESO PDR deliverables and QFITS -v- CFITSIO

JRL reported on his comparison of the functionality of QFITS compared 
with CFITSIO.  A start on this was made by going through the pipeline 
software modules and noting what additional functionality would be required 
in QFITS if the pipeline code had to be converted from CFITSIO.  The 
top-level overview list is:

(1) lack of Rice tile compression
(2) the ability to read a subset of columns or image pixels
(3) the ability to access single extensions and image subsets within a file
    using the following notation:
        fff.fits[1]
        fff.fits[1][100:150,200:237]
        (NB: This is not a FITS standard, but it seems to be common practice
             in many FITS packages: CFITSIO, FUNTOOLS, IRAF, etc)
(4) some, but not all, routines return an error status and there is no
    inherited status in the sense that subsequent calls ignore the current
    status
(5) lack of header keyword 'classes' -- allows you to exclude certain classes 
    of keywords when building a new file from an old one.
(6) a good user's guide. The reference manual is not particularly detailed and
    too much is left unexplained.
(7) no Perl interface (there is a restricted Python interface)
(8) no WCS interface

PSB pointed out that although these features are not be in the current 
version of QFITS, it is possible that perhaps some of it will be in the next 
version.  It was agreed that the way forward here was to make a detailed list
of the functionality required if CASU were to adopt CPL+QFITS and to include
it as requirements in the Data Reduction Specification Document. 

PSB also noted that CPL might have some of the required functionality.
He and JRL will continue the comparison.                                   <<<<

PSB at this point showed some plots of a simulated VISTA file that he had
generated and compared the DS9 browser (that seamlessly mosaics on-the-fly
given a WCS) with the RTD screenshots that SMB had obtained from the
simulated data.  SMB had determined from his tests that while the proposed 
VISTA data format was fully compatible with RTD, that RTD currently has a 
couple of shortcomings including not being able to zoom out far enough to 
display the whole mosaic in place.


6.   Data transfers to WFAU and Garching

MJI reported on email exchanges he has been having with Eckhard Sutorius.

WFAU want some form of external md5 checksum carried out on the processed
WFCAM files before they are transfered.  All agreed that this seems 
reasonable, even though standard TCP/IP transfer protocol does something 
similar anyway.  

There are various trivial alternative ways of implementing this,
"openssl md5 filename" (also "md5sum filename") and built-in routines for 
doing this also exist in standard Perl distributions.  MJI noted that
ESO have asked for something similar for raw data transfer to Garching,
but they wanted to provide their own checksum routine for doing this.

DWE noted that an additional use of an external checksum would be to verify
that files haven't been modified since the checksum had been generated.
This seemed like a good additional reason and will be implemented.

Other matters discussed were:

* There will be one directory per observing night. This is the way it will be
processed and structured ie. on a nightly basis.

* The issue of handling read-readiness will be done via a lock file set in
the directory while the data products are accumulating in it and after the
processing is complete the lock file will be removed. An alternate strategy
would be to have a go file created when the data is ready to process.
Another strategy suggested was to process the data in a temporary directory
and rename it when the data is ready.  Any of these would do.  For now
we will stick with the lock file and revisit the issue when we actually
get vast amounts of WFCAM data processed.

Linked to this was the request from CASU that a receipt file be created
after WFAU had copied and verified the data.

* The expectation is that reprocessing will occur very infrequently and
(probably) in large batches (eg. after bug fixes).  This will require CASU
to inform WFAU of the fact that reprocessing is happening and why. The data
will go into the same directories.  The file headers will contain all the 
versioning information.

* no processed data will be officially released until CASU are satisfied 
with it. The exception will be test data for ingestion experiments.


JRL reported that ESO have asked for some more extra keywords to be added 
eg. explicit RA and DEC keywords with keyword values in decimal degrees,
an AIRMASS keyword etc......  These will be inserted on-the-fly during 
data transfer.

 
7.   Status of ISAAC reductions

JRL has finished pipeline processing the OBs, bar the few missing ones.

MJI showed some images of raw data (6x20s coadd); the same dark-corrected 
reset-anomalied, and flat-fielded (limiting mag ~20); and the output of a 
single OB (40 such K-band frames suitably dithered and stacked, limiting
mag ~22).  His further stacking experiments using the ISAAC data revealed
a number of interesting low level remaining instrumental artefacts, that 
only become apparent on deep stacking.  These artefacts were removed
using extant parts of the pipeline toolkit - at the expense of any real
large scale features.  It was noted that the FIRES team had done a lot
more data massaging before stacking cf. Labbe et al.

This data covers the Hubble Deep Field South and is a ~3x3 arcminute square.  
The stack of all 44 K-band OBs (representing 60 hours of integration on an 
8m telescope) has a limiting magnitude fainter than 24th in K.  A 
pseudo-colour image using all the stacked J H K data was also shown.  

Visually it is difficult to compare this reduction with the FIRES project 
reduction since their version has been resampled onto a finer grid and 
also smoothed.  MJI will do further catalogue tests on the two datasets.

The data will be made available to the Subaru group in order for them to 
perform further stacking tests.

This work will be incorporated in the report actioned earlier.


8.   AOB
     ---

MJI said that there would be another APM building refurbishment planning
meeting tomorrow, which will be reported on at the next CASU meeting       <<<<

DWE showed the latest iteration of the documentation web page with the
document numbers on.  This will seamlessly become the official documentation 
page by the end of the month.  The only request was to reduce the overall 
lateral size by shrinking the description column slightly                  <<<<

If there are any more documents that need numbers allocating to them DWE
needs to know about them.                                                  <<<<

DWE will then send out an email to the authors of the documents that need 
a number inserted.                                                         <<<<

WJS's draft of a draft VDFS User Requirements Document was discussed. It 
was noted that this was the UK version and not an ESO requirements version.  
A number of issues were raised but it was decided not to discuss this in 
more detail there and then but rather arrange a meeting with WJS for next 
week to go through it page by page. 
[A slight change of plan from that discussed at the meeting.]              <<<<


Continuing Actions
------------------

STH  finish the bright star checking of the photometric standard fields
     (by end of next week)

STH  look at the spatial systematics in the UFTI standards data and new
     test data (by next meeting)

JRL  iterate on WFCAM example FITS headers and files with Adam and Benoit 

STH  acquire missing ISAAC data

MJI  compare FIRES project reduced data with ours

STH  calibrate FIRES data using NICMOS or the published FIRES work

JRL  write a report on ISAAC reductions
MJI


New Actions
-----------

JRL  add in CPL functionality to QFITS -v- CFITSIO comparison
PSB  

JRL  attend and report back from VoProc meeting in Leicester

MJI  attend APM building meeting (Wednesday) and relate saga next time
PSB

DWE  alter documentation web page (reduce the overall size and shrink 
     the description column)

ALL  inform DWE of any documents that require document numbers

DWE  email document authors of the numbers that should be inserted. 

ALL  read and make notes, if necessary, draft VDFS User Requirements 
     Document in readiness for a meeting with WJS next week