You are here: Home / Documents / Minutes of CASU Meetings / minutes_050505.txt

Plain Text icon — Plain Text, 15 KB (15612 bytes)

File contents

5th May 2005  Hoyle committee room 10.00 - 13:20

Present:    MR, MJI, DWE, EGS, STH, WJS, JRL, NAW
Apologies:  RGM, PSB

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  VDMT and Q2 tasks roundup
4.  Data archives status
5.  WFCAM processing update
6.  Optical processing update
7.  FDR VDFS documents
8.  Plone for software documentation
9.  AstroGrid liaison update
10. AOB

Minutes
-------

1.  Actions from last meeting

DWE  have set up an internal disc resources web page. It was decided that this
STH  would remain text only since it would be easier to maintain. The location
     is http://www.ast.cam.ac.uk/~wfcam/internal/hardware/hardware.html
     Alternatively you can find it off the documentation web page.
     
STH  have arranged a meeting with Kona Andrews about AstroGrid deployment
MR   (see item 9) 

JRL  has made the required changes in the pipeline production code ("Blanknn"
     FITS keywords).
     
PSB  have sorted out their flights to Munich
JRL
MJI

JRL  has started to train EGS in the use of the ING archive scripts (see 
     item 4)
     
RWA  has not taken up the offer of help from a student from the Perse School.
     This would not have happened until the end of May, by which time the
     work should be completed.

JRL  has sorted out the problem with the way that the pipeline is dealing 
     with Rice compressed data
     
MR   has set up a fault report page (see item 5)
     
MJI  has prepared a list of what Comm-I data is worth reprocessing (half of
     it)
     
STH  has completed the work progress part of the reporting needed for the
     VDMT

MJI  has asked JPE about the current status of the RIX Response Document.
     The draft of the board responses document has been sent to Michele
     Peron

MJI  has finished the quarterly plan for 05Q2 

PSB  sent the updated ESO documents to Michele Peron on 3rd May

PSB  has upgraded cass123 and RGM's machine to Solaris 10

EGS  have looked into using Plone for the software documentation
MR

DWE  has provided section links in the documentation web page

DWE  emailed STH for suggestions regarding sorted output from the 
     documentation Perl script. There were no apparent short cuts, but the
     script was successfully modified anyway. An option now exists on the
     documentation web page for date-sorted output.

     
2.  Comments on WFAU minutes

The last WFAU minutes were noted.


3.  VDMT and Q2 tasks roundup

We were reminded that the next VDMT telecon would be this afternoon.
It was pointed out that the style of monthly reporting had changed yet again
and that in future when reporting effort used for a work package this
would be in terms of staff weeks (where 1 SW = 37 hours).

MJI then did a brief run through of everyone's primary responsibilities.

EGS optical data processing (will include VST)
    ING archive infrastructure scripts, internal SV for WFCAM
MR  WFCAM data processing (STH will be the backup for this)
    WFCAM transfers to ESO
JRL pipeline infrastructure and recoding
PSB VISTA documentation (50% - other 50% system management)
    (in future some recoding eg. ETC Perl -> C, testing CPL interface)
STH management documentation, WFCAM photometry and processing
DWE WFCAM documentation and PSF development (50% - other 50% Gaia)
MJI management, algorithm development, internal DV and SV for WFCAM


4.  Data archives status

JRL said that a WFCAM raw data archive has been set up using the Comm-2/SV
data. This has been set up as a basic single table with about 30 keys. This
is similar to the UKIRT archive. One of the remaining tasks is to arrange
how to restrict access to the data.

There was then a discussion regarding the availability of WFCAM observing
logfiles and how you find out what was observed on a particular night.
MJI pointed out that the index files for each night, that are created directly
from the FITS headers, could be used as the basis for logfiles.  These are
directly available from the raw wfcam data store and could also be made
available from the raw data archive web page                               <<<<

MR reported on the transfer of data to ESO. Only UKIDSS data and calibration 
files are being sent to ESO. It would be convenient if there was a simple 
way to pick these files out.  However, one of the problems is that currently 
the entries for standard star observations in the PROJECT header keyword 
are not unique, sometimes being assigned the engineering code eg. U/EC/10
and at other times the main observing programme code.  It was anticipated 
that when the "hourly" standard stars are observed, they would be done
with a separate identifier.  This issue needs to be raised at the next JAC 
telecon which needs arranging as soon as possible.                         <<<<

MJI said that JMI had volunteered to clone a WFCAM version of the INT WFC 
processed product archive (for internal use only, and mainly for 
troubleshooting and keeping tabs on the processing).

The UKIRT on-line archive has been resurrected over the last month. The main
problem has been the necessity to rename files in a more sensible fashion. 
RWA is still in the process of reading in the DVDs but should be finished 
by the end of next week.

It was also stressed that the AAT raw data archive needs to be up and 
running by the end of the month.  About a 1/3 of the data has already been 
read from DVD (occassional problems with scratches notwithstanding).  These
are unique online archives and expediting their availability is crucial
to the CASU archive work.

EGS has been looking at the ING archive scripts. For future transfers the 
Internet is being considered since the link from La Palma is being upgraded. 
In order to test transfer speed ING have been given an account on one of our
PCs.  Something along the lines of the method used for automatically
transferring data to ESO might be useful for this.

The task of putting the 7000 CDs from the INT archive on-line has been 
temporarily put on hold, due to the above work having a higher priority.


5.  WFCAM processing update

MR said that all the (useful) Comm-1 data has been reprocessed. This was for
the nights of 8-11, 14 and 26-30 November.  This still needs various checks
doing before it is released to WFAU over the next week.

The Comm-2/SV data has a number of problems that still need sorting out.  
Among these are the following:

A library of master darks, flats and confidence maps is needed.

A master sky will be generated for each filter for each night. STH asked
about the residual variation within a night and JRL said that this was 
below 1%.  This strategy is still being tested but if successful it 
alleviates the problem of sky correction (really mainly residual reset 
anomaly) - see later notes below.

JRL noted that the Comm-2/SV data seems to be linear ie. no non-linearity
correction was needed. This is due to various tweaking of the electronics 
that was carried out prior to on-sky Comm-2.  This is very good news since 
the full non-linearity correction is computationally intensive (currently
30s per pawprint).

There was a discussion on what observations were needed for regularly 
generating good master flats. This needs to be raised at the next 
JAC telecon.                                                               <<<<

MJI pointed out that there was a large apparent sensitivity variation on 
all detectors, whereby the response seems to vary by up to a factor of 2. 
This in turn causes problems for the generation of confidence maps, 
uniformity of surveys, and possibly also impacts on the calibration error 
budget. 

It was noted that this problem was not widely known.  It was generally felt 
that a new report is needed on the properties of the detectors as they
currently are. It was agreed that this time only one should be written and 
that it again would be HTML format with contributors putting in their own 
parts.   MJI and STH will edit the report into a consistent form.          <<<<

STH asked for clarification on what the current processing strategy is:

Dark correction -> Twilight Flatfield -> Decurtaining * -> Sky Subtraction

a.  Decurtaining also corrects for the majority of the reset anomaly.
b.  Data from point * is used to generate the Master Sky Frames via
    scaled combination and then the sky subtraction is carried out by
    rescaling first and then subtraction.
c.  Sky subtraction is not currently carried out in the summit pipeline

JRL said that he had rewritten the SDF to FITS converter and that it was 
4-5 times faster on his laptop than the previous version. However, it had 
not yet been tested in the summit pipeline.

MJI reported that he and JMI had done extensive I/O tests on the RAID 
file systems, the result being that all the CASU RAID EXT3 filestores 
(bar apm6_a) have been converted from EXT3 to XFS (designed for very big 
file systems). This has resulted in a factor 3 improvement in I/O performance
giving results close to the expected data transfer rates.

STH asked about the calibration files that are used on the summit. The
answer was that we create them here and update the summit ones periodically.
In the calibration directory only one version of each calibration file is
present in order to reduce confusion. The old ones are placed in a different
directory.

It was pointed out that MR needs to train STH in the use of the pipeline.  <<<<

In future a rota system will be in place to carry out the processing. MJI
will sort out getting a whiteboard for this and for general use.           <<<<

A colourful M51 picture is needed for publicity, MJI will organise.        <<<<


6.  Optical processing update

EGS had been asked to process Phil James's INT WFC data covering some of the
UKIDSS survey area.  This has been done.  The processed data is being copied
and sent back to Phil.  EGS noted that this had been very straightforward. 
This is good, since that's the goal of pipeline processing.

STH said that he currently has a visitor (a student of Sofia Feltzing) who
is working on processing 5 nights of INT imaging data. This was taken with
Stromgren filters, so it is likely that they will have to sort out the
calibration of the photometry themselves.

In the near future (from June onwards) we will be getting more IPHAS data. 
This will consist of runs of one week of data per month until December.

MJI said that the VST proposals are currently being assessed. There remains
an open issue regarding advanced data products from these surveys ie. what 
exactly does ESO want?


7.  FDR VDFS documents

These were sent off to ESO on Tuesday (3rd). PSB, JRL and MJI will be
going to ESO next week (Monday to Wednesday).


8.  Plone for software documentation

EGS gave a demonstration of a Plone version of our web site he has set up.
This is currently on http://apm14.ast.cam.ac.uk/. He said that it was
possible to migrate the whole VDFS documentation site onto this setup.
MJI noted that we may not want to migrate all of the VDFS documentation. 
Meanwhile, we were all invited to explore the site and provide comments.   <<<<
Usernames for the site will be sent out shortly.                           <<<<

DWE reported that the system managers were reluctant to install Plone on
the Solaris system due to the additional management overhead. Apparently,
the University Computing Service has looked into Plone and was not in favour
of it since it had too high a management cost.

EGS also showed the document search facility that came with Plone as
standard. This seemed quite impressive and definitely adds something that is
not currently available with out current system.

The initial reaction was that Plone contains some impressive features and 
should definitely be given a longer trial for the software documentation,
and pipeline progress web pages, fault reporting and so on.  A few of the 
luddites among us felt that Plone was possibly a bit off-putting and maybe 
an overkill for the simpler tasks such as the reports documentation.  
However, the consensus was that it was generally an impressive demonstration.

MJI also felt that the look-and-feel of the Ploned web pages was a bit 
cluttered and asked if the side bar information could be optionally turned 
off.  EGS said he would investigate doing this.                            <<<<

MR demonstrated the fault reporting web site that had been set up:
http://apm14.ast.cam.ac.uk/devel/collector/. This was positively received.

MJI and NAW raised the issue of backups of these web sites. Since they 
are not on the Solaris home disk file system network they have to be backed 
up specfically.  NAW suggested rsnapshot as a good candidate for software.
MJI pointed out that a mirror harware setup was needed as a backup since
we are otherwise vulnerable to single point hardware failures. 


9.  AstroGrid liaison update

STH reported on a meeting he and MR had with Kona Andrews, Guy Rixon and RGM
last week. AstroGrid want to set up 2 tools (sextractor and hyperz) that can
be used on the fly on our WFS data. The timescale for this is by the end of
July.

Guy, Kona and STH will have another meeting on Tuesday afternoon in order for
STH to be shown how to configure cass123 so that it can do this.           <<<<

STH suggested that cass123 should go down in the dungeon and pointed out 
that the system managers would continue to manage the Solaris side of this 
machine. He has been volunteered to broach the relocation topic with the 
system managers.                                                           <<<<


10. AOB

STH announced that his bet with PSB has now been paid off (bar some pud) 
and that a very good time was had at Grantchester last Friday lunchtime.

WJS raised some VISTA items of news including the availability of a 
new version of the UK URD.  There is a push for randomisation of the order
of the pawprints for generating a filled tile to minimise the effects of
persistence in deep stacks.  This is an SDT issue.  WJS also showed some
interesting pictures of cosmic ray events on some of the VISTA detectors.

EGS asked for clarification regarding the high-order products that WFAU will
be providing (stacking etc.). MJI said that CASU will be providing WFAU
with software to perform various tasks including stacking and mosaicing,
difference imaging, list-driven photometry and so on.  He also pointed out 
that global large area calibration of surveys is a work package that will be 
provided by WFAU.


Continuing Actions
------------------

None!


New Actions
-----------

JRL  include access to "observing" log index files in raw wfcam archive

JRL  arrange next JAC telecon as soon as possible

MJI  raise issue of flagging observations of standard stars and other 
     calibration frames at JAC telecon

MJI  raise issue of what is needed for the generation of master flats at
     JAC telecon

MJI  produce a new report on the detector properties as they currently are 
JRL  and illustrate with processing examples
EGS

MR   train STH in the use of the pipeline

MJI  sort out getting a whiteboard

MJI  organise producing M51 colour picture needed for publicity

EGS  generate and distribute usernames and passwords for Plone website

ALL  explore Plone version of our website and provide comments

EGS  generate option for hiding side bars in Plone-generated web pages

STH  meet with Guy Rixon and Kona Andrews on Tuesday afternoon to configure 
     cass123 
     
STH  approach system managers with view of placing cass123 in dungeon and
     it being managed by the system managers