You are here: Home / Documents / Minutes of CASU Meetings / minutes_050517.txt

Plain Text icon — Plain Text, 15 KB (15867 bytes)

File contents

17th May 2005  Hoyle committee room 10.00 - 13:30

Present:    MJI, RGM, PSB, DWE, EGS, STH, WJS, JRL
Apologies:  MR

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Global calibration of UKIDSS data
4.  VDMT report
5.  Data archives update
6.  Optical processing update - VST surveys news
7.  WFCAM telecon and processing update
8.  Post-FDR VDFS meeting at ESO
9.  Plone for documentation
10. AstroGrid liaison update
11. AOB - GAIA news

Minutes
-------

1.  Actions from last meeting

JRL  has still to include access to "observing" log index files in the
     raw WFCAM archive - ongoing                                           <<<<

JRL  arranged next JAC telecon (see item 7)

MJI  raised issue of flagging observations of standard stars and other 
     calibration frames at JAC telecon (see item 7)
     
MJI  raised issue of what is needed for the generation of master flats at
     JAC telecon (see item 7)
	  
MJI  are in the process of writing a new report on the detector properties 
JRL  as they currently are - ongoing                                       <<<<
EGS

MR   trained STH in the use of the pipeline

MJI  contacted Judith Moss re. getting a whiteboard

MJI  has not had a chance to produce a M51 colour picture - ongoing        <<<<

EGS  has generated and distributed usernames and passwords for the Plone 
     website

ALL  the discussion regarding Plone was postploned until item 9

EGS  generated an option for hiding the side bars in the Plone-generated 
     web pages

STH  had a meet with Guy Rixon and Kona Andrews to configure cass123
     (see item 10)
          
STH  approached the system managers with a view to placing cass123 in dungeon, 
     which has happened.  Further discussion was deferred to item 10.


2.  Comments on WFAU minutes

It was noted that the WFAU minutes were getting shorter as ours were getting
longer. This led to the postulation of a law of conservation of length of
minutes within a project.

MJI noted that there will be a UKIRT Board Meeting in Cambridge on 26/27 May 
at which MJI and NCH will give presentations.  MJI said that he would find 
out who will be attending this meeting.                                    <<<<


3.  Global calibration of UKIDSS data

EGS reported on a WFAU document on the global calibration of UKIDSS
data. The approach was to gather all the information available (observation
of standards, overlaps, etc.) and do a big non-linear solution all at once.
For this they apparently need PSF magnitudes and illumination maps.

There then followed a discussion on extinction and the best observing
strategy for determining extinction. Views were also given on the merits of
global versus non-global solution methods. It was decided that there was a
need to respond to this document within a timescale of a couple of weeks.
MJI said that he would place this document on our internal web page.       <<<<
EGS will collate the comments (aiming for about a page in size).           <<<<

There was a short discussion on the use of 2MASS and photometric standard
fields for calibrating WFCAM frames.  The effectiveness of calibration 
methods should be reviewed after the first period of WFCAM science 
observations finishes in June.


4.  VDMT report

The last VDMT meeting was on May 5th and a number of issues were raised and 
discussed:

MJI reminded us that the VDMT UK Review will be held on 16 September and
that we need to start assessing the amount of work required for the various
extras needed.

The data rate from VISTA has now been agreed and it is between a factor of 
1-2 more than WFCAM. This is mainly due to the shorter than expected exposures
of WFCAM and the longer exposures of VISTA.  This suggests that issues of 
scalability of the pipeline are irrelevant.

The UK design documents have been frozen (cryogenically), but are still 
awaiting signatures.  This could still take some time.

STH is still in the process of adjusting the style of the management 
reporting documents.  Although a phone call with JMS has clarified many of 
the issues, a written example would guarantee that the format is ok.

WFAU are working on the access rights to the science archive data.

STH reminded us that we need to send in our monthly reports to him before
31st May.                                                                  <<<<


5.  Data archives update

UKIRT: this is now back online and fully accessible (hopefully). The data 
occupies 1.5 Tbytes of space, consists of millions of I-node gobbling tiny 
files and covers the period up to mid-2004.  It has also all been completely
re-organised in a simpler and more consistent manner and has been backed 
up on 5 LTO-II tapes.

WFCAM: the data is online and JRL said that the query interface is 75%
working. This will look like the ING archive interface and for now have
straightforward password protection. At the moment JRL is tidying up the 
download facility and including the option of retrieving Rice tile compressed
or uncompressed data.  There was some discussion as to what the period data
should remain in the transfer disk buffer.  JRL and MJI suggested 48 hours, 
while STH and RGM suggested 1 week.  The problem, as usual, is limited disk
capacity in the staging area where the data can be accesses via protected 
http.  It was decided that we start with 1 week and see how
things go. It was agreed that there would be a 0.25 TB limit for each
request. 

AAO: most of the data (~1 Tbyte) is now online.  JRL will start working on 
the download side of the access.  The user interface already exists.       <<<<

ING: a start has been made reading the DVDs. However the favoured DVD 
reader/writer has now expired under the strain and either needs fixing or 
replacing.

ESO transfers: these have not started yet but will as soon as MR returns.


6.  Optical processing update - VST surveys news

STH reported that the processing of the INT WFC data for Anna Arnadottir 
had been completed.

MJI said that the public surveys for VST have been chosen. The 3 core
surveys are ATLAS (Shanks et al), VPHAS (Drew et al) and KIDS (Kuijken et
al). CASU are signed up for the processing for the first two projects. The 
last will presumably be processed by Astrowise. There are also 3 secondary 
surveys: UVEX (Groot et al.), z<2 census (Oliver et al.) and UKIDSS LAS/GCS 
(Warren et al.).  There will be a meeting at ESO on 22/23 June to discuss 
these projects.  The goal of the meeting is "to coordinate the various teams 
and maximise the science products of the surveys" and from our point of view
hopefully find out more about the data products required by ESO. 

MJI will attend representing VPHAS since Janet Drew can't make those dates.


7.  JAC telecon and WFCAM processing update

The last telecon was on 12 May. MJI reported on the issues raised:

The project ID for observations standard stars should be UKIRTCAL and
CAL for other calibration observations.  This has not quite happened yet
for standard star observations, but will be done soon.  STH said that there 
should also be a standard flag keyword set to TRUE, however, this also
hasn't happened yet.  It looks like the 1st month of science data at least
will have these problems with headers.  MJI pointed out that it is not a
CASU responsibility to alter PROJECT ID keywords in the headers so these
will remain as received.

There are occassionally missing files, up to now for single extensions.
Somtimes this occurs when a single DAS crashed and there just wasn't a file. 
But there have also been a few cases of files mysteriously vanishing before 
making it to tape. This is currently a low rate (roughly 1 in 10,000) and 
has been spotted when the MEFs are being created.  Various improvements to the 
checking procedure at JAC (and in Cambridge) are now being implemented to 
minimise the occurence even further and to ensure no files go entirely
missing.

The few single extension missing files that do exist have been recovered 
and included in the on-line data here (thanks to PH at JAC).

JRL pointed out that about half a night of data (1/3 end 11th and first 2
hours of 12 April) was taken with 1 detector (#2) not functioning ie. all 
zeros in the data file. This probably occurred after a reboot of the data 
system and was not noticed at the time.

There are routine procedures for taking twilight flats in all broadband
filters and a full set will be generated at roughly weekly intervals.

There have been some hardware problems with the summit LTO drives that are
causing delays in shipping out tapes to us.  So far this has not caused
any problems at our end.

MJI pointed out that any University of Hawaii and Japanese data will be
processed by CASU and they can access this processed data in the usual way
from Edinburgh. They will also be able to access the raw data from the CASU 
raw data WFCAM archive if they really feel the need.

The HAWK-I project was mentioned (see the recent ESO Messenger).  The NIR
camera has a  7.5'x 7.5' field of view using a 2x2 array of 2kx2k Rockwell 
3-edge buttable devices for use on the VLT (giving 0.1 arcsec pixel sampling).
This is scheduled for introduction at the end of next year.  Although the
GRASP is still a factor 3 lower than WFCAM if the image seeing is identical, 
MJI pointed out that the GRASP also depends on seeing**-2.

STH said that the processing was progressing well and that the pipeline was
becoming more stable. Four nights (8-11 April) of Science Verification data 
are currently being analysed. It takes about 24 hours to process one night 
of data on one CPU on one PC.  Since we have 8 CPUs available for processing,
this timescale is acceptable.

The dark frames have been found to be stable over the night. The variation
from night to night is about 10 counts in the DC level but the structure
is stable at the few counts level.

MJI again pointed out that the apparent sensitivity variation in the flats 
can be as large as a factor of 2. This appears to be a property of the 
detectors and not the filters. This will make producing a uniform survey 
very difficult.

STH noted that we were still tweaking the reduction strategy, particularly
the sky subtraction.


8.  Post-FDR VDFS meeting at ESO

(PSB's notes from this meeting are available as a separate document).

PSB reported that most things got sorted out during the visit to ESO 10/11
May. ESO did not get a chance to review all the documents before the
meeting, so some small changes may still be needed. The Board Dispositions 
are almost all done.

One of the changes is that the SDT has now become the SADT (Survey Area
Definition Tool or SAD Tool). WJS was asked to acquire the latest version 
of the SADT document for us.                                               <<<<

During a meeting with the DFS group and the DICB chairman (Adam Dobrzycki) 
it was suggested that the keyword for observations of standard stars be 
changed from SCIENCE to CALIB. It was originally thought that since science 
could be carried out with these observations this would be a reasonable 
descriptor, however due to issues related to ESO release of data CALIB is 
preferred.  This needs to be reviewed by the VISTA project as a whole.

A new opportunity for pipeline processing at Garching was revealed, a 
super-recipe. The processing for this is not triggered at the template level, 
but by someone "pushing a daily button". This idea might involve CASU 
having to provide some more recipes but it enormously simplfies the 
maintenance of calibration files.

JRL and PSB were given a CPL tutorial which went well.

It was pointed out that we will have to deliver v0.1 of the DRL by the
end of the year and the ETC C modules.


9.  Plone for documentation

EGS said that the transfer of the Plone system has been fully completed from
apm14 to apm15, which has two processors. Backups are done nightly and are
kept for one week. If apm15 goes down apm14 can take over.

DWE said that he had spent some time exploring the system and although there
were many useful features, the overall impression was some simple tasks, 
like document writing, were harder.  EGS said that one of the problems 
encountered by DWE (the bulk transfer of plots) could be overcome by using 
FTP.

STH said that he liked it, but that it did need some reorganizing since he
found it difficult finding things. He also didn't like the name "Issue
Collector". Of the forthcoming suggestions, the use of Bug and Fault were
rejected since the this tool covers more than this. In the end it was
decided to rename it the Reporting Database.                               <<<<

PSB asked what the Gaia group were intending to do with Plone. DWE
said that Plone had been installed on Vasily Belokurov's Linux machine and
that he was in the process of setting something up.


10. AstroGrid liaison update

Clarification was provided on the status of cass123. This machine has now
been physically moved into the dungeon. GTR/STH had directly contacted Hardip
Sanghera regarding the managing of cass123. It was agreed that the system
managers would continue to manage and upgrade the Solaris side of this
machine and that the AstroGrid software would be managed by STH. The only
additional work for the system managers would be to add cass123 onto the
backup schedule.

PSB said that the stated action had been very woolly and was not clear as to
what exactly the system managers were being asked to take on. In order to
make sure that everything was clear, PSB and STH will discuss this
further.                                                                   <<<<

RGM said that he would distribute a document by Tony Linde on service support
for VO.                                                                    <<<<

RGM also asked if we needed any more hardware for AstroGrid purposes. There
is a small amount of money available, but it must be spent by July. 
RGM, PSB and STH will have a meeting to discuss what should be bought 
with this money.                                                           <<<<


11. AOB - GAIA news

STH said that AstroGrid finally gave their presentation to the Staff Meeting
and that they were generally impressed with it. The questions asked were
mainly to do with how would it be supported and will any other system
duplicate its functionality.

DWE reported on a recent Gaia group meeting in which Gerry Gilmore reported
on his presentation to the PPARC science committee. This was apparently well
received and the committee seemed to be extremely enthusiastic about Gaia.

MJI said that the heating problems in the APM building have still not been 
sorted out but that he has found a way to temporarily override the problem
(too smart central heating and boiler controllers)


Continuing Actions
------------------

JRL  include access to "observing" log index files in raw WFCAM archive

MJI  produce a new report on the detector properties as they currently are 
JRL  and illustrate with processing examples
EGS

MJI  organise producing M51 colour picture needed for publicity


New Actions
-----------

MJI  find out who will be attending the UKIRT Board Meeting

MJI  place the WSA Photometric Recalibration Design Document on our internal
     web page

ALL  read and provide comments on above document

EGS  collate above comments

ALL  send in monthly reports to STH by 31st May

JRL  progress download side of AAO archive scripts

WJS  acquire the latest version of the SADT Definition Document

MR   rename the Issue Collector in the Plone system to Reporting Database
EGS

PSB  make sure all responsibilities regarding cass123 are understood
STH 

RGM  distribute document by Tony Linde on service support for VO

RGM  have meeting to discuss what equipment needs to be bought from the
PSB  AstroGrid money 
STH