File contents

12th June 2006  Hoyle committee room 13.30 - 15:30

Present:    DWE, MJI, JRL, PSB, STH, MR, EGS, RGM (to 14:30)
Apologies:  NAW, WJS

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Data archives update
4.  Optical processing update
5.  WFCAM news
6.  VDFS news
7.  Plone update
8.  AstroGrid liaison update
9.  AOB

Minutes
-------

1.  Actions from last meeting

MJI  has made no further progress investigating the merging of 
     classification statistics - ongoing                                   <<<<
     (It was noted that an interim solution had been sent to WFAU some
     time ago.)

WJS  it is not known if contact has been made with the VPO to confirm that 
     we will get no more ETC-related digitized data from them (WJS not
     present) - ongoing                                                    <<<<
     MJI said that he would email WJS to remind him about this.            <<<<
     
MJI  has emailed Nigel Hambly regarding the non-implementation of the 
     saturation flag in the classifier
     
JRL  has not implemented the fix to the problem with the UKIRT archive
     (missing header information problem) - ongoing                        <<<<

MJI  placed the submitted UKIRT Board report on our web site

JRL  provided MJI with the original data for the non-linearity graphs

JRL  requested further dome sequences to be taken. These are now in
     Cambridge (see item 5) and being analysed

STH  produced 2MASS comparison plots on a per detector basis

DWE  have not yet helped JRL in setting up an ADASS 2007 web site. This
EGS  will require copying and cloning various style sheets from previous 
     Plone versions of the ADASS web site

JRL  has released version 0.2 of the DRL. The current version is 0.3 and is
     about to be released


2.  Comments on WFAU minutes - transfer tests and UKLight update

PSB said that Johann Bryant has sent off the application form for UKLight
on 1 June. A response is now awaited from UKLight. The timescale of this 
and the implementation "roadmap" is unclear.

PSB also noted that we still wait expectantly for WFAU to carry out a transfer
test using single-thread large-buffer zero-encryption high-performance scp.

MJI pointed out that the NaNs found in the catalogues that had been noted in 
the WFAU minutes had now been fixed and the catalogues regenerated.  These
were a minute fraction of the total detected objects and only affected mainly 
flux values and errors for the largest radius apertures.


3.  Data archives update     - AAT, ING, WFCAM, UKIRT

WFCAM - MR said that all the data from 28 April to 31 May have been copied to
disc, reformatted to MEFs, verified and are awaiting ingestion to the raw 
archive. The latter is contingent on final tests on the updated verification
and flagging procedure (see later).

AAT - no change

UKIRT - no change

ING - EGS said that the restructuring of the archive from a tape/CD/DVD 
numbered system to a date-based directory structure is progressing well. The 
period 1996-2006 ie. the present is almost complete. Duplicate data, usually
due to several versions of data on multiple CDs/DVDs, is being tracked and 
corrected at the same time.  MJI noted that the internet transfers, already 
date-based, form part of this and are complete? from 20050101 to 20060531.
There is a small gap toward the end of 2004 that requires ingestion from
DVDs and that will be taken care of next.  After this the next stage is to 
investigate the pre-1996 data and see how far back it is feasible to push 
this. 

In parallel, the re-ingestion of the header information into a new version
of the database is progressing well. There is a need to finalise checks for 
any missing data.  The total archive is about 20-25 Tbytes in size and
has been backed up onto tape (~100 Ultrium LTO-II tapes).


4.  Optical processing update

EGS, STH and MJI have been working on data from various optical telescopes
(INT WFC, CFH and Subaru). RGM pointed out that it would be useful if EGS 
could update the INT WFC web pages to contain links to the latest 
flat-field, fringe and bias frames.                                        <<<<


5.  WFCAM news

MJI reported on the reprocessing of 05A data (Version 3.0). There are still
a couple of nights that are causing problems due to jitter keyword FUs, but 
these should be sorted as far as is reasonable in a few days.  11 nights 
with interesting sky file and sky correction problems (pipelines CAN lie) 
near the start of the 05A reprocessing were found after following up a
fault reported by Mike Read. This was traced as to a script update FU in the
CVS.  The affected nights have now been reprocessed and flagged as Version 3.1

In addition to the broken files discussed in the last minutes, WFAU reported
some 05B catalogue files with a few columns with a small number of NANs,  
mainly in large aperture flux and error estimates columns.

The catalogue software bug fix reported in minutes a while back, seems to have 
cured the problem and the catalogues were reprocessed and made available for
copying to WFAU.

MJI reported that there was also a mysterious problem with virtually all the
sky estimation and subtraction for the night 20051015.  This was picked up at 
UKIDSS QC checking.  MJI investigated what could have caused the problem but
other than possibly non-optimal choice of sky stratgey by the pipeline there
were no obvious faults.  However, as a final check, the night was reprocessed
using the latest pipeline version which uses an improved sky correction
strategy and this seems to have resulted in much better performance for that 
night - though final confirmation of this is still awaited.

MR went through some of the new features that have appeared in the 06A
raw data. In the period 28 April to 31 May (33 nights) there have been
many files with randomly missing extensions. Of the 50,000 files, 400 are 
incomplete (0.8%) and half of these are due to chip 2.  Additionally, and 
independently, the chip 2 extension is sometimes populated with null values.  
Around 3.5% of the MEFs suffer from this.  However, it is not quite that 
bad, since at least 1/2 of these occur on 1 night.  JAC suspect that this is
a controller problem.  About half the nights have at least one null
chip 2 file and these are spread evenly throughout the period. These files
were causing problems during ingestion and subsequent processing so a more
robust and thorough scheme has now been implemented to deal with them
and flag them by adding new keywords.  Subsequent propagation of this 
corrupted data (eg. interleaving and stacking) is already handled by
the confidence maps and provenance track record in the headers.
[AJA reported after the meeting that the problem had now been "fixed", as 
of early June, such that WFCAM is now dropping less than a frame a night.]

STH reported on the non-linearity saga. Further checking of the original
dome flats showed that all was OK at the 1% level. However, to recap, 
comparison with 2MASS showed that there was a 2-3% level departure from 
linearity over the full ~5 magnitude range of overlap.  More dome flat 
sequence data has now been taken by Sandy Leggett on 26 May to investigate 
this.  In addition exposures of standard stars covering a range of integration
times have also been acquired.   The processing of these is now underway and
we are even considering after a suggestion from Sandy Leggett that this might 
be a rate-dependent non-linearity (cf. NICMOS).

MJI said that the presentation he gave at the UKIRT Board meeting (23 May,
University of Hertfordshire) was well received. Mike Reid and Steve Warren
were also present at the meeting. The main debating point was the release of
WFCAM data with the PI data being the main issue raised.

STH said that he had no news regarding acquiring data from the 2MASS
touchstone fields. He said that he would chase this up.                    <<<<

MJI noted that further 7 and 10 TB fibre-channel RAID arrays have been added 
to the system and that we will be continuing with this upgrade route. It was 
suggested that DWE and MJI update the hardware web page.                   <<<<

MJI and PSB have had a discussion about bulk data storage requirements with a 
representative from HP who is an expert on Storage Area Networks. He seemed 
suitably impressed with our homegrown data storage system but, of course, 
followed up his visit by sending some literature for alternative (more 
expensive but setup and maintained by HP) bespoke solutions.

The planned upgrade for an extra air conditioning system in the "store room" 
was stalled for a week due to delays in delivery of the condenser unit. This 
is now expected tomorrow and will take about half a day to install. [Note
added on Wednesday: it was installed and is now operational].  This is all 
part of our proactive infrastructure upgrade programme and should now give 
us sufficient capacity (and backup) for the foreseeable future.


6.  VDFS news

There was not much new news on VIRCAM since WJS was away. PSB said that he 
had received a file from RAL since the last meeting, but it only consisted of
headers and no data.

PSB welcomed the release of the AIT Requirements document from JPE. This
mainly lists the lab tests that CASU asked to be carried out on the VIRCAM 
system . It was noted again that some Cambridge presence at these tests 
would be desirable.

The DRLD is now up to Version 1.4pre12 and final debuggery tests are ongoing 
on Version 0.3 of the DRL. These mainly involve the jitter/microstep recipe 
- the main science recipe. It is anticipated that V0.3 will be released at 
the end of this week. PSB, JRL and MJI said that they would have a short 
meeting around coffee time tomorrow (13 June) to finalise changes to the 
accompanying DRLD.                                                         <<<<

We were reminded that the community was in the process of discussing the
desired observing modes for VISTA. It was stressed that it was important for
us to be involved in this discussion so that the decisions made are
consistent with template and pipeline processing designs. These will be
further discussed at the VISAS meeting in Edinburgh (20/21 June) which JRL
will be attending to represent CASU.  RGM will also be there in his PS PI
role.


7.  Plone update

DWE described the latest progress on the Plone redesign of the web site.
There was a meeting of interested parties earlier today. The overall design
seems to have settled down, but most of the technical summaries have still
to be written. The plan is to get as much of these written by the end of   <<<<
this month (when EGS is away) so that the site is almost ready to go live.
This would be followed by extensive testing and comments from the whole
group with the intention of making the site live sometime in August (2006).

There was some discussion regarding the setting up of a common mail account
(or other system) for VDFS questions. It was decided given the low current
attrition rate that the current "system" seemed to work fine so no change was 
needed ie. generally MJI gets emailed and the work gets distributed.

A short demo was given of the site as it currently stands and various comments
were made.


8.  AstroGrid liaison update

PSB noted that the Cambridge AstroGrid twiki had been hacked again. Most of
the links had been redirected to a site of a gynaecological interest (that's
one way of putting it).


9. AOB

DWE and JRL will be going to the image processing workshop in Dubrovnik.

MJI noted that there will be a workshop at ESO in January and a CASU
presence would be desirable.


Continuing Actions
------------------

MJI  finish investigating merged classification statistics

WJS  contact VPO to confirm that we will get no more ETC-related digitized
     data from them
     
JRL  implement fix to problem with UKIRT archive (missing header 
     information problem)

New Actions
-----------

MJI  remind WJS regarding above action

EGS  update INT WFC web pages with latest flat-field and fringe frames

STH  acquire data from the 2MASS Touchstone fields 

DWE  update hardware web page
MJI

PSB  have short meeting to finalise updates to DRLD
JRL
MJI

DWE  finalize content of Plone site by end of month
EGS
STH
MR