File contents

22nd August 2007  Adhoc meeting in Hoyle Committee Room

Present:          JRL, STH, EGS, PSB, MR, MJI, DWE
Apologies:    

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Data archives update     - AAT, ING, WFCAM, UKIRT
4.  Optical processing update
5.  WFCAM news               - raw data transfer status
                             - WFCAM network transfers
                             - processing status
                             - reprocessing requests
                             - photometric calibration update
                             - operational lessons learnt
6.  VISTA news               - pipeline progress
                             - cold camera test data from Paranal
                             - VISTA QC and operations
7.  ADASS 2007               - latest news
8.  AOB                      - retiring cass36
                             - system management

Minutes
-------

1.  Actions from last meeting

RGM  have not completed the work on the classification aspect of cross-talk 
EGS  weeding and classification generally - ongoing                        <<<<
MJI  

MJI  has still not put the OMP log comments into the processed directories
     - ongoing                                                             <<<<

JRL  are still working out a scheme for doing header updates for ING archive
EGS  - ongoing                                                             <<<<

PSB  has moved office to APM building and has ordered and installed another 
     Gbit switch 

 
2.  Comments on WFAU minutes

PSB reported briefly on his further investigations into data transfer rates 
to WFAU noting that WFAU have recognised the djedefre bottleneck.  He also
commented that a few WFCAM transfer sessions have taken place recently over
UKLight at the usual 3MB/s, compared to realistic test WFCAM transfer rates 
(ie. a ~50 GByte whole night of real processed WFCAM 06B data) of ~30MB/s 
using un-encrypted hpn-ssh to seaforth connected over UKLight.

STH commented on the release schedule for DR3 and wanted to clarify when
T0 was likely to be and were the units in the minutes really weeks since 
they implied DR3 would be 14.5 weeks after T0.  MJI thought they were 
unlikely to be anything other than weeks but noted that some of them 
happened in parallel so the time after T0 was unclear.  He also noted that 
since all of the 06B data had finally been processed, checked and flagged, 
T0 should be imminent.  


3.  Data archives update

Most of the off-line primary archive tapes have now been moved to their 
temporary location viz the lockable store cupboard in the Observatory 
meeting room.  MJI noted that the "cleared" shelves in the APM store room
could be used for storing personal backup tapes and so on.

ING:
Data continues to flow seamlessly from the INT and WHT via automated network 
transfers and a night's observing is invariably completely transferred 
to Cambridge before the observer has made it back down to bed.

AAO:
New nights of data are transferred over the Internet as they are made 
available. The new access interface will be publicised in the next AAO 
newsletter. Small bug fixes are still being made, but the system seems to 
be working well on the whole.

UKIRT:
This is up-to-date.

WFCAM:
All 06B data that are available at JAC have now been received and ingested
into the raw data on-line archive.  07A data was received long ago and 07B 
data has not yet started to flow.

MR noted that roughly 2 weeks of 07B data (allowing for stoppages due to 
Flossie) had been taken and he would inquire when the first tapes would
be shipped.                                                                <<<< 

Transfer of remaining UKIDSS and calibration raw data for 06B to ESO is 
ongoing.


4.  Optical processing update

The IPHAS processing and DB ingestion for June and July has been completed 
amid much excitement due to jammily capturing pre-Nova r,i,Ha images of 
Nova Vul 2007.  

EGS has developed and released a further trial component (alpha stage) for 
simplifying browsing of the Astrogrid-enabled IPHAS Early Data Release.


5.  WFCAM news

All of the available WFCAM raw data for 06B has arrived.

PSB has had an email exchange with Andy Adamson regarding the possibility
of transmitting raw WFCAM data to Cambridge via the Internet.  An outline
plan was sent to JACH and we are waiting for Brad Cavanagh to reply. 
(It is hoped to reduce manpower overheads, increase data integrity and 
improve lead time to science delivery.)

The final few nights of 06B data have now been processed and as of today all 
of the data for 06B that is processable has been processed, checked and
flagged as ready to go.  There are a couple of problem nights, in particular
20061116 and 20061118.  The former has most of the data missing but has
been processed as well as possible.  The latter has no #3 data for the entire
night and has been flagged as un-processed after a brief email exchange
to verify this with WFAU.

This completes the processing of 06B. The quality of the October/November 
data up to the 20th was in general quite poor due to: bad weather, lack of 
enough good calibration frames (especially darks), many missing frames 
(caused mainly by camera problems), and a lot of interrupted incomplete
group MSBs.

MJI and MR summarised the state of play with reprocessing requests:

The problem GPS data for 06A has all been reprocessed and Phil Lucas has
checked it. Most of the requests for reprocessing concerned fixing the
known feature in the extant (at that time) cross-talk correcting algorithm
that made a pigs ear in regions of highly varying nebulosity.  A few were
related to poor sky sbutraction but as these were invariably on poor nights
interrupted by bad weather these were harder to deal with (see previous
minutes for more details).

The UDS saga continues. JRL will test a special purpose sky background 
modelling algorithm involving iterative object masking on some of the UDS 
data that has been requested for reprocessing (as above, see previous 
minutes for more details).  

STH reported that he will shortly (end of the week) circulate a final  
analysis of the WFCAM Z and Y-band zeropoint offsets to the  
calibration WG for comment, prior to implementing the adjustment to  
the processed catalogues.  MJI noted that to avoid confusion he would not 
apply the upcoming new offset corrections to the processed WFCAM data until 
the remainder of 06B has been transferred.  He also commented that since 
this is currently planned to be purely a zero-point offset for these two 
bands only it should be trivial to implement at CASU and WFAU without the
need for any transfers.

MJI kicked off a general discussion about lessons learnt from the operational
side of running the WFCAM pipeline for several semesters.  
PSB and JRL commented they knew it all already and had already built in
the necessary infrastructure to the VIRCAM pipeline - we shall see.

There was discussion at length about software updates, tests and the 
ability to trace changes made in the pipeline infrastructure. In spite of
the previous optimistic statement, JRL agreed to go away and think about 
this some more over the coming month or so.

MR contacted the Durham group (Jim Geach) about their progress in processing 
raw WFCAM data and obtained some example processed data.  Now that 06B 
processing is finished he will undertake a comparison.                     <<<<


6.  VISTA news

JRL commented that he had received some requests from ESO for updates to 
the pipeline.  One of these comes about from the change to CFITSIO and its
impact on reading binary tables.  The others were for minor (but irritating)
name changes to various parameters etc....

PSB reported around 2,300 frames were obtained from the July Paranal
engineering run of VIRCAM, with apparently only two being lost
semi-permanently.  Because of some technical difficulties with the camera,
the data needs reformatting to be able to be processed (semi)automatically.  
This is in progress.

JRL and PSB are currently running tests on this camera data.  This has 
already turned up a few bugs, albeit minor ones. A report on the data will 
appear in due course.  MJI suggested that a further independent 
examination of the VIRCAM test data was probably worthwhile and volunteered
EGS to look at it as well since MJI and MR were busy.                      <<<<


7.  ADASS 2007

The final programme is currently being constructed and will be sent to the 
printers next week. A floor plan and final decisions regarding catering
will also need to be done in the next week or so. The conference looks as
though it will be well attended with probably a little over 300 people.


8. AOB

MJI and PSB noted that cass36 was now (definitely honest) going to be
retired, albeit in gradual installments.

All thanked JMI for his expert system management over the past few years.
PSB and MJI were tasked with acquiring a system brain dump from him before 
he leaves to take up a postdoc position at Harvard.                        <<<<

All congratulated Simon on his recent 40th birthday.


Continuing Actions
------------------

RGM  report back on classification aspect of cross-talk weeding
EGS
MJI

MJI  place OMP log comments into processed directories
     
JRL  work out scheme for doing header updates for ING archive
EGS


New Actions
-----------

MR   find out when the first 07B tapes will be shipped

MR   undertake comparison of Jim Geach processed data with pipeline output

EGS  help with assessment of VIRCAM lab data

MJI  find out how little they know about CASU system management
PSB