File contents

8th August 2007  Virtual Meeting Room

Present:          STH, EGS, PSB, MR, MJI, RWA
Apologies:        DWE, JRL, NAW, RGM

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  News from meetings       - CWG, VDMT
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical processing update
6.  WFCAM news               - raw data transfer status
                             - processing status
                             - photometric calibration update
7.  VISTA news               - pipeline progress
                             - summit processing hardware
                             - cold camera test data from lab and telescope
8.  ADASS 2007               - latest
9.  AOB                      


Minutes
-------

1.  Actions from last meeting

RGM  have not completed the work on the classification aspect of cross-talk 
EGS  weeding and classification generally - ongoing                        <<<<
MJI  

MR   contacted the Durham group (Jim Geach) about their progress in
     processing raw WFCAM data and has obtained some example processed
     data. 

EGS  has investigated the photometric redshifts issue in DXS after 
     conversations with Michael Rowan Robinson and Steve Warren about this. 
     MRR was comparing our WFCAM aperture magnitudes with 2MASS isophotal 
     magnitudes from the extended catalogue and that is why they did not 
     agree. When using the correct magnitudes the WFCAM and 2MASS apertures 
     agree well. There is still the issue that when introducing the WFCAM 
     magnitudes in the photoz code the specz and photoz show a large 
     disagreement, but this is probably due to the fact that MRR has been 
     using Petrosian magnitudes from the WFCAM catalogue. EGS has suggested 
     using aperture magnitudes to measure fluxes with apertures the same size 
     as used in in the optical.  As a general rule the default should be 
     specific radius aperture mags, worries about PSF variation corrections 
     are probably 2nd order and are also qualitatively constrainable using
     PSF aperture correction information.

PSB  the missing NDR data has been transferred via FTP

JRL  have successfully carried out tests on a demonstration copy of the 
PSB  Kognitio software using a minimal parallel CPU hardware setup

MJI  has still not put the OMP log comments into the processed directories
     - ongoing                                                             <<<<

JRL  are still working out a scheme for doing header updates for ING archive
EGS  - ongoing                                                             <<<<

RWA  asked Peter Weeks if blinds could be fitted to interconnecting annex but 
     this was considered to be unworkable and an alternative was found (see
     later)

JRL  is in discussion with Chris Rammage about how best to proceed with 
     advertising the AAO archive

MJI  asked Paul Aslin about the Carlsberg extinction payments and found out
     that it has been paid

PSB  emailed JPE regarding version 1.5pre1 of Calibration Plan 

PSB  is moving office to APM building and will order another Gbit switch 
     when the dust has settled - ongoing                                   <<<<

 
2.  Comments on WFAU minutes

PSB was asked for his thoughts on WFCAM transfer problems (see wfau.minutes.
070622.txt). In response he asked for a (none-privileged) account on 
djedefre.roe.ac.uk, but there seems to be some difficulty setting this up.  
So ...... in order to follow up on CASU->ROE networking performance issues, 
PSB worked closely with Horst Meyerdierks to experiment with a JANET 
connection between CASU and ATC. HM installed seaforth.roe.ac.uk and carried 
out some requested software installation and tuning.  Results were 25MB/s 
using hpn-ssh and 45MB/s using ftp (much more lightweight protocol). This 
compares very favourably with the best ~12MB/s using the multi-threading 
technique used originally by WFAU and the current rock-steady ~3MB/s over 
UKLight.  Further tests are planned.


3.  News from meetings

There has been one VDMT meeting (27th July) via telecon since the last CASU 
meeting. 

Present were STH, PMW, JMS, NCH. Agenda items included:

DR3 release schedule: NCH has not yet fixed a release date for DR3, 
contingent on the completion of the processing of NDR data from 06B. A 
release date some 6 weeks after this data is released to ROE was suggested.

JPE asked that the Memorandum of Understanding from JAC be discussed with 
him before responding.

JMS reported that agreement had been reached with ESO on the procurement of 
a single workstation to run the VISTA summit pipeline.  This seems to have 
already been ordered and will consist of a PowerEdge 2950 with 2xCPU 
Quad-Core and 8GB RAM with 4x300 GB RAID5 of onboard data storage disks, 
plus a similar user offline workstation.

Some concern was expressed over the modest data storage available on this 
machine, not to mention the limited amount of CPU.  

JMS was keen to learn how JRL was getting on with the VIRCAM data.

Date of next meeting: Fri 7th Sept @ 14:00 by telecon, which MJI noted he
cannot make since he is taking part in a thesis defense in Groningen then.


4.  Data archives update

The store room in the SPO containing our primary backup tapes (UKIRT, WFCAM
AAT, ING) and plates is being gradually emptied. The backup tapes will be 
stored for the duration of the building works in the Observatory meeting 
room in a large lockable cupboard therein.

ING: 
The duplicate CDs and DAT tapes are being skipped and should make enough room 
in the CASU storeroom to accommodate the remainder of the CDs still in 
Room 20 in the SPO. 

JRL and EGS are discussing moving the archive to the Sybase server on apm20. 

RWA noted that there are still a few more early tapes to be read and the
conversion of those VMS Backup tapes read on the La Palma Alpha recently
from STF to FITS awaits discussion with JRL. Another 100 tapes or so
remain to be read on LP.

AAO:
The raw data in the AAO archive is now fully up to date. New nights
of data are being transferred as they are made available. JRL is in
discussion with Chris Ramage on how best to publicise the new interface.

UKIRT:
This is up-to-date.

WFCAM:

All 06B data, bar a few rogue files, have now been received and ingested.

A WFCAM calibration meeting was held between MJI, STH, PH and SJW in  
Cambridge on Mon July 2nd. Discussion focussed on preparation for DR3  
and completion of the analysis needed for the paper. A timescale was  
agreed for the paper (draft to be circulated end of August, with  
submission aimed for by end September), and STH has since circulated  
a draft outline for comment. MJI presented a summary of the WFCAM  
photometry spatial systematics, broken down into their components:

varying pixel scale across the FoV (already accounted for);

PSF variations as a function of focus, a ~1% effect for AperMag3 from 05B
onwards and negligible for larger apertures.  This can be readily measured 
from comparisons between small and large aperture photometry.  Correction
tables would need to be on a per science product frame basis and unclear
if worth correcting for general use;

detector-to-detector zeropoint error (to be included in DR3), this  
comprises about half the size of the current residual spatial systematics;

residual residuals. No clear handle on what these relate to, but  
appear to be reasonably stable over semester-length timescales. Look  
up tables could be provided for optional use but need not be part of DR3.


5.  Optical processing update

IPHAS processing for June July almost complete.

CASU are also still carrying out the extinction calculation derived from the
Carlsberg Meridian Telescope data but as this requires only minimal effort
it is ongoing.


6.  WFCAM news

All the 06B data is now present, except for a few files from 20061031 and 
20061122, and has been ingested into the raw data archive. All UKIDSS data 
for 07A has been transferred to ESO and we are now transferring the 06B nights
prior to 20th December.

We are aiming to finish off the remaining 06B processing before 07B data 
arrives in earnest mid-August.

NDR:
All the NDR data was downloaded over the network in about a week.
Some additional time was necessary because for a few nights not all the
data had been copied due to some cases where the observing log Brad was
using to dig out the NDR data did not match ours, but this was just a
minor hiccup. To date, all the December data is processed plus a few nights
of November (up to 27th). December nights looked quite reasonable but for 
the first tranche of November (counting backwards) the quality dropped 
significantly with loads of incomplete images and observations taken in very 
poor weather.  Additionally we are having to use December flats and confidence 
maps also for November nights because the flats taken in the earlier period 
had a very bright star in the center of chip #1 and the dithering was not 
large enough to get rid of it (this was prior to changes in the flat observing 
that we discussed with JAC).

UDS reprocessing:
After reprocessing the UDS data for 20061227 and making it available to
Sebastien and Omar, they came back about a week/10 days after saying
that, although the re-reduction made the sky background smoother, there
were some weird features (especially on chip #2) that Swarp could not
cope with. The main feature is a darkish vertical stripe on the left half
of chip #2 that extends almost from the bottom to the top of the image.
The fact that it does not show any quadrant symmetry or channel aligment
suggests that it is not caused by a glitch in the instrument electronics. 
After looking into this in detail, JRL and MR came to the conclusion that 
the feature is not introduced by a poor sky correction per se but is actually 
present in the data itself. It can be clearly seen, although at a lower level, 
in the sky frame as well. This, together with the fact that the feature is 
still present after the sky correction, means that the feature was varying in
intensity (but not in position) during the night.

JRL is investigating if it is possible to get around this problem by
using a finer-grained sky correction for cases like this. However, this 
will require masking the real objects to avoid problems when the dithering 
size is not large enough to guarantee a proper object reject (like in the 
case of some of the UDS data) and may lead to problems with larger objects
where no pixels are then left for sky estimation.  

GPS reprocessing:
MR has reprocessed all the data in the list provided by Phil Lucas. Most
of the problems were due to a known feature of the cross talk correction 
in regions of highly varying nebulosity that had already been fixed and did 
not affect any of 06B and 07A data. In some other cases there seemed to be 
problems with a "patchy sky background" or with "stars being subtracted". 
The reprocessing fixed the crosstalk problem satisfactorily.  

In most of the cases where the sky correction was not good the reason was 
simply due to poor weather conditions.  

The "subtracted stars" mystery was solved by looking at the 8 images for the 
2-point dithering + 2x2 microstep sequences. In all cases at least one of 
the two microstep sequences had an image with a faulty detector showing all 
sorts of weird patterns. All the data is available from the WSA so we 
strongly encourage the PIs to have a harder look at the data
when there appear to be problems and also take account the weather conditions 
(e.g. from http://www.cfht.hawaii.edu/cgi-bin/uncgi/elixir/skyprobe.pl?list)
and the nightly summary QC plots before asking for reprocessing.

External comparisons:
Jim Geach has made available some example processed data for comparison with 
the pipeline output but this is on hold for now until 06B processed data has 
been kicked out of the door.


7.  VISTA news

Reports on the current likely VISTA timescales were received.

ESO have settled on the workstation specification for the Paranal QC
pipeline (briefly, a 2x4-core system) which, although a top-end workstation,
is below the 16-cpu blade system suggested by the Garching pipeline
benchmarks.

A RAL team with JPE in support carried out a cold run on VIRCAM in July
and despite typical commissioning difficulties obtained around 2300
data frames including EMC on-telescope tests and a series of lab tests
including linearity sequences.  Some near-time analysis was possible
back in the UK by using the Paranal (2x2Mb/s) network link within
agreed restricted limits (about 4% of the bandwidth), and data was also
returned on portable disks by SMB and JPE. All data is now transferred to
Cambridge for more detailed analysis and tests.


8.  ADASS 2007

The oral programme has now been published on the ADASS website. The
poster programme will appear in due course. Financial aid awards have been
announced and Elinor is making the arrangements for the hotel rooms
for the successful applicants. Ian Bell continues to deal with the credit 
card details. PSB has convinced Amanda to do the work for the conference 
photograph. RWA is making arrangements for the registration packs.

Following the expected flurry of last-minute registrations and alterations,
the numbers are around the break-even 300 mark (phew).

The network (4x8Mb BT lines, secure wireless, guaranteed demo wired bandwidth)
has been ordered.

The initial program is now available and generally, the planning is well on
track.


9. AOB

MJI reported that the VDFS 6 month extension grant had finally been received.

PSB noted that Fibre re-routing in preparation for demolition and
subsequent building works is causing a series of short LAN outages. The
opportunity is being taken to re-equip with a 10G-capable backbone fibre.


Continuing Actions
------------------

RGM  report back on classification aspect of cross-talk weeding
EGS
MJI

MJI  place OMP log comments into processed directories
     
JRL  work out scheme for doing header updates for ING archive
EGS

PSB  order another Gbit switch for APM building