You are here: Home / Documents / Minutes of CASU Meetings / minutes_100324.txt

Plain Text icon — Plain Text, 14 KB (15095 bytes)

File contents

Wednesday 24th March 2010   APM Meeting tables    11:30 - 12:40

Present:        MJI, RWA, EGS, JRL, MR, SC, STH 
Apologies:      NAW, RGM

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent & upcoming meets
4.  Data archives update     - AAT, ING, WFCAM, UKIRT, VISTA
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, VST
6.  WFCAM update
7.  VISTA update
8.  AOB


1. Actions from the last meeting

MJI  stage I software release ongoing but the following will be included:  
EGS  JRL's standalone catalogue software tarball together with the 
     example piece of code for applying all the corrections for the 
     catalogue binary FITS tables.                                         <<<<

     STH asked about procedures for testing pipeline upgrades.  The short
     answer is that it is non-trivial, the slightly longer answer is that
     there are two parallel software development systems which are written 
     and prototyped in different languages (toolkit and pipeline) and the
     result intercompared, test datasets are also used to check results
     when bug fixing or smaller upgrades occur.  Checks are non-trivial 
     different compilers produce slightly different results and this has
     to be factored in to any comparisons.

MJI  noted that the dome flat comparison is still ongoing                 <<<<

JRL  finished the low light level tests and found a small non-linearity in 
     WFCAM amounting to ~0.5% at 500 ADU and a similar level of non-linearity 
     at 30K ADU, neither of which is worth correcting for. 

STH  has been updating the Calibration Plan document to better reflect the 
JRL  actual method in use for photometric calibration for VISTA.  The
     latest colour equations, including any Vega zero-point offsets, AB 
     magnitude conversions, comparison with WFCAM photometry etc..  still 
     need further work.                                                   <<<<
     MJI emphasised that although ESO will no doubt mangle this information
     in the fullness of time, the CASU work done now will form a good basis
     for the VISTA calibration paper.

RGM  is still testing the catalogue conversion software to destruction    <<<<

SC   has TLC'd the network monitoring web pages and not only do they now 
     monitor the relevant network traffic, they have also been very useful
     in helping to diagnose a gross-and-persistent network problem

SC   a policy for non-cluster machine backups is in place and everyone present 
     was reminded to use it or lose it.
MJI  contacted NCH about PMW's retirement and is awaiting developments

JRL  signed up for and sent in an abstract on VISTA pipeline processing

JRL  external object masks: Marijn Franx et Ultras have been doing some 
     testing with object masks.  A trial tile mask (32-bit reals ????) for
     UltraVISTA has been sent to JRL who, of course, converted it to a 
     compressed byte bit mask (>100 times smaller) and he reports that this
     works well due to the sparse UltraVISTA field and the large (60) number
     of dithers being combined.  This method will not work well for normal 
     observations and is indeed quite a time-consuming procedure as each 
     frame has to be accurately WCS aligned, prior to most of the processing.
     MJI and JRL are pondering whether or not this procedure can be invoked
     post normal pipeline processing.

EGS  has not yet plonked our versions of the ESO-released PR images on 
     the CASU web pages                                                   <<<<

2. WFAU minutes

There have been various recent discussions about UKlight network transfer 
speeds becoming slower over the last few months.  In parallel MJI and SC
have been investigating a curious internal network bottleneck that has
proven extremely difficult to pin down.  To cut a very long story short:
one of our 10G-bit -to- 10G-bit fibre module connectors between the two
computer rooms was found to be deteriorating to the point where 
communications between two 10Gbit switches were throttling some network 
connectivity.  Direct tests revealed no problems ! but a more indirect 
conceptual thought experiment did.  The recalcitrant piece of kit
(one of the "transducers" that converts laser light to e-) was swapped out 
for a spare and everything magically sprang back to full life.

The network monitoring web page proved very useful in eventually tracking
down the fault which inspired EGS to ask SC to add his machine to the 
traffic monitoring.                                                       <<<<

It was noted that DR7 was released on February 25th and contains data up to 
the end of 09A for LAS, GCS and DXS.  

3. Recent and upcoming meetings

Comments on the recent 2+ hour VISTA IOT telecon included the words 'paint' 
and 'drying' and not a lot else.  There was a "complaint" by STH that the 
VIRCAM exploder for e-mail included only JRL and MJI, so STH will attempt to 
get his name added (though questions about his sanity were raised at this 

A few whinges about FITS headers, raised in the papers for the IOT, were 
discussed but it was unanimously agreed that if that was all there was
wrong with the science products then we can retire now.

Apparently there have been no requests from PIs to re-do OBs according to 
JRL's recollection of discussions.  There are various putative mechanisms
for feedback about bum OBs, including our own web pages, but it looks like
the dot-joining artist is on vacation.

Results from each survey are being compared to a survey simulation tool to 
compare real and virtual progress.  As usual there were some gripes about 
priority tweaking (sound familiar?) negating some of the point of the 

The next action-packed meeting is Friday April 23rd.

JRL and STH are going to be attending NAM next month in various roles.

4. Archives.

UKIRT:   - finally done apart from 3 nights and it is now back where it belongs

AAT:     - smooth progress as usual

ING:     - ditto

WFCAM:   - up to date bar an interesting faux pas that was discovered by an 
           eager user (see later item)

5. Optical/NIR processing

HAWKI    - nothing to report. 
INT      - processing mostly on hold until the VISTA catchup is ketchup'd 
MegaCam  - after a Mega effort to process the 09b and early 10b PAndAS data 
           all is quiet in the Hobyar House
Subaru   - likewise
VST      - ars longa vita brevis

6. WFCAM update

Processing and data receipt (almost) up to date apart from two things:

first the faux pas - about 2/3 of the data for the nights of 12,13
July 2009 had mysteriously never made it here - in retrospect the run nos.
ending at 999 for both nights was a bit of a gimme but ......  quick checks
at JAC and CASU revealed this to be a one-off (two-off?) event.  The data
was retrieved by AJA at the JAC end, transferred over and the nights'
incomplete MSBs and missing data were (re)processed.  Most of the missing
data was for a PI programme but there were a few odds and ends of GPS and 
DXS data too.  All of the relevant bits will need: 
a.) factoring in to UKIDSS survey releases at some point
b.) releasing to the eager PI 
b.) transferring to ESO and ingesting to the raw WFCAM archive -> MR      <<<<
c.) to be included in the raw WFCAM tape archive ->  MJI                  <<<<

second the curve ball from left field - the WFCAM transfer machine, apm6, 
blew its last working power supply overnight on March 22nd/23rd.  When 
the other two power supplies had popped off late last year MJI had managed 
to find a supplier in the US and over Christmas had ordered three replacement 
power supplies with EXACTLY the same part nos. and spec.  After Santa had 
delivered the package he was therefore somewhat surprised to find they were
in-line power supplies with NO external mains connectors - d'er!  Another
long story, but after much discussion during the meeting about alternatives
(these power supplies don't seem to be available any more), the good news is 
that after cannibalising the backs off the old power supplies and the front 
end including circuit board off the new ones, plus a bit of judicious
clipping, rewiring and soldering, apm6 is back up and running, and, we have 
a spare power supply again.

The STFC's "managed withdrawal" from UKIRT has been better defined recently
and it seems that from late 2010, UKIRT will be run in "minimalist mode".
MJI's guess is that this will precipitate an increased demand for observers
to see out the UKIDSS surveys to completion end of March 2012.  From our
point-of-view it means that our squeezed-down-to 1 FTE of effort will be
required until at least then too.

The new waveforms introduced on WFCAM around February 20th have not lead
to any noticeable changes in the science products. As mentioned previously,
the new linearity data taken with them also suggest no change in the
pipeline is required either.

The UDS team in Nottingham copied over the J band reprocessed 05b-07b data
via apm3.  Due to the network problems this took around a week to complete.
During final checks MJI noticed one night had mysteriously got no data in
it and is redoing that one before letting WFAU copy it.

After being bombarded with several queries about the workings of the 
nebulosity filter MJI decided to write up the details for the upcoming UKIRT 

A few small fixes have been required with a particular sample of stacked 
products for a PI programme which showed the well-known (but fortunately rare) 
astrometric registration problem.  The latest pipeline version is more immune
to these and produced the goods without any intervention.

7. VISTA update

Raw data have been received, ingested and reduced to the pawprint level up to 
February 24th. (As usual the next two raw data disks arrived while writing 
this.)  Restacking and recataloging of data taken between October 15th 2009 
and January 13th 2010 has been completed and Version 0.8 released to PIs and 

All VISTA data from January 14th onwards has been reduced with Version 0.8 
of the pipeline and is therefore xyzaxbc.  The final stages of Rice compression
and transfer to final resting place are still ongoing but are expected to
be completed by the end of the week.

MJI noted that with the end of "Science Verification", we need to put a break 
point in the progress web pages.  SV finished on February 14th and P85 started
(early especially for VISTA) on February 15th.                             <<<<

EGS during progress DB ingest spotted an interesting problem with data from 
one night (20100107) whereby a DFU caused an EFU from detector #11 onwards
for some OBs.  Apart from the fact "that can't happen" this was rapidly
diagnosed and fixed.  EGS will inform the PI who has downloaded data from 
that night.                                                                <<<<

Apparemmente de temps en temps the Paranal (and Garching?) QC pipeline 
reports large scatter in the derived photometric zero points and the image 
ellipticities are also on occasion somewhat larger than life. 
It transpired that the summit pipeline was not using the deblending option
and therefore unsurprisingly, in crowded regions results were a bit flaky.
If the deblending option is switched on the ellpticity problems goes away
and we suspect the zero-point problem does too - though it is proving hard
to establish what datasets show this problem.  The science pipeline in
Cambridge does not seem to suffer from these issues so in addition to 
deblending, other possibilities to investigate are poor weather conditions, 
Galactic extinction effects for Z and Y and so on.  Visual inspection of
our progress web pages shows the rms scatter on photometric nights to be  
at the 2% level.

All of this discussion was what prompted STH to anounce that he would like to 
have access to the error reports (tickets) and wondered how he could achieve
that.  JRL said he would help him get set up to do so if that was what he 
really really wanted.

This reminded MJI of an outstanding (sic!) ticket from WHU regarding a funny
P-Cygni-like shadow on the sky frames after processing with an H-band set of
flats taken only 4 hours earlier on the same night.  WHU has independently
discovered that the filter wheel positioning is not as good as it should
be - the shadowing is clearly due to a small differential misalignment of 
the filter wheel position occuring over the 4 hour period.  Not ending
up with this problem is something we have been banging on about for years -
must have been the wrong drum. Fortunately the science impact is generally 
small and the sky correction stage fixes its obvious visual manifestation
in science product data.

JRL noted that there was still a problem with the raw frame CD matrix not 
being rotated properly during jitter offsets.  The fix implemented in the 
science pipeline works 99% of the time but the other 1% requires manual 
intervention.  Steven Beard has been looking into this but quite where any
fix has got to down on the front-line is unbeknown to us.

Now that Version 0.8 has finally appeared it is time to revisit the SV data.
EGS, MJI and STH were actioned to resurrect their efforts.                 <<<< 

In addition to the network constipation there was also a problem with a bad 
disk on the VISTA raw data raid in early March but this has now also been 
sorted and the fat electron kicked out of the building.

8. AOB

The motion sensor in STH's office was on the blink and playing games with the
scurity alarm system.  The sensor has now been replaced and (we think) the 
alarm is no longer tripping out at night.

Our first choice for filling PSB's post accepted the position and will start 
on July 1st and will finally bring the VISTA team up to the required 

Another new addition to CASU starts after Easter and will be housed in
STH's office using the not-so-spare spare desk.  STH's other regular
visitors will have to hot-swap with others in the IoA's Visitor Office.

Suggestions and negotiations with ESO for in-kind payment schemes rumble on 
with no clear sign of convergence.

Continuing actions

EGS  - add catalogue software and binary table conversion software to 
MJI    software release pages.  

MJI  - finish off dome flat comparison

JRL  - carry on with calibration plan doc updates and produce a new draft
STH    for discussion at next CASU meeting

RGM  - soak test the catalogue conversion software

EGS  - plonk CASU versions of VISTA PR (released) colour images on web pages

New actions

SC   - add EGS's machine to the network traffic monitoring web pages   

MR   - ensure any relevant bits of July 12,13 get transferred to ESO etc..

MJI  - make sure the B's get backed up

MR   - put the break on SV and introduce P85 to VISTA progress web pages

EGS  - inform the lucky PI who has downloaded data from 20100107 

EGS  - unbutt and revisit the SV data now that Version 0.8 has arrived