—
Plain Text,
14 KB (15095 bytes)
File contents
Wednesday 24th March 2010 APM Meeting tables 11:30 - 12:40
Present: MJI, RWA, EGS, JRL, MR, SC, STH
Apologies: NAW, RGM
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent & upcoming meets
4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST
6. WFCAM update
7. VISTA update
8. AOB
Minutes
-------
1. Actions from the last meeting
MJI stage I software release ongoing but the following will be included:
EGS JRL's standalone catalogue software tarball together with the
example piece of code for applying all the corrections for the
catalogue binary FITS tables. <<<<
STH asked about procedures for testing pipeline upgrades. The short
answer is that it is non-trivial, the slightly longer answer is that
there are two parallel software development systems which are written
and prototyped in different languages (toolkit and pipeline) and the
result intercompared, test datasets are also used to check results
when bug fixing or smaller upgrades occur. Checks are non-trivial
different compilers produce slightly different results and this has
to be factored in to any comparisons.
MJI noted that the dome flat comparison is still ongoing <<<<
JRL finished the low light level tests and found a small non-linearity in
WFCAM amounting to ~0.5% at 500 ADU and a similar level of non-linearity
at 30K ADU, neither of which is worth correcting for.
STH has been updating the Calibration Plan document to better reflect the
JRL actual method in use for photometric calibration for VISTA. The
latest colour equations, including any Vega zero-point offsets, AB
magnitude conversions, comparison with WFCAM photometry etc.. still
need further work. <<<<
MJI emphasised that although ESO will no doubt mangle this information
in the fullness of time, the CASU work done now will form a good basis
for the VISTA calibration paper.
RGM is still testing the catalogue conversion software to destruction <<<<
SC has TLC'd the network monitoring web pages and not only do they now
monitor the relevant network traffic, they have also been very useful
in helping to diagnose a gross-and-persistent network problem
SC a policy for non-cluster machine backups is in place and everyone present
was reminded to use it or lose it.
MJI contacted NCH about PMW's retirement and is awaiting developments
JRL signed up for and sent in an abstract on VISTA pipeline processing
JRL external object masks: Marijn Franx et Ultras have been doing some
testing with object masks. A trial tile mask (32-bit reals ????) for
UltraVISTA has been sent to JRL who, of course, converted it to a
compressed byte bit mask (>100 times smaller) and he reports that this
works well due to the sparse UltraVISTA field and the large (60) number
of dithers being combined. This method will not work well for normal
observations and is indeed quite a time-consuming procedure as each
frame has to be accurately WCS aligned, prior to most of the processing.
MJI and JRL are pondering whether or not this procedure can be invoked
post normal pipeline processing.
EGS has not yet plonked our versions of the ESO-released PR images on
the CASU web pages <<<<
2. WFAU minutes
There have been various recent discussions about UKlight network transfer
speeds becoming slower over the last few months. In parallel MJI and SC
have been investigating a curious internal network bottleneck that has
proven extremely difficult to pin down. To cut a very long story short:
one of our 10G-bit -to- 10G-bit fibre module connectors between the two
computer rooms was found to be deteriorating to the point where
communications between two 10Gbit switches were throttling some network
connectivity. Direct tests revealed no problems ! but a more indirect
conceptual thought experiment did. The recalcitrant piece of kit
(one of the "transducers" that converts laser light to e-) was swapped out
for a spare and everything magically sprang back to full life.
The network monitoring web page proved very useful in eventually tracking
down the fault which inspired EGS to ask SC to add his machine to the
traffic monitoring. <<<<
It was noted that DR7 was released on February 25th and contains data up to
the end of 09A for LAS, GCS and DXS.
3. Recent and upcoming meetings
Comments on the recent 2+ hour VISTA IOT telecon included the words 'paint'
and 'drying' and not a lot else. There was a "complaint" by STH that the
VIRCAM exploder for e-mail included only JRL and MJI, so STH will attempt to
get his name added (though questions about his sanity were raised at this
point.)
A few whinges about FITS headers, raised in the papers for the IOT, were
discussed but it was unanimously agreed that if that was all there was
wrong with the science products then we can retire now.
Apparently there have been no requests from PIs to re-do OBs according to
JRL's recollection of discussions. There are various putative mechanisms
for feedback about bum OBs, including our own web pages, but it looks like
the dot-joining artist is on vacation.
Results from each survey are being compared to a survey simulation tool to
compare real and virtual progress. As usual there were some gripes about
priority tweaking (sound familiar?) negating some of the point of the
comparison.
The next action-packed meeting is Friday April 23rd.
JRL and STH are going to be attending NAM next month in various roles.
4. Archives.
UKIRT: - finally done apart from 3 nights and it is now back where it belongs
AAT: - smooth progress as usual
ING: - ditto
WFCAM: - up to date bar an interesting faux pas that was discovered by an
eager user (see later item)
5. Optical/NIR processing
HAWKI - nothing to report.
INT - processing mostly on hold until the VISTA catchup is ketchup'd
MegaCam - after a Mega effort to process the 09b and early 10b PAndAS data
all is quiet in the Hobyar House
Subaru - likewise
VST - ars longa vita brevis
6. WFCAM update
Processing and data receipt (almost) up to date apart from two things:
first the faux pas - about 2/3 of the data for the nights of 12,13
July 2009 had mysteriously never made it here - in retrospect the run nos.
ending at 999 for both nights was a bit of a gimme but ...... quick checks
at JAC and CASU revealed this to be a one-off (two-off?) event. The data
was retrieved by AJA at the JAC end, transferred over and the nights'
incomplete MSBs and missing data were (re)processed. Most of the missing
data was for a PI programme but there were a few odds and ends of GPS and
DXS data too. All of the relevant bits will need:
a.) factoring in to UKIDSS survey releases at some point
b.) releasing to the eager PI
b.) transferring to ESO and ingesting to the raw WFCAM archive -> MR <<<<
c.) to be included in the raw WFCAM tape archive -> MJI <<<<
second the curve ball from left field - the WFCAM transfer machine, apm6,
blew its last working power supply overnight on March 22nd/23rd. When
the other two power supplies had popped off late last year MJI had managed
to find a supplier in the US and over Christmas had ordered three replacement
power supplies with EXACTLY the same part nos. and spec. After Santa had
delivered the package he was therefore somewhat surprised to find they were
in-line power supplies with NO external mains connectors - d'er! Another
long story, but after much discussion during the meeting about alternatives
(these power supplies don't seem to be available any more), the good news is
that after cannibalising the backs off the old power supplies and the front
end including circuit board off the new ones, plus a bit of judicious
clipping, rewiring and soldering, apm6 is back up and running, and, we have
a spare power supply again.
The STFC's "managed withdrawal" from UKIRT has been better defined recently
and it seems that from late 2010, UKIRT will be run in "minimalist mode".
MJI's guess is that this will precipitate an increased demand for observers
to see out the UKIDSS surveys to completion end of March 2012. From our
point-of-view it means that our squeezed-down-to 1 FTE of effort will be
required until at least then too.
The new waveforms introduced on WFCAM around February 20th have not lead
to any noticeable changes in the science products. As mentioned previously,
the new linearity data taken with them also suggest no change in the
pipeline is required either.
The UDS team in Nottingham copied over the J band reprocessed 05b-07b data
via apm3. Due to the network problems this took around a week to complete.
During final checks MJI noticed one night had mysteriously got no data in
it and is redoing that one before letting WFAU copy it.
After being bombarded with several queries about the workings of the
nebulosity filter MJI decided to write up the details for the upcoming UKIRT
newsletter.
A few small fixes have been required with a particular sample of stacked
products for a PI programme which showed the well-known (but fortunately rare)
astrometric registration problem. The latest pipeline version is more immune
to these and produced the goods without any intervention.
7. VISTA update
Raw data have been received, ingested and reduced to the pawprint level up to
February 24th. (As usual the next two raw data disks arrived while writing
this.) Restacking and recataloging of data taken between October 15th 2009
and January 13th 2010 has been completed and Version 0.8 released to PIs and
WFAU.
All VISTA data from January 14th onwards has been reduced with Version 0.8
of the pipeline and is therefore xyzaxbc. The final stages of Rice compression
and transfer to final resting place are still ongoing but are expected to
be completed by the end of the week.
MJI noted that with the end of "Science Verification", we need to put a break
point in the progress web pages. SV finished on February 14th and P85 started
(early especially for VISTA) on February 15th. <<<<
EGS during progress DB ingest spotted an interesting problem with data from
one night (20100107) whereby a DFU caused an EFU from detector #11 onwards
for some OBs. Apart from the fact "that can't happen" this was rapidly
diagnosed and fixed. EGS will inform the PI who has downloaded data from
that night. <<<<
Apparemmente de temps en temps the Paranal (and Garching?) QC pipeline
reports large scatter in the derived photometric zero points and the image
ellipticities are also on occasion somewhat larger than life.
It transpired that the summit pipeline was not using the deblending option
and therefore unsurprisingly, in crowded regions results were a bit flaky.
If the deblending option is switched on the ellpticity problems goes away
and we suspect the zero-point problem does too - though it is proving hard
to establish what datasets show this problem. The science pipeline in
Cambridge does not seem to suffer from these issues so in addition to
deblending, other possibilities to investigate are poor weather conditions,
Galactic extinction effects for Z and Y and so on. Visual inspection of
our progress web pages shows the rms scatter on photometric nights to be
at the 2% level.
All of this discussion was what prompted STH to anounce that he would like to
have access to the error reports (tickets) and wondered how he could achieve
that. JRL said he would help him get set up to do so if that was what he
really really wanted.
This reminded MJI of an outstanding (sic!) ticket from WHU regarding a funny
P-Cygni-like shadow on the sky frames after processing with an H-band set of
flats taken only 4 hours earlier on the same night. WHU has independently
discovered that the filter wheel positioning is not as good as it should
be - the shadowing is clearly due to a small differential misalignment of
the filter wheel position occuring over the 4 hour period. Not ending
up with this problem is something we have been banging on about for years -
must have been the wrong drum. Fortunately the science impact is generally
small and the sky correction stage fixes its obvious visual manifestation
in science product data.
JRL noted that there was still a problem with the raw frame CD matrix not
being rotated properly during jitter offsets. The fix implemented in the
science pipeline works 99% of the time but the other 1% requires manual
intervention. Steven Beard has been looking into this but quite where any
fix has got to down on the front-line is unbeknown to us.
Now that Version 0.8 has finally appeared it is time to revisit the SV data.
EGS, MJI and STH were actioned to resurrect their efforts. <<<<
In addition to the network constipation there was also a problem with a bad
disk on the VISTA raw data raid in early March but this has now also been
sorted and the fat electron kicked out of the building.
8. AOB
The motion sensor in STH's office was on the blink and playing games with the
scurity alarm system. The sensor has now been replaced and (we think) the
alarm is no longer tripping out at night.
Our first choice for filling PSB's post accepted the position and will start
on July 1st and will finally bring the VISTA team up to the required
complement.
Another new addition to CASU starts after Easter and will be housed in
STH's office using the not-so-spare spare desk. STH's other regular
visitors will have to hot-swap with others in the IoA's Visitor Office.
Suggestions and negotiations with ESO for in-kind payment schemes rumble on
with no clear sign of convergence.
Continuing actions
------------------
EGS - add catalogue software and binary table conversion software to
MJI software release pages.
MJI - finish off dome flat comparison
JRL - carry on with calibration plan doc updates and produce a new draft
STH for discussion at next CASU meeting
RGM - soak test the catalogue conversion software
EGS - plonk CASU versions of VISTA PR (released) colour images on web pages
New actions
-----------
SC - add EGS's machine to the network traffic monitoring web pages
MR - ensure any relevant bits of July 12,13 get transferred to ESO etc..
MJI - make sure the B's get backed up
MR - put the break on SV and introduce P85 to VISTA progress web pages
EGS - inform the lucky PI who has downloaded data from 20100107
EGS - unbutt and revisit the SV data now that Version 0.8 has arrived
MJI
STH
