Monday 18th May 2009 APM Meeting tables 11:30-13:00
Present: MJI, JRL, STH, SC, RWA
Apologies: NAW, MR, EGS, RGM, DWE
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent and upcoming meetings
4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, Gemini, VIMOS
6. WFCAM update
7. VISTA update
1. Actions from the last meeting
STH PLONE pages. Sky subtraction and dribbling text updates have been added
RWA (it later transpired that the sky subtraction text should also be on
the WFCAM as well as the VISTA technical pages.) RWA will continue
checking the technical pages - instances of poor alignment noted. <<<<
STH HAWK-I colour image of Carina. STH has put this on a private web page
with password and demo'd zoomifying at the meeting. JRL still has
to write some text to go with it. <<<<
MJI says (re)photometry for HAWK-I still to be done <<<<
MJI Stage 1 release software - still ongoing <<<<
MJI updates to technical paper started, but not yet finished <<<<
RGM liaised with JRL about HAWKI - data processed
JRL has sorted out the VISTA docs and pdfs so that this is essentially
complete and they are all now on the VDFS internal web pages.
MJI write utility to convert binary tables with all corrections applied
- in the queue but not high priority <<<<
JRL has sent the latest catalogue software to WFAU
JRL implemented keyword in sky FITS files and back copied it to all 09A
EGS has not yet sent off example code to UVEX team <<<<
NAW polite but firm e-mail to VIZIER about IPHAS - ongoing <<<<
MR sorted out WFCAM transfer script with result that MJI now gets two
e-mails per day from MR rather than PSB
STH inquired about linearity sequence status and noted nice new screen
but no lamps yet
MJI wrote and sent the two reports to JPE for distribution
MJI has put the minutes he has of the IOT telecons on the internal web
pages, though he noted that the minutes of the previous April meeting
have not appeared yet
STH is liaising with ASTEP folk and has started to construct a webpage <<<<
MR status of comparison of FITS file header problems on 20090312 with
original sdf files unknown - ongoing <<<<
MJI compiled and sent off report to AJA for the upcoming UKIRT Board meeting
and put a copy of it and the UKIDSS submission on the internal web page
JRL has been in communication with JAC about cfitsio version in use there
JRL VISTA processing status webpage. A lot of action on this and it is now,
MJI thanks to MR, doing live updates. JRL has also has made access to
reduced data possible by our usual password-protected web page access.
MJI is steadily verifying, correcting astrometry and producing
summary QC plots for the live web page. 1st pass photometric ce's and
ZPs have been generated based on STH touchstone field observations. <<<<
MJI sorting out the VISTA i/o disks is on hold until the GAIA team finish
assorted i/o tests on the new RAID system. <<<<
2. Comments on WFAU minutes
MJI noted that as a result of the recent computer attack on the IoA cluster,
all private authorized keys on the APM cluster were disabled as a security
measure. It wasn't clear at the time who was using this method but it turned
out to be the ING incoming transfers and the WFAU outgoing transfers. The
relevant keys have now been regenerated.
There have been several comments about processing and reprocessing status of
various things - see the WFCAM section later for the current state-of-play.
3. Recent and upcoming meetings
There will be a UKIRT Board Meeting in Nottingham on Thursday, and SJW, NCH
and MJI will attend for the pm session.
EGS later reported via email that he gave a CASU presentation in the VO and
Distributed Computing session at NAM 2009. He noted that it looks like
Europe and the US are taking the VO seriously.
VISTA IOT telecon on April 28. JRL and MJI attended from Cambridge and STH
from Mauna Kea (albeit briefly due to inclement weather). JRL and MJI
submitted reports on sky subtraction strategies and science verification
results and they can be found under the VISTA commissioning internal web
The next telecon is due in about a week. There is a definite need for a sky
offset template (HAWKI uses 4 different flavours, but 1 would do for VISTA)
as noted during the previous telecon. After a brief discussion on possible
practical ways to implement this it was felt that this should be taken
offline and the constraints discussed with JPE and Steven Beard before
proceeding further. STH noted that as he was imminently about to fly out
to Paranal he would discuss this further with JPE when he arrives.
MJI noted that the PLATO Science consortium will meet in Belfast on June 11.
STH can't go but NAW will attend and further impart some wisdom.
JRL commented that ADASS 19 is now open for registration (Sapporo, Japan) and
he thinks we ought to contribute something on VISTA processing. MJI agreed
but pointed out that the travel budget is currently zero and still awaiting
a delivery from STFC.
At the JENAM meeting Tim de Zeeuw said that first light for VST would be
this September - watch this space, but don't hold your breath for the
first survey data.
4. Data archives
AAT - Jim says that the AAO now write to DVD in real time. He has written a
script to download the data as each DVD is completed and we are now running
~1 night behind. MJI noted it might not be a bad idea to use a semaphore
system of some sort rather than dead reckoning to ensure synchronicity. <<<<
The RAID array that this is being stored on seems to be having a hard time
of it recently and has had two different disk drive units going awol within
the space of a week. No obvious reason other than stochasticity but at
the time of writing another (different again) channel is going awol - d'er!
ING - data keeps on rolling in and being verified, ingested and backed up
WFCAM - MR has more fully automated the MEFing and verification step, which
together with the automatic data transfer stage from JAC to CASU has further
minimised overheads for this aspect (good job too!), but still needs manual
tape archiving further downstream.
UKIRT - remaining few DLTs and LTOs still to read
VISTA - MJI noted a report in the latest ESO news letter that from mid-2009
bulk ESO archive users need to be ready to receive Blu-Ray disks (sort of
souped up DVD) and it warns users to have suitable hardware in place.
MJI asked JRL to double check that this is not going to affect us and that
the disk bricks will still continue to be the VISTA transport medium for
the near future. <<<<
5. Optical/NIR processing
HAWKI - as noted previously, JRL has processed the latest HawkI data for RGM
and MJI still hasn't finished off regenerating the Carina catalogues. Yet
another excursion for the nebulosity filter.
INT WFC - MJI had a pleasant observing run on the INT WFC towards the end of
April, generated and analysed some new non-linearity observations and
finished the processing of data from his observing run before he got back.
MegaCam - nothing to report
Subaru - EGS says he has finished processing deep optical data from Subaru,
pipeline runs smoothly and produced beautiful stacks.
Gemini - MJI processed some SLUGS imaging g,r,i data for an IoA-led lensing
VIMOS - a pile of venerable P74 and P75 deep VIMOS imaging data on nearby
galaxies has also been processed to final stack stage. All that remains is
to trim the crud, remove the assorted reflections and generate some
6. WFCAM update.
09A processing is going smoothly after a slight pause to generate a full
set of May flats before starting the May processing. MJI is doing the honours
while MR takes a well-earned break observing with WFCAM, or maybe the break
is the holiday bit after. As usual when at full steam the pipeline processing
to completion runs about 5-7 days behind the date of observation (but recall
the observation "date" is UT-based and it takes ~24 hours from observing
finishing for the data to be transferred, verified and ingested here).
At least one logjam could finally be cleared now that sufficient 09A data has
been accrued to compute the photometry corrections. The final checks and
photometry udpates were made to January-April 09A data over the w/e and this
has now been flagged as ready for transfer to WFAU.
A disk hardware failure on a RAID unit on one of the processing engines
required a disk drive to be replaced. The bad news is that this flavour
is now out of warranty, the good news is that we have around 8 spares
from a now defunct older RAID array.
Reprocessing is currently stymied somewhat by insufficient hours in the week
but moaning aside the current status is:
the UDS K-band reprocessed data awaits the final thumbs up from the UDS team
but is looking good. MJI was asked to help make a quick assessment of the
depth (expected -v- reality) and predicted from purely photon noise
limitations due to sky, a 5-sigma depth of K=22.3 within a 2 arcsec diameter
aperture from the no. of components in the latest stack, or ~22.1 for an
average point source - i.e. accounting for aperture loss. His analysis of
the random noise component (allowing for interpolation correlated noise) of
a 10kx10k chunk of the proposed deep stack suggested K=22.2, in reasonable
agreement. Larger scale correlations present in the deep stack will conspire
to lower this limit by ~0.1 mags but this is now fairly close to what is
being measured by the UDS folk. MJI will double check with Omar at the
Board meeting before making this available for transfer and also discuss
with NCH the simplest approach to this.
MJI finally started the GPS nebulosity reprocessing after writing a script
to do all the hard work. The filtering took a few days and the catalogue
regeneration several more. It now needs assorted tests running in
conjunction with GPS folk.
Lack of available effort has meant that the Petrosian etc. reprocessing/
updates for 05A-08A still haven't been started and neither has supplying
WFAU with the relevant script/details.
MJI noted that there is an upcoming PPRP meeting in Glasgow at which, among
other things, a proposal to build a planet-finder spectrograph for UKIRT will
be discussed. He noted that if successful we could be involved in the
There has also been a flurry of e-mails about an old hot potato, flushing the
detectors after changing filters. In practice this amounts to a single
"quick look" recipe exposure of similar parameters to the upcoming science
observations after each filter change. For some projects this adds
negligible overheads (~1%) for others it can be up to ~5% or more. The need
for this is going to be reassessed. It mainly (solely?) seems to affect
detector#1 and was first spotted in SV data taken in April 2005 by SJW.
7. VISTA update
CASU have received complementary remarks from Thomas Szeifert on the quality
of the output from the summit pipeline.
JRL notes he has made a few more bug fixes (Shirley not) to VIRCAM software
and is making good progress on the UK science pipeline. No new useful science
data has yet been received but we anticpate once the detectors are running
at the correct operating temperature and the camera is back on the telescope
that characterisation will begin in earnest once more. STH has flown out to
Paranal to help JPE and WJS with this process.
MJI asked about the progress with the tiling software, tests with MONTAGE,
and in particular what the contents of the tile FITS headers will be, i.e.
what gets inherited, what doesn't, what extra keywords etc... JRL said he
would produce an example tile FITS headers that could be used as the basis
for discussion with interested UK parties. <<<<
As noted earlier the VISTA processing status webpages are now live after
a lot of recent work. They are accessible remotely but deliberately are
hidden from casual view. All of the useful processed VISTA data up to the
end of January is being checked astrometrically and photometrically and
the usual WFCAM style QC information is being generated. The first tranche
of this is available through the web page and the first pass colour equations
and default zero-points have been produced. New raw data is picked up
and logged on these pages as it comes in after being copied, verified and
MJI commented that JPE was planning to spend his upcoming sabbatical at the
IoA and given the current STFC funding crisis we can put him to good work
helping to run the VISTA pipeline.
The TV/wall mounted monitor for the CASU meeting area has been in abeyance
for a while and as SC raised this topic MJI asked SC to investigate the
possibility of a display similar to that on the wall outside the Sackler
lecture theatre -v- a large format LCD TV (which was the original plan). <<<<
RWA continue checking through the technical pages looking for glitches et al.
JRL produce web page blurb for HawkI
MJI finish off HAWKI reprocessed data by nebulosity filtering and
MJI finish off stage-I software release pages
MJI finish off technical paper updates
MJI write utility to convert binary tables with all corrections
EGS send off example code to UVEX team
NAW polite but firm e-mail to VIZIER about IPHAS reqired
STH put ASTEP info on webpage
MR check with JAC to see if SDF files have same detector#4 header problem
MJI continue producing summaries a la WFCAM of reduced VISTA data
MJI sort out VISTA i/o disks
JRL set up semaphore system for AAO transfers
JRL check any impact of ESO archive decision to go to Blu-Ray disks
JRL produce draft FITS headers for tile products
SC investigate TV/monitor similar to that in the Hoyle for CASU meeting area.