—
Plain Text,
18 KB (18557 bytes)
File contents
Wednesday 21st April 2010 APM Meeting tables 11:30 - 13:00
Present: JCR, MJI, EGS, JRL, MR, STH, RGM, SC, JPE
Apologies: NAW
Agenda
=======
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent & upcoming meets
4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST
6. WFCAM update
7. VISTA update
8. AOB
Minutes
========
1. Actions from the last meeting
--------------------------------
EGS stage I software release partially done but still more to do. Will
include JRL's standalone catalogue software tarball together
with the already included example piece of code for applying all the
corrections for the catalogue binary FITS tables. <<<<
MJI asked that the stacking code not be included yet since it contains
some sneaky tricks that are (honest!) going to be published in the
WFCAM and VISTA technical papers.
MJI the dome flat comparison is still to be done. <<<<
STH updating the Calibration Plan document is progressing well. It
will better reflect the actual method in use for photometric
calibration for VISTA with: the latest colour equations, Vega
zero-point offsets from standard stars, AB magnitude conversions,
comparison with WFCAM photometry etc.. There's enough high
latitude data to do overlap tests. Should be ready by next meeting. <<<<
JPE noted an issue with the VISTA NB980/990 split filter which
hasn't been tested on the sky yet. MJI pointed out that currently
the filter keyword is only present in the PHU and for this we really
need to get filter keywords into the FITS extensions. This issue will
be raised at the IOT meeting on Friday. <<<<
RGM is still testing the catalogue conversion software to destruction <<<<
EGS plonked our png versions of the ESO-released PR images on the
ALL CASU web pages. Now we need to write a small amount of text for each
picture. EGS will put in links/cross references to the ESO versions. <<<<
SC has added EGS's machine to the traffic monitoring web page. MJI noted
that since the bottleneck was fixed internal traffic now gets close to
Gb/s connectivity
MR added the extra odds and ends of the rediscovered missing July WFCAM
data to the ESO transfers and
MJI made sure the raw data was archived to tape
MR has put a break point in the progress web pages denoting the end of
"Science Verification" and the beginning of the Dark Ages
EGS informed the VISTA survey PI, who downloaded data from the night of
20100107 about the fix, whereby a DFU caused an EFU from detector #11
onwards for a small number of OBs
EGS are revisiting the SV data and regenerating all the science products
MJI encore une fois. They gave a brief update of progress so far. EGS
STH noted the meeting in ESO of the Orion team on May 26th and MJI bet
that the other M's would want a similar get together for the NGC253
SV project. EGS is making good progress with the Orion SV data and
MJI is re-stacking NGC253 data and cursing the intermittent problems
(~30% of the time) with channel#14 detector#6, particularly bad for
the J-band which is the main bit left to finish off.
2. WFAU minutes
----------------
MJI was a bit puzzled about the comments regarding tardy 10a WFCAM releases
seemingly triggered by a Japanese PI who wanted fast access to his processed
WFCAM data taken 15th-19th March. The deal has always been to release pipeline
processed WFCAM data directly from CASU when asked by the PI for more urgent
access than normal. As it happened the monthly photometry updates for March
were being done as the email came in on 15th April. These are obviously not
usually included in fast access products. MJI then added that some of the
delay in releasing February data was caused by one night having to be
completely reprocessed after failing final checks done here. And in case
you are wondering, even after the monthly updates are done there is one final
set of hurdles the data has to overcome, error-free database ingest here,
which takes another day or two depending if problems are found or not.
In the interests of clarity regarding CASU software we have cobbled together
a timely brief history and commented on one or two other aspects that have
cropped up recently.
The software modules created by CASU for the pipelines exist in only two
forms. First are MJI's Fortran versions which are generally used as
standalone applications and which MJI also uses to prototype new ideas.
The C versions written by JRL are the same algorithmically but are
structured differently as they are meant to be part of a pipeline and exist
within the CASU C/Perl pipeline infrastructure (cirdr) in addition to
being part of the ESO software deliverables. JRL also directly prototypes
pipeline modules, but in C and/or Perl rather than Fortran. The sum total
of these C/Perl modules are what is used to pipeline process WFCAM and VISTA
data.
Back when WFCAM was first starting to rock'n roll out serious amounts of
data, WFAU requested a copy of the stacking and mosaicing routines which
could be use to do deep stacks and large area mosaics. Jonathan Irwin had
already produced standalone C routines using the existing toolkit software
development routines as algorithmic templates. These were released
and updated as needed to keep pace with functionality and toolkit upgrades
for WFCAM processing, together with the catalogue generation software and
list-driven photometry code (from within cirdr). The stacking routine was
functionally equivalent to what the pipeline did (since the pipeline stacker
in turn was developed from the toolkit code) and gave essentially (mainly bar
rounding errors) identical results. So, although it was not "the pipeline
module", differences were negligible. These stacking and mosaicing routines
however cannot be used with VISTA data since among other things they lack
recognition of the new (now standard) catalogue FITS table WCS format and
have no interpolation options, hence the need for newer versions.
Although the cataloguing routines came within the full cirdr environment
this was never intended for export since it is a peine dans le cul to build
and maintain outside of CASU. So, to make things easier for all involved
CASU agreed to shift the stacking, WCS fitting and source extraction software
from cirdr into a standalone package (casutools). The advantage is that the
processing modules in casutools are identical to those in cirdr. The only
difference is that the casutools modules are wrapped in C main routines
with a slightly different command line interface from the perl routines
that are used in cirdr. Interface issues notwithstanding these standalone
routines should yield identical results to the pipeline.
Inevitably there will be a few teething problems (e.g. local -v- networked
copies of 2MASS) with this slightly different approach but in the longer
term it should make future upgrades much simpler.
3. Recent and Upcoming Meetings
-------------------------------
The two J's gave VISTA presentations at NAM in Glasgow last week:
VISTA Performance - Jim Emerson (Queen Mary, Univ of London)
The VISTA science pipeline - James Lewis (Institute of Astronomy)
as did a bunch of VISTA survey PIs. There followed a bit of discussion
on extra non-VDFS processing for the various PI survey programmes. As far
as we know, Terrapix are doing the deep stacking, swarping, cataloguing
etc... for Ultra-VISTA and may be requested to do likewise for VIDEO;
Astrowise seem to be doing something similar for VIKING. The rest seem to
be more or less using the contents straight out of the VDFS processing tin.
There is an upcoming VISTA IOT telecon on Friday (23rd) 14:00 - 16:00
which JPE and MJI will take part in, but although MJI asked for significant
recollections from STH and JRL about the previous VISTA IOT he drew a blank.
A not-the-VDMT meeting took place on 7th April. Some of the issues MJI wanted
to raise were deferred but are listed here in case he forgets them namely:
updated ToR, membership to better reflect VISTA processing and archive
expertise, information access (somewhat eased by ESO web publishing some
of the oft-requested information) and ....... d'oh!
This triggered an alternative latent memory of a statement MJI made at the
affable CASU-WFAU information exchange which took the place of the VDMT,
regarding data rates of WFCAM cf. to VISTA. Now updated to include the first
3 months data for 2010, the raw Rice-compressed data volume received from
WFCAM was 4.0TB whilst that from VISTA was 4.3TB *. Uncompressed these would
be roughly 16TB and 17TB respectively. Since the processed output is 2x the
raw volume for WFCAM (blame interleaving) and only about 1.5x for VISTA
(blame tiling) both survey systems generate comparable data volume handling
requirements. In particular, this means that transport infastructure (e.g.
UKlight), computing infrastructure and disk storage requirements are similar
too and therefore do not require radically different solutions to those
already in place.
[* In case anyone is puzzled by this, among other differences, VISTA users
tend to do more coadds and use longer integrations.]
4. Data Archive
---------------
WFCAM, INT, MegaCam, AAT, UKIRT, VISTA all up to date..
5. Optical / NIR Processing
---------------------------
HAWKI - nothing to report apart from another draft paper by Thomas
Preibisch, bets on MM eruption timescale were laid.
INT - STH doing some processing for Spanish collaborators and EGS
is processing 2009 season IPHAS data
MegaCam - odds and ends of PAndAS survey data has been re-massaged
Subaru - upcoming data on MJI's to do list
VST - que ?
6. WFCAM Update
---------------
Processing and raw data ingest is up to date. As almost noted previously,
February and March 10a processed data have now passed their final checks and
have been flagged as ready to copy.
Remaining issues arising from 999 file runs from 12,13 July have been dealt
with (there were really 3186 and 3329 files), though MR will double-double
check the raw WFCAM archive has the extra frames ingested properly.
MJI wrote up the details of the nebulosity filter for the upcoming UKIRT
newsletter. EGS noted that he had been using the filter with some success
for Planck and Herschel data too. MJI commented that it is quite general
purpose for image feature separation based on scale size and that
the new filtering option, proper 2D cross-shape cf. to the original (faster)
consecutive 1D operations allows even more flexiblity.
MJI noted that AJA was leaving the building after 10 years and would be
relocating to Gemini. We want to thank him for all his hard work toward
making WFCAM the success that it is and for making our job easier by
providing an excellent interface for all WFCAM-related issues. We all wish
him well in his new enterprise and noted with amusement that he will once
again be Paul Hirst's boss.
One enterprising PhD student, David Sobral, requested, and received (after
a few sanity checks) all the raw WFCAM data for CMP/3. As usual we will be
interested to see what he makes of it.
MJI and STH briefly summarised their knowledge of the state-of-play of
the ongoing parallax measures for T720. The team are on the ball, contacting
CASU to get access to the data within minutes of a night being flagged as
processed. Must be interesting !
7. VISTA Update
---------------
Ingestion and processing data to pawprint level (version 0.8) up to date and
even all done up to 30th March.
The raw data disks are a bit slow to arrive at the moment (last one came on
Monday 12th), perhaps Marco and Simon's ash clouds are delaying post.
EGS then briefly outlined the post-processing stages. After processing we
run various extras, which includes astrometry checks, photometric calibration
and file Rice-tile compression, visual inspection of QC plots and some sample
images. The data is then copied to the final processed data directory and
FITS headers ingested in a database for further checks. If everything is ok
we then update the version of the fits file, keyword CASUVERS, and create a
VERSION_* file for the subdir which flags the night as being ready to copy
for WFAU.
Regarding data transfers, we have received ~100 direct PI requests for data
now, totalling some 4TB of (compressed) processed data.
JPE gave an update on the progress of the G-bit fibre to connect Paranal to
Antofagasta and noted an upcoming SPIE abstract/paper describing the
system and performance. Various transfer tests between Santiago and Garching
have been made to tune system performance and further tests including to
Cambridge are planned.
MJI and JRL have finished developing the standard VISTA filtering and
mosaicing software and this is now being implemented in the stage II
pipeline, which also includes recataloguing from the tiles etc...
It was agreed that it was important to know what PIs need from tile products
and check also that they understand what they will get. However, demonstrating
the results from the method we are proposing is key here and that is still
some way off. Tiling without filtering is an option (e.g. to make colourful
PR images) but we suspect that most punters will not find these much use
for the majority of their science requirements. For now we are proposing to
deal with this latter type of tiling on an adhoc request basis and see how
it goes.
It was noted that the narrowband Fynbo DDT data, which had an exposure
time of 5 minutes per image and 2.5 hours per OB, was hard to deal with
because the background sky level varies too much. JRL is doing the best he
can but in future exposures should aim at ~1 minute (and are still not in
danger of being sky background-limited). JRL has implemented the algorithm
to use externally generated sky masks and this is another possibility to try
here since the region observed is part of the Ultra-VISTA survey field for
which we have already received a tile mask.
There was a discussion about how best to deal with the overlaps ("ears")
between tiles, which is generally more of a problem for larger area surveys
aiming for efficient sky coverage. Mosaicing the overlapping bits of "ears"
from adjacent tiles and recataloguing from them is clearly the best solution
but is not something that can be implemented in a nightly pipeline as it
requires a non-causal solution.
Several PIs have been unable to download VISTA data easily from MACS (simple
CLI tools like wget and lftp are apparently not standard on MAC distributions
), so EGS has written Python scripts to help.
"Short wave depression" is not a new psychosis but a more inventive hardware
excuse for the problems with the top part of detector#16. Apparently a week
after cooldown and all will be well. Slight problem with this theory, so
we've opted for the CASU SWFUD technical term. Until this situation changes
the best option will probably be to mask out the top 1/3 of the detector
for Z -> J bands when tiling. This will be done by zeroing the confidence
map in that region when creating the tiles. [The confidence map does not pick
this up automatically as poor since over short time periods e.g. during
linearity sequences, from which the basic confidence/bad pixel map is made
it behaves generally ok.]
In a significant fraction of cases the CRVALS of raw VISTA files are set
incorrectly to 0.0 in the FITS headers at the telescope. Unsurprisingly
this is a problem for dither stacking. The science pipeline here has had
a workaround for this (e.g. check and use RA Dec instead), and other similar
problems, but the Paranal and Garching pipelines currently don't, causing
lots of "tickets". JRL estimates that this happens several times a night.
he has already been in contact with Steven Beard about this but the simulator
software does not show this problem on long soak tests making it hard to
nail down. JPE will raise the issue at the IOT on Friday. <<<<
There were a bunch of fascinating but trivial FITS header issues like
EXTNAMEs not making it into the science products. JRL has tracked these
down and fixed the problems, not that it affects the products in any
sentient way.
8. AOB
-------
JPE++ and SC-- may be testing streaming VISTA data directly here (and back)
to test for any potential bottlenecks in VISTA data transfer.
5 new BLADES (20 cores) have been acquired and added to the VISTA processing
kit and SC is in the process of connecting them up and installing an
operating system on them.
Not much progress on the ESO in-kind-payment front - don't watch this space.
EGS pointed out that apm25 & apm26 system software needs to be upgraded soon
since the current Ubuntu software version is reaching the end of its 4 year
support cycle. The next Ubuntu release is due the end of the month. We
may try it out on JRL's machine first which as usual refuses to do his
bidding.
MJI noted that the UKIRT board report is due by the beginning of May and he
will concoct some suitable fabrications to include. <<<<
MJI moaned about lack of disk space on /data/apm29_a and hopes that soon
Jim'll fix it ..... oh nearly forgot ...... and the recently introduced
minor FUD'd IAMFUDs in the WFCAM catalogues. <<<<
Continuing Actions:
===================
EGS - finish adding catalogue software to release pages
MJI - get around to finishing dome flats comparison
STH - finish off last bits of calibration plan document update
JRL
RGM - test catalogure conversion software
ALL - contribute text to go with CASU versions of VISTA PR images
EGS - having unbutted finish off the SV data reprocessing in time for
MJI end of May chinwag
STH
New Actions:
============
JPE - raise issue of NB980/990 split filter and FITS headers at IOT
EGS - to put in links/cross references to the ESO versions of PR images
JPE - raise the issue of absent (actually 0.0) CRVALS at the IOT
MJI - write and submit UKIRT Board report
JRL - tidy up the detritus on /data/apm29_a
JRL - fix recently introduced minor perturbations in IAMFUD in catalogues
