You are here: Home / Documents / Minutes of CASU Meetings / minutes_100421.txt

Plain Text icon — Plain Text, 18 KB (18557 bytes)

File contents

Wednesday 21st April 2010   APM Meeting tables    11:30 - 13:00

Present:     JCR, MJI, EGS, JRL, MR, STH, RGM, SC, JPE
Apologies:   NAW


1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent & upcoming meets
4.  Data archives update     - AAT, ING, WFCAM, UKIRT, VISTA
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, VST
6.  WFCAM update
7.  VISTA update
8.  AOB


1. Actions from the last meeting

EGS  stage I software release partially done but still more to do. Will
     include JRL's standalone catalogue software tarball together
     with the already included example piece of code for applying all the 
     corrections for the catalogue binary FITS tables.                     <<<<
     MJI asked that the stacking code not be included yet since it contains
     some sneaky tricks that are (honest!) going to be published in the
     WFCAM and VISTA technical papers.

MJI  the dome flat comparison is still to be done.                         <<<<

STH  updating the Calibration Plan document is progressing well. It
     will better reflect the actual method in use for photometric
     calibration for VISTA with: the latest colour equations, Vega
     zero-point offsets from standard stars, AB magnitude conversions,
     comparison with WFCAM photometry etc.. There's enough high
     latitude data to do overlap tests. Should be ready by next meeting.   <<<<

     JPE noted an issue with the VISTA NB980/990 split filter which 
     hasn't been tested on the sky yet.   MJI pointed out that currently
     the filter keyword is only present in the PHU and for this we really
     need to get filter keywords into the FITS extensions. This issue will
     be raised at the IOT meeting on Friday.                               <<<<

RGM  is still testing the catalogue conversion software to destruction     <<<<

EGS  plonked our png versions of the ESO-released PR images on the
ALL  CASU web pages. Now we need to write a small amount of text for each 
     picture.  EGS will put in links/cross references to the ESO versions. <<<<

SC   has added EGS's machine to the traffic monitoring web page. MJI noted
     that since the bottleneck was fixed internal traffic now gets close to 
     Gb/s connectivity

MR   added the extra odds and ends of the rediscovered missing July WFCAM 
     data to the ESO transfers and 

MJI  made sure the raw data was archived to tape

MR   has put a break point in the progress web pages denoting the end of
     "Science Verification" and the beginning of the Dark Ages

EGS  informed the VISTA survey PI, who downloaded data from the night of 
     20100107 about the fix, whereby a DFU caused an EFU from detector #11 
     onwards for a small number of OBs

EGS  are revisiting the SV data and regenerating all the science products
MJI  encore une fois.  They gave a brief update of progress so far.  EGS
STH  noted the meeting in ESO of the Orion team on May 26th and MJI bet
     that the other M's would want a similar get together for the NGC253 
     SV project.  EGS is making good progress with the Orion SV data and 
     MJI is re-stacking NGC253 data and cursing the intermittent problems 
     (~30% of the time) with channel#14 detector#6, particularly bad for 
     the J-band which is the main bit left to finish off.

2. WFAU minutes

MJI was a bit puzzled about the comments regarding tardy 10a WFCAM releases
seemingly triggered by a Japanese PI who wanted fast access to his processed 
WFCAM data taken 15th-19th March. The deal has always been to release pipeline 
processed WFCAM data directly from CASU when asked by the PI for more urgent 
access than normal.  As it happened the monthly photometry updates for March 
were being done as the email came in on 15th April.  These are obviously not 
usually included in fast access products.  MJI then added that some of the 
delay in releasing February data was caused by one night having to be 
completely reprocessed after failing final checks done here.  And in case
you are wondering, even after the monthly updates are done there is one final
set of hurdles the data has to overcome, error-free database ingest here,
which takes another day or two depending if problems are found or not.

In the interests of clarity regarding CASU software we have cobbled together 
a timely brief history and commented on one or two other aspects that have
cropped up recently.

The software modules created by CASU for the pipelines exist in only two
forms. First are MJI's Fortran versions which are generally used as
standalone applications and which MJI also uses to prototype new ideas. 
The C versions written by JRL are the same algorithmically but are 
structured differently as they are meant to be part of a pipeline and exist 
within the CASU C/Perl pipeline infrastructure (cirdr) in addition to
being part of the ESO software deliverables. JRL also directly prototypes
pipeline modules, but in C and/or Perl rather than Fortran. The sum total
of these C/Perl modules are what is used to pipeline process WFCAM and VISTA

Back when WFCAM was first starting to rock'n roll out serious amounts of 
data, WFAU requested a copy of the stacking and mosaicing routines which 
could be use to do deep stacks and large area mosaics.  Jonathan Irwin had
already produced standalone C routines using the existing toolkit software 
development routines as algorithmic templates.  These were released
and updated as needed to keep pace with functionality and toolkit upgrades 
for WFCAM processing, together with the catalogue generation software and
list-driven photometry code (from within cirdr). The stacking routine was 
functionally equivalent to what the pipeline did (since the pipeline stacker
in turn was developed from the toolkit code) and gave essentially (mainly bar 
rounding errors) identical results. So, although it was not "the pipeline 
module", differences were negligible.  These stacking and mosaicing routines 
however cannot be used with VISTA data since among other things they lack 
recognition of the new (now standard) catalogue FITS table WCS format and 
have no interpolation options, hence the need for newer versions.

Although the cataloguing routines came within the full cirdr environment 
this was never intended for export since it is a peine dans le cul to build
and maintain outside of CASU.  So, to make things easier for all involved 
CASU agreed to shift the stacking, WCS fitting and source extraction software 
from cirdr into a standalone package (casutools). The advantage is that the
processing modules in casutools are identical to those in cirdr. The only 
difference is that the casutools modules are wrapped in C main routines 
with a slightly different command line interface from the perl routines 
that are used in cirdr.  Interface issues notwithstanding these standalone 
routines should yield identical results to the pipeline.

Inevitably there will be a few teething problems (e.g. local -v- networked
copies of 2MASS) with this slightly different approach but in the longer 
term it should make future upgrades much simpler.

3. Recent and Upcoming Meetings

The two J's gave VISTA presentations at NAM in Glasgow last week:

   VISTA Performance - Jim Emerson (Queen Mary, Univ of London)

   The VISTA science pipeline - James Lewis (Institute of Astronomy)

as did a bunch of VISTA survey PIs.  There followed a bit of discussion
on extra non-VDFS processing for the various PI survey programmes.  As far 
as we know, Terrapix are doing the deep stacking, swarping, cataloguing 
etc... for Ultra-VISTA and may be requested to do likewise for VIDEO; 
Astrowise seem to be doing something similar for VIKING.  The rest seem to
be more or less using the contents straight out of the VDFS processing tin.

There is an upcoming VISTA IOT telecon on Friday (23rd) 14:00 - 16:00
which JPE and MJI will take part in, but although MJI asked for significant 
recollections from STH and JRL about the previous VISTA IOT he drew a blank.

A not-the-VDMT meeting took place on 7th April. Some of the issues MJI wanted
to raise were deferred but are listed here in case he forgets them namely:
updated ToR, membership to better reflect VISTA processing and archive
expertise, information access (somewhat eased by ESO web publishing some
of the oft-requested information) and ....... d'oh!

This triggered an alternative latent memory of a statement MJI made at the 
affable CASU-WFAU information exchange which took the place of the VDMT,
regarding data rates of WFCAM cf. to VISTA.  Now updated to include the first 
3 months data for 2010, the raw Rice-compressed data volume received from
WFCAM was 4.0TB whilst that from VISTA was 4.3TB *. Uncompressed these would
be roughly 16TB and 17TB respectively. Since the processed output is 2x the
raw volume for WFCAM (blame interleaving) and only about 1.5x for VISTA
(blame tiling) both survey systems generate comparable data volume handling 
requirements. In particular, this means that transport infastructure (e.g. 
UKlight), computing infrastructure and disk storage requirements are similar 
too and therefore do not require radically different solutions to those 
already in place.

[* In case anyone is puzzled by this, among other differences, VISTA users 
   tend to do more coadds and use longer integrations.]

4. Data Archive

WFCAM, INT, MegaCam, AAT, UKIRT, VISTA  all up to date..

5. Optical / NIR Processing

HAWKI    - nothing to report apart from another draft paper by Thomas 
           Preibisch, bets on MM eruption timescale were laid.
INT      - STH doing some processing for Spanish collaborators and EGS
           is processing 2009 season IPHAS data
MegaCam  - odds and ends of PAndAS survey data has been re-massaged
Subaru   - upcoming data on MJI's to do list
VST      - que ?

6. WFCAM Update

Processing and raw data ingest is up to date.  As almost noted previously, 
February and March 10a processed data have now passed their final checks and
have been flagged as ready to copy.

Remaining issues arising from 999 file runs from 12,13 July have been dealt 
with (there were really 3186 and 3329 files), though MR will double-double 
check the raw WFCAM archive has the extra frames ingested properly.

MJI wrote up the details of the nebulosity filter for the upcoming UKIRT 
newsletter.  EGS noted that he had been using the filter with some success 
for Planck and Herschel data too.  MJI commented that it is quite general 
purpose for image feature separation based on scale size and that 
the new filtering option, proper 2D cross-shape cf. to the original (faster) 
consecutive 1D operations allows even more flexiblity. 

MJI noted that AJA was leaving the building after 10 years and would be 
relocating to Gemini.  We want to thank him for all his hard work toward
making WFCAM the success that it is and for making our job easier by 
providing an excellent interface for all WFCAM-related issues.  We all wish 
him well in his new enterprise and noted with amusement that he will once 
again be Paul Hirst's boss.

One enterprising PhD student, David Sobral, requested, and received (after
a few sanity checks) all the raw WFCAM data for CMP/3.  As usual we will be 
interested to see what he makes of it.

MJI and STH briefly summarised their knowledge of the state-of-play of
the ongoing parallax measures for T720.  The team are on the ball, contacting
CASU to get access to the data within minutes of a night being flagged as 
processed.  Must be interesting !

7. VISTA Update

Ingestion and processing data to pawprint level (version 0.8) up to date and 
even all done up to 30th March.

The raw data disks are a bit slow to arrive at the moment (last one came on
Monday 12th), perhaps Marco and Simon's ash clouds are delaying post.

EGS then briefly outlined the post-processing stages.  After processing we 
run various extras, which includes astrometry checks, photometric calibration 
and file Rice-tile compression, visual inspection of QC plots and some sample
images.  The data is then copied to the final processed data directory and 
FITS headers ingested in a database for further checks. If everything is ok 
we then update the version of the fits file, keyword CASUVERS, and create a 
VERSION_*  file for the subdir which flags the night as being ready to copy 
for WFAU.

Regarding data transfers, we have received ~100 direct PI requests for data 
now, totalling some 4TB of (compressed) processed data.  

JPE gave an update on the progress of the G-bit fibre to connect Paranal to 
Antofagasta and noted an upcoming SPIE abstract/paper describing the
system and performance.  Various transfer tests between Santiago and Garching
have been made to tune system performance and further tests including to
Cambridge are planned.

MJI and JRL have finished developing the standard VISTA filtering and 
mosaicing software and this is now being implemented in the stage II
pipeline, which also includes recataloguing from the tiles etc...
It was agreed that it was important to know what PIs need from tile products
and check also that they understand what they will get. However, demonstrating
the results from the method we are proposing is key here and that is still
some way off.  Tiling without filtering is an option (e.g. to make colourful
PR images) but we suspect that most punters will not find these much use
for the majority of their science requirements.  For now we are proposing to
deal with this latter type of tiling on an adhoc request basis and see how 
it goes.

It was noted that the narrowband Fynbo DDT data, which had an exposure
time of 5 minutes per image and 2.5 hours per OB, was hard to deal with 
because the background sky level varies too much. JRL is doing the best he 
can but in future exposures should aim at ~1 minute (and are still not in
danger of being sky background-limited).  JRL has implemented the algorithm 
to use externally generated sky masks and this is another possibility to try 
here since the region observed is part of the Ultra-VISTA survey field for
which we have already received a tile mask.

There was a discussion about how best to deal with the overlaps ("ears")
between tiles, which is generally more of a problem for larger area surveys
aiming for efficient sky coverage.  Mosaicing the overlapping bits of "ears"
from adjacent tiles and recataloguing from them is clearly the best solution 
but is not something that can be implemented in a nightly pipeline as it 
requires a non-causal solution.

Several PIs have been unable to download VISTA data easily from MACS (simple
CLI tools like wget and lftp are apparently not standard on MAC distributions
), so EGS has written Python scripts to help.

"Short wave depression" is not a new psychosis but a more inventive hardware
excuse for the problems with the top part of detector#16.  Apparently a week 
after cooldown and all will be well.  Slight problem with this theory, so 
we've opted for the CASU SWFUD technical term.  Until this situation changes
the best option will probably be to mask out the top 1/3 of the detector
for Z -> J bands when tiling.  This will be done by zeroing the confidence 
map in that region when creating the tiles. [The confidence map does not pick 
this up automatically as poor since over short time periods e.g. during 
linearity sequences, from which the basic confidence/bad pixel map is made 
it behaves generally ok.]

In a significant fraction of cases the CRVALS of raw VISTA files are set 
incorrectly to 0.0 in the FITS headers at the telescope.  Unsurprisingly 
this is a problem for dither stacking.  The science pipeline here has had
a workaround for this (e.g. check and use RA Dec instead), and other similar 
problems, but the Paranal and Garching pipelines currently don't, causing
lots of "tickets".  JRL estimates that this happens several times a night.
he has already been in contact with Steven Beard about this but the simulator
software does not show this problem on long soak tests making it hard to
nail down.  JPE will raise the issue at the IOT on Friday.                 <<<<

There were a bunch of fascinating but trivial FITS header issues like 
EXTNAMEs not making it into the science products.  JRL has tracked these
down and fixed the problems, not that it affects the products in any 
sentient way.

8. AOB

JPE++ and SC-- may be testing streaming VISTA data directly here (and back) 
to test for any potential bottlenecks in VISTA data transfer.

5 new BLADES (20 cores) have been acquired and added to the VISTA processing 
kit and SC is in the process of connecting them up and installing an 
operating system on them.

Not much progress on the ESO in-kind-payment front - don't watch this space.

EGS pointed out that apm25 & apm26 system software needs to be upgraded soon 
since the current Ubuntu software version is reaching the end of its 4 year
support cycle.  The next Ubuntu release is due the end of the month.  We
may try it out on JRL's machine first which as usual refuses to do his 

MJI noted that the UKIRT board report is due by the beginning of May and he
will concoct some suitable fabrications to include.                        <<<<

MJI moaned about lack of disk space on /data/apm29_a and hopes that soon 
Jim'll fix it ..... oh nearly forgot ...... and the recently introduced 
minor FUD'd IAMFUDs in the WFCAM catalogues.                               <<<<

Continuing Actions:

EGS  - finish adding catalogue software to release pages

MJI  - get around to finishing dome flats comparison

STH  - finish off last bits of calibration plan document update

RGM  - test catalogure conversion software

ALL  - contribute text to go with CASU versions of VISTA PR images

EGS  - having unbutted finish off the SV data reprocessing in time for 
MJI    end of May chinwag

New Actions:

JPE  - raise issue of NB980/990 split filter and FITS headers at IOT

EGS  - to put in links/cross references to the ESO versions of PR images

JPE  - raise the issue of absent (actually 0.0) CRVALS at the IOT 

MJI  - write and submit UKIRT Board report

JRL  - tidy up the detritus on /data/apm29_a

JRL  - fix recently introduced minor perturbations in IAMFUD in catalogues