You are here: Home / Documents / Minutes of CASU Meetings / minutes_100728.txt


Plain Text icon minutes_100728.txt — Plain Text, 11 KB (11364 bytes)

File contents

Wednesday 28th July 2010   APM Meeting tables    11:30 - 13:00

Present:     MJI, MR, EGS, JRL, AKY, RGM, NAW, JPE
Apologies:   STH, JCR


1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent & upcoming meets
4.  Data archives update     - AAT, ING, WFCAM, UKIRT, VISTA
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, VST
6.  WFCAM update
7.  VISTA update
8.  AOB


MJI welcomed AKY (Aybuke Kupcu Yoldas) to her first CASU meeting.  Aybuke
is currently helping JRL and EGS run the VISTA pipeline.

1. Actions from the last meeting

EGS  stage I software release progressed via test build release to WFAU
JRL  will be released on web pages next                                    <<<<

STH  still finishing off last bits of calibration plan document update     <<<<

RGM  still testing but wants illumination table option adding to enable    <<<<
     complete end-to-end test.  MJI will make it so.                       <<<<

ALL  contribute text to go with CASU versions of VISTA PR images. JPE
     volunteered to contribute some text - ongoing                         <<<<
JPE  contact ESO re: signing off pipeline deliverables - still ongoing     <<<<

MJI  made sure that WFAU are aware of tile photometric issues and also 
     produced a brief write up of the relevant bit for the web pages

ALL  made sure the internal review of VISTA processing was both fascinating
     and interesting.  Unfortunately, the interesting bits weren't
     fascinating and the fascinating bits weren't interesting.

EGS  updated the Zpt trend plot and made sure the NB118 was visible too

JRL  fixed the EXTNAME duplication in the catalogues and sorted out the
EGS  missing info in the SHDUs of the tile catalogues prior to v1.0 release
AKY  The ESO QC info and OB constraints have also been added (see version
     log on web pages for more info)

There ensued a brief discussion about the desirability or otherwise of adding
a TOTEXP keyword to the tiles.  Cons: the info is there in the HDU already to 
work it out; only EXPTIME matches the MAGZPT keyword possibly causing 
confusion if two exposure time keywords are present; there is a quite a range
(~1-6) of "total" exposure times for each pixel depending on location (e.g.
see confidence map); the info is listed in the web page summary files.  
Pros: you could define an "average" total exposure time; users will get it 
wrong and/or confused otherwise !

MJI  one has started and the other has thought about starting the final SV 
EGS  processing - ongoing                                                  <<<<

SC   investigated VPN for remote access and found it relatively easy to setup

SC   ZFS sluggishness on simple disk utlities still under investigation    <<<<

MJI  two out of three briefly got together after the meeting and agreed on
NAW  how to proceed

SC   ordered some more tapes, mais malheureusement pas de vouchers, and
     some longer longer ethernet cables                                

2. WFAU minutes

CASUrtools release/upgrades featured a few times as did saturation 
corrections, flagging of "extra" objects around bright stars, and
data products delivery to ESO.

JRL noted that he had finally finished his build script mods and had released
a new version of the CASU software (mainly stacker, cataloguer, wcsfitter, 
and list-driver) to WFAU.  The nebulosity filter and tiling software still
need a few add ons and further checking/feedback about tile products,
before being included.

MJI supplied a suggested correction for saturated objects along the lines
of the UKIRT Newletter article but wasn't entirely convinced about the
desirability of directly replacing apermag3 at ingest with a corrected 
value for saturated stars.

It was felt that flagging extra spurious objects around bright star halos,
diffraction spikes, or other artefacts, is an archive or user-end issue 
mainly because how do you know which objects are real/false without 
cross-matching to other bands or repeat observations.

Data products delivery to ESO is currently rather open-ended and awaits
more enlightenment as to process.  The promised Phase III workshop
in the Autumn is probably as good a place as any to make real progress
on this.

3. Recent and Upcoming Meetings

Wolfgang Hummel circulated the persistence report noted in the previous
IOT telecon.  Unfortunately the test involved flooding all the detectors
and noting the decay time of the overall background.  Even this appeared 
to be unpredictable when coupled with repeat additive events.  A more
realistic test involving saturated isolated star tests would be more 
relevant to normal operations but also unlikely to yield much of further 

To reiterate the conclusion from the last set of minutes: .... the consensus 
is that, like other NIR arrays, the persistence is not quantitatively 
predictable and that correcting for it is therefore impractical. Since a 
sensible observing sequence precludes it being a problem .....

Maria-Rosa and Brent have tried their damndest to find examples of residual
persistence in the VMC data, including the deep stacks and tiles of 30Dor.
A poxy bit of red nebulosity possibly caused by a 3rd magnitude star on one
of the pawprint sets, but possibly also just another bit of nebulosity in 
the region, is the sum total thus far.  So ..... no worries then.

MJI and JPE attended the UKIDSS and VISTA Galactic surveys meeting in Leeds
on July 8th to find out what folk are up to.  Eamonn Kerrins presented
an nice update on his DIA analysis for the VVV team.  We suspect that
this will be a good complement to list-driven methods but perhaps not the
only required show in town.

MJI and NAW hosted an EGAPS meeting in Cambridge on 12th July followed
by a half day discussion on the IPHAS global calibration: 
problems - traditional anchors too restrictive; 
solution - Bayesian version of Glazebrook's method 

NAW presented a brief update on PLATO and EUCLID news.

4. Data Archive

ING, AAO, UKIRT, VISTA  all up to date and still no complaints other than
whichever company now owns Sybase wanting 1/2 million in back payments.
JRL will inquire of CADC and CDS if they are also facing bankruptcy.       <<<<

5. Optical / NIR Processing

A bit of IPHAS'ing re: global calibration, a bit of Subaru processing from 
EGS and MJI, and some HAWKIng from JRL, otherwise tranquility rules.

6. WFCAM Update

WFCAM was back up and running on June 10th after its summer vacation, and
data flow and processing resumed as normal.

Raw data ingest and verification is up to date and WFCAM processing has 
caught up after a delay awaiting a full set of twilight flats.  The odd
confused MSB has been intercepted and lobbed through the pipeline but
otherwise all is proceeding smoothly.

MJI used the nebuliser to erase the GPS DR7 left overs and notified
WFAU re: transfers.

Chris Davis turned up another mysterious vanishing set of camera data 
(not #3 for once) from 29th July 2009.  The first 2/3 of the night had
no #4 data but the last 1/3 did.  Investigations are afoot, but fortunately
these occurences are very rare and there are unlikely to be many (any?)

SJW has noted a mysterious couple of examples of stacked interleaved data
for DR8 whereby some detectors look fine and others looked crap.  JRL will 
investigate further since this is not supposed to happen anymore.          <<<<

7. VISTA Update

Ingestion and verification of raw data up to 13th July is in place and
the next tape backup is due.  The "new" calibration unit is in place and
producing the goods re: linearity sequences.  Apart from the odd whinge
from JRL about 1s twilight flats and sometimes no darks to match the
flats (there's no pleasing some people), everything is rocking along fine.

Version 1.0 of VISTA data products containing all reprocessed data up to end 
of February were released to the PIs and VSA on 21st of July. A version log
summarising the changes as well as a description of the tiles and changes in
the sky background method for VVV and UltraVista is available from our web
pages.  The reprocessing of data up to the end of May is also finished and
will shortly be released together with, shortly thereafter, v1.0 to end
of June.  This has all been processed & post-processed bar the odd data
relocation.  The final chunk is awaiting ESO keywords and database 
ingestion here (and final verification) prior to being released in about a 
weeks time.

Stuart Sale popped over for the EGAPS meeting and sent us a couple of 2TB 
disks for the Chilean VVV end, to which we are copying stacked frames
and tile images and catalogues to be sent to Phil Lucas who will make sure 
they get handed over for direct transport to Chile. (Internet transfers to 
Chile are not reliable enough for this volume of data.)

Data transfer of processed data to PIs has now reached 9TB for the last 6 
months and will no doubt increase rapidly again with the release of v1.0.

Recent updates to skysub strategy (see web pages) include using 2 tiles if
possible for VVV to improve smoothness of sky frames viz-a-viz leftover bits
of GP stars; and incorporating an external deep object mask for UltraVista 
(and NB118).  The mask is based on stacking a season of observations.

MJI then initiated a quick review of recent fixes prior to v1.0 release
all of which were minor and involved trivial header manipulations. He was
assured (honest!) that all have been done including the ESO QC info and OB 
constraints where available (see earlier comment in actions section).

Finally a brief discussion about junk data and what to do about it ensued.
All VISTA data including junk is stored in the raw suppository.  At
pipeline processing preview stage, best efforts are made to recover
interrupted or otherwise screwed up OB structures, partial OBs (that get
repeated later) with enough observations present are processed as normal,
OBs with the odd file only therein are quietly ignored.

8. AOB

Disk space issues have eased somewhat with the end of the reprocessing 
effort but still needs monitoring.  For now the latest fibre-channel 14TB 
XFS system is being used as scratch space for VISTA processing but will 
eventually revert to WFCAM use.  At the next meeting we need to discuss 
acquiring more VISTA data storage capacity, the main question given our less 
than enthusiastic impression of ZFS systems, being what sort ?             <<<<

Continuing Actions:

EGS  - finish adding catalogue software to release pages

STH  - finish off last bits of calibration plan document update

RGM  - finish off testing catalogue conversion software

JPE  - contribute text to go with CASU versions of VISTA PR images

JPE  - contact ESO re: signing off pipeline deliverables

EGS  - SV time again still

SC   - does our ZFS need a good tune ?

New Actions:

MJI  - add in optional illumination table correction to catalogue software

JRL  - contact CDS and CADC about Sybase licenses

JRL  - figure out why some WFCAM stacked interleaves are FU'd

SC   - investigate options for disk storage for VISTA