You are here: Home / Documents / Minutes of CASU Meetings / minutes_081127.txt

Plain Text icon — Plain Text, 16 KB (16889 bytes)

File contents

27th November 2008  APM meeting tables  11:30-12:50

Present:          MJI, JRL, STH, MR, DWE, EGS, RGM, RWA
Apologies:        NAW, PSB


1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru
6.  WFCAM update            
7.  VISTA update            
8.  AOB                     


1.  Actions from last meeting

STH  has not finished checking that the Plone astrometry technical pages 
     are correct, however, the photometry pages have been updated and 
     the astrometry ones are "mid-edit" - ongoing                         <<<<

STH  has included the graphic showing the size of the photometric radial 
     distortion correction in the Plone technical pages 
JRL  has not produced a web page of HawkI publicity for CASU - ongoing     <<<<

MJI  have not finished off the stage-I software release pages - ongoing    <<<<

JRL  attended the Cambridge VOTC meeting

STH  has not made the photometric calibration paper available on the Plone 
     and VDFS sites since he is waiting for the referee's response on the
     re-submitted paper. Apart from holding off placing this on astro-ph 
     it was not thought necessary to wait further - make it so             <<<<

MJI  has not finished off the technical paper - ongoing                    <<<<

MJI  has passed on feedback regarding CASU contentment with proposed directory 
     structure for transferring data on USB disks
MJI  has informed ESO that CASU would like to acquire the first few nights 
     of SV data as soon as possible after it is taken
PSB  has made some processed VIRCAM test data available to Edinburgh

STH  acquired some decent calibration data for VIRCAM
MJI  made sure JPE/WJS are aware of the Ndit=1 issue for twilight flats

2.  Comments on WFAU minutes

There were no specifc comments other than to note the plans for the next
UKIDSS data release, DR5.

3.  Recent and upcoming meetings

EGS was asked to check if the CASU presentation from the ESO Public Surveys
meeting in September is available on their website.                        <<<<

ESO meetings:
MJI was over at ESO for the LP workshop and had a meeting with the ESO 
members of the VISTA SV teams. Among other things discussed were action 
items on fast SV data access and the format (apparently Rice-tile compressed) 
of the 500 Gbyte USB disks which ESO will deliver VISTA data on.  He also
confirmed that PIs of VISTA surveys will get access to the raw data, if
required, direct from ESO. The ESO Garching pipeline will run on all 
calibration frames to derive QC information but will apparently only sample
a subset of the science frames for QC purposes !

QC0 run at Paranal is mainly concerned with conditions of the observations 
and visual checks. Real time OB grades are only stored locally in logs/DBs
which are heavily firewalled => summit OB grade will not make it into the 
FITS headers at least initially.  A processing status web page similar
to that used for WFCAM could be used for feedback of OB information derived
at CASU, but there is still an open question of how to close the loop on 
bad OBs, though most should get spotted at Paranal ?  We need to emphasise
that the onus is on the survey PIs and ESO to access and make use of the
information on our web pages.

MJI was also over at ESO for the OPC meeting in November (17th-20th) and 
discussed further VISTA narrow-band NB118 processing and assessment issues
(bandpass dependence on radius, sky fringing, calibration etc.) with Palle 
Moller and Wolfram Freudling.  Our default scheme for calibrating
the NB118 data, i.e. treat it like any other band and calibrate on 2MASS,
may work well enough for their immediate science needs.  A simple test is
to monitor the stability of the difference between ZPs derived from
consecutive J and NB118 data. So far this looks promising.

Estimating the likely size of illumination corrections from existing VIRCAM
data may be possible but no firm promises have been made since this is still
TBD.  To ease their worries about NB118 and ZY calibration STH has made the
WFCAM calibration paper available to them.   We also need to check what
the Calibration Plan document currently says about these issues.           <<<<

One further outcome of these meetings was the need to arrange another meeting
~February with the SV teams to discuss results from first pass processing 
and their initial assessment of the products.  It was agreed that we would 
try and provide rapid pawprint level access to these data products from the 
first two nights to facilitate the assessment process.  It was agreed that 
access via a passwd-protected web page would be fine for this, but we also 
need to ensure that there is easy access to the DP description linked from 
this access point.

We also discussed deep stacking the N4945 SV data, which includes NB118, 
and examining band-merged catalogues across the Orion SV region to check 
for lower level artefacts and other gotchas.  A further possiblity is to use
the many repeat observations in the N4945 SV data to examine time series 
analysis problems in conjuntion with NJC.  Tiling will come second, since 
it is more important to iterate quickly rather than wait for fully polished 

WFCAM meeting:
MJI observed with WFCAM at UKIRT on 3rd-6th October and whilst there 
had a meeting with Andy and Chris to finalise the format and content
of the MSBTid logs.  These are now generated after the first stage 
nightly data processing and provide a more rapid feedback on MSB 
quality with respect to cloud cover, seeing and sky brightness, if these
are specified by the PI. The Cl-range was clarified by defining 3 categories.
As the nominal ZP varies by ~+/-2% due to a mixture of measuring and
natural variation due to mirror cleaning cycles, these are now:
<5% variation in the nominal ZP represents photometric; 5-20% is 
thin cirrus, and >20% represents thick cirrus.  The MSBTid log also
flags those MSBs rejected at the summit.

The hopefully final iteration on the offset sky strategy was also discussed
and a full set of recipes are now in place (JITTER_FLIP12, 13, and 
JITTER_SKY), and, as data subsequently taken shows, working.

MJI gave a talk at ADASS and JRL talked.

MJI also noted that there is an upcoming UKIRT board meeting and that
the submitted report is available in the usual place on the internal
web pages. 

Likely attendance at the December Science with UKIDSS RAS meeting was also
discussed.  MJI will attempt to provide JRL with results from the new
and improved bug-fixed deblended Petrosian et al. measures prior to the 
meeting to pre-empt expected criticism from the unusual suspects.          >>>>

4.  Data archives update     - AAT, ING, WFCAM, UKIRT

RWA reported on an email from a satisfied customer using the ING archives.
A taxidermist has been contacted.

All archives are up to date though JRL noted that he is still sorting old
UKIRT CASS data header problems.

5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, ESO WFI

EGS reported that he has been processing data from the ESO Spitzer Imaging 
Survey (P.I. Berta) obtained with WFI (ESO). This is a survey over 5.5 sq.deg. 
of the Spitzer/SWIRE S1 area obtained in  B, V and R from 2002 to 2007 in 155 
nights in total. Only the central part has been processed before and the idea  
is to process everything with the same pipeline used also to process  
the optical data in the rest of SWIRE (i.e. WFS and Mosaic I and II).

MJI noted that it was time to get some mileage out of our processing of
HAWKI SV data and asked STH if he could contact Mark McCaughrean and see
what the status was. JRL commented that he was going to revisit the sky
subtraction before we produce the next iteration of the catalogues.
There was also some discussion about the usual radial distortion model that 
we generated to do the astrometry.  Apparently this is newsworthy !
MJI will contact Mark Casali and see if it really is.                      <<<<

RGM noted that there was some HAWKI PI data that he would like processing. <<<<

EGS has taken over finishing off the EGAPS UVEX data products from 07B now
that the U-band and HeI-band offset calibration has been sorted out.

MJI noted that various people including Google Sky and MicroSoft are 
interested in IPHAS mosaics of the Galactic Plane.  This is currently a
no work on our part project, which is fine if it stays that way.

Assorted new panoramic M31 survey data from MegaCam for 08B has arrived and
is being processed and the first data with the new Hamamatsu deep depletion
CCDs on SuprimeCam has been processed.  These detectors are early production
versions of those for HyperSuperimeCam and give a much improved i and z-band
response whilst maintaining the good blue passband sensitivity.  The cosmic-
ray tracks are also much more entertaining.

6.  WFCAM update             

The photometric calibration paper has been updated to incorporate the 
responses to the referee comments and has been resubmitted to MNRAS.

MR has been beavering away running the nightly pipeline for WFCAM data as
it comes in and is comfortably keeping up with the data flow.  The raw data
arrives and is checked, MEF'd and ingested within 24 hours of it being taken.
The nightly processing is started shortly thereafter.  Checking the products,
fixing the parquet floor pattern and bundling up the monthly photometry 
updates takes the rest of the time.  

The new data transfer setup from JAC now allows us to offer a (limited) 
fast track processing for urgent data requests where residual channel 
offsets are unimportant and for which photometric calibration to 2% 
accuracy is sufficient.  This option has already been trialled and can
deliver data products to the PI within 48 hours of the data being observed.
The service will be necessarily limited in scope and, after discussion with 
JAC, will probably only be offered through the PATT form via a tick box 
requiring additional justification for its use.  The extra load imposed by
this is expected to be marginal, but will be monitored and re-asssessed as

Assorted reprocessing issues have cropped up or been stalled since our last 
get together.  Three years downstream a long standing bug with estimated 
Petrosian radii and fluxes (also affecting Kron and Hall measures) for 
deblended objects has been reported to us. This affects all processed WFCAM 
data.  Fortunately, due to the rather large number of flux measures in the 
catalogues, a fix has been found for this that can be run on the existing 
catalogues.  Although the update can potentially be generated over a w/e run, 
progressing it all the way through the various data products in the middle 
of DR5 preparations did not seem like a good idea.  This update will therefore 
be rolled out for DR6 with the added benefit of allowing much more time for 

The UDS team came in with an unexpected request to reprocess all the UDS
data for 05B as this semester currently has a high attrition rate due
to background variations.  A few tests on typical nights unfortunately 
confirmed that the latest "pawprint" sky estimation strategy, with iterative
object masking, does appear to do a much better job.  This reprocessing is
scheduled after completion of 08b sometime mid-December.

A much shorter request for reprocessing a limited amount of Orion data taken
for U/05a/100 recently came in.  The original sky estimation and subtraction 
for this was so bad that this was pushed through quickly. This data also 
provides an interesting test case for the nebulosity filter developed for 
GPS photometry in regions of high nebulosity.

This leaves us with the extant GPS reprocessing in nebulous regions to improve
both detection and photometry, also on hold due to lack of available effort.

MJI commented that there still appeared to be a problem with scaling for 
nexp > 1 data. Although not irrepairable, this has been noted many time 
previously and he asked JRL to check with Brad at JAC to find out why this 
issue still persists.                                                      <<<<

7.  VISTA update            

STH gave a brief report on his "observing" trip to VISTA and summarised
the data he had taken whilst there.  As a result we now have available a 
series of pointings (pawprints) in Y,J,H,Ks,NB118 on several 2MASS 
touchstone fields and a few assorted other science fields plus the
corresponding series of twilight flats, darks and linearity sequences. Apart 
from a few problems due a TCS dither offset bug, this data has been processed 
with a preliminary Cambridge pipeline and has yielded some useful science

A first pass analysis of these and some earlier data has provided a good 
first stab at the astrometric distortion, including the need for, and 
definition of, an r**5 term (PV2_5).  It has also yielded estimates of the
system zero-points in these passbands and will be used to check the overall
system throughput, sky brightness in various bands and provide a first pass 
estimate of the colour equations with respect to 2MASS.  STH was asked to 
look at this available VISTA data and further investigate this data.       <<<<

To make the data easier to digest JRL and STH will also sort out the object 
names of the various OBs according to their RA and Dec (all currently have 
the same name).                                                            <<<<

MJI noted that all VIRCAM data products use the ESO HIERARCH system for
recording processing information (seeing, sky level, noise etc.) and also
use the "legal" table form of the WCS for recording the astrometry transform
parameters in the catalogues. This is currently incompatible with all the 
WFCAM post-processing analysis software operating on the catalogues.  As an 
interim solution this has been sidestepped by running a translating script to 
convert them to WFCAM style keyword names, but will need updating properly 
later on.
JRL noted that the latest release v0.8.4 of the VIRCAM pipeline software and 
updated docs were delivered to ESO the other day.  Ironically some requested
CPL updates related to FITS header problems and efficient MEF file writing 
that have been incorporated by ESO recently, have caused problems with
running the VISTA summit pipeline at Paranal due to them not being 
incorporated there yet.

Apparently a USB disk with all the VISTA data on up to early November is
en route to us via the correct channels, and currently is being ingested 
(slowly) into the ESO archive in Garching.  It hasn't arrived here yet but
watch this space.

PSB reports that he has acquired a VISTA Trac account and also that against 
the odds network transfers from Paranal are making good progress.  So we
already have access to the new data on the USB disk before it got here.

MJI reported that a sample of narrow band NB118 and J-band data has been 
made available to the UDS narrow-banders to enable them to check assorted
properties.  This had been previously cleared with JPE and WJS.

8.  AOB                  

MJI gave a brief update on the (lack of) news on the rolling grant application
of 18 months ago.  Various parties with vested interests in the data 
processing have been made aware of the issues and asked to press the case
for some action.

RWA has done a sterling job organising the APM and friends Christmas party,
lunchtime 9th December, consequently a good afternoon for inspired software

Finally, MJI reported on the recent changes proposed, nay happening, in the 
IoA computer support group and the impact it will have on CASU.

Continuing actions

STH  check that Plone astrometry technical pages are correct

JRL  produce web page of HawkI publicity for CASU

MJI  finish off stage-I software release pages

STH  make photometric calibration paper available on Plone and VDFS sites

MJI  finish off technical paper

New actions

PSB  compare Calibration Plan with current expectation for calibration

EGS  check if the CASU presentation from the ESO Public Surveys meeting in 
     September is available on ESO website.                   

MJI  pre-empt expected criticism re: deblended Petrosian fluxes           

STH  contact Mark McCaughrean about HAWKI Eta Carina data status

MJI  contact Mark Casali about HawkI radial distortion

RGM  liaise with JRL about processing HAWKI PI data

JRL  contact Brad at JAC regarding the Nexp > 1 issue

STH  further investigate photometric features in available VISTA data 
     (throughput, colour equations, sky brightness)

JRL  sort out project names of VISTA MSBs taken by STH