You are here: Home / Documents / Minutes of CASU Meetings / minutes_100216.txt

Plain Text icon — Plain Text, 16 KB (16508 bytes)

File contents

Tuesday 16th February 2010   APM Meeting tables    12:00-13:00

Present:        MJI, RWA, EGS, NAW, RGM, MR, SC, STH 
Apologies:      JRL, JPE

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings - VISTA IOTs
                                 - VISTA PI meeting in Cambridge
4.  Data archives update         - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing       - HAWKI, INT WFC, MegaCam, Subaru, VST .....
6.  WFCAM update                
7.  VISTA update                
8.  AOB 


1. Actions from the last meeting

MJI  stage I software release still on hold - ongoing                      <<<<
EGS  JRL has produced a standalone catalogue software tarball which will
     be incorporated together with the example piece of code for applying
     all the corrections from the catalogue binary FITS tables.

MJI  technical paper - no further progress but there is a cunning plan ........

MJI  reported that the dome flat comparison has not yet been finished      <<<<

JRL  likewise has not finished the low light level linearity tests         <<<<
     He has done some related work with the new waveform test data 
     but the linearity tests require further effort and/or better data.

SC   reviewed the backup plan for the main cluster and outlined a possible
     solution for the remaining kit.  MJI noted that most backup tapes are 
     now housed separately in the Observatory building so other than for a
     major catastrophe (in which case we wouldn't care anyway) we are sorted.
EGS  confirmed that the example of persistence in VISTA, and other recently
     derived information discussed during the PSPI meeting in Cambridge, have 
     been put on the technical web pages.

STH  said that he and JRL are working on the updates to the calibration 
JRL  document.  JRL managed to find the latest copy from a backup of PSB's
     and STH will update the plan to include 2MASS calibration details.
     He also wanted to know how much information to include on the more
     subtle aspects of this.  MJI suggested that he includes a brief 
     description of these and then refers the interested reader to the WFCAM
     MNRAS paper (Hodgkin et al. 2009) for more details.  This would also
     be a good opportunity to compute and publish the latest colour equations
     including any Vega zero-point offsets, AB magnitude conversions etc..  
     We would like to be close to signing this one off by the next meeting <<<<

RGM  has not yet destructively tested the catalogue conversion software    <<<<

EGS  added the Rice compression information to the VISTA web pages for those
     users who are not familiar with the WFCAM ones.  There was a brief
     discussion about which other software packages seamlessly supported
     this and the, now standard, table WCS format.

MJI  checked the status of the VISTA transfer to WFAU and JPE and EGS collated
     and supplied the list of project IDs

SC   reported that apmcolour has now been replaced with ... err ... apmcolour.
     The cost would be split between VDFS and GAIA, and Dafydd.

JPE  forwarded details of the VISTA session at NAM and succeeded in confusing
     everybody about the timing of the session.  Fortunately the website is
     now online and in real time RGM checked that the VISTA session is during
     the afternoon of the 13th April.

SC   still needs to dish out some TLC to the assorted web pages for remote 
     monitoring.  Whilst on this topic she noted that there had recently 
     been a call from security to say that one of the high temperature alarms 
     was going off.  This was great apart from the fact that the temperature
     was fine and nobody told us (until this meeting). Clearly a few wrinkles
     to iron out then !                                                    <<<<

2. Comments on the WFAU minutes

It was noted that PMW is due to retire ~end of April and we thought it would 
be a good excuse to mark this occasion with an appropriate knees-up.  MJI 
will contact NCH and see what plans there are afoot.                       <<<<

MJI noted that NJC's WFCAM standards star timeseries database is now 
available and encouraged STH, among others, to take a peek.

JRL reported a comment from the minutes of 20100115 suggesting that 
provenance keywords are missing in VISTA confidence maps. He will look into 
this, but would appreciate being directly informed about it.

RGM noted that the date for the next DR data release (DR7) was a tad vague
and wondered if there was a target date set.

3. Recent and upcoming meetings

MJI said that there was nothing of import to report from the last IOT and 
JRL concurred (offline). The next one will be in two weeks or so.

MJI commented that the PSPI meeting in Cambridge seemed to be very successful
and offered up the following list of items that needed investigating and/or
acting upon. In no particular order and by no means complete:

  versioning in each FITS file to avoid ambiguities
  half-light radius and flux also useful as classification check
  the CASU ETC seems to give erroneous Z-band predictions
  other differences between ESO and CASU ETC most likely aperture corrections
  FITS header items missing/desirable e.g. containers, concatenations, recipes
  conversion to AB mags cf.Blanton et al. 2005 2MASS; Hewett et al. 2006 WFCAM
  colour equations and extinction corrections (for latter see zpts)
  how to deal with the "ears" of tiles
  what to do with the top 1/3 of detector #16
  seeing -v- sec(z); sky SB -v- moon phase/distance -v- twilight
  is resampling within dither stacks to finer grid worthwhile
  completeness estimates - scaling relations ??
  tiling strategy - nebulosity filter, tile, then catalogue
  how many PI surveys really need "naked" tiles ?

There will be a session on VISTA (status and science) at the upcoming NAM 
in Glasgow in April.  It would be good to have a CASU presence as this even
though the main focus will be on early survey science results. Maybe the
SV results which CASU are involved in would be a contender.                <<<<

4. Data archives update

AAT, ING, WFCAM, VISTA: all going smoothly

UKIRT: JRL reported that he was still awaiting the last 20 or so nights of 
missing data. The rest of the port is in fact done (consumed?) and being 
backed up even as we speak. Once JAC have come through with the last bits of 
missing data he will dot the i's and cross the t's and underline the z's.

5. Optical /NIR processing

JRL reported he has just done some restacking of Hawki data for RGM + student. 
Apparently there are a lot more waiting to be done - que?

MJI relayed some news about the VST received via Janet Drew. Apparently the 
next attempt to get the M1 mirror cell shipped to Chile is due to be in May, 
and the stated cunning plan is to have the telescope assembled and pretty 
much commissioned by the end of 2010.  The summoning of survey representatives 
to ESO Garching is presently being pencilled in for June/July !

A pile of PAndAS project MegaCam data has been slowly winding its way through
the system and has now been stacked, pummelled, catalogued, and otherwise 
processed out.

Further IPHAS and UVEX processing is on hold until we get a breather.

6. WFCAM update

MJI and JRL report that JAC have been experimenting with subtle changes to 
waveforms for the the WFCAM detectors in an attempt to minimise channel-edge
effects, parquet floor patterns etc....  A set of test data from 20100129
was fast-track processed by MJI and JRL spent a while straining his eyeballs
to assess the effects.  Of the 10 different waveforms, a mixture of 4 CDS and
6 NDR, there were only two sets (current and a new one) that were worth
pursuing further. Further darks and linearity sequences were done with these 
and reported back on.  We are currently waiting now to see whether/when these 
new waveforms will be implemented.  In parallel a small slew of other hardware
work has also been carried out in an attempt to correct some of the 
long-standing quirks of the detector arrays.  It is not yet clear what, if 
any, the knock-on effects of these hardware changes will be but it will be 
worth keeping an eye out for any changes as of 1st February processed data.
WFCAM is currently running on the "old" waveforms, the implementation of the
new has been deferred for now (note added in writing up:  now = 20100218).

MR reports slight delays, partly because of lack of available disk space, 
partly due to the large amounts of data being received - recent nights have 
contained more than 3000 exposures, and partly due to time spent looking at a 
lot of extraneous interrupted LAS files.  MJI added that there had been a 
slight upus-cockus with delivery of some 1TB disks which are going to be used
as a cost-effective (and Green) measure to swap out 2 sets of 12 x 400 GByte 
disks to increase WFCAM storage space.

[As usual when reporting these things, the disks turned up shortly after the
 meeting concluded].

Meanwhile a chunk of VISTA storage space has been purloined temporarily and
normal business will be resumed as soon as possible.

There have also been the odd few problems with the recipe keyword in the 
fits headers but nothing untoward otherwise.  MR added that Brad, who among 
other things, had been handling the data transfers at the JAC end, had now 
left and his replacement Anubhav Agarwal (of the philosophical email musings) 
has taken over this aspect.

MJI noted the STFC intent to withdraw from UKIRT at the end of 2010 but
given both the difficulties of achieving this and the community desire to 
continue the UKIDSS surveys, somehow doubted that the end of 2010 would be 
the last we see of WFCAM data.  Current CASU planning, both financial and 
staff effort are in place to deal with WFCAM data processing and user support 
until 31st March 2012.

MJI added that the J-band UDS reprocessing for 05B-07B was finished.  It just
needs some of his CPU time to finish of the monthly calibration updates.
There remains the equivalent 56 nights of H-band data to do but this is a 
long way down the pecking order.

DR7 jpeg checking is appparently proceeding apace.  The only feedback received
so far has been from SJW who noted that 1.5% of LAS data were being deprecated,
but the vast majority of this was due to trailed images.

7. VISTA update

EGS reported that we have now added a new keyword in the primary header of 
all fits products, CASUVERS, which stores the CASU product release number. 
Since the release of version 0.6 to WFAU and to the PIs we have made several 
improvements and fixes: the sky rms is now corrected for the average noise
covariance properties; a small problem with the astrometry of stacked VISTA
images has been fixed; the Hall flux and radius have been replaced by the
half-light flux and radius; we have added the average confidence in the 
default aperture for each object (column #58) to the tables; and we have 
fixed a problem with the rejection algorithm on the stacks. Unfortunately
this took two iterations (version 0.7 was deprecated for internal use only)
and we are currently (re)reprocessing all the data from Oct 15 2009 and when 
finished it will be tagged as version 0.8.

JRL sent in a short report which after removing all the expletives was even
shorter: we now have raw data upto and including 20100203 but because of the 
reprocessing are a couple of weeks behind with the normal processing. The ever
prescient MJI had already asked SC to get a quote for more Sun Blade CPU's 
to fill the remaining 5 slots in the VISTA processing kit. SC noted that 
as Oracle have now taken over Sun, all the educational discounts have to be 
(re)checked which is delaying receipt of a quote.  At some (TBD) future stage 
we will also need to conjure up some more VISTA disk space but will delay 
making that purchase as long as possible.

JRL noted that he has also received an update from ESO on how the information 
about linked/concatenated/cojoined OBs will be propagated via the FITS headers
and will hopefully enlighten the rest of us.  He also reported that version 
1.0 of the summit and Garching pipelines and version 2.0 of the DRLD have 
been released. All of the important ARS tickets have been resolved (good job

RWA reconfirmed that data up to Feb 3 had been received and ingested and that 
data up to Feb 2 has been backed up onto tape. ESO apparently want the disks 
returned more quickly since they have only bought a small no. of them (d'er!)
We had, of course, been sending them back ~3 at a time to save posting costs.  

There have been quite a large number of upgrades to the technical pages  
and MJI urged the assembled hordes to check that the right data is available 
in the right place e.g. there was some discussion during the meeting regarding
the whereabouts of the colour equations and zero-points.

MJI noted that chip #14 has a marginally dodgy channel #11 amounting to a few 
ADU, but although it is being monitored, it is not considered a significant 
issue at this stage.

MJI said that the UltraVISTA team are worried about residual haloes around 
bright stars, among other things, and want to trial external sky object masks.
JRL has been working on this and will send then a "template" for how this
can be most easily achieved.  MJI is still skeptical about the utiliity of
this, particularly for halos of bright stars done using a pawprint sky
method, since we will end up with regions that are not sky-corrected at 
all.  As a glance at virtually any sky correction map will reveal, this is 
generally not a good idea.                                                 <<<<

MJI and Henry McCracken have been discussing AB magnitude conversions.  As
a starting point, M sent H the conversions for 2MASS and WFCAM since we expect
only minor differences for VISTA.  Henry worked out the VISTA Ks offset to
AB using the filter and detector QE curves, which turned out to be 0.01 mag
different to the 2MASS published Ks conversion - yup minor.  STH will revisit 
this and incorporate all of it in the VISTA photometric calibration paper

MJI reported verbally about an ongoing STFC-ESO exchange relating to 
contributions in kind in lieu of dosh for VISTA late delivery. Anybody
fancy living in Garching ?

ESO had another PR release of the CASU-generated colour images, this one was
the M42 beaut. Presumably this means we can now make our colour versions of
this and the other released PR images available.  EGS will make it so.     <<<<

8. AOB

MJI gave a summary of the news about the various grants that CASU people are
involved in this round and those that have been reviewed as part of the STFC
hatchet job.  The only ones vaguely smelling of roses are the CASU roller
and the VISTA exploitation grant, and even these are subject to 15% cuts -
though quite what that means in practice is still unclear.

After the job interviews in early Februrary an offer has been made to fill 
PSB's post and we hope to have more definitive news next week.

There was some further discussion about backup strategies for software on
assorted machines and SC will continue looking into various options for

In an attempt to be even greener than usual, and to forestall ever spiralling 
electricity costs, Peter Weeks has been asked to investigate installing a 
a louvred fan to externally cool the main computer room in cooler weather.
If this works the ~5K/yr likely savings should provide enough beer money
even for us.

Continuing actions

EGS  - add catalogue software and binary table conversion software to 
MJI    software release pages.  

MJI  - finish off dome flat comparison

JRL  - finish off low light level comparison and linearity assessment

JRL  - carry on with calibration plan doc updates and produce a new draft
STH    for discussion at next meeting

RGM  - soak test the catalogue conversion software

SC   - provide a bit of TLC for the remote monitoring web pages

SC   - report at next meeting on cunning backup plan

New actions

MJI  - contact NCH and see what plans are afoot for Peredur's retirement

ALL  - consider suitable talks for VISTA session at NAM and discuss with
       MJI well before abstract submission deadline

JRL  - make a pre-emptive strike to fix the format of external object masks

EGS  - plonk CASU versions of VISTA PR (released) colour images on web pages