You are here: Home / Documents / Minutes of CASU Meetings / minutes_090204.txt

Plain Text icon — Plain Text, 18 KB (18556 bytes)

File contents

4th February 2009  APM meeting tables  11:30-13:00

Present:          MJI, JRL, STH, MR, EGS, RGM, RWA, NAW, SC
Apologies:        DWE


1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru
6.  WFCAM update            
7.  VISTA update            
8.  AOB                     


1.  Actions from last meeting

STH  the technical pages are still not up to date and other inconsistencies
     have been pointed out in terms of current WFCAM processing practice
     jarring with existing information, MJI will help with the updates     <<<<

JRL  has not produced a web page of HawkI publicity for CASU, deferred
     due to other factors but still - ongoing                              <<<<

MJI  have not finished off the stage-I software release pages - ongoing    <<<<

STH  has put the WFCAM calibration paper on astro-ph, but RGM pointed out
     that the astro-ph version of the photometry paper has a couple of 
     messed up figures. STH will fix this and put a high resolution 
     version of the paper on our own website(s).              - ongoing    <<<<

MJI  has not finished off the technical paper - ongoing                    <<<<

PSB  compared Calibration Plan with current expectation for calibration
     and found it needed updating (see later item)

EGS  made sure the CASU presentation from the ESO Public Surveys meeting in 
     September was available on the ESO website.                   

MJI  pre-empted expected criticism re: deblended Petrosian fluxes and 
     publicised the fix

STH  contacted Mark McCaughrean about HAWKI Eta Carina data status

MJI  forgot to contact Mark Casali about HawkI radial distortion - ongoing <<<<

RGM  liaise with JRL about processing HAWKI PI data - still ongoing        <<<<

JRL  contact Brad at JAC regarding the Nexp > 1 issue - happened and
     is now solved

STH  further investigate photometric features in available VISTA data 
     (throughput, colour equations, sky brightness) - deferred due to
     catalogue bug affecting VISTA catalogues as well, they need 
     regenerating prior to doing this.  Dropped from actions since part
     of longer term WP.

JRL  sorted out project names of VISTA MSBs taken by STH as well as they
STH  could

2.  Comments on WFAU minutes

CASU would like to thank Mike Read for spotting the faux pas with the
catalogues.  Next time we meet up the drinks are on us.

MJI noted the comments on the UKlight ups and downs and crushings and
pointed out the resubmission of the bid and the new contact names
involved.  He also noted the hardware server moves to the new room over
last w/e and wondered if the ensuing silence was good news or bad news.

RGM raised an issue about version control and the current difficulty of
knowing which files were processed with what version of the software.
There was a fair bit of discussion on the best way to improve things
since the current versioning relies on FITS header items from each
software component used in the processing and for bulk reprocessing
relies on a global version flag set in the datesubdir.  There are two
main problems with the current strategy: the CVS id as currently implemented
fails to pick up lower level changes in embedded subroutines resulting
in header processing id's that do not reflect ALL the changes; the lack
of a simple overall version flag in each processed product file i.e. 
without a bit of detective work if only a few files in a datesubdir have been
reprocessed its not completely straightforward to recover the event sequence.
JRL has been figuring out a way to get around the id problem and this 
should point to a solution for the version issue too.                      <<<<

3.  Recent and upcoming meetings

EGS reported on a recent EGAPS meeting at the IAC, Tenerife 22-23 Jan 2009. 
Every IPHAS field has been observed at least once now. Fields with very poor 
data quality will be included for re-observation in the hopefully final 
proposal to be submitted for period 09B. Regarding the data release the 
global calibration of this GP surveys is not going to happen in the near 
future due to delays in recruiting, so it was decided to extend the IDR with 
all the currently available data.  Better statistics regarding who is using 
it are required and this will be done via suitable tagging when the data is
downloaded.  The consortium also want to improve the visibility of IPHAS in 
the astronomical community by contacting Vizier and asking them to make the 
IDR and subsequent releases available from their web interface. There was
some discussion about the best way to proceed here given the IPHAS IDR is 
already published through the VO.  

From the UVEX perspective they also asked to have data access similar to the 
one provided by the IPHAS IDR and preferably with the IPHAS and UVEX data 
already cross-matched.  MJI and EGS had already discussed the best way to
do this and decided to do the cross-matching when the band-merged catalogues
are produced.

MJI noted that a UKIRT board meeting had taken place since our last get 
together and thanked the Board for pressing STFC to progress the then
stalled CASU and WFAU rolling grant issue.

MJI, STH and RGM attended the UKIDSS RAS meeting held just before Christmas.
All the presentations from the meeting are available at

PLATO - STH+NAW attended the last PLATO consortium meeting at ESTEC  
(Nov. 28th 2008), and will be attending the next Data Centre meeting  
to be held in Paris on Feb 12-13. The IoA were awarded some modest funds by 
STFC to enable us to continue investigating the potential of the mission  
and our role within it.

NAW pointed out that there is a Euro VO workshop March 30-April 2,2009 in
Garching (
Deadline for registration is February 15, 2009 if anyone is interested 
in going.

4.  Data archives update     - AAT, ING, WFCAM, UKIRT

An easy one, all archives are up to date, more or less.

MJI asked about recent backups for the raw data archives. AAT is ok, as are 
UKIRT, WFCAM and ING.  The latter two are received over the internet and
a hard copy is made to tape and stored in suitable chunks after they have
been ingested and verified.  We will adopt a similar strategy for VISTA.

MJI asked EGS to separately backup the ING 2008 data since although the
raw transferred files (*.gz) are backed up to tape monthly they are 
reformatted to Rice-compressed FITS and assorted illegal FITS headers 
fixed before archiving them on-line.                                       <<<<

The WFCAM'ers noted their surprise at getting an email from PSB on
January 30th, and subsequent days, but it turned out to be from his cron job 
running on apm6 collecting WFCAM raw data.  MR has already been in contact 
with JAC about taking over the responsibility for the transfer scripts and
while that is being set up in parallel we will continue with PSB prompting us.

5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, ESO WFI

Optical processing: EGS continues processing the ESIS data with the 
usual customers for the data (VISTA/VIDEO, Herschel/Hermes, Spitzer/Servs).

HAWKI: JRL is regenerating the catalogues in light of the catalogue bug
though we note that image nebulosity filtered versions of the catalogues 
(already shared with the Carina team) are free from de'bug since they were 
generated with the toolkit catalogue software. 

STH is investigating some possible issues with the spatial distribution for 
colour-selected sources raised by the Carina team. JRL will also reprocess 
the narrowband data to try and improve the sky subtraction; the broad-band <<<<
data is as good as its going to get.  Meanwhile MJI will try and remember 
what he was going to do next for this project.                         

All the new panoramic M31/M33 survey data from MegaCam for 08B arrived and
have been processed, stacked catalogued and glued together and the results 
are awesome.

The usual INT WFC camera processing for UVEX, IPHAS and the odd PI prog
has been done.

6.  WFCAM update             

WFCAM news: its back on and taking data.  Version 3 (sic!) of the flat 
screen will become available this semester and we have suggested taking
some linearity sequences to check how WFCAM fares.

Data transfers of 09A have commenced, the nexp > 1 scaling problem has
been fixed closer to source, the OMP DB is communicating and the first
few nights are sitting ready to go ...... well almost since we are still
eagerly awaiting some sky flatfield data so we can begin.

MJI moaned that he could find no trace of the updated compress routine in
the header, but it turned out he was looking in the wrong place and
it is there in the SHDUs - the PHU one being misleading.

As he was in rant mode he also whinged about the fact that ds9 still bombs
out when uncompressing the raw images even though there is nothing wrong
with them.  JRL opined it was to do with some ancient cfitsio version
being used at JAC.  He will check and help sort this out since it is a
an operational pain having to uncompress raw files first before being able 
to examine them in ds9.                                                    <<<<

Bug news: a rather subtle bug to the catalogue generator was introduced on 
15th April when some very minor fix to deal better with heavily saturated 
stars (sic!) was introduced.  Unfortunately due to its unpredictable (and
hard to spot) consequences this has required us to regenerate all the 
catalogues from 20080312 onward (i.e. the rest of 08A and all of 08B) and
has lead to some delay in the release of DR5.

It has also required recalibration of the photometry, detector zero-point
offsets and illumination corrections, though the only changes to the
images have been updated photometry coefficients in the headers (typically
only at the 1-2% level).  Generally all the smaller aperture fluxes were ok 
hence the overall photometry was only marginally off contributing to why this
wasn't picked up during our normal checking procedure.

The bug was found and fixed in short order and some good news in this 
saga is that the VISTA processing kit regenerated all the required 08A and 
08B catalogues far faster than we expected.  All the reprocessed data has
now been checked off and awaits retransfer to WFAU.

How to avoid this happening again (or at least minimise the chances) was
discussed in detail offline prior to the meeting, though at least one of the
issues raised there, better version control in the headers, is commented
on in 2. above.

As part of the bug fix we also implemented, as an extended test for the 08B 
recataloguing, the mod to improve the Petrosian, Kron and Hall (PKH) radii 
and fluxes for deblended objects.  The other piece of good news is that this 
mod can be back-propagated to all the earlier catalogues from 05A ->, i.e.
these parameters for deblended objects can be fixed in place without having 
to regenerate the catalogues.

Reprocessing news: the UDS K-band reprocessing from 05B was completed over 
the Christmas period and made available to the UDS folk for assessment on 
2nd Jan 2009 - is no news good news ?

The PKH flux/radii fixes for 05a-08a inclusive are ready to go as soon as
the recatalogued 08A data has been copied.  

The Orion nebula data for U/05a/100 (taken in 05B and 06B for Catarina Alves 
de Oliveira and Mark Casali) was reprocessed from scratch to improve the sky 
subtraction. This data also provided a good independent test of the nebulosity 
filter developed for GPS photometry in regions of high nebulosity.  We await
results with bated breath.

Ricky Smart contacted us at the UKIDSS meeting about the parallax programme.
Although satisifed with the results he was curious that his own detailed 
attempt at PSF-based astrometry actually gave worse results than the pipeline 
astrometry.  MJI suggested we try "dribbling" the 2x2 interleaved stacked
data to improve the PSF characterstics and provided him with the appropriately
dribbled frames.  Although the PSFs were now well-behaved (i.e. not "spiky")
RS's PSF fitting still gave results not as good as the pipeline astrometry 
(which just shows you how damn fine it is and how well the sneaky ML 
shortcut we implemented several semesters back works).

This still leaves us with the extant GPS reprocessing in nebulous regions to 
improve both detection and photometry, stalled due to the recataloguing of
08A and 08B noted earleir.

Other news: the photometric calibration paper has been accepted, posted on 
astro-ph (figure faux pas not-withstanding) and awaits proofing.

MR noted that one intrepid user (Fraser Clarke in Oxford) had asked for a copy
of the processed UKIDSS/B2 data he had taken during 08B since he wanted to
select by eye the best frames to stack and stack them himself. Aforesaid 
individual was presented with the data and hasn't been heard from since.

7.  VISTA update            

VISTA news: VISTA data from Paranal has been regularly arriving each week 
via Garching. Each external 250 Gbyte USB disk contains, oddly enough, a 
week's worth of Rice-compressed data.  After the usual initial delivery hiatus 
this mode of operation is now running smoothly and a dedicated ingest machine 
has been set up at this end to slurp the data off and plonk it on the VISTA 
raw data RAID6 array.  It slurps at up to ~30 Mbytes/s and only take a few 
hours to plonk.  There have been a few problems with missing files (TBF'd) and 
occassionally with missing FITS header keywords but in general data transfers
are working well (10 weeks worth of disks and no problems reading them and
we are on average, as expected, about 2 weeks behind the observations).

ESO took over operating the telescope and ingesting data into their Paranal
archive around the beginning of November.  Consequently MR, with input from
JRL and EGS, has set up a VISTA processing status web page in the same vein 
as the WFCAM web pages, which tracks data from the 1st November.  This page 
will shortly go live on the Plone site and will, a bit further down line, have 
the usual DQC information available to allow external users to see some of 
what we are up to and monitor assorted parameters and processing status.
(Earlier pre-ESO "commissioning" data is stored on-line but remains 
invisible to external users.)

Assorted hardware foibles have delayed the SV phase for VISTA and one of them,
the filter wheel positioning problem, has made generating useful science
output, or even good publicity pictures, rather difficult. This was
compounded by the previously reported cataloguing bug affecting all the 
VISTA science frames catalogues too.  JRL is busy regenerating them prior to 
any further analysis of the CEs, throughput and sky brightness levels etc....  
Although, as for WFCAM, the bug had only a marginal effect on any of these 
measures its a case of "if a job's worth doing it's worth doing properly".

VISTA tasks: Pete was leading the VISTA effort at CASU and although a large 
fraction of the Christmas period was spent going through and sorting out 
VISTA documents, software, hardware and other things, there are still a lot 
of open issues related to assorted VISTA tasks that Pete was responsible for. 
Despite recent progress on the grant issue and Sue Cowell's (SC) appointment 
here for VISTA and GAIA system support, we are still seriously short-staffed 
for VISTA work and have had to spread some of the load around the group to
get things moving.

Computer support, progress pages and data ingest tasks are already in hand. 
Other jobs that need some TLC are: the ETC software, associated data tables 
and data files, and queries relating to SADT/P2PP issues which STH will 
take over; RWA is dealing with the data ingest and shipments of disks
and will provide help with the documentation updates; JRL is the sole
CPL and main ESO software support and development bod; while for now
MJI will be the point of contact for science-related and general 
processing-related queries; JRL, STH and RWA will review the current status 
of the VISTA documentation set and ensure that all the latest versions are <<<<
indeed up to date and avaialble on the internal web pages (MJI has
hard copies of them purloined from PSB's collection). In addition to this 
we will all have to muck in with science and QC assement as and when it is 

VISTA data: STH commented that he had been asked about early access to 
processed VISTA test frames by various survey PIs and was tasked with telling
them to get clearance through JPE before proceeding further.               <<<<

This reminded MJI that nothing had been heard from the narrow-banders
since our last meeting and that he recalled also that Maryjn Franx had asked 
for, and received, some processed VISTA data taken in the proposed 
UltraVISTA survey region to help select the best pointing.

8.  AOB                  

MJI gave a brief update on the rolling grant saga.  After much discussion 
with STFC a revised JES form was agreed upon and submitted mid-January and 
as a consequence of that a job advert for two positions at CASU was 
announced a week later with closing date 16th February.

The meeting closed with a brief discussion of the new computer support

Continuing actions

STH  check that Plone technical description pages are up to date 

JRL  produce web page of HawkI publicity for CASU

MJI  finish off stage-I software release pages

STH  make photometric calibration paper available on Plone and VDFS sites

MJI  finish off technical paper

MJI  contact Mark Casali about HawkI radial distortion

RGM  liaise with JRL about processing HAWKI PI data

New actions

JRL  produce a definitive solution to the pipeline versioning issues and
     implement it for any new processing

EGS  backup online 2008 ING archive data

JRL  redo the HAWKI NB image data processing to improve sky subtraction

JRL  contact JAC about possible old cfitsio version in conversion software
     causing ds9 reading problems for raw data

JRL  sort out VISTA docuementation set and make sure latest copies are
STH  available on internal web pages

STH  contact VISTA PSPIs re: need to get permission through JPE for
     access to reduced VISTA test frames