—
Plain Text,
18 KB (18556 bytes)
File contents
4th February 2009 APM meeting tables 11:30-13:00
Present: MJI, JRL, STH, MR, EGS, RGM, RWA, NAW, SC
Apologies: DWE
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent and upcoming meetings
4. Data archives update - AAT, ING, WFCAM, UKIRT
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru
6. WFCAM update
7. VISTA update
8. AOB
Minutes
-------
1. Actions from last meeting
STH the technical pages are still not up to date and other inconsistencies
have been pointed out in terms of current WFCAM processing practice
jarring with existing information, MJI will help with the updates <<<<
JRL has not produced a web page of HawkI publicity for CASU, deferred
due to other factors but still - ongoing <<<<
MJI have not finished off the stage-I software release pages - ongoing <<<<
EGS
STH has put the WFCAM calibration paper on astro-ph, but RGM pointed out
that the astro-ph version of the photometry paper has a couple of
messed up figures. STH will fix this and put a high resolution
version of the paper on our own website(s). - ongoing <<<<
MJI has not finished off the technical paper - ongoing <<<<
PSB compared Calibration Plan with current expectation for calibration
and found it needed updating (see later item)
EGS made sure the CASU presentation from the ESO Public Surveys meeting in
September was available on the ESO website.
MJI pre-empted expected criticism re: deblended Petrosian fluxes and
publicised the fix
STH contacted Mark McCaughrean about HAWKI Eta Carina data status
MJI forgot to contact Mark Casali about HawkI radial distortion - ongoing <<<<
RGM liaise with JRL about processing HAWKI PI data - still ongoing <<<<
JRL contact Brad at JAC regarding the Nexp > 1 issue - happened and
is now solved
STH further investigate photometric features in available VISTA data
(throughput, colour equations, sky brightness) - deferred due to
catalogue bug affecting VISTA catalogues as well, they need
regenerating prior to doing this. Dropped from actions since part
of longer term WP.
JRL sorted out project names of VISTA MSBs taken by STH as well as they
STH could
2. Comments on WFAU minutes
CASU would like to thank Mike Read for spotting the faux pas with the
catalogues. Next time we meet up the drinks are on us.
MJI noted the comments on the UKlight ups and downs and crushings and
pointed out the resubmission of the bid and the new contact names
involved. He also noted the hardware server moves to the new room over
last w/e and wondered if the ensuing silence was good news or bad news.
RGM raised an issue about version control and the current difficulty of
knowing which files were processed with what version of the software.
There was a fair bit of discussion on the best way to improve things
since the current versioning relies on FITS header items from each
software component used in the processing and for bulk reprocessing
relies on a global version flag set in the datesubdir. There are two
main problems with the current strategy: the CVS id as currently implemented
fails to pick up lower level changes in embedded subroutines resulting
in header processing id's that do not reflect ALL the changes; the lack
of a simple overall version flag in each processed product file i.e.
without a bit of detective work if only a few files in a datesubdir have been
reprocessed its not completely straightforward to recover the event sequence.
JRL has been figuring out a way to get around the id problem and this
should point to a solution for the version issue too. <<<<
3. Recent and upcoming meetings
EGS reported on a recent EGAPS meeting at the IAC, Tenerife 22-23 Jan 2009.
Every IPHAS field has been observed at least once now. Fields with very poor
data quality will be included for re-observation in the hopefully final
proposal to be submitted for period 09B. Regarding the data release the
global calibration of this GP surveys is not going to happen in the near
future due to delays in recruiting, so it was decided to extend the IDR with
all the currently available data. Better statistics regarding who is using
it are required and this will be done via suitable tagging when the data is
downloaded. The consortium also want to improve the visibility of IPHAS in
the astronomical community by contacting Vizier and asking them to make the
IDR and subsequent releases available from their web interface. There was
some discussion about the best way to proceed here given the IPHAS IDR is
already published through the VO.
From the UVEX perspective they also asked to have data access similar to the
one provided by the IPHAS IDR and preferably with the IPHAS and UVEX data
already cross-matched. MJI and EGS had already discussed the best way to
do this and decided to do the cross-matching when the band-merged catalogues
are produced.
MJI noted that a UKIRT board meeting had taken place since our last get
together and thanked the Board for pressing STFC to progress the then
stalled CASU and WFAU rolling grant issue.
MJI, STH and RGM attended the UKIDSS RAS meeting held just before Christmas.
All the presentations from the meeting are available at
http://www.ukidss.org/rasworkshop08/talks.
PLATO - STH+NAW attended the last PLATO consortium meeting at ESTEC
(Nov. 28th 2008), and will be attending the next Data Centre meeting
to be held in Paris on Feb 12-13. The IoA were awarded some modest funds by
STFC to enable us to continue investigating the potential of the mission
and our role within it.
NAW pointed out that there is a Euro VO workshop March 30-April 2,2009 in
Garching (http://www.euro-vo.org/aidahandson2009/registration.html)
Deadline for registration is February 15, 2009 if anyone is interested
in going.
4. Data archives update - AAT, ING, WFCAM, UKIRT
An easy one, all archives are up to date, more or less.
MJI asked about recent backups for the raw data archives. AAT is ok, as are
UKIRT, WFCAM and ING. The latter two are received over the internet and
a hard copy is made to tape and stored in suitable chunks after they have
been ingested and verified. We will adopt a similar strategy for VISTA.
MJI asked EGS to separately backup the ING 2008 data since although the
raw transferred files (*.gz) are backed up to tape monthly they are
reformatted to Rice-compressed FITS and assorted illegal FITS headers
fixed before archiving them on-line. <<<<
The WFCAM'ers noted their surprise at getting an email from PSB on
January 30th, and subsequent days, but it turned out to be from his cron job
running on apm6 collecting WFCAM raw data. MR has already been in contact
with JAC about taking over the responsibility for the transfer scripts and
while that is being set up in parallel we will continue with PSB prompting us.
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, ESO WFI
Optical processing: EGS continues processing the ESIS data with the
usual customers for the data (VISTA/VIDEO, Herschel/Hermes, Spitzer/Servs).
HAWKI: JRL is regenerating the catalogues in light of the catalogue bug
though we note that image nebulosity filtered versions of the catalogues
(already shared with the Carina team) are free from de'bug since they were
generated with the toolkit catalogue software.
STH is investigating some possible issues with the spatial distribution for
colour-selected sources raised by the Carina team. JRL will also reprocess
the narrowband data to try and improve the sky subtraction; the broad-band <<<<
data is as good as its going to get. Meanwhile MJI will try and remember
what he was going to do next for this project.
All the new panoramic M31/M33 survey data from MegaCam for 08B arrived and
have been processed, stacked catalogued and glued together and the results
are awesome.
The usual INT WFC camera processing for UVEX, IPHAS and the odd PI prog
has been done.
6. WFCAM update
WFCAM news: its back on and taking data. Version 3 (sic!) of the flat
screen will become available this semester and we have suggested taking
some linearity sequences to check how WFCAM fares.
Data transfers of 09A have commenced, the nexp > 1 scaling problem has
been fixed closer to source, the OMP DB is communicating and the first
few nights are sitting ready to go ...... well almost since we are still
eagerly awaiting some sky flatfield data so we can begin.
MJI moaned that he could find no trace of the updated compress routine in
the header, but it turned out he was looking in the wrong place and
it is there in the SHDUs - the PHU one being misleading.
As he was in rant mode he also whinged about the fact that ds9 still bombs
out when uncompressing the raw images even though there is nothing wrong
with them. JRL opined it was to do with some ancient cfitsio version
being used at JAC. He will check and help sort this out since it is a
an operational pain having to uncompress raw files first before being able
to examine them in ds9. <<<<
Bug news: a rather subtle bug to the catalogue generator was introduced on
15th April when some very minor fix to deal better with heavily saturated
stars (sic!) was introduced. Unfortunately due to its unpredictable (and
hard to spot) consequences this has required us to regenerate all the
catalogues from 20080312 onward (i.e. the rest of 08A and all of 08B) and
has lead to some delay in the release of DR5.
It has also required recalibration of the photometry, detector zero-point
offsets and illumination corrections, though the only changes to the
images have been updated photometry coefficients in the headers (typically
only at the 1-2% level). Generally all the smaller aperture fluxes were ok
hence the overall photometry was only marginally off contributing to why this
wasn't picked up during our normal checking procedure.
The bug was found and fixed in short order and some good news in this
saga is that the VISTA processing kit regenerated all the required 08A and
08B catalogues far faster than we expected. All the reprocessed data has
now been checked off and awaits retransfer to WFAU.
How to avoid this happening again (or at least minimise the chances) was
discussed in detail offline prior to the meeting, though at least one of the
issues raised there, better version control in the headers, is commented
on in 2. above.
As part of the bug fix we also implemented, as an extended test for the 08B
recataloguing, the mod to improve the Petrosian, Kron and Hall (PKH) radii
and fluxes for deblended objects. The other piece of good news is that this
mod can be back-propagated to all the earlier catalogues from 05A ->, i.e.
these parameters for deblended objects can be fixed in place without having
to regenerate the catalogues.
Reprocessing news: the UDS K-band reprocessing from 05B was completed over
the Christmas period and made available to the UDS folk for assessment on
2nd Jan 2009 - is no news good news ?
The PKH flux/radii fixes for 05a-08a inclusive are ready to go as soon as
the recatalogued 08A data has been copied.
The Orion nebula data for U/05a/100 (taken in 05B and 06B for Catarina Alves
de Oliveira and Mark Casali) was reprocessed from scratch to improve the sky
subtraction. This data also provided a good independent test of the nebulosity
filter developed for GPS photometry in regions of high nebulosity. We await
results with bated breath.
Ricky Smart contacted us at the UKIDSS meeting about the parallax programme.
Although satisifed with the results he was curious that his own detailed
attempt at PSF-based astrometry actually gave worse results than the pipeline
astrometry. MJI suggested we try "dribbling" the 2x2 interleaved stacked
data to improve the PSF characterstics and provided him with the appropriately
dribbled frames. Although the PSFs were now well-behaved (i.e. not "spiky")
RS's PSF fitting still gave results not as good as the pipeline astrometry
(which just shows you how damn fine it is and how well the sneaky ML
shortcut we implemented several semesters back works).
This still leaves us with the extant GPS reprocessing in nebulous regions to
improve both detection and photometry, stalled due to the recataloguing of
08A and 08B noted earleir.
Other news: the photometric calibration paper has been accepted, posted on
astro-ph (figure faux pas not-withstanding) and awaits proofing.
MR noted that one intrepid user (Fraser Clarke in Oxford) had asked for a copy
of the processed UKIDSS/B2 data he had taken during 08B since he wanted to
select by eye the best frames to stack and stack them himself. Aforesaid
individual was presented with the data and hasn't been heard from since.
7. VISTA update
VISTA news: VISTA data from Paranal has been regularly arriving each week
via Garching. Each external 250 Gbyte USB disk contains, oddly enough, a
week's worth of Rice-compressed data. After the usual initial delivery hiatus
this mode of operation is now running smoothly and a dedicated ingest machine
has been set up at this end to slurp the data off and plonk it on the VISTA
raw data RAID6 array. It slurps at up to ~30 Mbytes/s and only take a few
hours to plonk. There have been a few problems with missing files (TBF'd) and
occassionally with missing FITS header keywords but in general data transfers
are working well (10 weeks worth of disks and no problems reading them and
we are on average, as expected, about 2 weeks behind the observations).
ESO took over operating the telescope and ingesting data into their Paranal
archive around the beginning of November. Consequently MR, with input from
JRL and EGS, has set up a VISTA processing status web page in the same vein
as the WFCAM web pages, which tracks data from the 1st November. This page
will shortly go live on the Plone site and will, a bit further down line, have
the usual DQC information available to allow external users to see some of
what we are up to and monitor assorted parameters and processing status.
(Earlier pre-ESO "commissioning" data is stored on-line but remains
invisible to external users.)
Assorted hardware foibles have delayed the SV phase for VISTA and one of them,
the filter wheel positioning problem, has made generating useful science
output, or even good publicity pictures, rather difficult. This was
compounded by the previously reported cataloguing bug affecting all the
VISTA science frames catalogues too. JRL is busy regenerating them prior to
any further analysis of the CEs, throughput and sky brightness levels etc....
Although, as for WFCAM, the bug had only a marginal effect on any of these
measures its a case of "if a job's worth doing it's worth doing properly".
VISTA tasks: Pete was leading the VISTA effort at CASU and although a large
fraction of the Christmas period was spent going through and sorting out
VISTA documents, software, hardware and other things, there are still a lot
of open issues related to assorted VISTA tasks that Pete was responsible for.
Despite recent progress on the grant issue and Sue Cowell's (SC) appointment
here for VISTA and GAIA system support, we are still seriously short-staffed
for VISTA work and have had to spread some of the load around the group to
get things moving.
Computer support, progress pages and data ingest tasks are already in hand.
Other jobs that need some TLC are: the ETC software, associated data tables
and data files, and queries relating to SADT/P2PP issues which STH will
take over; RWA is dealing with the data ingest and shipments of disks
and will provide help with the documentation updates; JRL is the sole
CPL and main ESO software support and development bod; while for now
MJI will be the point of contact for science-related and general
processing-related queries; JRL, STH and RWA will review the current status
of the VISTA documentation set and ensure that all the latest versions are <<<<
indeed up to date and avaialble on the internal web pages (MJI has
hard copies of them purloined from PSB's collection). In addition to this
we will all have to muck in with science and QC assement as and when it is
necessary.
VISTA data: STH commented that he had been asked about early access to
processed VISTA test frames by various survey PIs and was tasked with telling
them to get clearance through JPE before proceeding further. <<<<
This reminded MJI that nothing had been heard from the narrow-banders
since our last meeting and that he recalled also that Maryjn Franx had asked
for, and received, some processed VISTA data taken in the proposed
UltraVISTA survey region to help select the best pointing.
8. AOB
MJI gave a brief update on the rolling grant saga. After much discussion
with STFC a revised JES form was agreed upon and submitted mid-January and
as a consequence of that a job advert for two positions at CASU was
announced a week later with closing date 16th February.
The meeting closed with a brief discussion of the new computer support
arrangements.
Continuing actions
------------------
STH check that Plone technical description pages are up to date
MJI
JRL produce web page of HawkI publicity for CASU
MJI finish off stage-I software release pages
EGS
STH make photometric calibration paper available on Plone and VDFS sites
MJI finish off technical paper
MJI contact Mark Casali about HawkI radial distortion
RGM liaise with JRL about processing HAWKI PI data
New actions
-----------
JRL produce a definitive solution to the pipeline versioning issues and
implement it for any new processing
EGS backup online 2008 ING archive data
JRL redo the HAWKI NB image data processing to improve sky subtraction
JRL contact JAC about possible old cfitsio version in conversion software
causing ds9 reading problems for raw data
JRL sort out VISTA docuementation set and make sure latest copies are
STH available on internal web pages
RWA
STH contact VISTA PSPIs re: need to get permission through JPE for
access to reduced VISTA test frames
