—
Plain Text,
15 KB (15416 bytes)
File contents
2nd August 2006 Hoyle committee room 11.30 - 13:45
Present: DWE, MJI, JRL, PSB, MR, STH, EGS, NAW
Apologies: WJS, RGM
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. Report from VDMT
4. Report from UKIDSS meeting
5. Data archives update
6. Optical processing update
7. WFCAM news
8. VISTA news
9. AOB
Minutes
-------
1. Actions from last meeting
JRL has not implemented the fix to the problem with the UKIRT archive
(missing header information problem) - ongoing <<<<
[JRL noted that the Debianizing of apm12 required a workaround
for downloading data from the UKIRT archive.]
EGS updated the INT WFC web pages with the latest flat-field and fringe
frames
STH has not updated the Calibration Plan document. He said that he needed
to discuss this with PSB and MJI. His main concern was with the Z and Y
calibration. MJI felt that there was no practical alternative to the
currently proposed pipeline solutions. The draft of the updated
document would be needed by the first week of September. STH, PSB and
MJI will have a meeting to plan what is needed and then update the
document <<<<
DWE have not finalized the content of the Plone site. DWE said that not
EGS much progress has happened since the last meeting due to concentrating
STH on the PSF work. Small amounts of content have been added - even a page
MR from PSB on networking issues. The technical pages are about half
completed. MJI noted that the Plone site needs to be up and running
by the time of submitting the UK review documents - ongoing <<<<
DWE have set up a local ADASS 2007 web site using Plone
EGS
JRL contacted WFAU to clarify the problems with the catalogue generating
software (see later).
MJI compared old and new versions of the catalogue software and
MR also compared toolkit development version with pipeline version.
They found a few gotchas in the pipeline version and suggested
fixes (see below)
MJI has not emailed JAC to see if it would be possible to exclude
engineering data from the data sent to Cambridge - ongoing <<<<
JRL investigated the MSB ID checksums with a view for tagging bad MSBs.
MJI This complicated system was found to be not suitable. The ball is now
back in JAC's court.
JRL has fixed the pipeline problems (see above)
MJI raised the issue of VDFS document status at the VDMT meeting. JPE said
that he was waiting for the final installment of the SADT document to
be completed and then all FDR documents could be released at the same
time.
JRL arranged Debianizing apm12 with JMI who did
ALL sent in quarterly reports to STH
PSB sent in AIT and DRL reports to STH
JRL
PSB ordered the local link from the Computer Service for UKLight connection
STH circulated the SJW email regarding "deprecations"
2. Comments on WFAU minutes
PSB noted that AstroGrid had provided money for WFAU for hosting SDSS DR3.
PSB also noted that UKLight work was progressing. MJI said that he had asked
for the transfer of 06A to go on hold until the UKLight connection was up
and running, providing it does not take too long. 05A version 3 was
currently being transferred, and used for various timing tests.
STH said it was to be welcomed that there would be access to the flat files
from WSA. Access to these will be available in about a week to the whole ESO
community. He also noted the agreement to release the observations of
standard stars through the database.
STH noted that WFAU are working on a scheme to allow efficient photometric
recalibration updates from CASU-held catalogues. MJI pointed out that
when the dust has settled on the Y,Z calibration we will re-calibrate all
of 05A, 05B and 06A. His hope is that WFAU will then just transfer the
updated header information and apply it to their own catalogue and image
versions.
3. Report from VDMT
MJI and STH attended the VDMT meeting in Edinburgh on 24 July. MJI reported
on the main points:
The UK review of VDFS will be held in Cambridge on 24/25 October. The
documentation needed for this will have to be ready a month before that and
this implies that drafts need to be ready a month before this ie. start of
September. PSB reminded us that the UK URD had still not been officially
released. MJI added that now we know what the main public surveys on
VISTA are likely to be we need feedback from the survey PIs on the
requirements embedded in the UK URD.
An email from SJW to JPE had raised issues concerning sky subtraction
problems. Some of these were caused by observing too close to the moon
(sic!) but there were still a subset of science frames where the sky
subtraction was not optimal. The latest, and hopefully final, version of
the sky estimation alogrithm appears to have addressed the main issues and
was run on 05A version 3 and is being used in the current 06A processing.
While sympathising with the problems caused by poor sky subtraction, MJI
pointed out there is a practical limit to how well an automatic pipeline
can be expected to do and that it will never be perfect.
A further issue raised was the response of CASU to external email queries.
This was discussed at a previous meeting (Item 7 060612). Two possibilities
were discussed: email MJI and he either deals with them directly, or
allocates someone to deal with the task; or we set up a dedicated bug
tracking system. NAW suggested OMII-UK and DWE suggested the Mantis Bug
Tracker (as used by Gaia). There was also the IoA's helpdesk system written
by JRL. STH said that the acutual software was not important, but rather
the lines of responsibility and keeping track of the progress.
In the end it was decided that since the numbers of these emails is quite
low we would start with a much simpler system and revisit this issue in
a few months. PSB will set up an email account (casuhelp) which will <<<<
start life as an alias for MJI's account. MJI will then allocate someone
to deal with the task. MJI asked that any responses to these should be
CC'd to casuhelp where they will be saved for future reference. DWE will
make sure that this system is clearly visible on the web (and Plone) pages.<<<<
The concerns over staffing VST processing were discussed and Tom Shanks
joined in via telcon to discuss the shortly to be resubmitted Survey
Management Plan for ATLAS.
It was noted with some concern that the money for VDFS development runs out
in September 2007, well before the pipeline and archive will have had a
good shakedown with VISTA survey data. Behind the scenes lobbying is
in place to find a solution to this and to ensure a smooth transition to
VISTA operations.
4. Report from UKIDSS meeting
MJI and STH attended the UKIDSS meeting in Edinburgh on 25 July and gave
presentations. These are available from our web site. MJI said that MR
had updated the processing status pages to reflect the current state of the
pipeline. The main moans came from the UDS folk, with some justification.
The recent improvements to the catalogue generating software should go some
way to alleviate this problem but this still leaves open the question of
how to best form the deep stacks. Sebastien Foucaud and Omar Almaini said
they had spent the best part of 2 months fiddling with sky background and
excising dubious chunks of data before producing the final deep stack.
STH reported that in SJW's summary of survey progress (see presentation on
http://wiki.astrogrid.org/bin/view/UKIDSS/Jul06Meet) and noted that DXS
was rejecting 38% of the data for deep stacks due to having rather generous
sky and seeing constraints.
There has been a call for proposals for UKIRT beyond the end of the current
UKIDSS 2 year plan in response to the Ellis review. This includes new
surveys and even new instruments (sic!). The one new survey all present
agreed on was the idea of a Northern hemisphere survey.
5. Data archives update
AAT: No news is good news
ING: JRL and EGS reported that the last 10 years of data (1996-2006) had
now been ingested into Sybase. The frontend of the new archive interface
was now being written and should be at the beta-test stage by the end of
the month. EGS said that duplicate data was now sorted and could be removed
(which MJI duly did having first backed it up).
WFCAM: MR reported that tapes up to 13 July have been received, MEF'd and
ingested into the raw archive. ESO transfers have been completed up to the
end of June. It was noted that WFCAM had been taken off the telescope on
31 July. The frequency of occurrence of nulled detectors remains at the
same low level and appears to be mainly at the start of nights.
UKIRT: No further news
6. Optical processing update
MJI said that he had been processing 12 nights of data from the INT for the
UVEX project which is a precursor to VPHAS (VST). This amounts to about
1/2 TByte of u, g, r, i, He and Halpha data (ie. the equivalent of a few
pro-rata months of VPHAS+ data. According to ESO's survey-to-FTE
conversion he can now take the rest of the year off.
STH was very excited about an (astronomical) image he had seen recently
displayed as a poster. This was a tiled image of a supernova remnant taken
within the IPHAS project on the INT. He thought that we should have a copy
of this poster, since it was very striking. He will look into this. <<<<
7. WFCAM news
Catalogue software: MR and EGS have been investigating the problems reported
by UDS with the cataloguing software in a recent paper. The UDS EDR stack
was acquired and the latest version of the CASU software was compared with
the results from Sextractor. The end result is that the two catalogues now
agree on detection levels and star/galaxy separation and the resulting
colour-magnitude diagrams are similar.
MJI said that the three main improvements to the catalogue software were:
- being able to cope correctly with zero or negative sky
- removing the quantization in the noise estimation
- improvements to the matched filter for image detection.
A consequence of the latter is that another input parameter is needed. Both
the pipeline and toolkit versions of the programme now produce results
that agree with each other.
STH asked if we needed to make a response to the UDS EDR paper (currently on
astro-ph). EGS said that there was also another paper making similar comments.
It was decided that MJI and MR will deal with this issue making sure that
any email is CC'd to SJW and AL. <<<<
Non-linearity: STH wondered if stacking could be linked with the magnitude
scale difference seen in the 2MASS comparison. STH said that he would
follow this up. <<<<
Progress with 06A: Processing of the 06A data is complete to the end of June.
PSF fitting: DWE reported that the bug in the split PSF fitting routine had
been found and that the tests with simulated data were successful. Tests
with real data were in progress.
Offsets: MJI raised the issue of inconsistent offsets within dither stacks
and asked if a sort of "majority vote" could be applied to produce consistent
offsets for all detectors.
8. VISTA news
VIRCAM Status: No news
DRL status: PSB said that the main work that had been carried out over the
last few weeks was sorting out the interleaving part of the main science
recipe. He had also been in contact with Sandra Castro regarding
benchmarking tests to enable ESO to specify the hardware requirements for
the summit pipeline. The cataloguing module has been updated in line with
the changes to the WFCAM pipeline. The next task is to deal with dead chips.
VDFS docs status: PSB said that we are still waiting for the overhead
algorithm for the new ETC. His new document describing the data that has
been used for the ETC has been sent to the VISTA Project Office to see if
they have any objections.
There was some discussion regarding the file size of a single tile
(0.75 GBytes) and the fact that an interleaved tile would thus be bigger
than 2 GBytes and thus cause problems for some file systems.
Upcoming UK review: For the UK Review (Cambridge 24/25 October) various
documents and access to our plone web site will be needed. There was some
discussion as to what documentation was needed. PSB, STH, JRL and MJI
will have an offline meeting to plan this. <<<<
It was noted that JPE will be visiting Cambridge next week to discuss the
specification of the VISTA Z filter with Paul Hewett. It was suggested
that STH be present. <<<<
9. AOB
Hardware update: apm12 has been Debianized and its existing RAID5 system
converted to RAID0 since this is effectively used as processing scratch
space. The advantage is that the I/O performance is improved by a
factor of 2. If this makes a significant difference to overall processing
speed, then the other RAID arrays used as temporary data buffers by the
processing PCs will also be swapped over to RAID0.
MJI noted that another 10 TBytes of fibre channel RAID6 longer-term
storage had been ordered.
UKLight saga: PSB reported on the Edinburgh end plans for the UKlight link.
Further transfer tests have been carried out, the most notable of
which were single-threaded transfers which averaged 16 MBytes/s. Further
tests are being carried out and a summary will be collated.
Gaia: MJI said that there will be a Gaia meeting of CU5 (Photometric
Processing) in Cambridge on 25th and 26th of September. More details can be
found on the relevant GaiaWiki page:
http://www.rssd.esa.int/SA-general/Projects/GAIA/wiki/index.php? \
title=CU5:_Photometric_Processing:_Meetings:_CU5M02
Note that you will need a Gaia username and password to access this.
DWE, MJI, PSB and NAW already have these - any problems get in touch with
DWE. In the meantime, DWE will forward the email announcement to those who
were not on the original distribution. <<<<
ADASS 2006: EGS pointed out that that the deadline for submission of
abstracts to ADASS 2006 was August 11 - so get writing <<<<
Continuing Actions
------------------
JRL finish the fix to problem with UKIRT archive (missing header information)
STH update the Calibration Plan document (VIS-SPE-IOA-20000-0002). Have
PSB meeting soon to plan the document. Draft by start September.
MJI
DWE finalize content of Plone site by end of September
EGS
STH
MR
MJI email JAC to see if it would be possible to exclude engineering data
from the data sent to Cambridge
New Actions
-----------
PSB set up casuhelp email account
DWE ensure visibility of above email address on web/Plone pages
STH acquire tiled supernova remnant image
MJI draw up a response to UDS EDR paper
MR
STH investigate effects of stacking on 2MASS/WFCAM non-linearity comparison
PSB organize meeting soon to decide on what documents are necessary for UK
STH review
JRL
MJI
STH attend meeting regarding Z filter (with JPE and PCH)
DWE forward Gaia meeting email
ALL make sure ADASS 2006 abstracts are submitted
