—
Plain Text,
13 KB (14074 bytes)
File contents
6th April 2006 Hoyle committee room 10.30 - 12:30
Present: DWE, MJI, PSB, STH, MR, RGM, WJS
Apologies: NAW, JRL, EGS
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. VDMT Q1 reporting and WPs
4. Data archives update
5. Optical processing
6. WFCAM processing
7. VDFS VISTA update
8. AstroGrid liaison update
9. AOB
Minutes
-------
1. Actions from last meeting
JRL has sorted out the access problems with the OMP database mirror, however
there is still an outstanding issue regarding the MSB quality flag and
propogataing this information to the run number level. JRL is thinking
about how to implement it since it apparently non-trivial - ongoing <<<<
MJI has made no further progress investigating the merging of
classification statistics - ongoing <<<<
JRL has finished modifying the AAT archive interface including the
appropriate 2 year proprietary period
MJI has not contacted JPE regarding the CASU preferred method of data
transfer from ESO, but will raise it at the VDMT meeting - ongoing <<<<
MJI circulated the ASCII version of the 06Q1 work packages
ALL provided comments regarding the 06Q1 work packages (see item 3)
WJS has not been in touch with JPE regarding acquiring the numeric (rather
than graphic) versions of the data tables required by the ETC. PSB
listed what data was missing. WJS said that some of this may be in the
hands of commercial companies and it may not be possible to obtain, but
that he would see if he could get hold of the data. If this is not
possible then the best we can do is deliver digitized versions of
the graphics - ongoing <<<<
It was noted that the reason that the Z filter response was
not available was that the Z filter hasn't been ordered yet!
RGM is working on quantifying the foul-up causes in LAS outliers list.
This will be finished by the end of April - ongoing <<<<
STH has not found out what happened at the AstroGrid Implementation
meeting. RGM suggested that he email to find out what was happening
regarding the minutes. DWE suggested that he talk to FvL - ongoing <<<<
PSB has set up MRTG on the network switch to monitor network traffic
ALL not all reports were in yet covering Q1-06. This is needed by the
end of the week to provide a report in time for the VDMT meeting
on Tuesday (11 April) - ongoing <<<<
JRL have had a meeting with RWA and made progress with the ING archive
EGS issues
DWE has got the persistence report ready for release
http://www.ast.cam.ac.uk/~wfcam/docs/reports/persistence/
This is also announced in the dairy.
DWE has written a web page containing a few examples of persistence. This
is linked at the bottom of the persistence report
http://www.ast.cam.ac.uk/~wfcam/docs/reports/persistence/ \
persistence_examples.html
It should also be added to the known features Plone page.
RGM suggested that a link to the relevant 2MASS page should be added
to the report <<<<
JRL and MJI have investigated the deblending problems identified from the
LAS outliers list. The development software had no problem with this
data, but sadly the production pipeline version had a low-level bug
that caused very occasional problems. This bug was fixed prior to
commencing 05A reprocessing. Although multiple comparisons are
routinely carried out between the development toolkit software and
the pipeline version, this bug had slipped through due to its unusual
nature. Weird science outliers are, of course, a good way of finding
pathological problems.
MJI emailed Nigel's list of bad sky frames to JRL, again!
JRL implemented the bad frame list in the sky determination procedure
JRL fixes have been carried out for most of the reported header problems
MJI for the raw 05A data. A small number linked to the jitter sequence
keywords still need fixing - ongoing <<<<
MR checked the reprocessed data from 8/9 April and identified a few
MJI issues. The reprocessed data is definitely better but more checks
are needed - ongoing <<<<
DWE wrote a script to check for newly added files to the internal file
system. New files are checked to see they are in a sensible location
and then copied over to the Plone system.
DWE reorganized the internal file system on the same lines as the Plone
internal system
EGS is not giving a talk at NAM (nor presenting a poster)
MJI distributed the details of the Croatia workshop. DWE said that he had
signed up for it.
2. Comments on WFAU minutes
Surprise was expressed that WFAU appear to be going ahead with the UKLight
option already. This has resource implications, hardware and recurrent
costs, and staff effort for CASU. We felt that it was still not clear
that this will solve the bandwidth limitation problem. At this stage
we still need to ascertain where the bottleneck in network transfer speed
is and explore other aspects (eg. cost) of the UKLight option. This issue
will be raised at the VDMT meeting.
3. VDMT Q1 reporting and WPs
MJI said that he has received comments regarding the Q1/Q2 plan. He will
send out the updated plan to assist with the writing of the monthly
reports. <<<<
Summaries for the VDMT are being written eg PSB on the ETC and network issues.
These all have to be finished by the end of the week.
STH and MJI discussed the predicted VISTA transfer rate and came to an
agreement based on the expected mean daily data rates for VISTA.
MJI is considering the possibility of a script to push the processed WFCAM
data to WFAU rather than the current system. The data server having control
of the process is proving extremely effective and trouble-free for transfers
to the ESO data archive and from transfers from ING to the archive in
Cambridge. Both are much lower end-to-end bandwidth network systems, yet have
no problem maintaining an acceptable data transfer rate using simple transfer
protocols. This possibility will be raised at the VDMT meeting.
RGM raised the issue of public release of VDMT products. He was of the
opinion that the pipeline products (images and catalogue products) should
be released with next to no delay once they had been pipeline processed
and released by CASU to the WSA.
He recalled a requirement from JAC for this. This will also be raised at
the VDMT.
MJI summarised the issues that will be raised on Tuesday and asked if there
were any others. The alternative options for dealing with persistence were
discussed and it was felt that this also should be an item for discussion
at the VDMT.
All agreed that Tuesday would be a fun meeting. RGM will decide if he
is going to participate via telecon for some of the items. <<<<
4. Data archives update
AAT: has been upgraded (see earlier)
ING: the interface is being upgraded (see earlier)
WFCAM: no new data
UKIRT: Paul Hirst has reported that some files have got missing header
information JRL will look at this when he returns <<<<
5. Optical processing
VST survey management plans:
MJI reported that the VPHAS+ VST survey management plan has been submitted
and that he forwarded a copy of this to Tom Shanks to help him draft the
equivalent plan for the VST ATLAS survey.
It was noted that the committee for deciding the VISTA Public Surveys will
meet on 19 June after the initial pruning has been carried out in May. The
number of submitted surveys was 15 - the Dunlop UDS survey was the one
missing from our list. MJI said he would get a copy. <<<<
6. WFCAM processing
MR is currently leading the reprocessing 05A. The processed data will be
dumped in the same place as the earlier version due to pressure on disk
space. The previous version catalogues have been saved for direct science
comparison. The new sky processing strategy creates a lot (~500/night) of
intermediate step sky frames (xskys). Although these are currently being
saved for diagnostics purposes they should not be required for long term
archiving. MJI will double check this at the VDMT meeting.
STH reported on the ZY photometric calibrations. Comparisons have been
carried out with respect to Sloan for one globular cluster field (M3) and
a much larger scatter than expected has been found, although the zero-point
was ok. A WFCAM (night-to-night) comparison for M3 also showed a large
scatter. MJI wondered if this was simply due to crowding and suggested
inter-comparing other WFCAM Z-band data. This investigation is ongoing and
a summary report will be given to the photometry working group by the end of
the week.
The persistence problem was discussed. MJI said that we had a number of
options: ignore it; correct each frame (probably not viable or worth doing);
or flag suspect catalogue entries using 2MASS as the basis for the predictor.
The latter is the favoured option (if persistence is shown to cause science
problems) but would involve a fair bit of coding. The main advantage is that
it can be added on post-processing as a flag bit in the catalogues. However,
rather than rushing into this it was felt that this should be science
requirement-driven ie. has persistence been identified as a problem in
doing the science?
MJI noted that the main wfcam data discs were full. By the time we start
processing 06A we will need more disc storage. New options are being
considered - either 7TB or 11TB RAID6 (2 parity discs) systems that hang off
our existing disc servers. This will cost about 1k per TB (excl. VAT) and
about 50TB will be needed over the next year (not just for WFCAM).
STH mentioned a request from Sandy Leggett regarding linearity. JRL had
analysed series of dome flats to characterize this. MJI said that he
would contact JRL and compile a response. <<<<
RGM raised the issue of stacking problems. MJI said there were two main
aspects to this arising when doing deep stacking. The usual CASU pipeline
output uses 32-bit integer FITS and given enough files in a stack quantisation
noise can become an issue. The external (to the pipeline) stacking software
has an option to produce Real*4 output if required. This version has been
available for over a year. The other aspect was caused by a software limit
in the supplied pipeline catalogue software. A simple workaround for this
was suggested at the time and a more robust solution is under development.
7. VDFS VISTA update
WJS reported that the telescope has passed the acceptance test, including
(nearly all) the mirror supports. This will now be shipped from Houston over
Easter before the Hurricane season. WJS then reported on the schedule for
delivery of the mirrors and coating plant.
The camera is being tested as we speak. PSB said that we have not received
any AIT data yet [Gavin must have overheard him because some appeared the
next day].
It is expected that the full all-up testing for the camera will take place
in May. PSB stressed that we want all the data from each exposure, not
cutouts around the exposed spots etc.. WJS said that he would pass this
requirement on. <<<<
WJS also added that the optical test camera has arrived in Edinburgh.
RGM said that ESO now has an optical sky monitor at Paranal but was unsure
how accessible the data from this was.
PSB said that the number of pending VDFS documents has gone up by one. The
newly found document is called "VDFS Requirements for VISTA Data
Information" and is a QMU-led document. Nothing else has happened on the
documentation front.
The DRL has got bigger still and is making steady progress. The use of the
ETC has quietened down now that the survey deadline has passed. ESO has
started work on the production version of the ETC. This means that it is
becoming more important to supply them with the correct calibration tables.
MJI reported that the FITS extension reordering request (an SPR - Software
Problem Report) had gone in and will be implemented before commissioning,
but not before AITing. PSB will write a script to order any AIT FITS files
before letting them loose here. <<<<
8. AstroGrid liaison update
Nothing to report
9. AOB
None
Continuing Actions
------------------
JRL sort out how to transfer MSB quality information to processed files
MJI finish investigating merged classification statistics
MJI raise issue of CASU preferred method of data transfer from ESO at VDMT
meeting
WJS contact JPE regarding acquiring the numeric (rather than graphic)
versions of the data that is required by the ETC
RGM finish quantifying foul-up causes in LAS outliers list (by end April)
STH find out what happened at the AstroGrid Implementation meeting
ALL send reports to STH covering the entire quarter (by Friday 7th)
JRL check and if possible fix any remaining FU'd headers (JITTER keywords)
MR check reprocessed data from 05A for quality control
MJI
New Actions
-----------
DWE place link in persistence report to the relevant 2MASS persistence
web page
MJI distribute updated Q1/Q2 plan
RGM decide if telecon presence at VDMT meeting wanted
JRL investigate and fix problem with UKIRT archive
MJI acquire copy of Dunlop UDS proposal
MJI send email to Sandy Leggett regarding linearity
WJS pass on information to RAL that CASU want whole data frames not just
cutouts
PSB write a script to order the extensions sensibly for AIT FITS files
