—
Plain Text,
16 KB (16681 bytes)
File contents
18th April 2005 Hoyle committee room 10.00 - 12:30
Present: PSB, MR, RGM, MJI, DWE, EGS, STH, WJS, JRL, RWA
Apologies: STH
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. CASU grant review outcome
4. Data archives update
5. Report from UKIDSS SciVer meeting
6. JAC telecons and WFCAM update
7. Report from VDMT meeting
8. Status of FDR VDFS documents
9. AstroGrid liaison update
10. AOB
Minutes
-------
1. Actions from last meeting
STH it was not known whether STH had contacted JMS regarding the misdirected
links. Since it was not regarded as very important, it was decided to
drop the issue.
DWE has set up a new entry point to the CASU web pages, given them a
small makeover and checked links are up to date
(http://www.ast.cam.ac.uk/casu/)
EGS has finished the calibration of the large WFS surveys. MJI has also
updated the WCS information in the headers and will start updating
the magnitude zero-points.
STH has not been in Cambridge for most of the time since the last meeting
DWE so no progress has been made on setting up the internal disc resources
web page. This was regarded as being of minor importance - ongoing <<<<
JRL after numerous attempts to resurrect the DVD tower, it has finally been
PSB declared an ex-DVD tower. The process of transferring the ~700 DVDs of
MJI data from the UKIRT and AAT archives to RAID array has been started by
RWA. Problems with files names are being sorted out as copying
progresses and the UKIRT archive is gradually coming back to life.
It will take another 3-4 weeks for the UKIRT archive transfer to be
completed.
MJI neither had time to go to NAM nor write a poster for it
RGM
MJI has received an updated VDFS schedule. Two external dependencies are
being added to highlight CASU work package progress dependency on
simulated FITS files and on laboratory test data.
STH have not yet arranged a meeting with Kona Andrews about AstroGrid
MR deployment (see item 9) - ongoing <<<<
STH has placed a copy of Simon Dye's VDUC presentation in our internal
directory
JRL MJI said that he has modified his development code (currently being
used for catalogue generation for WFCAM transfer and ingest tests) so
that the catalogue FITS keywords for unused columns are now "Blanknn"
(ie. the names are now unique) plus he incorporated a few other
requested table name changes (to make the 16, out of 22, character name
subset more readable). JRL has still to propagate these
changes to the pipeline production code - ongoing <<<<
MJI has asked JRL about his desktop computing requirements and a suitable
system has been identified and ordered.
ALL have handed over their spare keys and they are now in a special key
cupboard in the workshop.
STH have updated the DRLD figures - well PSB did actually
JRL
PSB incorporated the new templates into the documentation and rationalised
MJI the whole template set, recipes, functions. CPL plugins, DPR keyword
design and so on
PSB has organized the hotel for the trip to ESO to discuss the changes to
JRL the CAL plan and DRLD (mid-May) including a technical discussion on CPL
MJI and DICB issues. The flights have yet to be sorted out. <<<<
RGM has not sorted out the installation of AstroGrid-in-a-box. Due to a lack
of AstroGrid code deployment activities this is not a high priority,
ongoing but subsumed into the regular Astrogrid liaison update agenda
item.
MR has registered for the Euro-VO workshop
DWE raised the issue of the Euro-VO workshop at the appropriate IoA
Gaia meeting
2. Comments on WFAU minutes
There were no comments other than MJI pointed out that WFAU were again
recruiting for the software position in addition to advertising a position
for a data mining guru.
3. CASU grant review outcome
The good news is that we have heard the outcome of the grant review and that
at least some of it was positive. The panel judged that 3 of the 4 requested
funding lines would be funded at more or less the requested levels bar a few
caveats. The flip side was the lack of enthusiasm shown for the unique
raw data archives maintained here (UKIRT, AAO and ING). This curtails any
development of these systems other than for the ongoing efforts to make them
all available online and clean up the many and assorted header problems,
particularly in the ING archive.
A puzzling decision in the light of the VO world, but not helped by the
unfortunate demise of the DVD tower early this year, and the lack of a clear
plan with well defined deliverables and milestones in the grant case.
[MJI blames having to deal with WFCAM commissioning, ESO FDR documentation
and grant proposal writing all at the same time].
4. Data archives update
RWA reported on recent activities with the archives. The JKT and WHT
archives have now all been transfered to RAID arrays. The UKIRT DVDs
from the tower are being manually transferred to RAID, the AAO collection
will follow. The tape store in the SPO is being/will be used for storing the
original tapes and DVDs for security.
Discussions have taken place with Don Carlos Abrams from the ING about what
medium to use in future for data transfer since DVDs are quite bulky and slow
and unreliable. The ING are considering several options including LTO-II tape
systems. Another possibility is to use the Internet. The link between La Palma
and Tenerife is being upgraded in the near future and automated internet
transfer (cf. WFCAM raw --> ESO) is a possibility. A slight snag with this
method is the lack of a backup offline archive at the Cambridge end. This
would have to be addressed.
RWA said that there have been many problems with the ING DVDs and that some
batches had a large number of failures. When this happens the ING sends
replacement DVDs but this all adds to the overheads of keeping the archive
viable.
It was suggested that JRL trains up MR (and/or EGS) in the mysteries of the
Perl Sybase archive scripts to relieve some of the archive management
burden on him. <<<<
RWA said that he had been approached by a student from the Perse School
regarding work experience in an astronomy environment. It was thought that
helping with transferring the DVD data to the RAID arrays might be a
suitable task. RWA was asked to chase this up and see if the student would
be interested in doing this for modest remuneration <<<<
5. Report from UKIDSS SciVer meeting
RGM and STH attended the UKIDSS Science Verification meeting held at
Imperial College on April 1st. The minutes of this meeting can be found on
http://wiki.astrogrid.org/bin/view/UKIDSS/SciVer. This has the planned WFCAM
schedule on it. Also on this page are the observing summaries from
the science verification phase ie. the observing that the two Simons are
carrying out now. We were all encouraged to read these.
The minutes contain a summary of the schedule UKIDSS seems to be expecting
for release of science verification (SV) data plus other deliverables.
The SV schedule agrees with the actions from the VDUC meeting, and bar the
fact that commissioning slipped 2-3 weeks, this is not unreasonable.
One or two other target dates for delivery of products though seem a bit
optimistic and the CASU WFCAMers are concerned about expectations being
raised too high by the dates' presence on a public web page.
6. JAC telecons and WFCAM update
Since the last CASU meeting there have been 3 JAC telecons:
9th March - MJI, STH at IoA and AJA, PH and JRL at JAC
17th March - MJI, STH at IoA and AJA, PH and JRL at JAC
14th April - MJI, JRL at IoA and PH, STH, SD at JAC
There are still issues of processing throughput with the summit pipeline.
These are actively being investigated and JRL has spent a lot of time
rewriting large sections of the summit pipeline in a more monolithic
fashion to try and get around some of these bottlenecks in conjunction with
assorted hardware updgrades and investigations of the summit processing PCs.
MJI and JRL are investigating alternative processing strategies to remove
the sky creation and subtraction step since a.) this is not an ideal solution
anyway, b.) the decurtaining algorithm they have developed also does a very
good job of removing residual reset anomaly and c.) the latter can be run
during data aquisition rather than having to wait for MSB termination to
be triggered.
One of the bottlenecks seems to be the conversion from SDF to FITS format.
This is taking 2-6s per frame, whereas it should be taking < 1s.
This may be a buffering problem with CFITSIO linked to a small blocksize
or it may be a RAID controller issue. JRL said he and PH were investigating.
Consideration has been given to trimming down the summit pipeline so that it
only does what is required for data quality control, but this is difficult
and as noted by STH impacts on observers perceptions of the processing.
It was noted (see above) that the decurtaining algorithm also removes the
reset anomaly, so there is no need for sky subtraction at the summit or
possibly in general elsewhere. However, the decurtaining algorithm is fairly
CPU intensive and needs tuning a bit to run effectively at the summit.
MJI and JRL pointed out that this seems to be a robust and viable alternative
that is much better than having to form local sky estimates with the
inherent problems that always causes (eg. -ve halos around bright objects).
Another problem relating to displaying summit pipelined data has been raised.
The browser of choice there, GAIA, is not very resilient to bad data value
ranges. This is undoubtedly related to not having a good enough quality bad
pixel map/confidence map in place there yet and is probably caused by flat
fielding bad pixels. This should be easily fixable and will be corrected
when the next set of master calibration files are produced.
MR reported that he had almost completed the processing for the first
commissioning run. Many segmentation faults and core dumps were caused by
the bad data (possibly a good stress test therefore?). In general, the
pipeline is working fine. A recent test showed that there was a time-related
problem with the way the pipeline is dealing with Rice compressed data.
JRL said that he understood the problem and will fix it shortly. <<<<
MR noted that a problem with the first commissioning run was that it contained
no narrow band twilight flatfield data. Consequently no narrow band master
flats could be created. Processing narrow-band data from comm-I is low
priority.
A summary status of WFCAM commissioning-I data reduction can be found at
http://apm15.ast.cam.ac.uk:9675/casu/docs/wfcam/commissioning
MJI pointed out that a fault report page would be useful. This shouldn't
necessarily be a full error reporting database yet, but a simple HTML page
with fault, action, who needs to do it, when was it done, comments .....
This needs setting up and maintaining. <<<<
MJI and MR will prepare a list of what comm-I data is worth reprocessing
(ie. potentially useful science content) after the thorough shakedown
the pipeline has had during the first pass through. This will be the
only comm-I data reprocessed. <<<<
STH is currently setting up the secondary photometric standards system
and taking lots of data for illumination correction tests.
7. Report from VDMT meeting
The VDMT meeting was held by telecon on 13th April. MJI reported there
were no major problems and the issue of external dependencies on the
VDFS pipeline developement schedule had been raised. STH still has to
complete the work progress aspect of the reporting. <<<<
There was still a problem with the signing off of the board
RIX response document that was now impacting on completing the changes
required. MJI said that he would ask JPE about the current status. <<<<
There were no quibbles regarding the quarterly progress. The plan for the
next quarter is still be be drawn up. MJI said that he hoped to finish
this by the end of the week and would circulate the list for comment. <<<<
An interesting misunderstanding has arisen about ESO's requirements on
the deliverable science products for the processed public survey VST data.
ESO appear to want much higher level data products than anticipated
eg. uniformly calibrated data over whole survey areas. However, this
requirement on its own should not directly impact on CASU pipeline
processing of VST data. The main unknowns from our point of view are the
requirements on individual processed products meeting the, as yet,
undefined science requirements.
8. Status of FDR VDFS documents
PSB said that he is still waiting for the RIX response document to finish
off many of the changes in hand. After a lot of work, we now think we
understand the constraints that we have to deliver within. It was decided
that the updated Cal plan and DRLD documents should be sent to Michele
Peron by the end of April so that ESO have enough time to study the
changes before the mid-May meeting in Garching. <<<<
PSB said that the ESO workstation was up and running and that he could
control the VIRCAM instruement plus VLT simulators from his office (sort of).
9. AstroGrid liaison update
RGM said that although AstroGrid have regular telecons, they are unminuted.
There is no recent news apart from noting that the release of AstroGrid 1.0
has slipped from 1 April to 1 May.
An account (agcasu) has been set up for CASU-related Grid use.
The software packages tomcat (Solaris 10) and mavan?? were mentioned as
installation requirements.
PSB said that he would upgrade cass123 and RGM's machine to Solaris 10 <<<<
so that he (RGM) could break it.
10. AOB
EGS said that Plone (http://plone.org/) was a useful tool that we should
consider using for CASU. It is a user-friendly open source content
management system. DWE said that the Gaia group was considering using it as
the basis of their webpage and information distribution (after MR had done
a demo for them).
[a comment found on the website "Wicked: wiki with no wiki aftertaste!"]
MJI suggested that EGS and MR package up the software documentation
using Plone as a demonstration to see how suitable it is for us. <<<<
DWE said that he had two requests from RGM for improving the documentation
webpage
1. adding section links (easy) <<<<
2. providing a sorted-on-date version so that you can see at a glance which
is the most recent document. From the way the Perl script is written, this
is not entirely straightforward. This was regarded as useful, so it was
suggested that DWE email STH (original author) to see if he had any
quick fixes. <<<<
Continuing Actions
------------------
STH set up the internal disc resources web page
DWE
MR arrange meeting with Kona Andrews about AstroGrid deployment
STH
JRL update the pipeline version of the catalogue generation software to
be compatible with the recent changes made to the development version
PSB book flights for the trip to ESO (mid-May)
MJI
JRL
RGM remind Andy Batey about the installation of AstroGrid-in-a-box
New Actions
-----------
JRL train MR and EGS about the mysteries of the archive scripts
RWA reply to Perse School student who might help with transferring the DVD
data onto RAID array
JRL fix compression timing problem with pipeline
MR generate a fault reporting page for the pipeline and populate it
MJI prepare a list of what comm-I science data is worth reprocessing
MJI ask JPE about the current status of the RIX Response Document
STH finish generating work package progress information and send to JMS
MJI finish generating the quarterly plan for 05Q2 and circulate for comment
PSB send updated ESO documents to Michele Peron by the end of April
PSB upgrade cass123 and RGM's machine to Solaris 10
EGS package up the software documentation using Plone
MR
DWE provide section links in documentation web page
DWE email STH for suggestions regarding sorted output from documentation
Perl script and provide date sorted output option
