—
Plain Text,
17 KB (17537 bytes)
File contents
6th November 2007 Hoyle Committee Room 10:00-12:50
Present: STH, EGS, PSB, MR, MJI, DWE, RWA
Apologies: JRL, NAW, RGM
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent meetings/visits/telecons
4. Data archives update - AAT, ING, WFCAM, UKIRT
5. Optical processing update
6. WFCAM news - raw data transfer status
- WFCAM network transfers status
- processing status
- reprocessing status
- photometric calibration status
7. VISTA news - pipeline progress
- camera test data
- letter to PIs
- science verification
8. ADASS 2007
9. AOB - windows machine++
Minutes
-------
1. Actions from last meeting
RGM have not completed the work on the classification aspect of cross-talk
EGS weeding and classification generally. It was decided to subsume this
MJI action into general checking since WFAU now flag potential cross-talk
affected objects with an error bit code.
JRL are still working out a scheme for doing header updates for ING archive.
EGS It was decided to remove this from the actions list as it is really a
work package.
MJI is now forwarding the weekly email JAC reports from Chris Davis to a
wider audience. It was also pointed out that these summaries are
available on a wiki page. MJI will forward access details. <<<<
MJI has checked the status of the reprocessed UDS data (see later).
PSB has contacted Brad Cavanagh at JAC for an update regarding the WFCAM
network transfer (see later).
MR figured out the best to way to advertize the illumination correction
information and will now implement this scheme <<<<
JRL has finished the self-flatting tests on VIRCAM data and emailed a
summary of points to note from the VIRCAM tests to the participants of
the telecon.
JRL the compilation of a wishlist for the late October VIRCAM cold camera
PSB tests was dealt with during a telecon, ie. run dark current template,
linearity template, and acquire one full r/o with TCS running to test
headers etc.. However this was later subsumed by events.
PSB reacquired the helpdesk laptop and PDF'ified the ADASS presentations
DWE is now updating the apm1 Windows machine regularly
ALL considered attending the upcoming UKIDSS meeting in December. Currently,
EGS, MJI, STH and JRL have registered. PSB was still thinking about it
subject to availability of dunkelbier.
MJI has bought a handheld vacuum for the APM building. This should only be
used for computers.
PSB investigated getting wireless for the APM building and will go get one
This would also be an opportunity to acquire a CASU order book from
Paul Aslin. <<<<
2. Comments on WFAU minutes
MJI noted the recent problems WFAU were having with reading old LTO tapes
and commented that although we haven't had any problems with LTO tapes it
was worth doing a specific test to check. MR will to dig out 4 old LTO
WFCAM raw tapes and verify, or otherwise, that they can be read. <<<<
3. Recent meetings/visits/telecons
Luca Rizzi visited on 22nd and 23rd October to discuss various WFCAM issues
and to get to know more about how the pipeline works. A set of notes
from the meeting are available and are now on the internal web pages.
MJI volunteered to make sure that the various reports to the UKIRT board
he has copies of are also available on the internal web pages. <<<<
There was a telecon on 9th October mainly to discuss VIRCAM test data
reports that JRL had produced assessing the properties of the detectors
from the July lab test data taken on Paranal. Much of the discussion
centred on the differences between the two readout modes. The CASU wish
list for the upcoming cold camera run was also briefly discussed.
STH reminded everyone of the upcoming VDMT meeting and the need to get
reports in on time.
4. Data archives update
ING archive:
The old DAT tapes will be ditched. At least one of the cabinets will be
kept in the IoA/Greenwich House interface for potential future storage
use for LTO backup tapes. RWA will skip the old data tapes when an
empty skip becomes available. <<<<
AAT archive:
The AAT CDs have now been incorporated into the collection in the
cabinets in the storeroom. It remains to insert the JKT CDs into the
drawers.
Another month of AAT data has been copied and ingested via the Internet.
Plates:
Elizabeth Griffin will visit IoA in the first week of December. Craig
Mackay's SPO office has been emptied and will be used to sort the plates
from the adjacent plate store. The high resolution Coude plates will
probably be sent to Canada and the archive Cambridge plates will go to
Laundry Farm in the first instance. Bob will ask whether the PDS machine
can be of use in one of the new plate repositories.
At Laundry Farm, there are about 1000 direct plates, mostly from US
observatories and 4 metre or Schmidt telescopes. About 400 of these plates
are objective prism exposures and will go to the new repository in the eastern
US. The original Palomar 14 x 14 plates are wanted at Palomar and the fate
of the other plates has not yet been decided. Attempts are being made to
find a new owner for the POSS 1st epoch plate copies and the custom-built
shelving that goes with them.
There was some discussion about offering the glass copies as souvenirs at
Open Days etc. if all else fails and nobody wants them as a set.
WFCAM:
MR reported that we have received complete nights up to the 3rd of October.
JRL is currently working on a new interface to the WFCAM raw data archive
that will allow unregistered users to search and request data that
under the current policy is public (ie. older than 18 months or whatever
this policy morphs into). PIs will be able to obtain access to data from
their project in exactly the same way as before.
Tapes are still coming in batches containing different nights for each chip.
The mismatch of detector nights and tapes was discussed during the recent
visit of Luca Rizzi.
Plans to perform direct FTP data transfer tests from JACH to CASU are
also progressing and it is hoped to test this later in November.
There is a G-bit link from the summit to JAC and it is intended to use
an FTP server at the summit to transfer some data directly to CASU after
some further tests and discussion on transfer protocols are completed.
Transfer of raw data to ESO is still intermittent due to problems at the
ESO end. This requires too much monitoring at our end which we would like
to minimise so MR will email ESO regarding the raw data transfers to see
what the status of their planned improvements is. <<<<
STH noted that one of his campaign proposal science frames was officially
the millionth observation taken with WFCAM, so presumably drinks are either
on him or JAC.
MJI noted that each executed MSB now has a unique transaction ID in the
FITS header to enable better tracking.
UKIRT:
no change
5. Optical processing update
MJI reported that he had received some positive news via Janet Drew about
the status of the VST including a realistic date for when it will begin
taking science data.
He also noted that he had found and fixed a bug in the INT WFC WCS
initialisation software that explains the recent spate of problems with
converging on the final WCS solution for detector #4.
6. WFCAM news
Processing:
We are currently processing September data (note the first 12 nights have
no data due to WFCAM engineering work). About half of the remaining nights
have already been reduced and the remaining ones are currently being
processed. Processing of September data was delayed whilst waiting for
another batch of tapes to arrive because we had only three complete nights
and an incomplete set of data to form decent flats. However once we received
the data covering the whole month of September (on October 26th) it turned
out that the flats could not be used to process the data because of an
interesting new instrumental feature (see below) that was particularly
strong on the nights when most of the flats were taken.
The effect manifests itself as a variable channel level "offset" on
detectors #2 and #3 and has both a repeatable and a random component, in
addition to being strongly, but not completely, anti-correlated between the
two detectors. An example of the repeatible part of the effect is available
at http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/
2007b-known-features. This shows a sky frame formed during the original
processing of one of the September nights. Unfortunately, the random
component is also about the same level (~1% of sky) and even after sky
subtraction affects some science product frames.
After some investigation it was clear that most of the created September
flats suffered from this effect as well (contributing to the problem when
used in processing) and after a series of tests to ensure that other flat
field characteristics had not changed during the camera engineering work we
decided to use the August flats for processing.
So far we have found that the pipeline-created sky frames are a good
diagnostic of the problem being present but are still investigating which
nights are affected. To date we have not seen this problem in master dark
frames produced by the pipeline.
With some misgivings about its general use, we have developed a prototype
sledge hammer software fix which if all else fails we will use on affected
data in an attempt to mitigate the problem. Unfortunately, this has to be
run at the individual frame level, ie. post-sky subtraction but pre-
interleaving and stacking, due to the random nature of some of the effect.
This will involve selective partial reprocessing of affected MBSs and
possibly significantly add to the overheads of running the pipeline.
Reprocessing:
In total 17 nights of UDS data (~700 Gbyte of products) for 06B has had to
be reprocessed to allow for more rapid sky tracking to remove some odd
features in the science product frames. The reprocessed data has been
supplied direct to the UDS consortium for them to finalise the deep
stacking and thence make the deep stacks available for DR3.
Photometric calibration:
STH has circulated a first "real" draft of the photometric calibration paper
to MJI, PCH, and SJW. He asked for group members to have a read and send in
comments (and presumably anyone else reading this). The paper can be found at
http://www.ast.cam.ac.uk/~sth/photom_draft.pdf. In the meeting, discussion
focussed on the interpretation and possible causes of the spatial systematics.
STH also asked for thoughts about how to better demonstrate the effects of
non-photometric conditions on the quality of the calibration. He showed a
plot of sigma_ZP (taken as the MAD of the MAGZPT values across all 4 chips)
vs ZP. DWE suggested investigating repeat photometry in the different
regimes of the figure. STH will investigate if he can use the per chip
scatter in ZP instead of the inter-chip scatter as a more 'informative'
measure of non-photometricity. MJI hopes not because he doesn't want to
update all the headers yet again. STH hopes to circulate the paper to a
wider audience for comments in a couple of weeks. <<<<
MJI noted that due to the new calibration protocol involving individual
detector photometric zero-points, that for a trial period data would be
flagged as ok_to_copy in monthly batches since it requires a month of
data to accurately compute the corrections. The advantage is that this
saves having to keep track off and to keep updating the photometric
zero-points, the disadvantage is that it slightly delays release of flat
files through WFAU.
MR noted that the WFCAM quality control database has been updated to
include the latest version of the photometric calibration and some of the
recently reprocessed nights (but not UDS).
7. VISTA news
Pipline Progress:
A number of changes were required to the DRL to make it CPL 4.0
compatible (the version which uses cfitsio, not qfits). This was fairly
urgently requested by Sandra Castro because ESO want to install this version
in the Paranal pipeline. v0.7 of the software and v1.9 of the DRLD has
been released. Problems with using FITS compression via CPL have been
identified and reported. There was also a new version of the 2MASS
catalogue released with this.
Camera Test data:
The July Paranal engineering run produced 2,500 files which have been used
for various tests (of the system for VPO use and for DRL development);
reports were issued and discussed by telecon. It was agreed that it would
be unwise to offer READ-RESET mode observing initially.
Naidu circulated an interesting conversion gain non-linearity paper prior
to the telecon which highlights some interesting features of NIR detectors
including VIRGO and Hawaii-I (the paper is available on the internal web
pages - mind you it still leaves several open questions related to
non-linearity).
Tiles:
Invesigation of the behaviour of the VIRCAM/VISTA software when doing
tile offsets showed up a number of problems: namely in the computation
of the WCS rotation matrix; a sign error; and a (just) significant
rounding error in the propagation of the sky position angle. It is
fully expected that first-light will demonstrate further geometrical
transpositions et autre faux pas.
Survey PIs and Science verification:
ESO have reminded survey PIs that the science verification program will not
simply be a case of performing part of the surveys, but will be an
independently constructed program. In light of this MJI should raise the
issue of science verification of VISTA data at the next VDMT meeting.
CASU would like to be in the loop on this if possible. <<<<
Letter to PIs:
No further enhancements have been proposed so we will do the enhancements
already planned. Some issues have been flagged in the present system which
we will need to consider for VISTA: a.) the survey overview pages are not
in sync with the processing pages, ie. the user goes to the processing pages
and see that the data is processed but that is not included in the survey
progress pages. This is due to the database design; b.) UDS frames (ie. with
UDS in the project ID) have been observed that are not part of UDS so some
statistics were wrong for this survey. Corrected now.
MJI noted that Henry McCracken at Terapix wanted access to a full VISTA
header set in preparation for dealing with the Ultra-VISTA data. He was
sent a full simulated raw MEF file.
8. ADASS 2007
ADASS XVII proceedings:
About 150 contributions have been received with 80 still missing. The
deadline passed on November 2nd. Meanwhile RWA will check all the
contributions received so far and will delete extraneous files and rename
those papers with the wrong ID - during this time it will be possible to
include late arrivals. A date of the end of normal working in December
(ie. ~20th) was set for the final compilation of the latex source prior to
shipping to the publishers. It was felt important to have the proceedings
published in time for the next ADASS announcement in April.
9. AOB
DWE said that apm1 was possibly not the best place for the Sky Server of
the Wide Field Survey object catalogue. Currently it is causing fragmentation
problems. Should we get a larger machine or move the location of this
catalogue to a more maintainable system? It was decided that the catalogue
should be served the same way as IPHAS is. Once this is done, Guy Rixon
should be informed so that the AstroGrid registry can be changed.
EGS was volunteered to email JMI to get the necessary information so that
this work can be done. <<<<
Scheduled updates. - apm25 and apm26 patched with latest updates and
rebooted. The remaining APM machines will be rebooted during quiet
spells (ho hum!).
There was some further discussion about security of the APM cluster. MJI
will review the user accounts and for now lock any unused ones prior to
removing them. <<<<
Membership of the new IoA Computing Advisory Committee was noted.
Continuing Actions
------------------
none !
New Actions
-----------
MJI forward access details of JAC wiki page for WFCAM observing summaries
MR advertize illumination correction information
PSB acquire wireless for APM building (and get order book at same time)
MR perform backup verification test using 4 old LTO tapes
MJI make latest reports to UKIRT board available on internal pages
RWA skip old data tapes
MR email ESO regarding raw data transfer interruptions
ALL provide comments on photometric calibration paper
MJI raise issue of science verification of VISTA data at next VDMT meeting
EGS email JMI regarding details of database on apm1
MJI review user accounts on APM cluster
