—
Plain Text,
18 KB (18720 bytes)
File contents
23rd June 2008 APM comfy chairs 11:30-13:05
Present: RWA, MJI, JRL, PSB, MR, DWE, STH, RGM
Apologies: EGS, NAW
Agenda
------
1. Actions from last meeting
2. Comments on WFAU minutes
3. Recent meetings - Video meeting U. Herts
- Plato meeting ESTEC
- LSST meeting Edinburgh
- CACC meeting
- UKIRT board meeting
4. Data archives update - AAT, ING, WFCAM, UKIRT
5. Optical/NIR processing - HawkI
- IPHAS/UVEX
- MegaCam/CFHTLS photometry
6. WFCAM update - illum corrs yet more tests
- photometric calibration paper
- data transfers
- flippin' jitters
- processing status 08A
- DR4 checking
- flagging rejected MSBs
7. VISTA update - general news
- SADT + P2PP workshop at ESO
- pipeline progress (random jitters)
- data connections to ESO; ftp tests
- JPE visit and Paranal camera data
- survey progress web page requirements
8. AOB - network upgrades
- hardware
- APM air con etc..
Minutes
-------
1. Actions from last meeting
STH has partially checked that the Plone astrometry technical pages are
correct - ongoing <<<<
MJI acquired the transmission curve for the VIRCAM Z filters. He is still
trying to obtain the N118 transmission curve - ongoing <<<<
MJI finished off checking and acquiring the latest SMPs
RGM
JRL postponed reviving the changes log web page for the WFCAM pipeline
until the WFCAM data had been reduced - ongoing <<<<
STH has dropped the action to investigate the new SADT since there will
now be an ESO workshop covering use of this for VISTA/VST public surveys.
MJI has obtained a GANTT chart of the latest VISTA integration schedule. He
will continue trying to acquire more information about the NIR camera
commissioning plans. <<<<
STH completed the illumination correction tests in conjunction with WFAU
MJI
STH has obtained the graphic showing the size of the photometric radial
distortion correction for the calibration paper, but has not added
this to the Plone technical pages - ongoing <<<<
MJI did not directly find out at the CACC meeting what the current network
costing model is, but checked after the event and found that it hasn't
changed much.
MJI after a bit of discussion about location and format suitable meeting
tables for the APM building will be shortly ordered
DWE found out from Andy Batey that the wall display screen would be far more
expensive than the simple expedient of buying a flat-screen LCD TV. Part
of this was the "need" to have wall display screen installed on the wall
by the Company. Plan B is to buy a TV with PC interface and have it
free-standing on the end of the table against the wall. MJI will
check compatibility with his recently acquired home equivalent to see if
displaying from the laptop works well. He will also measure/check
the dimensions of a suitable model. <<<<
2. Comments on WFAU minutes
Although there were a lot of WFAU minutes since the last CASU meeting there
were no substantive comments apart from further discussion of assorted
photometric corrections.
3. Recent meetings
Video meeting U. Herts:
MJI, EGS and RGM attended. JPE was reminded of our desire to transfer data
from Garching to Cambridge via ftp rather than NGAST disks. JPE presented a
summary of VISTA progress including a ppt summary slide of the schedule,
which would be useful to have. <<<<
Video requirements are similar to those of UltraVISTA. MJI noted that EGS
had signed up within the consortium to do the deep stacking, starting from OB
pipeline output that has passed further stringent QC checking at U. of Herts.
Plato meeting ESTEC:
STH and NAW attended a one-day meeting entitled 'the PLATO payload
consortium: kick-off meeting', held at ESTEC on June 2nd. Both felt that CASU
could make a meaningful contribution to the UK bid for involvement in the
mission, and NAW has submitted text to the UK SoI, PI Don Pollacco.
LSST meeting Edinburgh:
RGM attended this one day meeting and reported on the discussions for UK/
European access to LSST data.
CACC meeting:
Nothing much of note relevant to CASU activities other than news about
network charging models gleaned afterwards.
UKIRT board meeting:
MJI reported that he had given the usual upbeat WFCAM processing talk.
Ross Collins gave a similar one for WFAU and Steve Warren for UKIDSS.
The interesting snippet gleaned was that the official Board policy is that
the proprietary period for ALL processed data is the same as for the raw data,
ie. 12 months after end of semester in which data was taken it is World
accessible - if M. World wants it. He also noted that before the Board was
the proposal to allow UoH access to UKIDSS data in exchange for 50% of UoH
time, ~24 nights/yr.
4. Data archives update - AAT, ING, WFCAM, UKIRT
The last shipment of Schmidt survey plates was loaded on Friday morning
and transported to Germany the same day, along with most of the custom-built
wooden shelves. It now remains to organise the 1000+ assorted objective-prism
and prime-focus plates still in CDM's office in the SPO. Those plates
relevant to Palomar will return there and the remainder will go to the
plate archive in the eastern USA. The plate store in the SPO is now empty.
AAT:
A recently transferred batch of data has been ingested and the archive is up
to date.
ING:
The archive is up to date and slurping in ever more INT and WHT data via
ftp transfers.
WFCAM:
The last batch of 08A data (16th June) was received via ftp and ingested a
few days ago making the raw data archive complete for 08A. Transfers of
UKIDSS and CAL data to ESO proceed apace with about 20 nights of 08A to go.
UKIRT:
JRL is fixing a few minor foibles that slipped through the cracks in
conversion from SDF to FITS.
5. Optical/NIR processing
IPHAS/UVEX:
Most of the data up to end of 07B (Jan 2008) has now been processed and first
pass calibrated. Two more runs in 07B for UVEX need to be finished off.
Final band-merging of UVEX data is awaiting modified bootstrap zero-point
offsets for the HeI and u' filters - to be further discussed at the upcoming
EGAPS meeting later this week.
The proofs of the IPHAS paper have been received and corrections sent off.
HAWKI:
JRL has processed the HAWKI Carina and GOODS SV fields. He will investigate
making a large area PR image, while STH will focus on improving the quality
of the photometric calibration by deriving more accurate colour equations.
MJI will investigate regenerating the catalogues for the Carina data to take
better account of the considerable nebulosity in the region. The results <<<<
from this and the GOODS (image, calibrations, catalogues) will be
advertised via a web page. <<<<
MegaCam/CFHTLS photometry:
MJI noted that assorted problems he and Rodrigo Ibata had been having with
MegaCam data (mainly centre-to-edge effects at the ~3% level) were apparently
due to incorrect pixel-level illumination corrections computed and applied by
the CFH Elixir pipeline. This problem probably affects all MegaCam data.
To correct properly requires starting from scratch since the problem occurs
at the (super)flatfielding stage and varies with each new flat used. A
salutory warning against applying a correction-too-far.
INT WFC data:
RGM has a summer student (Tom Young) working on calibration of old INT WFC
survey data. This has required (and will require some further) updating of
the pipeline toolkit on the Solaris cluster which, due to lack of much use,
had fallen well behind. <<<<
6. WFCAM update
Data transfers via ftp have been extremely successful and painless (at least
at our end). As noted previously this has generally resulted in observed
nights of raw WFCAM data being available on disk here roughly 24 hours after
the data is taken. It is checked, ingested and read to go for processing
roughly another 24 hours later.
Standard processing has been running about a week further back and complete
processing to fix parquet floor problems and a brief spell of CFITSIO
compression problems (too long a saga to relate here) plus all the QC
checks, including monthly photometry updates, is the current bottleneck.
Several more tests on the illumination corrections have been completed with
good results. These were applied at this end post band-merging of catalogues
since the merging software also applies the rest of the calibration,
aperture corrections, detector zero-points and radial distortion,
on-the-fly. A comparison with a test implementation at the WFAU end, after
a few iterations, produced calibrated magnitudes agreeing to better than
1 millimag.
RGM suggested that it might be worth producing a standalone catalog-level
piece of software that applied all the corrections to the standard catalogue
flat files and could be made available to external flat file users.
MJI will investigate the best way to deal with this. <<<<
JRL has nursed the flippin' jitter nights through the pipeline using a
development version. This reminded MJI of the possible need for a telecon
with JAC staff to discuss lessons learnt and related issues of requirements
on data self-description. <<<<
[The possible need of an equivalent observing mode for VISTA was mooted by
JPE after our last set of minutes. Though JRL noted that with a 4x4 array
of detectors there are few extra flippin' possibilities]
Some feedback from DR4 QC eyeballing had been obtained, virtually all was
very positive about the quality of the processed data. The only reported
UKIDSS problem turned out not to be new DR4 data but a previously missed
astrometry problem affecting a small percentage of GPS data (affecting
~200 files over a 3 year period). This was tracked down to several nights
(mainly 07A) where one of the CASU nightly QC checks had been omitted by
mistake. This problem should normally have been picked up and fixed (via
further refinement of the astrometric solution) during these checks.
JRL will investigate implementing the flagging of summit-rejected MSBs using
the mirrored OMP database tables. If this is feasible a suitable keyword
will be inserted in the raw FITS data files prior to ingestion. <<<<
On a related note, MJI pointed out that from 08B we would be running an
MSBTid script to check if MSBs passed the observing criteria (if set) in the
header. A log for each processed night would be made available via the
reduction progress web pages for JAC to make use of for faster feedback. <<<<
EGS reported via email that he had met with Michael Rowan-Robinson and Seb
Oliver regarding photometry of galaxies from 2MASS vs UKIDSS. After some
discussion with STH and several tests he found that a comparison of 7"
apertures provides a good agreement between both systems
(http://apm14.ast.cam.ac.uk/mywiki/wfcam/PhotometryGalaxies). Smaller
apertures do not agree so well, and there is an offset probably
due to 2MASS aperture corrections.
He also reported that on the topic of mirroring of the UKIDSS data at IAC,
he had talked to Nigel who reaffirmed that there is no requirement on VDFS for
this kind of redistribution of so much processed data. He has suggested to
IAC personnel that they start by using the WSA interface to download some
data as a test and take it from there. If they still want to do an Oliver
Twist then other avenues may need exploring. He estimates the total amount
of data they are interested in is around 4Tbytes and has suggested that
the interested parties at IAC negotiate directly with WFAU to clarify their
science needs.
Photometric calibration paper:
STH noted that he has updated the paper significantly following input from
Paul Hewett, Steve Warren and Sandy Leggett. Steve's comments in particular
were very thorough and all the contributors have definitely improved the paper
in ways too numerous to mention here.
There are a few extra tests from Steve that STH is carrying out at the
moment to examine:-
1. the variation in the slope of the colour term as a function of
galactic longitude and latitude, rather than season.
2. section 6.1. Steve thinks we can do more to handle the uncertainties
within the SDSS photometry. It is not obvious that this is going to add
significantly, or change the main result for the Y-band offset, versus the
JHK filters. However, STH is still attempting to do the tests suggested to
see what they show.
These additional checks should be finished within a couple of days. Then the
paper will be submitted. (Being a pint or so to the good, MJI declined the
bet on this timescale proferred by PSB).
7. VISTA update
PSB interacted with Florian Heissenhuber, ESO Network Specialist
(according to his signature) to attempt to narrow down the performance
problems with WFCAM raw data uploads, which run at an aggregate speed
of ~650KBytes/s. Simple ftp tests achieved ~5.5MBytes/s.
Florian is not in a position to install "iperf", which was his original
suggestion to perform an objective test, on the server. However the
interim conclusion is that the standard Internet from ESO headquarters
to Cambridge is easily sufficient to carry Rice-compressed raw VISTA data
on the final leg to the UK Science Pipeline, provided no throttling
software is employed.
JPE visited a short while ago prior to leaving for Paranal and more
integration testing. There was some discussion of the Calibration plan
document, test data to be taken, and a pic of a double star from the
on-board test camera.
JPE will buy 12x250GByte firewire disks and plans to DHL camera test data
daily using them. There was some discussion of the idea of trying to run DRL
recipes manually but that was dropped and also further discussion of direct
Paranal-Cambridge bandwidth for quicker turnaround.
After a bit of iteration the desire of the deep survey folk to be able to
use random jitter patterns seems to have made it to the home straight.
The pipeline doesn't care what dither patterns are used within reason
(ie. within ~10% of a detector width is fine).
There is a planned 3 day SADT + P2PP workshop at ESO in September and both
STH and EGS are attending on behalf of CASU involvement in VISTA and VST
public surveys respectively. Each survey PI + 2 worker drones are invited
to attend: Day 1 demo, Day 2 trial Phase II, Day 3 discussion.
RGM had sent in some comments on survey progress web page requirements
to EGS and cc'd to MJI but further discussion of this was deferred until
the next meeting with EGS present.
MJI noted that the VMC consortium had emailed him directly about access to
VIRCAM filter response tables and detector QE data, and, as directed, he had
batted their request to JPE.
8. AOB
Network upgrades:
As predicted with the previous cack 10G-bit fibre routing to the APM building,
Rodents 1 - 0 Networks. After re-routing the 10G-bit fibre down the correct
orifice we anticipate fewer bouts of thuggery from local wildlife. Amazingly,
and thanks to stunningly good service from our local network engineers, the
APM building was only disconnected for a few hours after a quick switch back
to the old 1G-bit fibre while a new 10G-bit fibre was installed.
PS. The IoA is getting an additional 1G-bit link to CUDN; 1+1 = 2G-bit.
ADASS:
ADASS proceedings. RWA has made the last list of corrections suggested
by ASP and (hopefully) the final version will be sent back in the next
few days.
Hardware:
MJI acquired and installed an additional 14 Tbyte RAID6 fibre channel disk
system, bought at a discount rate, and will now investigate the mysterious
incommunicative raw WFCAM data disk currently known as "you ****".
APM air con:
The successful partition installation was followed up by a rearrangement
of the main room air conditioning units, by relocating one of them to the
store/computer room. This gives a much better balanced system and means
we can pack away our winter work gear.
MJI pointed out the upcoming Ringberg meeting, "Classification and discovery
with large astronomical surveys" to be held 14-17 October. The abstract
submission deadline is 30th June so if anyone is interested in attending
they need to unplug their dyke (see http://www.mpia.de/class2008/).
MJI briefly reported on the latest news in the CASU grant saga and on related
grant applications either recently submitted or in the pipeline.
Continuing actions
------------------
STH check that Plone astrometry technical pages are correct
MJI acquire transmission curves for VIRCAM N118 filter
JRL revive changes log web page for WFCAM pipeline
MJI continue stirling efforts to find out more re: VIRCAM commissioning plans
STH include graphic showing the size of the photometric radial distortion
correction in the Plone technical pages.
MJI order some suitable tables for the meeting area
New actions
-----------
MJI check the connections on LCD TV to see if displaying from laptop works
and measure external dimensions
MJI check with EGS what deep-stacking work has been promised for Video
MJI see if JPE is amenable to sending us presentation from Video meeting
STH investigate photometry affected by nebulosity in HawkI data
MJI
JRL produce web page of HawkI publicity for CASU
MJI update pipeline toolkit on Solaris cluster
MJI investigate ways of implementing photometric corrections for flat
file users
JRL decide if flipping jitters telecon with JAC is needed
JRL implement flagging of summit-rejected MSBs during raw WFCAM data ingest
MJI implement MSBTid logs for 08A as a trial
MR
