You are here: Home / Documents / Minutes of CASU Meetings / minutes_080332.txt

Plain Text icon — Plain Text, 15 KB (16358 bytes)

File contents

32nd March 2008  Hoyle Committee Room  10:00-12:15

Present:          STH, PSB, MJI, DWE, JRL, MR
Apologies:        RWA, NAW, RGM, EGS

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent meetings/visits   - Leiden UltraVista
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
                             - new archive server system
5.  Optical processing update- VST update
                             - revised IDR IPHAS paper
6.  WFCAM update             - photometric calibration
                             - ftp and tape transfers
                             - processing status 08A
7.  VISTA update             - linearity analysis doc
                             - filters, and filter responses NB118 and NB++
	                     - new SADT release
                             - new calibration plan, lab tests, commissioning
                             - pipeline progress
8.  ADASS 2007
9.  AOB                      - STFC programmatic review


Minutes
-------

1.  Actions from last meeting

STH  has checked the advertized illumination correction information using
     LAS for overlap tests and found that the same correction is required
     for both stars and galaxies whatever the magnitude.  MJI reckons this
     "proves" it is a genuine illumination correction but JRL bet MJI a 
     pint (is that all!) that these were really a reset anomaly problem
     in disguise.                                                          BET<

ALL  some have provided comments on the photometric calibration paper. A 
     final set of comments were requested before the paper gets 
     submitted - ongoing                                                   <<<<

JRL  is halfway through checking why the pipeline catalogue software and 
     toolkit give different results in a few specific cases - ongoing      <<<<

MJI  checked that the Plone technical pages (WCS howto) were correct.
     Further discussion led to uncertainty whether the Plone astrometry
     technical pages were correct. HTML errors were also being reported.
     http://casu.ast.cam.ac.uk/surveys-projects/wfcam/technical/astrometry
     eg. for ZPN, is PV2 being used rather than PROJP? STH was tasked with 
     finshing this off.  MJI will supply the relevant email                <<<<

MJI  has asked JPE for the transmission curves for VIRCAM Z and N118 filters,
     but has not received them yet.  These are needed to complete the ETC
     tech spec deliverables and finish off the prototype ETC - ongoing     <<<<

JRL  has backed up the UKIRT cass archive

MJI  has not checked what SDSS do about elliptical apertures - ongoing     <<<<

PSB  has sent off a wishlist of observations for the next cold VIRCAM run.

RGM  have acquired most of the  versions of the SMPs for the VISTA 
MJI  public surveys but are unsure of the status (ie. are they the final 
     version ?) of some of them - ongoing                                  <<<<

PSB  has investigated the new colour printer options and will conduct 
     further trials when the IoA has acquired its latest machines.


2.  Comments on WFAU minutes

No substantive comments though JRL noted the WFAU preparation of a response
to the STFC programmatic review. 


3.  Recent meetings/visits   - Leiden UltraVista

MJI and JRL part attended this meeting and gave presentations on the
plans for processing VISTA data.  Several issues came up during discussion
not the least of which were an understandable desire on the part of the
UltraVista team to have access to VIRCAM bad pixel maps to decide on
the detail of their dither offset strategy (a series of "random" offsets
within a 30 arcsec x 30 arcsec box).  MJI commented that for any
reasonable dither pattern within an OB, image resampling during stacking
was inevitable and that CASU currently planned to offer three alternative
interpolation schemes: nearest-neighbour; "dribble" a variant of drizzling; 
and an optimal cubic MOMS (eg. Blu, Thevenaz & Unser 2001) - subject to
further testing and feedback.  

This also raises the issue of the correct strategy to adopt during computing
accurate dither offsets from the on-sky data.  Because of the large radial
distortion overall delta_TPA and _TPDs for the entire array are required
and will be implemented in the pipeline stacking (the WFCAM pipeline 
currently uses pixel offsets within a dither stack).

One further important concern raised was the overall transfer speed of
raw data from Paranal to Cambridge via Garching, which might take up to a 
month.  Improvements at the Paranal end with a dedicated G-bit fibre 
connection will eventually offer possible alternative transport schemes but 
not for the first year or so of operation.  The usual alternative chestnuts 
to the 12 x 250Gbyte disk system proposed by ESO were aired but it is unclear 
at this stage how to progress further discussion of this.

MJI noted that all the talks for the UKIDSS public survey meeting held in
Garching in December 2007 were now available online at
http://wiki.astrogrid.org/bin/view/UKIDSS/Dec07Workshop.

STH commented that a planned VDMT meeting in March failed to materialise
but that he would no doubt be in touch later for updates for its future
reappearance.


4.  Data archives update     - AAT, ING, WFCAM, UKIRT
                             - new archive server system

JRL reported that the entire AAT archive has been backed up and sorted
onto one disk system. In the process he discovered some duplication in
the AAT data and has resolved that.  The UKIRT CASS archive is still backing
up even as we speak and has also been rearranged onto one disk system. 

ING transfers continue as normal and data up to the end of February has been
checked.  March data is awaiting a similar treat.

WFACM raw data updates for 08A are paused awaiting completion of installation 
of the new archive server system, as are ESO transfers.

MJI clarified for himself that apm25/apm26 will now be used solely for
the CASU plone web site, QC DB and survey progress pages.  The IPHAS IDR DB 
will be migrated to the new server.

PSB and JRL noted that the new archive server system (apm5) has been 
installed and is now in the shakedown phase.  When verification is complete
this will become the live interface replacing the current system.


5.  Optical processing update- VST update
                             - revised IDR IPHAS paper

MJI said that he had been informed that "the VST telescope is being integrated 
on Paranal and this phase should be completed this April.  Primary and 
secondary mirror support are still under revision, and INAF foresees the 
final integration in the telescope structure on Paranal starting end 2008 and 
completion by early 2009".  So, we may see some VST survey data in 2009 then.

EGS has revised the IPHAS IDR paper in light of the referee comments and
this has now been resubmitted to MNRAS.  MJI noted that he had seen some 
impressive on-the-fly demos of cross-matching of other large catalogues 
with the IPHAS IDR using the Astrogrid toolkit.  PSB commented that all
you needed was an EGS to construct the hieroglyphs.


6.  WFCAM update    
                    
07B status: all of the remaining 07B raw data had been transferred across, 
processed, checked, and copied up by WAFU.  

The reprocessed UDS data from 06B is also now in its final resting place.

Photometric calibration: a close to final draft of the photometric calibration 
paper has been generated.  Part of the changes relate to the tests noted 
earlier on the illumination correction.  This has been circulated around the 
authors for comment.

Acquisition of raw WFCAM data: PSB reported that after some set up and
debugging, automatic copying of data from UKIRT is working routinely.
Each night's data is placed on a JAC summit server as 1-chip Rice compressed 
FITS and a flag set; a cron job in Cambridge scans the flag and mirrors the 
data.  Transfer speed is around 8 MByte/s with a night's data taking typically 
two hours to transfer.  This currently uses a 32 thread transfer protocol, not 
ideal since it clobbers the server end more than necessary, but will do for 
now in lieu of further server system tuning.

At the time of writing (2nd April) WFCAM data up to and including 31st March 
have been transferred this way bar the first few weeks of February. These
eventually arrived on tape mid-March (well almost, data for one detector for 
20080221 arrived via ftp last night).

Data still arrives on a per detector basis and is checked and factored into 
MEFs as usual.  Since all WFCAM data from now on should arrive via ftp we have 
instigated writing our own archive tapes (currently using LTO2s) from the 
MEF'd raw 08A data as an off-line backup. Previously we just saved the 
incoming tapes.

08A processing: the sudden influx of 08A data was timed perfectly to coincide 
with the Easter break but processing has now commenced.  MJI commented that 
the parquet floor pattern was still present, at least in the twilight sky
flats, necessitating a cunning strategy to produce master flats.  Proto-
master flats are created in the usual way and are then flatfielded using
previous good master flats as a reference but sans gain correction.  The 
parquet floor pattern on #2 and #3 then shows up clearly and is removed with 
the usual bespoke software.  Flatfielding^{-1) the output of this process, 
also sans gain correction, with the reference flat then restores the
parquet floor-free new master twilight flats.  08A master flats for the 
February-March period for Z,Y,J,H,K,1-0S and BGamma have been produced this  
circuitous way.

Software issues: JRL said that he would revive the changes log for the WFCAM 
pipeline which seems to have been terminated with the recent changes to the 
internal plone web pages                                                   <<<<

MJI asked JRL to close the loop on the discussion related to offset sky 
strategies and pipeline requirements for keywords, before we encounter
08A data with this feature implemented.                                    <<<<

EGS noted that he has been asked by people at the IAC, Tenerife for a copy 
of all the UKIDSS processed imaging data. He has asked them to contact WFAU 
with their request since this is the proper access channel. 

This reminded MJI of a recent email response from JAC which indicated that 
all processed WFCAM data would be subject to the same proprietary periods as 
raw data.  This includes UK PI as well as survey and campaign data, plus 
Japanese PI data and possibly (TBC) UofH PI data.  This raises an interesting 
question: while the provenance of the raw data is relatively straightforward, 
who owns the "copyright" of the processed data ? and what does processed mean 
in this context, flat files output by the pipeline ? + extras like band-merged
products ?

As a bit of light relief MR and STH presented some interesting results from 
their investigations of long term trends (spatial and temporal) in the filter 
zero-points and the night sky brightness.  This is being worked into MR's sky 
properties paper, a draft of which will be forthcoming later, honest.


7.  VISTA update       

Filters: PSB noted we are still waiting to acquire the NB118 & Z filter 
transmission data to finish off the ETC technical specification deliverables
and complete the prototype ETC.

The issue of the on-sky calibration, including illumination correction, of 
the NB118 filter was raised at the UltraVista meeting in Leiden.  This is 
complicated by the variable bandpass of the filter over the field of view 
- the wavelength shift is a significant fraction of the filter FWHM.  The
same worries apply to other, more exotic, narrow-band filters that have
been suggested.  MJI posed the question of who is responsible for accurate 
(flux) calibration of specialist NB filter data, since experience with 
equivalent NB optical calibration shows it is a non-trivial task.

The recent release of the SAD Tool and related documents was discussed;
unfortunately we did not have a chance to contribute or comment before the 
deadline. It was noted that SMB had made a number of detailed technical 
comments. There is still some concern about how an investigator chooses the 
value for the RECIPE keyword (essential for offset sky reductions and tile/ 
pawprint stacks resampling/interpolation). We look forward to an end-to-end 
test from SADT to FITS file to see what comes out.  A forgotten action from 
the last minutes was to download the latest SADT and see what tiling patterns 
etc. are generated.  STH was volunteered to look at this since he reviewed 
earlier versions.                                                          <<<<

Oliver Twist: the need for more prep-room data was discussed. There has been
no feedback following the previous request which accompanied the updated 
Calibration Plan in mid-February. Issues to be analysed include filter 
repeatability and/or tartan patterning, linearity and/or persistence, 
production of high-fidelity bad-pixel maps ...... usw.  Given the amount of
discussion the recently released linearity report generated, the sooner
we get some more test data the better.

Looking further ahead, it is hoped that now M1 is safely in the enclosure,
a firm commissioning schedule will become available, this is needed by CASU
to plan development and data handling.  PSB wondered if it might be
possible to beg more bandwidth from Paranal during commissioning, given
the minimal impact the 2-hour 2Mb/s window used last summer had.  MJI will
attempt to find out what the schedule is and in particular what the 
commissioning plans are                                                    <<<<
			     
Software:  JRL commented that there have been no major software updates just
a few bug (surely not) fixes.  Having warmed up, JRL then noted that
in processing some of the SV data from HawkI on the VLT he had espied
OBs taken with offset skies included in them.  Naturally, he was curious
why this capability was ruled out for VIRCAM OBs. 

EGS has released a preview (flashy! pas moi) version of the survey progress 
and QC tool which addresses several comments received.  The current version 
is still active but will be superceded by the new one.


8.  ADASS 2007

The finances for the event have been signed, sealed,,and profit delivered.
There is a still a small amount of editorial effort required before the
tome gets kicked out of the door but at least it should be close to being 
shifted before the next ADASS announcement goes out.


9.  AOB                      

MJI filled in details of what we knew so far about the outcome of the STFC 
programmatic review and potential meanings of various official statements.
He also went through the grant review feedback in an attempt to second
guess where things were heading.

Construction work on the partition in the APM building main room has begun.
In theory this should not be too disruptive since it is basically a 
carpentry and glazing job, but we will see.

It was recently spotted that there is no overtemperature alarm in the APM 
building (other than the ultimate ones in the ceiling), unlike the rest of 
the critical computer rooms in the IoA.  We will see if collective digits
can be removed to expedite installing one or more.

JRL volunteered to host a spring barbeque. His prefered date would be a
Sunday. He will propose a few alternative dates for us to consider.        <<<<
Pete will supply (or was that drink) the beer.


Continuing actions
------------------

ALL  provide comments on latest draft of photometric calibration paper

JRL  finish off checking why pipeline catalogue software and toolkit give 
     different results in a few specific cases 

MJI  acquire transmission curves for VIRCAM Z and N118 filters

MJI  check what SDSS do about elliptical apertures

MJI  finish off checking and acquiring the latest SMPs
RGM


New actions
-----------

STH  check that Plone astrometry technical pages are correct

JRL  revive changes log web page for WFCAM pipeline

JRL  check that pipeline can cope with offset sky strategy

STH  download, install and investigate new SADT

MJI  find out latest VISTA schedule including commissioning plans
			     
JRL  suggest a few dates for proposed spring barbeque