You are here: Home / Documents / Minutes of CASU Meetings / minutes_090416.txt

Plain Text icon — Plain Text, 15 KB (15473 bytes)

File contents

16th April 2009   APM Meeting tables    11:30-13:00

Present:        MJI, JRL, STH, MR, EGS, RGM, SC, RWA
Apologies:      NAW, DWE

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Recent and upcoming meetings
4.  Data archives update     - AAT, ING, WFCAM, UKIRT
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru
6.  WFCAM update
7.  VISTA update
8.  AOB


RWA has taken over responsibility for arranging monthly group meetings and 
has also taken over as minute-taker extraordinaire.  Thus .......

1. Actions from the last meeting

STH  check that Plone technical description pages are up to date           <<<<
MJI  RWA will help edit the content, needs edit access sorting out.

JRL  reported that he had a beautiful image from HAWKI and STH said he
STH  is designing a website to put it (and others like it) on. He      <<<<
     will also add something about the HAWKI photometry. There is a 
     zoomable browser used for Spitzer images which seems to work very 
     well and which emphasizes the splendour of the HAWKI images which 
     are of very high quality. JRL has re-processed all the data now and 
     MJI said he would look at the photometry again.                       <<<<

MJI  finish off stage-I software release pages.                            <<<<
MJI  finish off technical paper. MJI said he was not going anywhere        <<<<
     in May and would devote a week of time to this task. 
RGM  liaise with JRL about processing HAWKI PI data - continues            <<<<

JRL  redid the HAWKI NB image data processing to improve sky subtraction

JRL  are still sorting out remaining VISTA documentation set and making 
STH  sure latest copies are available on internal web pages. JRL has all 
RWA  but one current document in place, the ETC. The current version cannot 
     be found online. However, the changes from the previous version are 
     minor and will be added.  MJI asked for all the docs and pdfs to be
     placed on the internal web pages. JRL exhorted everyone not to modify 
     them and noted that the lead author should be responsible for any
     updating.  There is still a lot of stuff in PSB's home directory not 
     completely sorted out - ongoing                                       <<<<

MJI  write utility to convert binary tables with all corrections 
     applied and dump as ASCII file - ongoing                              <<<<

JRL  send latest version of cataloguing software to WFAU - not yet done    <<<<

MJI  contacted JPE about CASU partaking in VISTA IOT March telecon 

NAW  submitted abstract for CASU talk at JENAM. EGS is also giving a talk

JRL  thinks he implemented the sky method FITS keyword in current pipeline
     and just had to back propagate it to all the 09A data                 <<<<

EGS  inquired what language UVEX team are using for coding. The answer is 
     C and he will provide an example demonstrating reading from merge     <<<<
     binary table.

NAW  send polite but firm email to IPHAS about VIZIER data release  ??     <<<<

MR   is chasing up wherewithall to produce duplicate WFCAM transfer script, 
     nice though it is to still get emails from PSB.                       <<<<

STH  will inquire about possibility/status of new WFCAM linearity measures
     when he goes out observing next week or so                            <<<<

MJI  forwarded UDS test results to JRL

STH  figured out the problem causing the apparent non-linearity when
     comparing VISTA with 2MASS: pawprint skies from limited dithers + 
     object masking + big aperture = FU

2. WFAU minutes

MJI noted there have been two sets of minutes since the last CASU meeting. 
DR5 happened on April 8th, GPS & UDS are not yet included but will be 
at some date still TBD.  WFAU have transferred and ingested all the 08B
data.  Prior to DR6 the assorted Petrosian et al. fluxes and radii for 
blended objects need fixing for 05A-08A,  08B and 09A -> already have the 
fix applied.  Timescales for DR6 are still uncertain.

RGM noted no LAS objects appear to be flagged as saturated (i.e. with class 
= -9) in the DR5 data release.  MJI commented that this was an issue raised 
well over a year ago and the solution put forward then was to use the 
detector-level saturation keyword/value in the FITS headers, together with
the peak intensity of an object and the local sky level to decide which 
detected objects are likely to be saturated.  This is deliberately not done 
in the catalogue table entries since the processed image saturation level is
not uniquely defined, making this a "fuzzy" decision.  CASU set this flag 
during band-merging using a simple conservative recipe.  MJI recalled some
email discussion with WFAU along these lines a few years ago but couldn't
recall the outcome.

3. Recent and upcoming meetings

There was a VISTA Instrument Operations Team (IOT) telecon on March 26th.
MJI and STH partly attended while JRL sat through the whole thing. CASU 
were asked to produce a brief report on their analysis of the VIRCAM data 
taken prior to the end of January and JRL was asked about his advice on how 
to proceed with enabling an offset sky strategy.  Ideally this report should 
be put together prior to the next IOT meeting scheduled for the 28th. MJI 
will mainly deal with astrometry, STH photometry and JRL image processing 
aspects.  This needs to go be sent to JPE/VPO in plenty of time for the IOT
meeting, which means in practice by Friday 24th if possible.               <<<<

The minutes of these IOT telecons will be placed on the VDFS internal 
web pages (VISTA/misc).                                                    <<<<

STH and NAW attended a PLATO meeting in Aarhus, but with beer at 8 quid a 
pint there the travel budget has shrunk considerably.

The same duo will attend a PLATO consortium meeting on June 5th in Rome
and while tpying these minutes news came in of a PLATO Science Consortium
meeting in Belfast on the 12th June, a busy time for Platonics.

EGS reported nothing of note happened at the Euro VO workshop last week held
at ESO, Garching.

STH had a week-long meeting with folks from ASTEP who seem happy with 
our contribution = astrometry and photometry sorted - not much then ! 
They have 8 x 8 degree fields taken unfiltered in the optical with exposures 
every 30 seconds, even with the Sun above the horizon, and also a nice movie 
of the Aurora Australis. Used TYCHO2 for the astrometry. Summary of this,
including movie, needs putting on a webpage, somewhere near you.           <<<<

4. Data archives

AAT: data trickles in

INT: more of a stream - usual occasional header problems, legal ASCII 
     characters set, moi!

WFCAM: data transfers (in and UKIDSS/CAL out to ESO) and ingests continue 
apace. MR noted an uncommon recent request for raw data, from Jim Geach,

UKIRT: 95% of DLTs re-read - just a few composite DLTs and LTOs to 
absorb. JRL noted the latest batch of CASS data has yet to arrive.

5. Optical/NIR processing : HAWKI, INT WFC, MegaCam, Subaru

HAWKI - reprocessed, recatalogued and awaiting magic from MJI

INT WFC - not much to report, processed some SNR data for Albert Zijlstra

Megacam - plus ca change

SUBARU - ditto

IPHAS - ditto

UVEX - processed and ingesting to database. EGS is dealing with export of 
data products to UVEXers.

6. WFCAM update

Semester 09A has been dominated by rubbish weather but at least this
means that it was easier than expected to keep abreast of the processing.
At the time of this meeting we were still waiting for nbH sky flats but
had decided to press on with the reductions using ones from last Autumn.
[By the time these minutes were written, 20090416! produced a good set of
nbH sky flats which have subsequently been used to update the cal files
for most of the April data.]  MR noted that one earlier night, 20090312, 
had been on hold for a while due to a rather odd set of channel#4 FITS 
headers.  Apart from the 1st 8 header keywords, the rest were all missing.
This was eventually bodged by a suitable workaround.  MR will check
with JAC if the same problem was apparent on the original SDF files.       <<<<
[e.g. see README in processed data directory for full set]

MJI noted he is compiling a UKIRT Board report for the upcoming May 
meeting and STH also noted up that he is submitting an update on his 
WFCAM campaign project.                                                    <<<<

MJI asked about the SDF->FITS conversion/compression feature - the raw
files still crash ds9 though have no other deleterious effects. JRL will 
check with JAC if the version of CFITSIO used in the conversion has been 
updated.                                                                   <<<<

MJI reported that WFAU had ingested both 08A and 08B without problems 
and that he has been talking to Nigel Hambly about encoding in the average 
confidence levels (for aperture 3) for objects in catalogue flat files,
in addition to the Petrosian et al. flux fix for blended objects noted 

On the reprocessing front, MJI reported that we have now finished UDS K-band 
reprocessing up to and including 07B with the plan that this will be stacked
by the UDSer's, by sometime in May and make it into the UDS DR5 release.  
J and H are on hold until the GPS nebulosity filtering exercise, required 
for GPS DR5, is sorted.

MJI verified with JRL that the overall computed offsets per pointing
were now routinely being used in the pipeline when dither stacking.
This is to eliminate the odd problem noted previously where one detector
goes awol while the rest are fine.  

MJI and JRL noted that Thomas Robitaille (CfA) had rediscovered the not
quite complete back propagation of the WCS for ....sf_st'd file component
parts (it only goes back one layer).  It turned out Thomas wanted to restack 
(essentially drizzle) the single processed images to get rid of the "hedgehog" 
PSFs arising in 2x2 interleave stacks.  MJI pointed out that there was a
simpler solution namely dribbling the final stacks, which works just as
well and of course is much faster and less trouble.  For the sake of 
posterity the solution implemented, and used quite frequently on request, 
in a CASU toolkit prog (fitsio_dribble) is outlined here: ................

If interleaving has been used, we optionally use a drizzle-like algorithm
to correct the PSF mismatches that occur in interleaved data, making use
of the fact that the real pixels are 0.4 arcsec in size and overlap the
neighbouring pixels in a 2x2 (or 3x3 if used) interleaved 0.2 (or 0.133)
arcsec grid.  This runs directly on the pipeline output interleaved stacks 
and gives much better behaved PSFs and, of course, correlated noise. Or put 
another way, each pixel in a stacked set of interleave images (e.g. 0.2 x 0.2 
arcsec) actually contains the flux of a pixel that is 0.4 x 0.4 arcsec 
centered on that position. So in effect, neighbouring pixels actually overlap, 
and we "dribble" these "big" pixels onto the underlying grid - comprenez ?

7. VISTA update

No new VISTA data has been received since the last CASU meeting since VIRCAM
has not been up and running on the telescope. Nonetheless disks still 
relentlessly flow from ESO each containing about 50 Mbyte of log files (of
what?) at a transport cost of about 70 pounds per disk.

It transpires that the summit pipeline at Paranal will look at the QC
info for all the detectors.  But due to real time speed constraints it will 
instead randomly sample the observation sequence rather than examine them 
all.  This is not dissimilar to what is planned for the Garching pipeline.

JRL gave us an update on the status of detector#16 and the assorted trials
and tribulations he has been through to demonstrate its properties.

He has also being doing some extended mosaicing tests with MONTAGE which 
could form the basis for at least a benchmark tiling algorithm for VIRCAM
(still TBD).  Related to this is the still open issue of what extra tiling
keywords (and inheritance of keywords) will go in the tile files.

Whilst he had the floor, he also reminded us of the ancient, and as of
yet, still unresolved question of RECIPE keywords for the different
surveys/eventual PI programmes, e.g. which of the three on offer 
interpolation schemes ?  At this rate these will be hard-wired in for 
each of the surveys after consultation with the PIs.

VISTA parameters webpage. JRL said this is currently stalled and needs some
action from MJI who will look at the VISTA reduced data and make sure 
that standard summaries a la WFCAM can be generated.                       <<<<

MJI noted a date for our diaries, JPE has booked the RAS Specialist 
Discussion Meeting slot on 12th November 2010 for VISTA science.

8. AOB 

MJI said that he anticipated no further news about the grant until after the 
Budget and is expecting a Healey tax 'em until the pips squeak. He noted
that it will be time to write the next rolling grant in 10 months time !

Interviews for the advertised posts have now been concluded and suitable
candidates chosen.  Start dates etc. are still being negotiated but are
unlikely to be for at least another 3 months.

Two additional RAID arrays have been installed. One fell off the back of 
the usual lorry and is otherwise known as /data/apm22_e (14 TByte of 
actual RAID 6 level storage). The other will be affectionately called 
/data/apm35_a (30 TByte of actual RAID z2) and is the other half
of the i/o for VISTA data for this decade.   This one will, after some 
shuffling be configured to hold the raw data, the other /data/apm29_a
after similar concommitant shuffling will house the reduced VISTA data.
MJI will sort out the shuffling                                            <<<<

Continuing actions

STH  with RWA's help check that the Plone technical description pages are 
MJI  up to date 

JRL  produce web page of HawkI publicity for CASU

MJI  finish off stage-I software release pages

MJI  finish off technical paper

RGM  liaise with JRL about processing HawkI PI data

JRL  sort out remaining VISTA documentation set and make sure latest 
STH  copies of docs and pdfs are available on internal VDFS web pages

MJI  write utility to convert binary tables with all corrections

JRL  send latest version of cataloguing software to WFAU

JRL  check if you have implemented the sky method keyword for 09A

NAW  send polite but firm email to IPHAS about VIZIER data release

EGS  will provide an example C program demonstrating reading from merge
     binary tables

MR   finish chasing up wherewithall to produce duplicate WFCAM transfer script 

STH  inquire about possibility/status of new WFCAM linearity measures during
     WFCAM visit


MJI  finish off HAWKI reprocessed data by nebulosity filtering and 

JRL  prepare report for Vista Project Office in advance of next IOT 
STH  telecon on April 28th

MJI  put IOT minutes on internal web pages

STH  put ASTEP info on webpage

MR   check with JAC to see if SDF files have same detector#4 header problem

MJI  prepare a report for UKIRT Board meeting by end of April

JRL  check if JAC version of CFITSIO has been updated. 

MJI  produce summaries a la WFCAM of reduced VISTA data

MJI  sort out VISTA i/o disks