16th April 2009 APM Meeting tables 11:30-13:00 Present: MJI, JRL, STH, MR, EGS, RGM, SC, RWA Apologies: NAW, DWE Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- RWA has taken over responsibility for arranging monthly group meetings and has also taken over as minute-taker extraordinaire. Thus ....... 1. Actions from the last meeting STH check that Plone technical description pages are up to date <<<< MJI RWA will help edit the content, needs edit access sorting out. JRL reported that he had a beautiful image from HAWKI and STH said he STH is designing a website to put it (and others like it) on. He <<<< will also add something about the HAWKI photometry. There is a zoomable browser used for Spitzer images which seems to work very well and which emphasizes the splendour of the HAWKI images which are of very high quality. JRL has re-processed all the data now and MJI said he would look at the photometry again. <<<< MJI finish off stage-I software release pages. <<<< EGS MJI finish off technical paper. MJI said he was not going anywhere <<<< in May and would devote a week of time to this task. RGM liaise with JRL about processing HAWKI PI data - continues <<<< JRL redid the HAWKI NB image data processing to improve sky subtraction JRL are still sorting out remaining VISTA documentation set and making STH sure latest copies are available on internal web pages. JRL has all RWA but one current document in place, the ETC. The current version cannot be found online. However, the changes from the previous version are minor and will be added. MJI asked for all the docs and pdfs to be placed on the internal web pages. JRL exhorted everyone not to modify them and noted that the lead author should be responsible for any updating. There is still a lot of stuff in PSB's home directory not completely sorted out - ongoing <<<< MJI write utility to convert binary tables with all corrections applied and dump as ASCII file - ongoing <<<< JRL send latest version of cataloguing software to WFAU - not yet done <<<< MJI contacted JPE about CASU partaking in VISTA IOT March telecon NAW submitted abstract for CASU talk at JENAM. EGS is also giving a talk JRL thinks he implemented the sky method FITS keyword in current pipeline and just had to back propagate it to all the 09A data <<<< EGS inquired what language UVEX team are using for coding. The answer is C and he will provide an example demonstrating reading from merge <<<< binary table. NAW send polite but firm email to IPHAS about VIZIER data release ?? <<<< MR is chasing up wherewithall to produce duplicate WFCAM transfer script, nice though it is to still get emails from PSB. <<<< STH will inquire about possibility/status of new WFCAM linearity measures when he goes out observing next week or so <<<< MJI forwarded UDS test results to JRL STH figured out the problem causing the apparent non-linearity when comparing VISTA with 2MASS: pawprint skies from limited dithers + object masking + big aperture = FU 2. WFAU minutes MJI noted there have been two sets of minutes since the last CASU meeting. DR5 happened on April 8th, GPS & UDS are not yet included but will be at some date still TBD. WFAU have transferred and ingested all the 08B data. Prior to DR6 the assorted Petrosian et al. fluxes and radii for blended objects need fixing for 05A-08A, 08B and 09A -> already have the fix applied. Timescales for DR6 are still uncertain. RGM noted no LAS objects appear to be flagged as saturated (i.e. with class = -9) in the DR5 data release. MJI commented that this was an issue raised well over a year ago and the solution put forward then was to use the detector-level saturation keyword/value in the FITS headers, together with the peak intensity of an object and the local sky level to decide which detected objects are likely to be saturated. This is deliberately not done in the catalogue table entries since the processed image saturation level is not uniquely defined, making this a "fuzzy" decision. CASU set this flag during band-merging using a simple conservative recipe. MJI recalled some email discussion with WFAU along these lines a few years ago but couldn't recall the outcome. 3. Recent and upcoming meetings There was a VISTA Instrument Operations Team (IOT) telecon on March 26th. MJI and STH partly attended while JRL sat through the whole thing. CASU were asked to produce a brief report on their analysis of the VIRCAM data taken prior to the end of January and JRL was asked about his advice on how to proceed with enabling an offset sky strategy. Ideally this report should be put together prior to the next IOT meeting scheduled for the 28th. MJI will mainly deal with astrometry, STH photometry and JRL image processing aspects. This needs to go be sent to JPE/VPO in plenty of time for the IOT meeting, which means in practice by Friday 24th if possible. <<<< The minutes of these IOT telecons will be placed on the VDFS internal web pages (VISTA/misc). <<<< STH and NAW attended a PLATO meeting in Aarhus, but with beer at 8 quid a pint there the travel budget has shrunk considerably. The same duo will attend a PLATO consortium meeting on June 5th in Rome and while tpying these minutes news came in of a PLATO Science Consortium meeting in Belfast on the 12th June, a busy time for Platonics. EGS reported nothing of note happened at the Euro VO workshop last week held at ESO, Garching. STH had a week-long meeting with folks from ASTEP who seem happy with our contribution = astrometry and photometry sorted - not much then ! They have 8 x 8 degree fields taken unfiltered in the optical with exposures every 30 seconds, even with the Sun above the horizon, and also a nice movie of the Aurora Australis. Used TYCHO2 for the astrometry. Summary of this, including movie, needs putting on a webpage, somewhere near you. <<<< 4. Data archives AAT: data trickles in INT: more of a stream - usual occasional header problems, legal ASCII characters set, moi! WFCAM: data transfers (in and UKIDSS/CAL out to ESO) and ingests continue apace. MR noted an uncommon recent request for raw data, from Jim Geach, again. UKIRT: 95% of DLTs re-read - just a few composite DLTs and LTOs to absorb. JRL noted the latest batch of CASS data has yet to arrive. 5. Optical/NIR processing : HAWKI, INT WFC, MegaCam, Subaru HAWKI - reprocessed, recatalogued and awaiting magic from MJI INT WFC - not much to report, processed some SNR data for Albert Zijlstra Megacam - plus ca change SUBARU - ditto IPHAS - ditto UVEX - processed and ingesting to database. EGS is dealing with export of data products to UVEXers. 6. WFCAM update Semester 09A has been dominated by rubbish weather but at least this means that it was easier than expected to keep abreast of the processing. At the time of this meeting we were still waiting for nbH sky flats but had decided to press on with the reductions using ones from last Autumn. [By the time these minutes were written, 20090416! produced a good set of nbH sky flats which have subsequently been used to update the cal files for most of the April data.] MR noted that one earlier night, 20090312, had been on hold for a while due to a rather odd set of channel#4 FITS headers. Apart from the 1st 8 header keywords, the rest were all missing. This was eventually bodged by a suitable workaround. MR will check with JAC if the same problem was apparent on the original SDF files. <<<< [e.g. w20090312_00245.fit see README in processed data directory for full set] MJI noted he is compiling a UKIRT Board report for the upcoming May meeting and STH also noted up that he is submitting an update on his WFCAM campaign project. <<<< MJI asked about the SDF->FITS conversion/compression feature - the raw files still crash ds9 though have no other deleterious effects. JRL will check with JAC if the version of CFITSIO used in the conversion has been updated. <<<< MJI reported that WFAU had ingested both 08A and 08B without problems and that he has been talking to Nigel Hambly about encoding in the average confidence levels (for aperture 3) for objects in catalogue flat files, in addition to the Petrosian et al. flux fix for blended objects noted earlier. On the reprocessing front, MJI reported that we have now finished UDS K-band reprocessing up to and including 07B with the plan that this will be stacked by the UDSer's, by sometime in May and make it into the UDS DR5 release. J and H are on hold until the GPS nebulosity filtering exercise, required for GPS DR5, is sorted. MJI verified with JRL that the overall computed offsets per pointing were now routinely being used in the pipeline when dither stacking. This is to eliminate the odd problem noted previously where one detector goes awol while the rest are fine. MJI and JRL noted that Thomas Robitaille (CfA) had rediscovered the not quite complete back propagation of the WCS for ....sf_st'd file component parts (it only goes back one layer). It turned out Thomas wanted to restack (essentially drizzle) the single processed images to get rid of the "hedgehog" PSFs arising in 2x2 interleave stacks. MJI pointed out that there was a simpler solution namely dribbling the final stacks, which works just as well and of course is much faster and less trouble. For the sake of posterity the solution implemented, and used quite frequently on request, in a CASU toolkit prog (fitsio_dribble) is outlined here: ................ If interleaving has been used, we optionally use a drizzle-like algorithm to correct the PSF mismatches that occur in interleaved data, making use of the fact that the real pixels are 0.4 arcsec in size and overlap the neighbouring pixels in a 2x2 (or 3x3 if used) interleaved 0.2 (or 0.133) arcsec grid. This runs directly on the pipeline output interleaved stacks and gives much better behaved PSFs and, of course, correlated noise. Or put another way, each pixel in a stacked set of interleave images (e.g. 0.2 x 0.2 arcsec) actually contains the flux of a pixel that is 0.4 x 0.4 arcsec centered on that position. So in effect, neighbouring pixels actually overlap, and we "dribble" these "big" pixels onto the underlying grid - comprenez ? 7. VISTA update No new VISTA data has been received since the last CASU meeting since VIRCAM has not been up and running on the telescope. Nonetheless disks still relentlessly flow from ESO each containing about 50 Mbyte of log files (of what?) at a transport cost of about 70 pounds per disk. It transpires that the summit pipeline at Paranal will look at the QC info for all the detectors. But due to real time speed constraints it will instead randomly sample the observation sequence rather than examine them all. This is not dissimilar to what is planned for the Garching pipeline. JRL gave us an update on the status of detector#16 and the assorted trials and tribulations he has been through to demonstrate its properties. He has also being doing some extended mosaicing tests with MONTAGE which could form the basis for at least a benchmark tiling algorithm for VIRCAM (still TBD). Related to this is the still open issue of what extra tiling keywords (and inheritance of keywords) will go in the tile files. Whilst he had the floor, he also reminded us of the ancient, and as of yet, still unresolved question of RECIPE keywords for the different surveys/eventual PI programmes, e.g. which of the three on offer interpolation schemes ? At this rate these will be hard-wired in for each of the surveys after consultation with the PIs. VISTA parameters webpage. JRL said this is currently stalled and needs some action from MJI who will look at the VISTA reduced data and make sure that standard summaries a la WFCAM can be generated. <<<< MJI noted a date for our diaries, JPE has booked the RAS Specialist Discussion Meeting slot on 12th November 2010 for VISTA science. 8. AOB MJI said that he anticipated no further news about the grant until after the Budget and is expecting a Healey tax 'em until the pips squeak. He noted that it will be time to write the next rolling grant in 10 months time ! Interviews for the advertised posts have now been concluded and suitable candidates chosen. Start dates etc. are still being negotiated but are unlikely to be for at least another 3 months. Two additional RAID arrays have been installed. One fell off the back of the usual lorry and is otherwise known as /data/apm22_e (14 TByte of actual RAID 6 level storage). The other will be affectionately called /data/apm35_a (30 TByte of actual RAID z2) and is the other half of the i/o for VISTA data for this decade. This one will, after some shuffling be configured to hold the raw data, the other /data/apm29_a after similar concommitant shuffling will house the reduced VISTA data. MJI will sort out the shuffling <<<< Continuing actions ------------------ STH with RWA's help check that the Plone technical description pages are MJI up to date JRL produce web page of HawkI publicity for CASU STH MJI finish off stage-I software release pages EGS MJI finish off technical paper RGM liaise with JRL about processing HawkI PI data JRL sort out remaining VISTA documentation set and make sure latest STH copies of docs and pdfs are available on internal VDFS web pages RWA MJI write utility to convert binary tables with all corrections JRL send latest version of cataloguing software to WFAU JRL check if you have implemented the sky method keyword for 09A NAW send polite but firm email to IPHAS about VIZIER data release EGS will provide an example C program demonstrating reading from merge binary tables MR finish chasing up wherewithall to produce duplicate WFCAM transfer script STH inquire about possibility/status of new WFCAM linearity measures during WFCAM visit Actions ------- MJI finish off HAWKI reprocessed data by nebulosity filtering and recataloguing JRL prepare report for Vista Project Office in advance of next IOT STH telecon on April 28th MJI MJI put IOT minutes on internal web pages STH put ASTEP info on webpage MR check with JAC to see if SDF files have same detector#4 header problem MJI prepare a report for UKIRT Board meeting by end of April JRL check if JAC version of CFITSIO has been updated. MJI produce summaries a la WFCAM of reduced VISTA data MJI sort out VISTA i/o disks