Wednesday 21st October 2009 APM Meeting tables 11:30-13:00 Present: MJI, RWA, EGS, JRL, JPE, MR, SC, STH Apologies: NAW, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST ..... 6. WFCAM update 7. VISTA update 8. AOB - grant status etc.. D'oh! ----- MJI forgot to officially welcome JPE to the CASU meetings and also to congratulate him on his choice of the IoA for his Sabbatical. Minutes ------- 1. Actions from the last meeting MJI stage I software release on hold due to other pressures - ongoing <<<< EGS MJI ditto technical paper - ongoing <<<< MJI ditto utility to convert binary tables plus corrections - ongoing <<<< JRL duly wrote his ADASS talk and delivered it in multiple ways STH briefly described the Palomar Transient factory. Managed by Nicholas Law, it uses the 48-inch Oschin Schmidt to do an optical survey of the sky in 8 square degree snapshots and has so far produced two papers. MJI still comparing WFCAM dome flats with twilight sky flats. Low light <<<< level linearity sequences have also been taken for both WFCAM and VISTA and it will be interesting to see how the linearity properties compare to WFCAM EGS has made the Plone system entry to the top level pages for VISTA similar to that of WFCAM EGS checked out and updated the WFCAM survey progress web pages RWA has inserted the VISTA performance specs emailed out by Magda on the VISTA technical pages SC said that the Chubb engineer has not yet visited the Kavli so no progress on the high temperature monitor. Another discussion with Paul Aslin is required. <<<< 2. Comments on the WFAU minutes DR6 happened about a week ago. There are still some holdups with the DR6 version of GPS and UDS, partly due to us and delayed reprocessing and partly due to one of life's eternal mysteries. WFAU are getting a bit itchy and scratchy about up to date examples of processed VISTA FITS files and after a brief discussion EGS was volunteered to make sure the night of September <<<< 27th (all calibration pawprint OBs as it happens) is available for slurping after it has been checked. After much CASU itching and scratching the format of the FITS tile images has finally been decided upon and MJI will ensure an example of this wends its way down the UKlight fibre shortly. <<<< It was noted that all the VISTA FITS files magically adhere to the FITS standard, even the catalogues. 3. Recent and upcoming meetings There was a VISTA IOT on October 6th in which among other things science verification and scheduling was discussed. SV started on October 15th and runs until the end of the month. ESO SV team members are attempting to produce science quality data at the telescope but JRL thought this was not going to happen without a lot of extras. We expect the first batch of SV data at the end of October and will process the data here from scratch as per usual. The plan is then to visit ESO late November to discuss the data processing science outputs directly with the ESO SV team. JRL reported that his ADASS trip was very successful, particularly in discussions after his talk when he was subjected to many questions about VISTA and the CASU pipelines. He also had many useful conversations with ESO attendees and even better, his mysterious back injury dissipated through a combination of immobility and Sake. MJI and NAW went to a EUCLID meeting in Durham. Although this is notionally a Dark Energy mission (weak lensing, BAOs etc..) its proposed three cameras are of interest to a range of auxilliary science, particularly a near-IR imager which will scan 20000 square degrees of high latitude sky down to 24th mag in Y,J,H (and possibly Ks) in 5 minutes per shot (JPE held head in hands here, but cheered up when he heard that the earliest possible launch is 2018). MJI, NAW and STH are investing a modicum of effort to keep an eye on this and PLATO, the other M-class space mission of interest. STH noted there was a PLATO meeting in early December in preparation for a funding bid to STFC for a Phase B study. The same is happening for EUCLID. 4. Data archives AAT, ING and WFCAM are up-to-date as usual. The Cassegrain UKIRT archive still awaits its final putsch involving (re)acquiring 2 to 3 dozen nights to complete the data archive. JRL promised to finish this by Christmas [which is odd since MJI recalled the same IoU in last year's minutes]. 5, Optical /NIR processing There is a lot of optical processing happening all directed at science research projects of various CASU individuals. Currently under the kosh are data from : Subaru SuprimeCam; KPNO 4m; CTIO 4m; and about to hit the road more PAndAS MegaCam survey data from the 2009 Autumn season. Ironically, VST rumoured to be aiming to begin surveying November 2010, will produce less data for CASU to process than the current crop of optical imagers already being dealt with. 6. WFCAM update WFCAM is back taking data after its two week vacation (much appreciated by all) and as usual, after all the loving care and attention it received, came back with a few extra bonus features. MJI was asked to check the effects of an increased reset anomaly on #4 seen in the darks (~10% bigger effect near edges) on the science products. Although no noticeable impact on processed science images, while checking this he noticed extra pickup noise on #3, but not the other detectors, readily visible in the science images. This visibly does not have the usual quadrant symmetry seen in the curtaining effect, otherwise it would have been removed in that part of the correction process. This took a while to intercept since a full set of broadband flats were not available until 20091015 due to uncooperative weather at UKIRT. Processing recommenced last week and should soon catch up to the incoming data flow. [Note added post meeting: after informing JAC of this new feature JAC staff tracked this down to pickup from a cooling fan in an electronics rack and a workaround was implemented to remove the problem as of 20091021.] MJI said that as of 20091001 processed data he had finally implemented the udpated CFITSIO Rice compression software for 16-bit integer data (using CFITSIO v3.14) now that most recent versions of DS9 correctly uncompress these on-the-fly. This is only relevant for the confidence maps. The most recent versions of CFITSIO and DS9 also seamlessly uncompress the earlier Rice compressed 16-bit data (which was actually converted to 32-bit integer during the compression process). The 16 bit-specific compression is slightly more efficient and produces slightly smaller (by a few %) compressed files. MJI noted the upcoming "six-monthly" UKIRT Board Meeting in early December and needs to compile a report on these and other matters as usual. <<<< Reprocessing is on hold while MR takes a well-earned pipeline break and also because of VISTA PR colour image massaging. The reprocessing queue is down to some nebulous GPS data and the the UDS J,H from 05B-07B. From 09A, UDS data gets its object masking pawsky fix automatically during pipeline processing. EGS produced at short notice some nice summary statistics on the various UKIDSS surveys for Steve Warren. By some miracle this agreed exactly with the WFAU stats confirming that transfer and ingestion does not suffer from a Black Hole in the UKlight network in the Leeds area. 7. VISTA update JPE hand delivered 5 USB disks of VISTA PR data containing nights from 27th September to 7th October to IoA on Sunday 11th October, where after some interesting shenanigans involving a fridge, some blocks of metal and much patience, it was read on for immediate processing. In parallel a speeded-up offcial copy of this arrived via the usual ESO delivery five days later. Virtually all of this is PR data and is, as expected, stress testing the pipeline and its occupants to destruction. As acceptable PR images sputter out the farside they are being made available to Richard Hook and the ESO PR team for suitable mastication. We always expected processing this data would be far worse than doing science with VISTA and indeed it has surpassed our expectations. JPE noted that as a breather prior to SV data arriving, the next ESO disk would be mostly air guitar data. The PR processing has yielded some nice images of the Fornax Cluster and the Flame Nebula in addition to the GC and M55 pictures already in the can. Various other PR images are still being worked on and assessed with the deadline for finishing off these the end of October in good time for handover. JPE described the ESOcasts which are propagated around the site (see http://www.eso.org/public/archives/esocast.html) and showed some of the publicity images. MJI said he was keen to get the PR exercise out of the way before the SV data comes in. JRL pointed out some problems with flat fields due to the close proximity of a 5th mag star whose diffraction spike was occasionally in the field of view when the telescope was jittered, so care needs to be taken when co-adding flats. Another problem arises when taking short exposures. Steven Beard produced a routine for calculating exposure times for given count rates but it does not allow for readout time (1 second) so at short exposures it is easy to overexpose the frame (particularly for #5). A workable rule-of-thumb is that if the flat field exposure time is less than 5 seconds to give ~10K ADUs/pxiel then the sky is too bright, so do not use short exposures or those already obtained which are affected. It was also noted that we would like flats taken in the half hour before sunrise to compare with the sunset twilight flats. JPE confirmed this was not happening at present. WFCAM suffers from serious persistence problems hence the pre-dawn twilight flats. Athough we expect VISTA to not suffer as badly from persistence effects as WFCAM, testing the characteristics on sky is still necessary. Quite a few nights of data had split OBs i.e. where an OB was interrupted and then restarted from where it left off. This currently happens when the AG drops out and JPE said that it was possible to restart a pawprint if this happens in the middle of the procedure. Unfortunately the headers do not reflect the connectivity between partial OBs in cases like this and JRL will have to implement his usual WFCAM-like association process as a workaround. The dome lights have been upgraded but not used in anger. There is some confusion about their readiness for use but since the last usable linearity sequence was taken way back in June and many things have been adjusted since then a new set of measures is highly desirable. JRL said he had already enquired about this and this topic was also raised at the recent IOT telecon and is high on our wishlist. (It turns out that the linearity sequences taken in August that were not quite deep enough for purpose were also meant to be used as a test of the stability of a trial light source. JRL will investigate.) <<<< The recent successful attempts to speed up delivery of VISTA data via ESO to CASU prompted suggestions that this should be repeated for SV data if possible. JRL noted that offset sky observations are still causing problems in associating desire with pipeline reality. Steven Beard's offset sky template, which is designed to circumvent these problems, cannot be tried until the camera software is rebuilt to include it. This is unlikely to occur until after SV is completed. MJI provided a brief mosaicing update starting with the different mosaicing/ tiling software. Up to now we have been using Montage for trialling. This generally works well but is a tad minimalist (and non-standard WCS) in its headers and also produces 64-bit floating point data. In parallel we have been developing (upgrading) our own mosaicing software to cope with VIRCAM tiles (96 correlated offsets to compute) and produce valid headers and 32-bit output. This is being stress tested by using it to produce the tiles for the colour images. JPE said that the Verifications Status review panel was worried about possible time-dependent astrometric distortion in the focal plane and asked for it to be monitored by the QC pipeline. MJI commented that during normal processing we continually monitor the astrometric performance and where required generate residual distortion patterns by stacking lots of pointings. He is generating some for the various filters anyway to assess wavelength- dependence of the distortion pattern. <<<< STH is checking colour equations and throughput and re-affirms his agreement with the numbers on the webpage. He confirms that the low value for WFCAM (16%) in the bluer passbands is correct and that this is mainly due to the much lower QE of the detectors. Photometric illumination corrections for Z,Y,J,H,Ks will be handled in a similar way to WFCAM i.e. computed from stacked 2MASS photometric solution residuals. However there are two sets of narrow band filters in the camera, that in addition to standard illumination corrections also have variable bandpass as a function of position in the field: the NB118 - that we knew about; and a hybrid NB980/985 split half and half over the detectors - that we didn't. The latter have never been exposed to on-sky light and Tim de Zeeuw has allocated 3 nights for them to be assessed. MJI asked about acquiring the filter transmission curves for the NB sets for feeding into the CASU ETC. JPE will send MJI this filter data. <<<< Whilst on the topic MJI reiterated that the CASU ETC (v1.4) gives plausible nos. for the broadband filters and that apparently the largest differences with the ESO ETC for VISTA are due to the way the expected sky background level is computed in the ESO version. Whilst rooting around with ETCs MJI noticed that the previous version (v1.3) was still directly accessible and had the system folk redirect the link to the latest version. MJI said that the stability of the filter wheel positioning was still being monitored. EGS is examining the generated sky correction frames for the tell-tale signature. The last time a problem was noticed was September 29/30. JPE commented he meant to re-datum the filter wheel each night to monitor accumulated drift but did not always do so. <<<< MJI reported on a slight panic about micro-stepping after a query from the VIDEO PI. Apparently the VISTA user manual is somewhat confusing on this point. He reiterated the CASU recommendation to not use micro-stepping and instead "drizzle" carefully chosen dither sequences onto a finer grid under-sampling is a concern. The experience with micro-stepping with WFCAM has been mixed. The basic problem is that the PSF is not stable on short timescales so you don't correctly sample the PSF by microstepping. One driver with WFCAM was to avoid pixel interpolation schemes in the processing so in this case microstepping is a viable option. With VISTA there is no choice, due to the larger field distortion you have to interpolate when combining deep stacks anyway so you may as well interpolate the stacks onto a finer grid if you so choose. STH said that he would look at image quality again when more recent good quality processed data is available. JPE expected that some of the SMC data taken October 5-7 would be suitable and JRL noted that this would soon be available. JPE said that VISTA's EED50 had been formally accepted as within specification, but EED80 was not and required a waiver. VPO were required to provide a report on EED over range of filter, alt, az, reversal positions and wind conditions in good seeing (without M2 micro-oscillations). There followed a discussion on the SIQ targets and how they depend on assumptions about the PSF profile. <<<< STH was keen to see trial survey OBs before they are used in anger and suggested that we should have access to them at the same time as ESO otherwise it is too late. JPE commented that ESO would be unlikely do this but that the PIs could anyway, however he would ask ESO. <<<< MJI and JRL are still chewing over Rice compression options for VISTA processed data - a recent poster paper presented at ADASS by Bill Pence is relevant - but almost certainly will plump for the WFCAM style. This means that all the current VISTA processed images are still uncompressed. 8. Any other business All of the GAIA hardware is now running and MJI said that 2 or 3 out of the 4 A/C units were running at any one time giving some welcome redundancy in capacity. Both JRL and EGS have had problems with mains spikes on the power supply to their PCs recently. A spare anti-surge unit is available and could be installed in their office upon receipt of suitable quantities of beer. <<<< MJI asked SC to investigate alternative backup strategies on non-cluster machines (i.e. apm 05, 25, 26, 28 etc..) for the next meeting. <<<< After much hassle with the rolling grant MJI noted that we are finally in a position to advertise for the vacancy required to properly cover VISTA pipeline processing. The advert is expected to be placed shortly on the AAS job register with closing date the end of November, interviews in early January, aiming for a start 1st April. Continuing actions ------------------ EGS add catalogue software and binary table conversion software to MJI software release pages MJI finish off technical paper updates MJI write utility to convert binary tables with all corrections MJI compare WFCAM dome flats with twilight sky flats SC progress the high temperature monitor installation New actions ----------- EGS email WFAU about VISTA processed data for night of September 27th MJI do the same with an example of a tiled image MJI compile and write the usual UKIRT Board report JRL investigate stability of trial light source from August data MJI assess wavelength-dependence of VISTA astrometric distortion JPE send MJI the NB filter transmission curves EGS monitor sky frames for signs of filter wheel problems STH reassess SIQ on the SMC data taken in good seeing JRL investigate alternatives for Rice compressing processed VISTA images MJI a la Bill Pence JRL supply some beer if they want the mains filter installing EGS SC investigate some strategies for backing up non-cluster machines