Wednesday 28th July 2010 APM Meeting tables 11:30 - 13:00 Present: MJI, MR, EGS, JRL, AKY, RGM, NAW, JPE Apologies: STH, JCR Agenda ======= 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent & upcoming meets 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST 6. WFCAM update 7. VISTA update 8. AOB Minutes ======== MJI welcomed AKY (Aybuke Kupcu Yoldas) to her first CASU meeting. Aybuke is currently helping JRL and EGS run the VISTA pipeline. 1. Actions from the last meeting -------------------------------- EGS stage I software release progressed via test build release to WFAU JRL will be released on web pages next <<<< STH still finishing off last bits of calibration plan document update <<<< RGM still testing but wants illumination table option adding to enable <<<< complete end-to-end test. MJI will make it so. <<<< ALL contribute text to go with CASU versions of VISTA PR images. JPE volunteered to contribute some text - ongoing <<<< JPE contact ESO re: signing off pipeline deliverables - still ongoing <<<< MJI made sure that WFAU are aware of tile photometric issues and also produced a brief write up of the relevant bit for the web pages ALL made sure the internal review of VISTA processing was both fascinating and interesting. Unfortunately, the interesting bits weren't fascinating and the fascinating bits weren't interesting. EGS updated the Zpt trend plot and made sure the NB118 was visible too JRL fixed the EXTNAME duplication in the catalogues and sorted out the EGS missing info in the SHDUs of the tile catalogues prior to v1.0 release AKY The ESO QC info and OB constraints have also been added (see version log on web pages for more info) There ensued a brief discussion about the desirability or otherwise of adding a TOTEXP keyword to the tiles. Cons: the info is there in the HDU already to work it out; only EXPTIME matches the MAGZPT keyword possibly causing confusion if two exposure time keywords are present; there is a quite a range (~1-6) of "total" exposure times for each pixel depending on location (e.g. see confidence map); the info is listed in the web page summary files. Pros: you could define an "average" total exposure time; users will get it wrong and/or confused otherwise ! MJI one has started and the other has thought about starting the final SV EGS processing - ongoing <<<< SC investigated VPN for remote access and found it relatively easy to setup SC ZFS sluggishness on simple disk utlities still under investigation <<<< MJI two out of three briefly got together after the meeting and agreed on NAW how to proceed SC SC ordered some more tapes, mais malheureusement pas de vouchers, and some longer longer ethernet cables 2. WFAU minutes ---------------- CASUrtools release/upgrades featured a few times as did saturation corrections, flagging of "extra" objects around bright stars, and data products delivery to ESO. JRL noted that he had finally finished his build script mods and had released a new version of the CASU software (mainly stacker, cataloguer, wcsfitter, and list-driver) to WFAU. The nebulosity filter and tiling software still need a few add ons and further checking/feedback about tile products, before being included. MJI supplied a suggested correction for saturated objects along the lines of the UKIRT Newletter article but wasn't entirely convinced about the desirability of directly replacing apermag3 at ingest with a corrected value for saturated stars. It was felt that flagging extra spurious objects around bright star halos, diffraction spikes, or other artefacts, is an archive or user-end issue mainly because how do you know which objects are real/false without cross-matching to other bands or repeat observations. Data products delivery to ESO is currently rather open-ended and awaits more enlightenment as to process. The promised Phase III workshop in the Autumn is probably as good a place as any to make real progress on this. 3. Recent and Upcoming Meetings ------------------------------- Wolfgang Hummel circulated the persistence report noted in the previous IOT telecon. Unfortunately the test involved flooding all the detectors and noting the decay time of the overall background. Even this appeared to be unpredictable when coupled with repeat additive events. A more realistic test involving saturated isolated star tests would be more relevant to normal operations but also unlikely to yield much of further use. To reiterate the conclusion from the last set of minutes: .... the consensus is that, like other NIR arrays, the persistence is not quantitatively predictable and that correcting for it is therefore impractical. Since a sensible observing sequence precludes it being a problem ..... Maria-Rosa and Brent have tried their damndest to find examples of residual persistence in the VMC data, including the deep stacks and tiles of 30Dor. A poxy bit of red nebulosity possibly caused by a 3rd magnitude star on one of the pawprint sets, but possibly also just another bit of nebulosity in the region, is the sum total thus far. So ..... no worries then. MJI and JPE attended the UKIDSS and VISTA Galactic surveys meeting in Leeds on July 8th to find out what folk are up to. Eamonn Kerrins presented an nice update on his DIA analysis for the VVV team. We suspect that this will be a good complement to list-driven methods but perhaps not the only required show in town. MJI and NAW hosted an EGAPS meeting in Cambridge on 12th July followed by a half day discussion on the IPHAS global calibration: problems - traditional anchors too restrictive; solution - Bayesian version of Glazebrook's method NAW presented a brief update on PLATO and EUCLID news. 4. Data Archive --------------- ING, AAO, UKIRT, VISTA all up to date and still no complaints other than whichever company now owns Sybase wanting 1/2 million in back payments. JRL will inquire of CADC and CDS if they are also facing bankruptcy. <<<< 5. Optical / NIR Processing --------------------------- A bit of IPHAS'ing re: global calibration, a bit of Subaru processing from EGS and MJI, and some HAWKIng from JRL, otherwise tranquility rules. 6. WFCAM Update --------------- WFCAM was back up and running on June 10th after its summer vacation, and data flow and processing resumed as normal. Raw data ingest and verification is up to date and WFCAM processing has caught up after a delay awaiting a full set of twilight flats. The odd confused MSB has been intercepted and lobbed through the pipeline but otherwise all is proceeding smoothly. MJI used the nebuliser to erase the GPS DR7 left overs and notified WFAU re: transfers. Chris Davis turned up another mysterious vanishing set of camera data (not #3 for once) from 29th July 2009. The first 2/3 of the night had no #4 data but the last 1/3 did. Investigations are afoot, but fortunately these occurences are very rare and there are unlikely to be many (any?) others. SJW has noted a mysterious couple of examples of stacked interleaved data for DR8 whereby some detectors look fine and others looked crap. JRL will investigate further since this is not supposed to happen anymore. <<<< 7. VISTA Update --------------- Ingestion and verification of raw data up to 13th July is in place and the next tape backup is due. The "new" calibration unit is in place and producing the goods re: linearity sequences. Apart from the odd whinge from JRL about 1s twilight flats and sometimes no darks to match the flats (there's no pleasing some people), everything is rocking along fine. Version 1.0 of VISTA data products containing all reprocessed data up to end of February were released to the PIs and VSA on 21st of July. A version log summarising the changes as well as a description of the tiles and changes in the sky background method for VVV and UltraVista is available from our web pages. The reprocessing of data up to the end of May is also finished and will shortly be released together with, shortly thereafter, v1.0 to end of June. This has all been processed & post-processed bar the odd data relocation. The final chunk is awaiting ESO keywords and database ingestion here (and final verification) prior to being released in about a weeks time. Stuart Sale popped over for the EGAPS meeting and sent us a couple of 2TB disks for the Chilean VVV end, to which we are copying stacked frames and tile images and catalogues to be sent to Phil Lucas who will make sure they get handed over for direct transport to Chile. (Internet transfers to Chile are not reliable enough for this volume of data.) Data transfer of processed data to PIs has now reached 9TB for the last 6 months and will no doubt increase rapidly again with the release of v1.0. Recent updates to skysub strategy (see web pages) include using 2 tiles if possible for VVV to improve smoothness of sky frames viz-a-viz leftover bits of GP stars; and incorporating an external deep object mask for UltraVista (and NB118). The mask is based on stacking a season of observations. MJI then initiated a quick review of recent fixes prior to v1.0 release all of which were minor and involved trivial header manipulations. He was assured (honest!) that all have been done including the ESO QC info and OB constraints where available (see earlier comment in actions section). Finally a brief discussion about junk data and what to do about it ensued. All VISTA data including junk is stored in the raw suppository. At pipeline processing preview stage, best efforts are made to recover interrupted or otherwise screwed up OB structures, partial OBs (that get repeated later) with enough observations present are processed as normal, OBs with the odd file only therein are quietly ignored. 8. AOB ------- Disk space issues have eased somewhat with the end of the reprocessing effort but still needs monitoring. For now the latest fibre-channel 14TB XFS system is being used as scratch space for VISTA processing but will eventually revert to WFCAM use. At the next meeting we need to discuss acquiring more VISTA data storage capacity, the main question given our less than enthusiastic impression of ZFS systems, being what sort ? <<<< Continuing Actions: =================== EGS - finish adding catalogue software to release pages JRL STH - finish off last bits of calibration plan document update JRL RGM - finish off testing catalogue conversion software JPE - contribute text to go with CASU versions of VISTA PR images JPE - contact ESO re: signing off pipeline deliverables EGS - SV time again still STH MJI SC - does our ZFS need a good tune ? New Actions: ============ MJI - add in optional illumination table correction to catalogue software JRL - contact CDS and CADC about Sybase licenses JRL - figure out why some WFCAM stacked interleaves are FU'd SC - investigate options for disk storage for VISTA MJI