Wednesday 22th September 2010 APM Meeting tables 11:30 - 13:00 Present: JCR, MJI, STH, EGS, JRL, AKY, MR, NAW Apologies: RGM, JPE Agenda ====== 1. Actions from last meeting 2. Comments on WFAU minutes 3. Meetings and Telecons 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST 6. WFCAM update 7. VISTA update 8. CASUhelps 9. AOB Minutes ======= 1. Actions from the last meeting -------------------------------- JRL CASU tool releases are up to date and at v1.06 JRL NBSIZE parameters have gone into the FITS header JRL has done all the updates he can on the calibration plan STH needs to revise the colour equations offline and finish off <<<< JRL the latest DRLD has been sent out MJI has added an illumination table option, action passed to EGS to link WFCAM tables to web pages <<<< RGM still testing catalogue listing software but sent in a note regarding astrometric differences at the 10mas level. MJI noted that these are undoubtedly caused by use of 32-bit arithmetic <<<< JPE text to go with CASU versions of VISTA PR images - ongoing <<<< JPE contact ESO re: signing off pipeline deliverables - ongoing <<<< though he is signing off the DRLD and sending it to ESO EGS have made good progress with the SV v1.0 reprocessing and will MJI discuss with the 3M's ++ at ESO as part of the VST meeting visit <<<< JRL has tied up his loose ends re: simplesky -> tilesky AKY is investigating various aspects of the ellipticity trends as part of an IOT action item (see below) - ongoing <<<< STH investigated the issue of CASU authorship on the Carina papers. It was decided to not be appropriate except on the initial papers. He will compose and send off a suitable email also commenting on the calibration errors <<<< JPE didn't quite manage to find out the date of the VISTA PSPI PhaseIII workshop but this was divulged during the VST meeting (30th November) and hence no longer an action item JRL found missing HAWKI data - where is it then ? MJI WFCAM calibrations in regions of high extinction investigation is STH still ongoing <<<< AKY have provided feedback to PIs on derived QC information etc ..... EGS in the form of FITS tables in addition to the usual sql dooberrys 2. Comments on WFAU minutes --------------------------- WFAU have been radio quiet for last week as a result of a power outage on Blackford Hill caused by some workmen with a drill and a bad aim. RGM informed us before the meeting that WFAU apparently need the CASU tiling software to make their band-merged catalogues, quite how and/or why we can't figure out, but the basic tiling software has now been made available anyway. 3. Meetings and Telecons ------------------------ On 07/09/10 there was a meeting to address GAIA computing requirements at the IoA. The upshot is that in ~a year there will (may) be 2x4 racks of computing space available in the old dog kennels at the Vet School (as opposed to the 2 X 16 racks available for finance !!). It will have a halon gas fire extinguishing system which means people will have to enter the building through the dog flap in pairs for 'elf & safety. STH said he had a very nice time in Portugal and enlightened JENAM'ers about VISTA processing, nebulisers, calibration, lightcurve generation and no doubt much else. On 28/29 September there is a VST review at ESO. EGS and MJI will be attending to talk about the CASU VST pipeline and processing experience en general, as well as talk to the VISTA SV teams. NAW pointed out that due to several upcoming large spectral surveys (e.g. FLAMES GIRAFFE/UVES, possible new wide-field MOS's etc....) there is a growing need for a half-decent spectral reduction pipeline. This is something that we have done in the past, at least to the extraction and wavelength calibration stage; while more advanced spectral products for GA are an active area of research within CASU. Let us see. 4. Data Archive --------------- ING, AAO, UKIRT, VISTA all up to date and motoring. JRL sorted out the delayed (at JAC request) few nights of CASS UKIRT archive and they are now also available. 5. Optical / NIR Processing --------------------------- Everything ticking along nicely apart from the IMACS data which has stalled due to advanced boredom with having to solve yet another astrometry setup. MJI has been investigating a scattered light issue in MegaCam, triggered by being a rapporteur for a SNLS PhD but also as part of the preparation for a PAndAS (M31 survey) meeting. He now understands the problem well enough to figure out a post-processing correction table to apply to the catalogues. EGS and Janet Drew have been exchanging some fascinating e-mails about which VEGAs system IPHAS is calibrated to. JRL has finished processing RGM's HAWKI data on the GOODS fields. Action on MJI and EGS to develop a VST pipeline plan before VST meeting, <<<< or at least be amazingly persuasive at same. 6. WFCAM Update --------------- All available 10B data has been received as expected and has been ingested into the raw data archive. Transfers to ESO are also up to date. September processing is a tad behind due to delays in getting a complete set of flats. Otherwise everything has been routinely pipelining and progressing as expected. The August data was flagged as ready to copy on September 16th and despite the best efforts of the workmen in Blackford Hill turned up at the WFAU archive end just over a week later. No requests for raw data and the only early access request for processed data was from Sandy Leggett. Omar and the UDS team noticed an interesting stripey pattern in J & K-band deep stacks in the UDS data prior to DR8 release. MJI investigated the problem, ruled out decurtaining due to the symmetry being wrong and tracked it down to a less than 1 part in 10000 effect visible in individual MSB stacks if you know what to look for. This looks like a (very minor) hardware feature whereby brightish stars in each quadrant cause a slight depression, or sometimes a slight rise, in the background level along nearby rows and/or columns. The effect is isolated in quadrants suggesting it is a readout feature. Mind you at this level if only becomes really noticeable when you make deep stacks. In the K-band, for example, the surface brightness of these stripes is between 23.5 and 24 mag/sq arcsec. Unfortunately as the effect is "attached" to the stars it does not go away on normal stacking and is probably best corrected during intermediate depth stacks (to minimise quadrant overlaps). This has resulted in a delay to UDS DR8 although begs the question of why this effect was not spotted before. 7. VISTA Update --------------- VISTA data up to the end of August has been received and processed data up to 7th August has been released. All data up to 20100901 has been processed and we are now up to the last couple of nights of post-processing. MJI noted that apm29a is fast running out of disk space and asked for a tidy up of non-VISTA processed data from there. The SV NGC253 data that was mistakenly processed with the 'simplesky' algorithm has now been reprocessed yet again, but this time with the original 'tilesky' option. This is hopefully it, and the latest latest version has been made available to ESO (Magda) and WFAU. Meanwhile EGS and MJI have been finishing off the rest of the SV post-pipeline processing and will discuss results with the 3M's during the ESO VST meeting. Matt Jarvis has observed 5 nights with the dual narrow band (975/985) filters. The data is expected at our end anytime soon. The odd few NaN/infinity/negative CRVAL issues++ have been investigated. Negative CRVALs crop up from the code not checking for offset sky exposures. The infinite values were caused by a bug in the overlapping routine. JRL has fixed both of these. The QC tables are now available on the CASU website with notes explaining column headers and the FITS files. We just need to tell the PIs where all this bounty is. AKY has been investigating why VVV and VHS sometimes have larger than acceptable ellipticities. It turns out that these occur most frequently during camera reshuffle periods (October last year, May this year..). Outside of this problem the other main culprits are trailed images due to poor tracking, primarily because the A/G hadn't settled before the OB sequence began. Meanwhile AKY will contact Marina and see what further information ESO want in this regard. VVV have obtained w/e use of a dedicated "large" bandwidth line from Santiago, Chile, during which they downloaded (or not) the newer tiles and catalogues not included on the shipped USB disks. It was decided that the data releases to PIs will be monthly from here on to simplify bookkeeping at both ends. JRL has been doing further work on the issue of apparent scattering in the detector zero-points raised by WHU at ESO. We are confident this is a non-issue and MJI sent a detailed explanation of why sometime ago. There followed a brief discussion about processing or otherwise of incomplete OBs and how to keep track of the process. Currently incomplete OBs are processed as usual providing there are enough frames to compute a reliable sky estimate. Grossly incomplete OBs are not processed. This raises two issues: how to keep track in our QC database of which ones are not processed; and flagging more directly in the science products that only "m" out of "n" jitters are present in the stack. Although the provenance information in the headers allows you to deduce this, it was felt that a specific keyword should be added to make it easier to "grep" for. <<<< Finally MJI suggested that a 'known features' page on the website might be useful for noting minor gremlins/gotchas and pre-empting some of the CASUhelp e-mails. EGS with helpers will make it so <<<< 8. CASUhelps ------------ Somehow MR mysteriously acquired exclusive ownership of CASUhelp and CASUreply email folders recently. This has been sorted but we have no idea how it happened. The other mr. confused everybody (CASU+WFAU) by managing to be browsing our web pages here as some June data was flagged (red -> green) as ready to go and then getting confused about dates as it turned out to be a repeat OB of earlier (Feb) data. ESO seem very keen on the PA of stellar ellipticities so we might have to give in and compute an average PA for each detector and plonk the results in the FITS headers. ...... and the contender for query of the month "it seems that the astrometric solution was only applied to the WCS positions (ra, dec) and not to the pixel positions. Would this be correct?" yup. 9. AOB -------- The new VISTA storage space disk server est arrivee (see pile of cardboard in the APM foyer). It should be accessible shortly and with any luck well before we run out of disk space for VISTA data products. A recalcitrant bit of extra metal bracket in the rack has now been removed to make way for it. One of the WFCAM RAID arrays (raw data) has lost one of its power supplies (careless of it). MJI is looking into getting it repaired as these seem to be no longer available. Meanwhile the data has been copied to apm19e as a precaution even though the RAID is limping along fine on one power supply module. MJI noted that home disk space was getting a bit tight again and asked for a bit of rm'ing. Continuing Actions: =================== STH - produce "final" VISTA colour equations and finish revising calibration plan document MJI - link WFCAM illumination correction tables and figs. to plone web pages EGS RGM - finish off testing catalogue conversion software JPE - produce some text to go with the VISTA PR images JPE - contact ESO re: signing off pipeline deliverables MJI - see if it is possible to finish off the SV processing saga EGS AKY - iterate a bit more with ESO about what ellipticity information they require, to be also further discussed at next IOT. MJI - finish off investigating WFCAM high extinction calibration issue STH New Actions: ============ STH - investigate large photometric errors quoted in Thomas' Carina paper (and others in the series) + compose a suitable email re: author lists MJI - produce cunning VST pipeline plan for VST meeting EGS ALL - move all non-VISTA processed data off apm29a JRL - add yet another keyword to the FITS headers (MJITTER?) EGS - to lead compiling a 'known features' web page ALL to help make it so