Thursday 26th August 2010 APM Meeting tables 11:30 - 13:00 Present: JCR, MJI, STH, EGS, JRL, AKY, RGM, JPE Apologies: MR, NAW Agenda ====== 1. Actions from last meeting 2. Comments on WFAU minutes 3. Meetings and Telecons 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST 6. WFCAM update 7. VISTA update 8. CASUhelps 9. AOB Minutes (powered by Hot Lava Java Extra Dark Roast - strength 6 YO!!) ======= 1. Actions from the last meeting -------------------------------- JRL the first two lots of on-line notes are done and dusted, the rest is EGS ongoing and will be completed soon for the CASUtools packages for cataloguing, list-driving, stacking and wcs-fitting. MJI suggested including the sample fortran code to read the catalogue files and apply the assorted corrections - ongoing but getting there <<<< RGM also reminded us that the NBSIZE parameter needs to be recorded in the FITS headers, all other relevant cataloguing parameters are. <<<< STH has computed the VEGA to AB conversions for VISTA and plonked them in the technical section within the filter set web pages. As a check he recomputed the WFCAM set and agreed with the Hewett et al. values to 1 mmag (using same Vega spectrum). Comparing with the appropriate WFCAM and 2MASS values shows general agreement at the 0.01mag level for these conversions. There was some discussion about including or otherwise Ax/E(B-V) values but it was generally felt that this was not a calibration issue per se since these values depend on which extinction law is being used. RGM pointed out the table in Schlegel et al. that is commonly used. Since we also provide the effective central wavelength of the filters on the CASU web pages it was thought that this was enough for users to make their own informed choice. These extras still need adding to the calibration plan document. <<<< MJI added an illumination table option. The illumination tables (WFCAM) need to be linked somewhere on the web pages though. <<<< RGM still testing <<<< JPE contribute text to go with CASU versions of VISTA PR images - ongoing <<<< JPE contact ESO re: signing off pipeline deliverables - still ongoing <<<< EGS has made good progress with SV processing, it will soon be back to where v0.8 was MJI made zero progress after discovering the v1.0 SV NGC253 data was worse than v0.8. To cut a long story short, this turned out to be due to a new and little used sky estimation option (simplesky - only used for the SV NGC253 project v1.0). This is being rectified even as we speak by reprocessing these with the tilesky option used previously - ongoing <<<< SC the ZFS issues have been bypassed by the Vogons ......... JRL has contacted Sybase regarding their hopeful demand for megabucks. CDS were also paying £1000 per year for their licenses and are as shocked as us. No answer from Sybase as yet. Decided best not to stick head above parapet. JRL has examined the reason for a set of peculiar stacked images in WFCAM DR8 LAS data. This was due to an inconsistent set of interleave offsets in a 2 dither 2x2 microstep sequecence. This problem has not been noticed before (nor after) and seems to be a rare occurence. The files have been left deprecated (SJW). SC have comprehensively explored VISTA disk storage options. ZFS and MJI thumpers, which now cost way too much, have been abandoned in favour of a different networked disk server option (see AOB). 2. Comments on WFAU minutes --------------------------- Fortnightly ! ...... way to go Possible flagging of spurious objects around bright sources isn't going to happen in the cataloguing software. Since most of these vanish when band-merging (or similar) it seems more like a user-end problem to us. WFAU have the latest CASUrtools software but this still does not include the nebuliser and tiling software. There was some discussion regarding the urgency of these final two bits of the puzzle, hereby boiled down to our current best guess of requirements for this: VHS doesn't need it urgently - though "ears" may later; UltraVISTA are doing their own thing as are probably VIDEO ?; the majority of this season's VVV gets tiled on-the-fly during processing and its only the variablility aspect that might want it, or have we missed something. But even here it would be a seriously bad idea to use tiles for time series photometry ......; so that leaves VMC and VIKING. We have no idea what VIKING is up to but note that we keep being bugged by Astrowise about processing details so we are guessing they are involved somewhere - but please enlighten us. Likewise VMC, conflicting requests from mr. are confusing us. There ensued a discussion led by RGM on apermags, corrected, not corrected, naming conventions ...... usw ...... that confused everybody even more. 3. Meetings and Telecons ------------------------ At the VISTA IOT on Tuesday 24th we learned that the camera has been recentred and tilted with respect to the optical axis to correct for the slight focus gradient. It is now hopefully re-aligned as the Prof ordered and everything seems to be working well, bar the usual few minor glitches. Prior to the IOT we generated an updated zpt trend plot which unambiguously shows a gradual decline in the throughput of the system. Although only a few % in the Ks band the effect is approaching 15-20% in the Y and Z-bands. The primary suspect is .... the primary and further investigations are in train to figure out the cause and ...... a solution. JPE noted also that the secondary mirror has not been recoated in its three year lifetime and its reflectivity (as a function of wavelength) also needs further monitoring. At ESO Marina Rejkuba is still fixing some recalcitrant OB grades that got mangled in their logging process. Apparently this problem will depart for new data when the new OT tool is installed but meanwhile expect a smallish proportion of the current OB grades in the FITS headers to need revising. Due to the uncertain timescale for these to be fixed, we have gone ahead with v1.0 data releases and the next monthly batches will be making an appearance shortly. The OB grade updates plus other minor header changes, updated photometric/astrometric calibration etc., will be saved up for a v1.1 release. There was the usual IOT digression on elliptical images and what, if any the correlations were with ......... things. MJI kindly volunteered AKY to investigate using our own database info. <<<< JPE raised a few things that had come up during a VMC telecon and that he had clarified during the telecon: first, the tiled images themselves are not nebulosity filtered but the tile catalogues are derived from tiles that are; second, CASU do not directly provide corrections for saturated star magnitudes. This is partly because even for pawprints there is no single correction that is optimal in all cases and for tiles it is of even more dubious value. Should someone from CASU listen in on these telecons ? About the only 1/2 decent use case so far for saturation corrections is mr.'s desire to study large amplitude saturated Mira's in the MC tiles - fair doos. In a related comment, MJI noted some meaningless (and misleading) plots of VISTA data products that accompanied the IOT agenda. All of the apparent problems in these plots (stripes, weird patterns etc.) were caused by saturation effects in tiled data, and from using poor quality data with extremely high counts in the sky background in at least one band. Plotting distributions of parameters for saturated objects, particularly from tiles, could obviously do with a health warning. As noted in the previous minutes there is a VST Public Surveys and GTO programs Review at ESO on 28/29th September. The agenda features various fascinating science things and also includes some items on process and processing viz talks on the ESO QC process, the Astrowise pipeline and the VSTube Naples (sleeps with the fishes) pipeline. JPE was tasked with attempting to find out when the VISTA PSPI PhaseIII meeting at ESO is likely to be. <<<< STH pointed out ('cos he's going) JENAM is in Portugal on 6 - 10 September, where he'll be talking about the Vista pipeline, nebulosity filtering, photometric calibration and lightcurves in the session: SYMPOSIUM 5 Star Clusters in the Era of Large Surveys. 4. Data Archive --------------- ING, AAO, UKIRT, VISTA all up to date and motoring 5. Optical / NIR Processing --------------------------- Everything ticking along nicely. EGS is processing some IPHAS data from October 2009 and MJI has taken up the cudgels on some Magellan IMACS imaging data. There was some discussion about HAWKI SV data of the Carina nebula that was processed by CASU over a year ago. This was caused by the appearance of draft papers on the Chandra study of Carina which need at the minimum browsing to establish which, if any, should include CASU folk in the author list. STH will investigate. <<<< Plus an action on JRL to locate the HAWKI data so we can a.) figure out what we did and b.) make some more colour pics <<<< 6. WFCAM Update --------------- All 10A data is fully reduced and the final lot of June, July data was flagged as ready for WFAU to copy on the 18th August. It had all finished being transfered on August 22nd ! All the available 10B data has been received as expected and has been ingested into the raw data archive. Processing of 10B is up to date with the latest batch of 3 nights chugging away. ESO transfers of UKIDSS and Calibration data are also up to date. No requests for raw data or early access to processed data have been made this month. A video board for one of the quadrants of detector#4 was replaced recently but no adverse effects have been noticed. Around 3 weeks ago, Chris Davis (JAC) noticed some missing #4 data that he had taken on 29th July 2009. About 2/3 of detector#4 data had gone walkabout prior to transfer for that night. It looks like this is a very unusual occurence and at a distance not clear what the underlying cause was. JAC put the relevant data for 20090729 online for all cameras and it was retransferred and reprocessed sans probleme. Phil Lucas has been browsing in his extinction cupboard and spotted a small problem with WFCAM calibration in regions of extremely high extinction, E(B-V) > ~10. The photometric calibration is out of spec in areas with high extinction. This is mainly due to the extinction coefficient term applied to all WFCAM data. This was derived from data with 0 < E'(B-V) < 2, where E'(B-V) is the Bonifiacio et al. (2000) modified Schlegel et al. (1998) extinction. The error in the slopes (coefficients) becomes significant by E(B-V)=10 leading to a systematic offset in the calibration in the J,H,K bands. With the recent data in the Galactic Plane in the Bulge region this problem has become more apparent. Phil is carrying out his own investigation for ways to improve the calibration in these regions and STH and MJI are investigating a better approximation to the coefficients for a much wider range of E(B-V) than was originally used. The total no. of all WFCAM pointings affected is around 150, i.e. b'all, so there are several possible ways to proceed. Watch this space . <<<< Narrow band observations of Jupiter tracked at the sidereal rate of Jupiter were happily processed by the pipeline in its complete ignorance. To, at least, MJI's surprise, even the stacked interleaved image products looked ok. 7. VISTA Update --------------- We will be up to date with releases by the end of this week. Up to the end of May v1.0 has already been released and up to the end of July will be released by tomorrow (Friday 27th August). We have data up to 11th August. There were a few bad nights with no data where the weather was too bad for anyone to get to the telescope. JRL and AKY are going to check that the raw data directories are correctly marked up when there is 'no data' or 'unprocessed data' i.e. no useful science data. There needs to be a consistent way of treating junk data. Following JRL's lead a new policy was converged upon. If the data is good enough to process (e.g. 4+ good frames) then it gets processed. If there's less than that (e.g. < 4 frames, or 4 low quality cloudy frames) then they get deleted and an explanation is entered into a README file. As with WFCAM, the individual images should be kept but the stack can be chucked. When this happens it should be recorded so that PIs are aware; though quite how this loop is best closed is a moot point. AKY will investigate producing a nightly log in the style of the WFCAM MSBTid.log to summarise the QC info we generate and making it available via the web pages. <<<< It was noted that the OB status' are in the headers (since 15th October) and are in a table on the web pages too. As commented earlier, there are still some problems with the values that ESO are supplying and these will be resolved ....... eventchewally. JPE and MJI noted that the next batch of ESO PR images will include the Monoceros scarab beetle pic and following that an ambitious 40kx30k colour image of a 3x3 tile region from the Orion SV data. CASU will supply the tiles and ESO will be doing the Orion gluing - best of luck. JRL relayed from Marijn Franx the compliments of the UltraVISTA team on the excellent quality of the v0.8 UltraVISTA data. Let's hope that v1.0 maintains the standard. 8. CASUhelps ------------ As a bit of light relief MJI relayed some examples of emails to, and replies from, Mme. CASUhelp. The commonest reply was a variant on RTFWP or RTFOL, while "why do tile catalogues have missing headers for TV5_1 TV5_3 and TV5_5 ?" raised the biggest grin, thereby proving what a sad lot we are. One of the more interesting questions related to "what happens if the extinction varies significantly during a tile OB ?". The short answer is nothing; slightly longer is that if the component pawprints of a tile have wildy varying extinction this isn't currently corrected for in the tiling; and the longer still answer is that the development tookit has an option for dealing with this using the derived zpts for each pawprint. This option would need to be automated along the lines of, if you believe the variation in zpts apply the correction. The issues of implementing this fully hands off are still being thought through though it is a moot point that, if the extinction really is varying significantly during a tile OB whether or not the tile data is reliable enough to be of use, and/or the best way to flag this in the tile headers. 9. AOB -------- SC has begun the process of trialling UBUNTU on one of the Blades. A solution has been found to provide more storage space for VISTA in the form of a 'Supermicro' system. Horrified at the price of purchasing Sun 'Thumpers' and the concomitant huge maintenance costs, we reversed the polarity for VISTA storage and are trialling a modern equivalent to what we originally designed in-house for WFCAM storage. Basically a 4U PC disk server stuffed, in this case, with 36 lots of easily accessible 2TB disks and future JBOD expansion module(s) capable of similar 36 disk capacity each. The server plus 1 expansion module will be configured to yield 4 RAID6 disk systems totalling ~120TB of usable RAID6 storage and works out around 250 pounds/TB of realised storage. It has the usual in-built spare power supplies, fans etc... We will start with the PC server unit (~64TB) and if ok get an extension unit (another ~64TB) which should cover VISTA storage needs until the end of the current main 3 year grant period. EMBS are in the loop regarding upgrading the motion detectors in each room from binary, anything that moves ****** ..... to no its a moth you idiot. Continuing Actions: =================== EGS - finish adding catalogue software description to release pages JRL and add link to fortran example code STH - finish off last bits of calibration plan document updates JRL RGM - finish off testing catalogue conversion software JPE - produce some text to go with the VISTA PR images JPE - contact ESO re: signing off pipeline deliverables EGS - SV time again still #2 STH MJI JRL - tie up loose ends with the sky subtraction. New Actions: ============ JRL - plonk NBSIZE background tracking parameter in FITS catalogue headers MJI - put in a link to the (WFCAM) illumination correct tables on the website AKY - investigate correlations of VISTA ellipticity measure with ?? JPE - find out when VISTA PSPI PhaseIII meeting is likely to be STH - go through Chandra/Carina papers and identify those CASU should be on JRL - find the HAWKI data so we can check how it was done etc.. STH - investigate gradual drift off spec of extinction corrections in high MJI extinction regions in WFCAM data (and possibly also VISTA). AKY - investigate producing a nightly summary QC log for VISTA