24th September 2008 APM not quite meeting tables 11:30-13:00 Present: MJI, JRL, PSB, MR, DWE, EGS, RGM Apologies: NAW, STH Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Telecon with JAC 4. ESO surveys workshop 5. AOB Minutes ------- 1. Actions from last meeting STH has not finished checking that the Plone astrometry technical pages are correct - ongoing <<<< (see minutes from 080332 for further details) STH has not included a graphic showing the size of the photometric radial distortion correction in the Plone technical pages nor checked that the rest of the photometry pages are up to date <<<< JRL has not produced a web page of HawkI publicity for CASU - ongoing <<<< MR has installed JRL's script to flag summit-rejected MSBs during raw WFCAM data ingestion before WFCAM08B ingestion commences MJI has discussed topics and arranged a telecon with JAC MJI have not finished off the stage-I software release pages - ongoing <<<< EGS EGS has written the presentation for the SADT+P2PP meeting and rammed home the appropriate points PSB helped STH prepare for his VISTA Paranal trip, but did not have anything to do with his laundry JRL have still to attend the Cambridge VOTC meeting since it hasn't MR happened yet - ongoing <<<< STH has not made the photometric calibration paper available on the Plone and VDFS sites - ongoing <<<< MJI has not finished off the technical paper - ongoing <<<< JRL investigated object masking sky strategy using 20080521 as a test night MJI has not investigated the WFCAM bad pixel map issue any further and has decided to wait for dome linearity sequences before proceeding - action dropped for now PSB has checked that the new machines' power consumption and fan speed are regulateable. Cooling occurs as it is needed. If you exercise a machine, then its fan whizzes around faster 2. Comments on WFAU minutes MJI noted the response to his query regarding relative time delay from DRn to world release of UKIDSS data products. 3. Telecon with JAC Slightly edited notes compiled by Chris Davis from telecon between CASU and JAC on 17th September at Jim's dog house. Technical changes: Just one filter change for 08B. BrG replaced with narrow-band H (1.619nbH). No changes to electronics - expect same behaviour from each array (channel edges and parquet structure in cameras 2 & 3, though occasionally in 1 & 4 - seems to be related to cable pairs). Derek Ives visiting this autumn - may do some work on WFCAM (though he's mainly here for CGS4 upgrade). PIs contacting CASU re early data access: Still relatively infrequent and not a problem. CASU happy with current informal arrangement; JAC support scientists will continue to "hand out" MJI's email address upon request! Pipeline at JAC: Raised as a possibility by Luca, who foresaw three advantages. - Fast-track reduction of PI data. - Reduction of projects with a JAC PI. - Testing the effects of any WFCAM configuration change before it reaches CASU. However, was felt that effort involved in installation and maintenance not probably not worth it yet. With FTP data transfers now taking just two hours, quick reductions on select data-sets (e.g. gamma-ray bursts) possibly seen as an option. Action on JAC to check which projects would have made use of quick reduction over the previous wfcam block (08A) and see if the expected delay time (24+24 hours) would be acceptable. Luca will talk to Mark who runs the service queue, which is where some of these requests come fomr. How to present this to future users is still an issue. Mostly likely will ask prospective users to present a good case for it via a tick box on the proposal form, with a text field in which they would justify the request. if the justification doesn't stand up, they don't get the quick reduction. Calibrations: Standards - no real requirement from CASU; JAC to decide how often we need these. Darks - at least 5 of each needed. Must have a set for each "coadds/expos time/readmode" setup. Biggest headache is when forget to take a set, or take them days/weeks afterwards. Flats - current schedule ok (at least one flat in each filter per month). Good to get all flats early in semester so CASU can get on with reductions. Repeat MSB handling: CASU to run script that will pick up MSB accept/reject status from OMP database at raw data ingestion stage in Cambridge and put suitable flag in headers (will be MSBFUD - with text reason). CASU will also monitor reduced MEFs with some sort of post-processing status (data within requirements etc.) and create MSBTID logs for each night giving two levels of accept/reject - summit and post-processing. Info to go in Marco's nightly status web-page, mainly for rapid feedback on MSBs for JAC. Not clear if WFAU need another flag in header for MSB reject after processing since suspect they already monitor assorted constraints derived from processing and compare with constraints specified (sometimes) in headers. Related to this, JRL raised the issue of more reliable OMP DB access (and access to the Archiver server). Sky subtraction and sky flipping: Established that method JAC is using to flip targets between two arrays is ok, though need to use as large a dither pattern as possible (3.2 arcsec often too small). JRL and MJI are still fine tuning some of the sky estimation stage parameters associated with this option. Also need recipe names for offsetting pairs of targets between pairs of arrays. JAC will propose some. Also investigate offsetting all four arrays to blank sky (for very large targets). JAC to set-up MSBs and get some test data. Need recipe names for this too. JRL will send an "ideal" data self-description for whole array offset sky mode. JAC will develop MSB for offsetting all four arrays to sky, and take some test data. Miscellaneous: New flat-field screen will hopefully be installed at UKIRT this year. CASU suggest taking linearity sequences; useful for IDing bad pixels as well as checking on the linearity. Comparison of dome and dawn flats presumably also a good idea. 4. ESO surveys workshop Summary mainly from notes compiled by EGS and STH at the workshop. Comments from this meeting in [ ]'s VST Status: * VST mount shipped on July 2007. Alt & Az axis are working and telescope is inside the enclosure. * Timescales: - Late October: mirror cell M2: conclusion of test campaign and delivery - Early November: mirror cell M2 arrives (by air) - Mid-November: mirror cell M1: conclusion of test campaign and delivery. - Mid-December: mirror cell M1 ships (by boat) - Feb 2009: Auxiliary Units ship (air) - Mid June 2009: delivery of VST and start of integration of OmegaCam. * ESO duties: procures transport, build infrastructure, design and commissioning of enclosure. Produce and deliver OmegaCam. Responsible for commissioning of camera + telescope. * Optics (M1, M2, corrector lenses) are currently in Paranal. * VST surveys starting probably end of 2009, beginning of 2010 * Current plans for SV undetermined. * Possible a smaller workshop similar to this one but still to decide when. * Note that in this case the PI will receive a copy of the raw data (how is open to discussion). VISTA Commissioning: * All subsystems functional and integrated but not yet fully optimised. * Template driven observing works but P2PP not tested. * Primary mirror as expected. Secondary has "trefoil" but corrected by the primary. * Time to write data to disk currently about 12 sec. This is larger than expected (it has been 4 sec). * Other overheads not completely determined at the moment. * Start of operations April 2009 as latest (but throughout the meeting it looked more like as soon as we can after SV). VISTA Data Flow: * Delivery using 500GB USB disks. This issue is not open to discussion now but ESO will take into account CASU suggestion of transferring the data by FTP from Garching. Note fibre link from Paranal to Antofagasta operational ~2012. [Good job we didn't buy and install Chenbro NGASTly lookalike then] * Diplobag takes about 10 days from Paranal to Garching and contains 1 week worth of data. Note extra constraints if Friday or holiday. * Disks will be normal external USB disks and estimate 3 disks per delivery. [3x500Gb is 3 Tbyte of raw data if compressed using bzip or gzip which we suspect is what will be used] Directories will be arranged by date and each directory will contain two subdirectories, one 'raw' with the raw data and another ones with additional files from ESO that can be ignored. CASU need to confirm that the arrangement of data in this way is ok for us. [We think we can manage to find the raw data] * Data will be shipped from Garching on Friday so expected to arrive here about Wednesdays but note that UK has to pay the costs of the shipment [sounds like ~3 weeks after observing before data arrives here] * Data will be pipeline processed at ESO to the pawprint level to produce QC parameters (using 40 nodes). * PI may access the processed data from CASU if they require it. * Expected monthly data flow is 2.5TB for OmegaCam and 7.5TB for VIRCAM [not clear if this includes realistic overheads since it equates to the idealised <250> Gbytes per night quoted over ayear ago]. * Expected 15800 OBs per year (11250 OBs the 1st year) -- this is larger than the combined OBs for the four VLT UTs. [~50 per night = short ?] * PIs will be asked to submit their OBs each semester. * PIs are ultimately responsible for the scientific value of the data products and they will need to send the data to ESO by a combination of Web interface plus FTP client (Phase 3). SADT: * The purpose of this is to generate the pointing position for each tile making up a survey finding also guide stars and active optics stars and write the result in a format that can be used by P2PP to generate the OBs. * Inputs are the boundaries of the survey area(s), the overlaps between tiles and the maximum amplitude in arcsec of the jitter pattern (max is 30"). [- why ?] * Issues relating to selection of angles for tiling but no show stoppers. P2PP: * The new thing here is the scheduling containers concept. - Timelink: sequence of OBs with minimum & maximum delay between them. If one OB of the containers fail the execution continues and only the failed OB is rescheduled. - Concatenation: in this case the OB is executed with no breaks but the execution order of the OBs is not specified. If one of the OBs fail then the execution fails and the whole container is rescheduled. - Group: this is used to specify that OBs should be executed close to each other but this constraint is desirable not mandatory. If an OB fails the execution continues. Once a group has started its priority increases in the queue. * Note that containers are constraints so reduce the likelihood of observations. * Sky offsets should be specified using concatenation containers. SADT and P2PP manuals at http://www.eso.org/sci/facilities/paranal/ instruments/vista/doc/ VISTA Science Verification: * Complementary observations on other systems, Hawk-I in VIDEO area and WFCAM in Orion (SV), should provide a good comparison [to the chagrin of some folk] * Aim of SV is twofold though unclear which has priority: - Produce scientific papers - Check a range of OBs and observing conditions requested in the public surveys (this is known as The Matrix). The Matrix is a table containing different parameters to be tested (e.g. linearity, persistence, deep stacking, photometric repeatability, effect of large galaxies, sky offsets) needed for the public surveys and if these will be either checked in commissioning or SV. [A copy of the Matrix was acquired, watched and discussed in some detail] * Despite the initial concerns about the SV from the public surveys PIs it looks like the Matrix is going to address the main issues that will affect the observing strategy of the surveys. * SV will apparently take place at the beginning of January 2009 for 11 nights. Both SV proposal teams have been made aware that they need sky offsets and they will take this into account when designing their OBs. * It is unclear what will happen after SV, probably more commissioning. * PIs have asked the survey team to perform some representative OB observations to have an idea of generally how they are implemented and in particular regarding overheads. This will be done in the SV phase or just after it. * There is a general concern in ESO that the timelines for data processing and delivery of data products are not well described. Obviously ESO want to know how long it takes from observation taken to final product delivered to them. [To rate cap this one we probably need in our progress status pages to show the date at which we receive the data since it looks like this will be the biggest contributor] Meeting with ESO Survey Team: * It is possible that only for the SV phase we can get the data on a daily basis with some delay (2 or 3 days). ESO want to know if doing this would help to speed up the processing of these SV data significantly (because it means a lot of work in their end with daily shipments -- and probably the astronomers having to take care of it due to holidays) [see same item below] * ESO asked for direct access to data from CASU. Note that transfer of data products supposedly only happens for those files which are not created by the pipeline at ESO, Garching, i.e. basically tiles and associated catalogues and confidence maps. [Unfortunately ESO seem to have a rather serious misconception about the Cambridge pipeline. The Garching pipeline is causal and triggers at the template level. This means it is very difficult to do good sky subtraction. In contrast the Cambridge pipeline has the flexibility to treat the whole night of data optimally, allowing use of offset skies, complete tiles for sky estimation and so on, in addition to making tiles etc..] * Issue of naming convention, how are we going to name the tiles, catalogues etc.. Need to describe this somewhere. [Our current working plan is to use a file naming convention akin to what we use for wfcam, with the prefix w -> vir to accomodate vst in the same unique name system and any future vis.] * Channels for email communication were set up. * ESO agreed to provide us with the information about the grading of each OB and also with the information needed for us to know if an OB has been aborted for whatever reason or if needs to be scheduled for re-observation due to some problem, so that we do not need to spend time processing it or wondering why it was problematic. ESO will check exactly how to do this but we suggested some keyword in the FITS headers. [ESO will deal with doing this via their own DBs - we hope] * Need to send some data to Edinburgh so that they test their ingestion scripts * We spoke of what happens after the VISTA surveys in several years in normal telescope operations [during rather than after presumably]. Need to handle proprietary periods but we already do this (e.g. UKIDSS, PIs). * General impression is that they are happy to work with us, and they are happy to help so that the survey is a success (and they expect the same from us) [ahhhhh] Other stuff: * Maria Rosa concerned about sky subtraction. She wants to be able to detect extended objects with low surface brightness [is that in the SMP ?] * Maria Rosa asked about mosaicing and how we are going to deal with the area which is not x6 [sic!] covered in each tile. Estimates ~7% of the area affected by this. Especially since they have been asked to produce an "homogenous survey". [Homogeneous is impossible for lots of reasons, but apart from that the cataloguing software is smart enough to not attempt to go deeper in say x4,6,8 exposed regions, so this particular worry should not be a problem] * Maria Rosa asked if she can come to Cambridge at some point and learn about data reduction. [maybe we should run a master class] * Somebody asked if there is a version of the tiling programs in CPL. ESO plan anyway to publicly release the software, source code and documentation. [no, but how much and hmmmmmm] EGS and STH met with Magda Arnaboldi, Marina Rejkuba, Martino Romaniello and Eckhard Sutorius on Wednesday 17th September split into two ~30 min sessions, notes from STH. Topics which came up included: 1. Would we prefer the SV data to come daily or in weekly chunks ? There were no promises that it could be done - but they could probably sort something out for the ~2 weeks of ESO SV. This would involve transferring the data on disk within Chile, then ftping it back to Garching. They estimate that we could have the data around 2 days after it was observed using this method, but it does involve significant extra effort on their behalf. Our response was that the time-saved would definitely be worth having, and that we would give them a more considered response after discussing with our group. [It was felt that certainly for the 1st few nights of SV data it would be very helpful to have faster access] 2. We discussed file-naming ...... we felt that the standard ESO naming scheme was fine. [We didn't, the names are not unique and wrap around every year. Neither is there any continuous run number, hence our earlier plan to rename them on ingest and adopt a wfcam-like system here.] 3. For SV, ESO would like to copy the flat files from CASU in advance of them becoming available through WFAU. We need to agree a protocol. [wget ...... via files soft-linked on web page] 4. For the SV data, Eckhard stated that there would be a 7-10 day delay between obtaining the flat files from CASU and there being available to copy from the archive. There would be a further 2 weeks for full ingestion and band-matching and construction of the tables etc.. 5. Will this data be made available in one block, or in installments ? [SV data - probably installments but depends on problems thrown up] 6. We asked them how we would communicate. They prefer us to use usd-help@eso.org, but they will also set up an email exploder for SV. We told them about casuhelp. 7. They would like the SV data to be mosaiced if possible (i.e. between tiles) [why ? catalogues can be "mosaiced" but what's the point of tiles-of-tiles for SV ? We feel latter is outside scope of SV process] 8. We asked how they implement QC file grading to accept and reject files. This happens at the summit in real time. The OBs are flagged as: A (all files are fine), B (most files are fine) .. and presumably C or F .. anything not A or B is immediately put back into the observing queue. It wasn't totally clear how they do this grading - it sounds like the responsibility of the ESO astronomer. Whatever, the OB grading will be put in a database (and/or possibly the files themselves). We said that we need access to these grades. [Aren't they going to be put in the FITS headers ? - see earlier]. We also said we need to close the loop for files that we cannot process and that WFAU will need to do the same, for files that do not pass the PI inspection. 9. We agreed that there should be another meeting (they thought June or July which seems rather far off) to discuss these issues and further issues arising from SV. [We probably need another meeting before SV and certainly will need one straight after SV has been first-pass processed.] 10. They asked how quickly we can make the SV data available. We reiterated one month from ingestion - but speculated that it "might" be possible to release a subset of some preliminary reductions on a shorter timetable if processing went well and we felt the data were good enough. A couple of extra things that came up in the general meeting from responses to questions STH asked: we will get access to the OBs for SV to comment on before the observations go ahead; the ESO DG will add of order ~nights of time for the surveys to complete additional SV tests not already included in the Matrix (provided by ESO from the SV program); there may be room to do dry runs using the survey MSBs in addition to these extra nights. Some actions from this lot below: The directory structure proposed by ESO for the transfer of data was thought reasonable. MJI will let them know via diplobag. <<<< We would like the first few nights of SV data as soon as possible so that we can provide prompt feedback. MJI will contact them as above. <<<< PSB will send some of the already processed VIRCAM test data to Edinburgh so that they can test ingestion. <<<< There was a discussion about calibration observations that we would like from STH's visit to Paranal and a brief diatribe about the necessity of <<<< "hard-coding" Ndit=1 into the twilight template, since the last thing you want with non-linear detectors and rapidly varying night sky is Ndit > 1. JPE/WJS should be made aware of this <<<< 6. AOB none worthy of comment Continuing actions ------------------ STH check that Plone astrometry technical pages are correct STH include graphic showing the size of the photometric radial distortion correction in the Plone technical pages and check rest of photometry pages are up to date JRL produce web page of HawkI publicity for CASU MJI finish off stage-I software release pages EGS JRL attend Cambridge VOTC meeting on 25-26 September MR STH make photometric calibration paper available on Plone and VDFS sites MJI finish off technical paper New actions ----------- MJI ensure feedback regarding CASU contentment with proposed directory structure for transferring data gets ...... err fedback. MJI ditto about acquiring first few nights of SV data as soon as possible after it is taken PSB send some processed VIRCAM test data to Edinburgh STH acquire the calibrations we desperately need for VIRCAM MJI make sure JPE/WJS are aware of the Ndit=1 issue for twilight flats