File contents

24th September 2008  APM not quite meeting tables  11:30-13:00

Present:          MJI, JRL, PSB, MR, DWE, EGS, RGM
Apologies:        NAW, STH

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Telecon with JAC
4.  ESO surveys workshop
5.  AOB

Minutes
-------

1.  Actions from last meeting

STH  has not finished checking that the Plone astrometry technical pages 
     are correct - ongoing                                                 <<<<
     (see minutes from 080332 for further details)
     
STH  has not included a graphic showing the size of the photometric radial 
     distortion correction in the Plone technical pages nor checked that the 
     rest of the photometry pages are up to date                           <<<<
	       
JRL  has not produced a web page of HawkI publicity for CASU - ongoing     <<<<

MR   has installed JRL's script to flag summit-rejected MSBs during raw 
     WFCAM data ingestion before WFCAM08B ingestion commences
     
MJI  has discussed topics and arranged a telecon with JAC 

MJI  have not finished off the stage-I software release pages - ongoing    <<<<
EGS  

EGS  has written the presentation for the SADT+P2PP meeting and rammed home
     the appropriate points

PSB  helped STH prepare for his VISTA Paranal trip, but did not have 
     anything to do with his laundry

JRL  have still to attend the Cambridge VOTC meeting since it hasn't 
MR   happened yet - ongoing                                                <<<<

STH  has not made the photometric calibration paper available on the Plone 
     and VDFS sites - ongoing                                              <<<<

MJI  has not finished off the technical paper - ongoing                    <<<<

JRL  investigated object masking sky strategy using 20080521 as a test night

MJI  has not investigated the WFCAM bad pixel map issue any further
     and has decided to wait for dome linearity sequences before proceeding
     - action dropped for now

PSB  has checked that the new machines' power consumption and fan speed 
     are regulateable. Cooling occurs as it is needed. If you exercise a
     machine, then its fan whizzes around faster


2.  Comments on WFAU minutes

MJI noted the response to his query regarding relative time delay from DRn
to world release of UKIDSS data products.


3.  Telecon with JAC

Slightly edited notes compiled by Chris Davis from telecon between CASU and 
JAC on 17th September at Jim's dog house.

Technical changes:

Just one filter change for 08B.  BrG replaced with narrow-band H (1.619nbH). 
No changes to electronics - expect same behaviour from each array (channel 
edges and parquet structure in cameras 2 & 3, though occasionally in 1 & 4 
- seems to be related to cable pairs).  Derek Ives visiting this autumn - 
may do some work on WFCAM (though he's mainly here for CGS4 upgrade).

PIs contacting CASU re early data access:

Still relatively infrequent and not a problem.  CASU happy with current 
informal arrangement; JAC support scientists will continue to "hand out" 
MJI's email address upon request!

Pipeline at JAC:

Raised as a possibility by Luca, who foresaw three advantages.
         - Fast-track reduction of PI data.
         - Reduction of projects with a JAC PI.
         - Testing the effects of any WFCAM configuration change before
           it reaches CASU.

However, was felt that effort involved in installation and maintenance not 
probably not worth it yet.  With FTP data transfers now taking just two hours, 
quick reductions on select data-sets (e.g. gamma-ray bursts) possibly seen 
as an option.  Action on JAC to check which projects would have made use 
of quick reduction over the previous wfcam block (08A) and see if the 
expected delay time (24+24 hours) would be acceptable. Luca will talk to 
Mark who runs the service queue, which is where some of these requests come
fomr.

How to present this to future users is still an issue.  Mostly likely will
ask prospective users to present a good case for it via a tick box on the
proposal form, with a text field in which they would justify the request.
if the justification doesn't stand up, they don't get the quick reduction.

Calibrations:

Standards - no real requirement from CASU; JAC to decide how often we
            need these.
Darks     - at least 5 of each needed.  Must have a set for each
            "coadds/expos time/readmode" setup. Biggest headache is when
            forget to take a set, or take them days/weeks afterwards. 
Flats     - current schedule ok (at least one flat in each filter per month).
            Good to get all flats early in semester so CASU can get on with
            reductions.

Repeat MSB handling:

CASU to run script that will pick up MSB accept/reject status from OMP 
database at raw data ingestion stage in Cambridge and put suitable flag
in headers (will be MSBFUD - with text reason).  CASU will also monitor
reduced MEFs with some sort of post-processing status (data within 
requirements etc.) and create MSBTID logs for each night giving two levels of 
accept/reject - summit and post-processing.  Info to go in Marco's nightly 
status web-page, mainly for rapid feedback on MSBs for JAC.

Not clear if WFAU need another flag in header for MSB reject after 
processing since suspect they already monitor assorted constraints derived 
from processing and compare with constraints specified (sometimes) in 
headers.

Related to this, JRL raised the issue of more reliable OMP DB access (and 
access to the Archiver server).

Sky subtraction and sky flipping:

Established that method JAC is using to flip targets between two arrays is 
ok, though need to use as large a dither pattern as possible (3.2 arcsec 
often too small).  JRL and MJI are still fine tuning some of the
sky estimation stage parameters associated with this option.

Also need recipe names for offsetting pairs of targets between pairs of 
arrays.  JAC will propose some.

Also investigate offsetting all four arrays to blank sky (for very large 
targets).  JAC to set-up MSBs and get some test data. Need recipe names for 
this too. JRL will send an "ideal" data self-description for whole array 
offset sky mode.  JAC will develop MSB for offsetting all four arrays to sky, 
and take some test data.

Miscellaneous:

New flat-field screen will hopefully be installed at UKIRT this year. CASU 
suggest taking linearity sequences; useful for IDing bad pixels as well as
checking on the linearity. Comparison of dome and dawn flats presumably 
also a good idea.


4.  ESO surveys workshop

Summary mainly from notes compiled by EGS and STH at the workshop.
Comments from this meeting in [ ]'s

VST Status:

 * VST mount shipped on July 2007. Alt & Az axis are working and telescope is 
   inside the enclosure. 
 * Timescales: 
      - Late October: mirror cell M2: conclusion of test campaign and delivery
      - Early November: mirror cell M2 arrives (by air)
      - Mid-November: mirror cell M1: conclusion of test campaign and delivery.
      - Mid-December: mirror cell M1 ships (by boat)
      - Feb 2009: Auxiliary Units ship (air)
      - Mid June 2009: delivery of VST and start of integration of OmegaCam.

 * ESO duties: procures transport, build infrastructure, design and 
   commissioning of enclosure. Produce and deliver OmegaCam. Responsible for 
   commissioning of camera + telescope.
 * Optics (M1, M2, corrector lenses) are currently in Paranal. 

 * VST surveys starting probably end of 2009, beginning of 2010 
 * Current plans for SV undetermined.
 * Possible a smaller workshop similar to this one but still to decide when.
 * Note that in this case the PI will receive a copy of the raw data (how is 
   open to discussion).

VISTA Commissioning:

 * All subsystems functional and integrated but not yet fully optimised.
 * Template driven observing works but P2PP not tested.
 * Primary mirror as expected. Secondary has "trefoil" but corrected by the 
   primary.
 * Time to write data to disk currently about 12 sec. This is larger than 
   expected (it has been 4 sec).
 * Other overheads not completely determined at the moment.
 * Start of operations April 2009 as latest (but throughout the meeting it 
   looked more like as soon as we can after SV).

VISTA Data Flow:

 * Delivery using 500GB USB disks. This issue is not open to discussion now 
   but ESO will take into account CASU suggestion of transferring the data by 
   FTP from Garching. Note fibre link from Paranal to Antofagasta operational 
   ~2012. [Good job we didn't buy and install Chenbro NGASTly lookalike then]
 * Diplobag takes about 10 days from Paranal to Garching and contains 1 week 
   worth of data. Note extra constraints if Friday or holiday.
 * Disks will be normal external USB disks and estimate 3 disks per delivery.
   [3x500Gb is 3 Tbyte of raw data if compressed using bzip or gzip which
   we suspect is what will be used] 
   Directories will be arranged by date and each directory will contain two 
   subdirectories, one 'raw' with the raw data and another ones with additional
   files from ESO that can be ignored. CASU need to confirm that the 
   arrangement of data in this way is ok for us. [We think we can manage to
   find the raw data]
 * Data will be shipped from Garching on Friday so expected to arrive here 
   about Wednesdays but note that UK has to pay the costs of the shipment 
   [sounds like ~3 weeks after observing before data arrives here] 
 * Data will be pipeline processed at ESO to the pawprint level to produce QC 
   parameters (using 40 nodes).
 * PI may access the processed data from CASU if they require it.
 * Expected monthly data flow is 2.5TB for OmegaCam and 7.5TB for VIRCAM 
   [not clear if this includes realistic overheads since it equates to the 
   idealised <250> Gbytes per night quoted over ayear ago].
 * Expected 15800 OBs per year (11250 OBs the 1st year) -- this is larger 
   than the combined OBs for the four VLT UTs. [~50 per night = short <OBs>?]
 * PIs will be asked to submit their OBs each semester. 
 * PIs are ultimately responsible for the scientific value of the data 
   products and they will need to send the data to ESO by a combination of 
   Web interface plus FTP client (Phase 3).

SADT:

* The purpose of this is to generate the pointing position for each tile 
   making up a survey finding also guide stars and active optics stars and 
   write the result in a format that can be used by P2PP to generate the OBs.
 * Inputs are the boundaries of the survey area(s), the overlaps between 
   tiles and the maximum amplitude in arcsec of the jitter pattern (max 
   is 30"). [- why ?]
 * Issues relating to selection of angles for tiling but no show stoppers.

P2PP:

 * The new thing here is the scheduling containers concept.
   - Timelink: sequence of OBs with minimum & maximum delay between them. If 
   one OB of the containers fail the execution continues and only the failed 
   OB is rescheduled.
   - Concatenation: in this case the OB is executed with no breaks but the 
   execution order of the OBs is not specified. If one of the OBs fail then 
   the execution fails and the whole container is rescheduled.
   - Group: this is used to specify that OBs should be executed close to 
   each other but this constraint is desirable not mandatory. If an OB fails 
   the execution continues. Once a group has started its priority increases 
   in the queue.
 * Note that containers are constraints so reduce the likelihood of 
   observations.
 * Sky offsets should be specified using concatenation containers.

SADT and P2PP manuals at http://www.eso.org/sci/facilities/paranal/
instruments/vista/doc/

VISTA Science Verification:

 * Complementary observations on other systems, Hawk-I in VIDEO area and
   WFCAM in Orion (SV), should provide a good comparison [to the chagrin
   of some folk]
 * Aim of SV is twofold though unclear which has priority:
    - Produce scientific papers
    - Check a range of OBs and observing conditions requested in the public 
   surveys (this is known as The Matrix). The Matrix is a table containing 
   different parameters to be tested (e.g. linearity, persistence, deep 
   stacking, photometric repeatability, effect of large galaxies, sky offsets) 
   needed for the public surveys and if these will be either checked in 
   commissioning or SV. 
   [A copy of the Matrix was acquired, watched and discussed in some detail]
 * Despite the initial concerns about the SV from the public surveys PIs it 
   looks like the Matrix is going to address the main issues that will affect 
   the observing strategy of the surveys. 
 * SV will apparently take place at the beginning of January 2009 for 11 
   nights. Both SV proposal teams have been made aware that they need sky 
   offsets and they will take this into account when designing their OBs.
 * It is unclear what will happen after SV, probably more commissioning.
 * PIs have asked the survey team to perform some representative OB 
   observations to have an idea of generally how they are implemented and in 
   particular regarding overheads. This will be done in the SV phase or just 
   after it.
 * There is a general concern in ESO that the timelines for data processing 
   and delivery of data products are not well described. Obviously ESO want 
   to know how long it takes from observation taken to final product delivered
   to them. 
   [To rate cap this one we probably need in our progress status pages to 
    show the date at which we receive the data since it looks like this will
    be the biggest contributor]

Meeting with ESO Survey Team:

 * It is possible that only for the SV phase we can get the data on a daily 
   basis with some delay (2 or 3 days). ESO want to know if doing this  
   would help to speed up the processing of these SV data significantly 
   (because it means a lot of work in their end with daily shipments -- and 
   probably the astronomers having to take care of it due to holidays)
   [see same item below]
 * ESO asked for direct access to data from CASU. Note that transfer of data 
   products supposedly only happens for those files which are not created by 
   the pipeline at ESO, Garching, i.e. basically tiles and associated 
   catalogues and confidence maps.  [Unfortunately ESO seem to have a
   rather serious misconception about the Cambridge pipeline. The Garching
   pipeline is causal and triggers at the template level.  This means it
   is very difficult to do good sky subtraction.  In contrast the
   Cambridge pipeline has the flexibility to treat the whole night of data
   optimally, allowing use of offset skies, complete tiles for sky estimation
   and so on, in addition to making tiles etc..]
 * Issue of naming convention, how are we going to name the tiles, catalogues 
   etc.. Need to describe this somewhere.  [Our current working plan is to
   use a file naming convention akin to what we use for wfcam, with the prefix
   w -> vir to accomodate vst in the same unique name system and any future
   vis.]
 * Channels for email communication were set up.
 * ESO agreed to provide us with the information about the grading of each OB 
   and also with the information needed for us to know if an OB has been 
   aborted for whatever reason or if needs to be scheduled for re-observation 
   due to some problem, so that we do not need to spend time processing 
   it or wondering why it was problematic.  ESO will check exactly 
   how to do this but we suggested some keyword in the FITS headers.
   [ESO will deal with doing this via their own DBs - we hope]
 * Need to send some data to Edinburgh so that they test their ingestion 
   scripts               
 * We spoke of what happens after the VISTA surveys in several years in normal
   telescope operations [during rather than after presumably]. Need to handle 
   proprietary periods but we already do this (e.g. UKIDSS, PIs).
 * General impression is that they are happy to work with us, and they are 
   happy to help so that the survey is a success (and they expect the same 
   from us) [ahhhhh]

Other stuff:

 * Maria Rosa concerned about sky subtraction. She wants to be able to detect 
   extended objects with low surface brightness [is that in the SMP ?] 
 * Maria Rosa asked about mosaicing and how we are going to deal with the 
   area which is not x6 [sic!] covered in each tile. Estimates ~7% of the 
   area affected by this. Especially since they have been asked to produce an 
   "homogenous survey". [Homogeneous is impossible for lots of reasons, but 
   apart from that the cataloguing software is smart enough to not attempt to 
   go deeper in say x4,6,8 exposed regions, so this particular worry should 
   not be a problem]
 * Maria Rosa asked if she can come to Cambridge at some point and learn 
   about data reduction. [maybe we should run a master class]
 * Somebody asked if there is a version of the tiling programs in CPL. ESO 
   plan anyway to publicly release the software, source code and 
   documentation. [no, but how much and hmmmmmm]


EGS and STH met with Magda Arnaboldi, Marina Rejkuba, Martino Romaniello 
and Eckhard Sutorius on Wednesday 17th September split into two  
~30 min sessions, notes from STH.

Topics which came up included:

1. Would we prefer the SV data to come daily or in weekly chunks ?  There 
were no promises that it could be done - but they could probably sort 
something out for the ~2 weeks of ESO SV. This would involve transferring 
the data on disk within Chile, then ftping it back to Garching. They estimate 
that we could have the data around 2 days after it was observed using this 
method, but it does involve significant extra effort on their behalf. Our 
response was that the time-saved would definitely be worth having, and that 
we would give them a more considered response after discussing with our group.
[It was felt that certainly for the 1st few nights of SV data it would be
very helpful to have faster access]

2. We discussed file-naming ...... we felt that the standard ESO naming  
scheme was fine. [We didn't, the names are not unique and wrap around
every year.  Neither is there any continuous run number, hence our earlier
plan to rename them on ingest and adopt a wfcam-like system here.]

3. For SV, ESO would like to copy the flat files from CASU in advance of them 
becoming available through WFAU. We need to agree a protocol.
[wget ...... via files soft-linked on web page]

4. For the SV data, Eckhard stated that there would be a 7-10 day delay  
between obtaining the flat files from CASU and there being available to copy 
from the archive. There would be a further 2 weeks for full ingestion and 
band-matching and construction of the tables etc..

5. Will this data be made available in one block, or in installments ?
[SV data - probably installments but depends on problems thrown up]

6. We asked them how we would communicate. They prefer us to use 
usd-help@eso.org, but they will also set up an email exploder for SV. 
We told them about casuhelp.

7. They would like the SV data to be mosaiced if possible (i.e. between tiles) 
[why ? catalogues can be "mosaiced" but what's the point of tiles-of-tiles
 for SV ?  We feel latter is outside scope of SV process]

8. We asked how they implement QC file grading to accept and reject files. 
This happens at the summit in real time. The OBs are flagged as: A (all files 
are fine), B (most files are fine) .. and presumably C or F .. anything not 
A or B is immediately put back into the observing  queue. It wasn't totally 
clear how they do this grading - it sounds like the responsibility of the ESO 
astronomer. Whatever, the OB grading will be put in a database (and/or 
possibly the files themselves). We said that we need access to these grades. 
[Aren't they going to be put in the FITS headers ? - see earlier].  
We also said we need to close the loop for files that we cannot process and 
that WFAU will need to do the same, for files that do not pass the PI 
inspection.

9. We agreed that there should be another meeting (they thought June or July 
which seems rather far off) to discuss these issues and further issues 
arising from SV.  [We probably need another meeting before SV and certainly 
will need one straight after SV has been first-pass processed.]

10. They asked how quickly we can make the SV data available. We reiterated 
one month from ingestion - but speculated that it "might" be possible to 
release a subset of some preliminary reductions on a shorter timetable if  
processing went well and we felt the data were good enough.

A couple of extra things that came up in the general meeting from responses 
to questions STH asked:

  we will get access to the OBs for SV to comment on before the observations 
  go ahead;

  the ESO DG will add of order ~nights of time for the surveys to complete 
  additional SV tests not already included in the Matrix (provided by ESO 
  from the SV program);

  there may be room to do dry runs using the survey MSBs in addition to  
  these extra nights.


Some actions from this lot below:

The directory structure proposed by ESO for the transfer of data was thought
reasonable. MJI will let them know via diplobag.                           <<<<

We would like the first few nights of SV data as soon as possible so that we
can provide prompt feedback. MJI will contact them as above.               <<<<

PSB will send some of the already processed VIRCAM test data to Edinburgh so
that they can test ingestion.                                              <<<<

There was a discussion about calibration observations that we would like from
STH's visit to Paranal and a brief diatribe about the necessity of         <<<<
"hard-coding" Ndit=1 into the twilight template, since the last thing you 
want with non-linear detectors and rapidly varying night sky is  Ndit > 1.  
JPE/WJS should be made aware of this                                       <<<<


6.  AOB

none worthy of comment


Continuing actions
------------------

STH  check that Plone astrometry technical pages are correct

STH  include graphic showing the size of the photometric radial distortion
     correction in the Plone technical pages and check rest of photometry
     pages are up to date                              
	       
JRL  produce web page of HawkI publicity for CASU
	       
MJI  finish off stage-I software release pages
EGS  

JRL  attend Cambridge VOTC meeting on 25-26 September
MR

STH  make photometric calibration paper available on Plone and VDFS sites

MJI  finish off technical paper


New actions
-----------

MJI  ensure feedback regarding CASU contentment with proposed directory 
     structure for transferring data gets ...... err fedback.

MJI  ditto about acquiring first few nights of SV data as soon as 
     possible after it is taken

PSB  send some processed VIRCAM test data to Edinburgh

STH  acquire the calibrations we desperately need for VIRCAM

MJI  make sure JPE/WJS are aware of the Ndit=1 issue for twilight flats