You are here: Home / Documents / Minutes of CASU Meetings / minutes_030417.txt

Plain Text icon — Plain Text, 12 KB (12518 bytes)

File contents

17 April 2003 

Present  : MJI, JRL, STH, PSB, DWE
Apologies:
Attending: JPE

Agenda
------

1)  Actions from last meeting
2)  Comments on WFAU minutes and CDR papers 
3)  Report from Edinburgh CDR (JRL,MJI)
4)  Results and implications of JAC summit hardware review (JRL,MJI)
5)  Hardcopy of raw data to ESO who ? how ? (MJI)
6)  Grant status (MJI)
7)  Management documentation and tasks/deliverables (STH)
8)  Update on work done for WPs
9)  Status of web pages (DWE)
10) Group name 
11) ESO deliverables/responsibles - what do we need for PDR ? 
12) AOB 


Minutes
-------

1)  Actions from previous meeting

* PSB has liaised with JPE.  Item 11 on agenda will cover this.
* DWE has restored the internal directory protections 
* STH has produced PS versions of the Gantt chart (with help from Microsoft
      newsgroups). This has been placed in ~wfcam/docs/ and been agreed.
* ALL the .away files have been filled in a bit more. It would be useful if 
      bank holidays could be filled in as well.
* MJI drafts of the VPB monthly reports will be made available to CASU.
* STH distributed suggestions for name change (see item 10). 
* DWE has updated the web pages (see item 9).
* ALL project documents have been checked and agreed as OK.
* MJI there has been progress on JAC-CASU data transfer (see item 4).
  JRL

2)  Comments on WFAU minutes and CDR papers 

The people in, and associated with, the WFAU group were described for PSB 
so that he could get a better idea of who does what and who to contact if 
needed.

As there were no further specific comments raised based on the CDR papers 
prior to the CDR review, issues arising from the CDR are listed under
the next agenda item.


3)  Report from Edinburgh CDR (JRL,MJI)

An impressive body of documents were presented at this meeting and are 
available through our web link (and on /data/cass49_b/mike/VDF*.ps).  
There will be a written report by the review panel later.  MJI and JRL 
were suitably impressed by the amount of effort and preparation made for 
the CDR and the stepwise plans to the real WSA.  The schedule is very 
tight but let's wait and see.  The following specific issues that impact
directly on CASU activities were either raised at the CDR or arose from 
MJI/JRL discussions at the meeting:

In the Critical Design Review, document 7, there is a brief section on 
photometric calibration plans.  STH should have a look at this (liaising
with NCH) and investigate if using the 2MASS model is sensible.            <<<<

MJI pointed out that WFAUs Version 2 now seems to end after Q1 of 2005. 
He thought that this was probably related to aligning with the grant 
funding.  JPE queried this since V2 is supposed to end Q4 2004 and that 
this should not have changed (JPE will investigate).

Ian Bond showed some demos of difference imaging software using the
UKST H-alpha survey. DWE should contact him to see what he's done          <<<<
(and read 2001MNRAS.327..868B) and try and get a copy of the software.  
Interpolation schemes may still be an issue.  This has an impact on
WP 15.

MJI got the impression that WFAU expect the CASU-supplied mosaicing 
software to be able to cope with "arbitrarily large mosaics" when it is
run as part of the WSA.  Overlap calibration and several other issues
are implict here which MJI thought were WFAU responsibilities.  MJI to 
clarify this.                                                              <<<<

Eckhard Sutorius has been investigating the data transfer bottleneck
between ROE and Cambridge (ftp-like from Solaris ftp server here gives 
1 MB/s; scp from apm3 gives 2 MB/s - whereas MJIs scp test from Leicester 
to here gave 4-5 MB/s).  This is probably related to the firewall that 
they have. Eckhard suggested that the default buffer TCP-IP buffer size 
be increased from 64KB to 256KB.  It was felt that putting a fast linux 
machine outside their firewall and repeating the test would be useful. It 
was generally thought that this was an ROE problem.  Help from CASU
would be available if they ask for it eg. buffer size increase.

Luc Simard (CADC) pointed out the use of Rice compression (built into 
CFITSIO) MJI and JRL did some tests and were impressed with the ~<factor>
4 achieved. Although lossy with real nos. compression, it is not lossy 
with integers and is much faster and compresses more than gzip or bzip2. 
There are CFITSIO routines (internal and external eg. imcopy) for this 
such that the header remains uncompressed AND the level of loss is 
selectable.  STH was not too sure about this since it would be unlikely 
that any (current) image display programmes would be able to cope with 
these files (true but they could be Perl-wrapped).  MJI/JRL/STH need to 
investigate this further for generic CASU data storage issues.             <<<<

There was some concern about the naming and propagation of the master 
calibration files.  More thought is needed here eg. if more that one 
master dark, flats, etc. are generated in a night, ditto the skys
generated throughout the night.  JRL to engage brain.                      <<<<

There was some discussion regarding the roles of the JAC-based UKIDSS 
Survey Manager and the Survey Scientist (SJW).  Where are the 
responsibilities split particularly with regard to survey progress/
planning tasks?  Do we know who will be the survey manager?  DWE was 
asked to contact Steve Warren to try and clarify this.                     <<<<


4)  Results and implications of JAC summit hardware review (JRL,MJI)

The hardware setup was explained. There would be 4 DAS systems and 4
data processing systems, one for each detector - plus one spare and
a JAC real time processing system.  Each processing unit is a PC + disk 
+ tape unit.  The 4 streams of data will be independently processed and 
independently output to tape.  For export to CASU the summit pipeline 
will dump the raw SEFs to tape.  JAC will dump the raw NDFs to tape for
their local copy.

The issue of the observer comments that go into the summit database
was raised.  These should go into the FITS headers.  Where ? At the
summit (probably not) at ingestion and conversion to MEFs in Cambridge
(probably since we will mirror the DB here).  JRL to ponder.               <<<<

The tape systems will be Ultrium LTO-2 with a native capacity of 200 GB. 
and a sustained data transfer rate of 100 GB/hr.  We would get the tapes 
roughly on a weekly cycle in batches of 4-8.   The tapes cost $69 
each, a third of the cost of the IDE option.  Individual tape drives cost 
about 6k pounds each and can be combined in a "library" jukebox system 
based on 8 tape cartridge cassettes to give 1.6TB capacity per cassette.
Cambridge would probably buy a 2 drive jukebox system and also use it
for backups.  MJI and JRL were satisified with this solution since it 
automatically gives us an offline raw archive solution.  The only caveat
is reliability of tape.  Need to instigate some tests next quarter.

This is also a fallback option for sending data to ROE (if bottleneck 
remains or network usage costs are high) since they are using a similar
setup for WSA backups.  MTB will be asked to investigate these drives.     <<<<


5)  Hardcopy of raw data to ESO who ? how ? (MJI)

Network -v- tape -v- disks

JPE said that he would seeing Peter Quinn (ESO) on 30th April and will 
discuss the matter then. It is likely that ESO will prefer IDE drives. 
Although there will probably be hardware costs (who pays?) associated 
with whatever solution used, manpower will also be a significant cost.

It was felt that delivering the raw Multi-Extension FITS (MEF) files to
ESO was desirable.  These are only generated at ingestion of tape data
into the CASU processing setup.


6)  Grant status (MJI)

This has still not finished!  PPARC et al. are sorting out problems at 
the detail level. It is expected that this will be finalised in time for 
next months pay deadline and backdated to 1st April.


7)  Management documentation and tasks/deliverables (STH)

This was combined with 8.

8)  Update on work done for WPs

STH said that the documents have been time-stamped and top level copies 
made available on the web page.  Updates and alterations will be carried 
out in a systematic way.  

It was agreed that before each of these meetings we would report any
progress on the work packages to STH.  He would update the documents and 
give an overall report on progress and point out any problem areas for
discussion.                                                                <<<<

STH asked about naming conventions for the documents.

JPE pointed out that ESO have a naming/signing convention and won't accept 
any documents that don't follow this.  WFAU have decided not to follow this
convention for philosophical reasons. JPE would like us to follow the ESO
path.  STH will ponder this and report back.                               <<<<


9)  Status of web pages (DWE)

DWE reported that the web pages have been reorganized. No password is
now required for the pages except for an internal section intended for
Cambridge only. The other main change is that the diary has been moved 
from the front page to its own page.

STH suggested that the ordering of the diary page be reversed so that when
you enter the page you get the most up-to-date information. Another
suggestion was to reverse the months but keep the dates in order. DWE
suggested a compromise where all links to the diary page would take you to
the end of the page. This was agreed as an interim measure, but we should
all think about how the diary page is organized.                           <<<<


10) Group name 

No consensus has been reached about this. MJI (although he wanted a name
change) pointed out that there were a lot of consequences with a name change
and that a number of documents and diagrams would have to be altered. Some
of these don't originate from our group implying repercussions in 
various (unforeseen?) places.  This item will be deferred until a general 
group meeting is held when the official grant announcement arrives.


11) ESO deliverables/responsibles - what do we need for PDR ? 

There was a lot of discussion regarding what documents were needed for 
the Preliminary Design Review for the VDFS.  Filling these documents will 
be the responsibility of PSB, with help from various others, and needs to 
be completed by 1 July.                                                    <<<<

The three doorstops are:

DFS user requirements document
Calibration plan document
Data reduction specification document

Will Sutherland can help PSB with the science case summary and system 
properties with input also from the WFCAM docs.  The WFCAM pipeline CDR 
document can also be used as a basis for the calibration plan and
data reduction plan.  JPE has provided ESO template documents to PSB
which are also available on our internal page.

There is an ESO requirement that all submitted documents are in PDF format.
However, there is no restriction as to how they might be generated (LaTeX 
or Word). All VISTA documentation is in Word format.

For VISTA, RAL/ATC/Durham are building the camera, while the telescope is
being built by Vertex-RS.  ATC are in charge of managing both projects.  
PSB will need to liaise with the camera team (Camera Scientist Gavin Dalton) 
and with Malcom Stewart and Steven Beard from the VPO.


12) AOB 

None - exhaustion and Easter had set in.


Actions
-------

STH  look at CDR document 7 (photometric calibration) and contact NCH
     about photometric calibration plans

DWE  contact Ian Bond at ROE re. difference imaging and read his MN paper

MJI  find out what the mosaicing requirements really are and who is doing what

MJI  investigate image compression some more and see how we could make use 
JRL  of it in generic CASU data processing activities
STH

JRL  deep thought about the naming of the master calibration files.

DWE  contact Steve Warren to get clarification regarding the responsibilities
     of the UKIDSS Survey Manager and Survey Scientist (and find out what's
     happening re. survey planning in general)

MTB  investigate Ultrium LTO-2 tape drives

JRL  observer comment database, plan to ingest in FITS headers, where ? how ?

ALL  report progress on work packages prior to meetings

STH  naming conventions for documents

DWE  alter links to diary page to take you to the bottom

All  think about organization of diary page

PSB  collect info needed for VDFS documents for PDR