You are here: Home / Documents / Minutes of CASU Meetings / minutes_070426.txt

Plain Text icon — Plain Text, 16 KB (16818 bytes)

File contents

26th April 2007  Hoyle Committee Room 11:00 - 13:25

Present:          DWE, STH, JRL, PSB, MR, MJI, EGS, NAW
Apologies:        RGM
Guest appearance: Robert Greimel (RS)

Agenda
------

1.  Actions from last meeting
2.  Comments on WFAU minutes  - UKLight saga update
3.  NAM news
4.  Report from Second Generation VLT Instruments meeting
5.  Data archives update      - AAT, ING, WFCAM, UKIRT
6.  Optical processing update - IPHAS release
7.  WFCAM news                - raw data status
                              - processing status
                              - photometric calibration update
8.  VISTA news                - pipeline progress
9.  ADASS 2007                - latest
10. AOB                       - upcoming VDUC meeting
                              - cass38 and cass36
                              - rolling grant application


Minutes
-------

1.  Actions from last meeting

JRL  has not repaired the AAT archive mirroring (PostgreSQL problems),
     however some progress has been made since Stuart Ryder has been in
     touch. Since this issue seems very much out of our hands, it
     has been decided to drop this as an action and pursue it as part
     of the archive redevelopment.
      
MJI  did not email Matthew Colless regarding the above problem. This action
     has now been superceded by the above events

RGM  has not reported back on the classification aspect of cross-talk weeding.
     EGS and MJI are looking at classifcation generally and will contribute
     to this action - ongoing                                              <<<<

EGS  prepared the ADASS 2007 preliminary announcement web page -
MR   http://www.adass.org:8080/Conferences/2007/Venue/

MR   continues to investigate credit card payments for ADASS 2007
     - ongoing                                                             <<<<

PSB  is waiting for a quote for wireless provision for ADASS 2007 from a
     local (Cambridge area) contact. 

WJS  the status of field rotation flat field tests and VISTA integration 
     is unknown.  MJI will raise this issue at the VDMT meeting tomorrow.  <<<<
     MJI noted that WJS has now officially left the IoA as of 31st March
     and all wished him well in his new job.

JRL  has not found out more about the previous providers of wireless services 
     at Kensington Town Hall. This action has been superceded by a previous
     action.
      
RGM  has emailed Tim Naylor, JPE and WJS about discussing the updated VDFS 
     UK User Requirements Document as part of the VDUC agenda.  MJI said that 
     he would also raise this directly at the VDMT meeting on Friday.      <<<<
     The VDUC meeting will be held in Edinburgh on 10th May.  STH said that 
     there will be a UKIDSS Survey Heads meeting in Oxford on 9th May. There 
     will also be a calibration meeting tagged onto this (before or after).
     NAW said that he would email Tim Naylor to obtain the agenda for the
     VDUC meeting and the minutes from the previous meeting.               <<<<

MR   has not contacted the Durham group about raw WFCAM data processing
     progress. It was intended that this not be done until the end of April 
     to give them a clear month to work on it - ongoing                    <<<<

MJI  distributed drafts of conference proceedings of ESO calibration
JRL  workshop
MR

ALL  commented on above drafts

MJI  will ask about the possibility of obtaining more VISTA test data at 
     next VDMT meeting - ongoing                                           <<<<


2.  Comments on WFAU minutes

PSB reported that the UKLIGHT connection is almost "up and running" between
Edinburgh and Cambridge.  Some problems still had to be sorted out after the
original announcement and there is still no SSH connectivity. Ping works ok,
but physical tests have shown that the signal is at the low end of the spec.

There then followed a discussion on fast links to ESO and ESAC (for GAIA). 
It was felt that making progress on a fast link to ESO is currently akin to 
flogging a dead horse.  However,  NAW and PSB agreed to discuss the ESAC
link possibilities offline to see how they might progress this angle.      <<<<

STH noted that WSA had installed a new toolkit and asked if this included
list driven photometry. MJI said that it did not since that has not yet
been supplied. It was also noted that DR2 for GPS had now been released.


3.  NAM news

EGS, RS and NAW attended NAM last week. EGS reported that Michael
Rowan-Robinson had said that photometric redshifts "don't work for the WFCAM
filters in the DXS, but that it did for 2MASS data". This was regarded as odd
since WFCAM photometry is calibrated from 2MASS and the colour terms are 
small. EGS said that he will look into this.                               <<<<

Also mentioned at the meeting was that the funding priorities for STFC (nee
PPARC) will be for science-based projects that are UK-led.


4.  Report from Second Generation VLT Instruments meeting

JRL and PSB reported on the 2nd Generation Instrumentation Pipelines meeting
held at the Max-Planck-Institut fur extraterrestrische Physikon
18 April (http://www.mpe.mpg.de/ir/meetings/vlt2gi.php). Many instrument
groups were present and they almost outnumbered the ESO personnel present.

Many CPL issues were discussed. One of the main criticisms of CPL is that
you can't update a file - you have to completely rewrite it.

QFITS will be replaced by CFITSIO in version 4.0 of CPL and access to WCSLIB 
will be included. This FITS i/o change will not make much difference to our 
software deliverables since the QFITS layer is hidden underneath the CPL 
interface.  This change may open some interesting possibilities regarding
use of Rice tile compression.

NAW asked about Reflex (lead by Richard Hook).  This is a modular system 
unlike the monolithic recipe-based CPL approach.  It was noted that this is 
only funded until November and will then be reassessed.


5.  Data archives update

AAT:
JRL reported that we have recently received some more data (on DVDs) which
has now been read onto disk.   However, it was noted that this archive is 
currently offline due to problems with mirroring the AAT end of the data base 
(which mainly contains an ingest of the header information).

UKIRT:
JRL said that more data has just been received and that it is in the process
of being ingested.

ING:
MJI reported that internet transfers from La Palma are proceeding routinely.
RWA has gone to La Palma to try reading in the remaining 300 or so early 
vintage (VMS backup) tapes that are causing problems with our tape readers.

WFCAM:
MR said that all the raw data (all chips) has been read in up to the end of 
5th April except for 6 nights in January and a few other possible problem
nights with apparently missing files in November.   These are still being 
chased up.  JRL mentioned that there is a group keyword problem with some 
of this early 06B data.

PSB is carrying out FTP tests with Brad at JAC and obtaining 64kBytes/s
with a single-threaded transfer.  This seems to be a software limit, since 
multiple FTP sessions can be set up, each getting the same data rate. It is 
likely that a solution can be found to increase this.  Even with an increase 
to the "wire limit", this may be much slower than say internal UK transfer 
and may only be useful for occasional transfers eg. missing files.  
Investigations are still ongoing.  Meanwhile PSB was asked to email a copy 
of his findings to Brad (CC'd to Andy Adamson).                            <<<<

ESO transfers have been carried out up to 24th March with the ocassional
hiatus with what appears to be disk changeovers occuring.  MR will ask
for more details since its a tad inconvenient having to keep stopping and
restarting the transfers.                                                  <<<<

JRL said that there was a press release about the Kognitio data base system
that he has tested with 2MASS data. This system uses multiple processors 
and highly parallelised queries to avoid indexing tables.  It is impressively
fast at dealing with queries.  Kognitio are keen to carry out further tests 
with the IPHAS dataset.  


6.  Optical processing update - IPHAS release

EGS reported that the data from 2003 to 2005 has been ingested and that the
aim is to have a public release on 1 June. The documentation for this is
being written by EGS, NAW and RS and a paper will be placed on astroph
soon. NAW said that there should also be a press release. There was then
some discussion about how the Kognitio system would cope with a merged data
base of IPHAS, 2MASS.  A complete cross-federation with 2MASS was suggested 
as an interesting, and useful, challenge.  This is definitely worth pursuing 
given the results so far.  JRL said that he would get in touch with
Kognitio to see if they would be interested in doing some further tests.   <<<<


7.  WFCAM news

MJI noted that it is time to think about getting some more disk storage 
space, probably more of the same fibre channel RAID6 systems.  There was also 
some discussion about what to do with the two extra spare 750 Gb disks. 
EGS said that he could use these as backups for the IPHAS data base.  MJI
said that he would double check that there were two extra spares first.    <<<<

MR said that, ignoring data before 20 December and the incomplete 6 nights 
in January, only a few more nights of 06B (up to 31st March) data remain to 
be processed.

MJI reported that our request to JAC for doubling the dither offsets for 
twilight flat sequences is now happening.  This helps get around the 
occassional mega bright star screwing up the flatfield frame stacking.

MR noted that there are still a lot of incomplete groups and that typically
this occurs because the 1st exposure in a group sequence is badly trailed
causing the sequence to be interrupted and restarted.

MJI noted that we now should be putting the night log comments into the 
processed directories for completeness.                                    <<<<

He also noted that he had received an email from AJA about long exposure
(generally NDR) light leaks in dark frames.  He and MR will keep a weather
eye out for these problems.                                                <<<<

STH gave us an update on the photometric calibration. The Y band has been
done to +/-1%, but the Z band has not been finalised yet.  MJI asked about 
the consequences of not using SDSS preselection and STH noted that the same 
numbers were obtained. A meeting of the photometry working group in early 
May is planned.

MJI said that there will be a UKIRT board meeting in late May and that it
would be good to keep the web pages containing the relevant photometry 
calibration information updated since these are referred to in his draft
report to the Board.                                                       <<<<

STH said that investigations have been carried out into Eddington bias.
This is apparent when you compare deeply stacked data with shallow survey
data due to rms fluctuations biasing fainter than average objects above the
detection threshold.   However, since we cunningly use a 10-sigma detection 
threshold for the photometric calibration from 2MASS, the bias at the faint
end is only 1% and the overall bias in the derived zero-point is much
less than this.  This work is still ongoing but the effect, as expected, 
seems to be negligible.

MR was congratulated for his article on the properties of NIR sky brightness
in the UKIRT newsletter.

During some tests on LAS data in DR2 RGM and EGS noted a small effect (an 
offset averaging ~1%) when comparing DR2 band-merged products with CASU 
generated band-merged products.  EGS asked if the WFCAM photometry has been 
corrected to allow for the astrometric distortion.  MJI noted that this is 
allowed for directly when computing the photometric zero-points but is not 
applied directly to the catalogue fluxes since the principle has always been 
that the fluxes should just be the sum of the relevant pixel contents. As 
with the aperture corrections the intent has always been to allow for this 
on-the-fly when forming magnitudes.  The WCS information in the catalogue 
headers provides the necessary information.  MJI noted that the CASU 
band-merging software does compute these and apply them during the merging 
process and that this explains the small offset (~1%) discovered between 
the local band-merged products and those available in DR2.  MJI said that 
he would email NCH about this.                                             <<<<


8.  VISTA news

JRL said that he had received comments back from Sandra Castro regarding 
the 0.5 release.

The ongoing issue of continuing changes to the content of PAF files was 
discussed.  Various email exchanges have happened to try and put a cap
on this by getting ESO to define exactly what they want in the PAF files.
Convergence may be an illusion.

PSB raised the issue of what the exposure time value -v- NDITS contained
within the VIRCAM FITS headers means.  MJI said that he would also raise 
this item at the VDMT.                                                     <<<<

PSB was congratulated on writing a web page explaining how to put WCS
information within FITS table headers, as opposed to FITS image headers,
http://casu.ast.cam.ac.uk/documents/vista/astrometry/tables-wcs/view
is the path to enlightenment.

It was noted that from 07B, WFCAM catalogues will adhere to this format 
within their headers.  Although two parts of llareggub software can
interpret this, it is now the agreed FITS standard and does at least pass
fverify.  We expect WCSLIB to soon adhere to this form and when it does
we will adopt it for WCS info i/o.

MJI said that an FP7 proposal is being lead by ESO to improve the data link
between Paranal and Antofagasta in Chile. JPE is currently at ESO helping
to draft this.

There was no further news regarding hardware requirement/procurement for 
Paranal.


9.  ADASS 2007

JRL said that 3 proposals for tutorials have been submitted (Data mining, 
3d visualization of data, Relational data bases). Thus, there will be a
tutorial. Should there be a charge for this?  Yes otherwise = chaos. 
This is the usual precedent and will encourage people who have signed
up to turn up.

The next milestone is 1st May, when the registration page needs to be ready.


10. AOB

MJI said that cass38 has now been retired properly and that cass36 was next
on the list. There was some discussion as to what functionality from cass36
needs to be preserved.

MJI said that the rolling grant application will be submitted late May and
that input was required for various parts of the grant bid.                <<<<

Use of office space was discussed.  Due to the upcoming building work on
site pressure on space will be more acute.  PSB will be sharing with JRL
and this has to happen before September.  Earlier than this is desirable
since the process of clearing out the SPO will be very disruptive.
Further discussions took place concerning use of the "storage" room and
main central area, in particular making the central part more functional
for visitor use.


Continuing Actions
------------------

RGM  report back on classification aspect of cross-talk weeding
EGS
MJI

MR   sort out how ADASS 2007 credit card payments are to be made

MR   contact Durham group about raw WFCAM data progress (~end of April)

MJI  ask about obtaining more VISTA test data at next VDMT meeting

     
New Actions
-----------

MJI  ask about status of flatfield rotation tests aka VISTA integration

MJI  raise issue of updating the VDFS UK User Requirements Document 
     at VDMT meeting
     
NAW  email Tim Naylor to obtain agenda for VDUC meeting and minutes from 
     previous meeting

NAW  investigate possibility of fast links to ESAC (for GAIA)
PSB

EGS  investigate photometric redshifts in DXS

PSB  send FTP test results to Brad (CC'd to Andy Adamson)

MR   inquire about causes of intermittent nature of WFCAM transfers

JRL  contact Kognitio regarding further tests involving merged data base
     containing IPHAS and 2MASS 

EGS  sort out use of spare 750 Gb disks as backups for the IPHAS data base
MJI

MJI  OMP log comments in processed directories, make it so

MJI  asses impact of light leaks in dark frames on pipeline processing
MR

STH  update web page on the recent photometric calibration developments
     in lieu of the upcoming UKIRT board meeting (17/18 May)

MJI  email NCH to remind him about the photometric corrections needed to
     account for the astrometric distortion

MJI  raise issue at VDMT meeting of what is meant by exposure time value 
     contained within VISTA FITS headers
     
ALL  provide input on various activities for rolling grant application