You are here: Home / Documents / Minutes of CASU Meetings / minutes_100826.txt


Plain Text icon minutes_100826.txt — Plain Text, 17 KB (17644 bytes)

File contents

Thursday 26th August 2010  APM Meeting tables 11:30 - 13:00

Apologies: MR, NAW


1.  Actions from last meeting
2.  Comments on WFAU minutes
3.  Meetings and Telecons
4.  Data archives update     - AAT, ING, WFCAM, UKIRT, VISTA
5.  Optical/NIR processing   - HAWKI, INT WFC, MegaCam, Subaru, VST
6.  WFCAM update
7.  VISTA update
8.  CASUhelps
9.  AOB

Minutes  (powered by Hot Lava Java Extra Dark Roast - strength 6  YO!!)

1. Actions from the last meeting

JRL  the first two lots of on-line notes are done and dusted, the rest is 
EGS  ongoing and will be completed soon for the CASUtools packages for
     cataloguing, list-driving, stacking and wcs-fitting.  MJI suggested
     including the sample fortran code to read the catalogue files and
     apply the assorted corrections - ongoing but getting there            <<<<

     RGM also reminded us that the NBSIZE parameter needs to be recorded
     in the FITS headers, all other relevant cataloguing parameters are.   <<<<

STH  has computed the VEGA to AB conversions for VISTA and plonked them in
     the technical section within the filter set web pages.  As a check he 
     recomputed the WFCAM set and agreed with the Hewett et al. values to
     1 mmag (using same Vega spectrum).  Comparing with the appropriate
     WFCAM and 2MASS values shows general agreement at the 0.01mag level
     for these conversions.  There was some discussion about including or
     otherwise Ax/E(B-V) values but it was generally felt that this was
     not a calibration issue per se since these values depend on which
     extinction law is being used.  RGM pointed out the table in Schlegel 
     et al. that is commonly used.  Since we also provide the effective 
     central wavelength of the filters on the CASU web pages it was thought 
     that this was enough for users to make their own informed choice.
     These extras still need adding to the calibration plan document.      <<<<

MJI  added an illumination table option.  The illumination tables (WFCAM)
     need to be linked somewhere on the web pages though.                  <<<<

RGM  still testing                                                         <<<<

JPE  contribute text to go with CASU versions of VISTA PR images - ongoing <<<<

JPE  contact ESO re: signing off pipeline deliverables - still ongoing     <<<<

EGS  has made good progress with SV processing, it will soon be back to 
     where v0.8 was
MJI  made zero progress after discovering the v1.0 SV NGC253 data was worse
     than v0.8.  To cut a long story short, this turned out to be due to a
     new and little used sky estimation option (simplesky - only used for the
     SV NGC253 project v1.0). This is being rectified even as we speak by
     reprocessing these with the tilesky option used previously - ongoing  <<<<

SC   the ZFS issues have been bypassed by the Vogons ......... 

JRL  has contacted Sybase regarding their hopeful demand for megabucks. CDS
     were also paying �1000 per year for their licenses and are as shocked as 
     us. No answer from Sybase as yet. Decided best not to stick head above 

JRL  has examined the reason for a set of peculiar stacked images in WFCAM 
     DR8 LAS data.  This was due to an inconsistent set of interleave offsets
     in a 2 dither 2x2 microstep sequecence.  This problem has not been 
     noticed before (nor after) and seems to be a rare occurence.  The files 
     have been left deprecated (SJW).

SC   have comprehensively explored VISTA disk storage options. ZFS and 
MJI  thumpers, which now cost way too much, have been abandoned in favour of 
     a different networked disk server option (see AOB).

2. Comments on WFAU minutes

Fortnightly ! ...... way to go

Possible flagging of spurious objects around bright sources isn't going
to happen in the cataloguing software.  Since most of these vanish when
band-merging (or similar) it seems more like a user-end problem to us.

WFAU have the latest CASUrtools software but this still does not include
the nebuliser and tiling software.  There was some discussion regarding the 
urgency of these final two bits of the puzzle, hereby boiled down to our 
current best guess of requirements for this: 

VHS doesn't need it urgently - though "ears" may later;
UltraVISTA are doing their own thing as are probably VIDEO ?;  
the majority of this season's VVV gets tiled on-the-fly during processing 
and its only the variablility aspect that might want it, or have we missed 
something. But even here it would be a seriously bad idea to use tiles for 
time series photometry ......;
so that leaves VMC and VIKING.  

We have no idea what VIKING is up to but note that we keep being bugged by 
Astrowise about processing details so we are guessing they are involved 
somewhere - but please enlighten us.  Likewise VMC, conflicting requests 
from mr. are confusing us.

There ensued a discussion led by RGM on apermags, corrected, not corrected,
naming conventions ...... usw ...... that confused everybody even more.

3. Meetings and Telecons

At the VISTA IOT on Tuesday 24th we learned that the camera has been 
recentred and tilted with respect to the optical axis to correct for the 
slight focus gradient. It is now hopefully re-aligned as the Prof ordered 
and everything seems to be working well, bar the usual few minor glitches.

Prior to the IOT we generated an updated zpt trend plot which unambiguously
shows a gradual decline in the throughput of the system.  Although only
a few % in the Ks band the effect is approaching 15-20% in the Y and Z-bands.
The primary suspect is .... the primary and further investigations are in
train to figure out the cause and ...... a solution.  JPE noted also that
the secondary mirror has not been recoated in its three year lifetime and
its reflectivity (as a function of wavelength) also needs further monitoring.

At ESO Marina Rejkuba is still fixing some recalcitrant OB grades that got
mangled in their logging process.  Apparently this problem will depart
for new data when the new OT tool is installed but meanwhile expect a
smallish proportion of the current OB grades in the FITS headers to need 
revising.  Due to the uncertain timescale for these to be fixed, we have 
gone ahead with v1.0 data releases and the next monthly batches will be
making an appearance shortly.   The OB grade updates plus other minor header 
changes, updated photometric/astrometric calibration etc., will be saved 
up for a v1.1 release.

There was the usual IOT digression on elliptical images and what, if any
the correlations were with ......... things.  MJI kindly volunteered
AKY to investigate using our own database info.                            <<<<

JPE raised a few things that had come up during a VMC telecon and that
he had clarified during the telecon: first, the tiled images themselves are 
not nebulosity filtered but the tile catalogues are derived from tiles that 
are; second, CASU do not directly provide corrections for saturated star 
magnitudes.  This is partly because even for pawprints there is no single 
correction that is optimal in all cases and for tiles it is of even more 
dubious value.  Should someone from CASU listen in on these telecons ?

About the only 1/2 decent use case so far for saturation corrections is mr.'s 
desire to study large amplitude saturated Mira's in the MC tiles - fair doos.  

In a related comment, MJI noted some meaningless (and misleading) plots of 
VISTA data products that accompanied the IOT agenda.  All of the apparent 
problems in these plots (stripes, weird patterns etc.)  were caused by 
saturation effects in tiled data, and from using poor quality data with
extremely high counts in the sky background in at least one band.  Plotting
distributions of parameters for saturated objects, particularly from tiles, 
could obviously do with a health warning.

As noted in the previous minutes there is a VST Public Surveys and GTO 
programs Review at ESO on 28/29th September.  The agenda features various
fascinating science things and also includes some items on process and
processing viz talks on the ESO QC process, the Astrowise pipeline and
the VSTube Naples (sleeps with the fishes) pipeline.

JPE was tasked with attempting to find out when the VISTA PSPI PhaseIII 
meeting at ESO is likely to be.                                            <<<<

STH pointed out ('cos he's going) JENAM is in Portugal on 6 - 10 September,
where he'll be talking about the Vista pipeline, nebulosity filtering, 
photometric calibration and lightcurves in the session: 
SYMPOSIUM 5 Star Clusters in the Era of Large Surveys.

4. Data Archive

ING, AAO, UKIRT, VISTA  all up to date and motoring

5. Optical / NIR Processing

Everything ticking along nicely. EGS is processing some IPHAS data from
October 2009 and MJI has taken up the cudgels on some Magellan IMACS
imaging data.

There was some discussion about HAWKI SV data of the Carina nebula that 
was processed by CASU over a year ago.  This was caused by the appearance 
of draft papers on the Chandra study of Carina which need at the minimum
browsing to establish which, if any, should include CASU folk in the 
author list.  STH will investigate.                                        <<<<

Plus an action on JRL to locate the HAWKI data so we can a.) figure out what
we did and b.) make some more colour pics                                  <<<<

6. WFCAM Update

All 10A data is fully reduced and the final lot of June, July data was 
flagged as ready for WFAU to copy on the 18th August.  It had all finished 
being transfered on August 22nd !

All the available 10B data has been received as expected and has been  
ingested into the raw data archive.  Processing of 10B is up to date with 
the latest batch of 3 nights chugging away.

ESO transfers of UKIDSS and Calibration data are also up to date.

No requests for raw data or early access to processed data have been  
made this month.

A video board for one of the quadrants of detector#4 was replaced recently
but no adverse effects have been noticed.

Around 3 weeks ago, Chris Davis (JAC) noticed some missing #4 data that he 
had taken on 29th July 2009. About 2/3 of detector#4 data had gone walkabout
prior to transfer for that night.  It looks like this is a very unusual
occurence and at a distance not clear what the underlying cause was.
JAC put the relevant data for 20090729 online for all cameras and it was
retransferred and reprocessed sans probleme.

Phil Lucas has been browsing in his extinction cupboard and spotted a small
problem with WFCAM calibration in regions of extremely high extinction,
E(B-V) > ~10.  The photometric calibration is out of spec in areas with 
high extinction. This is mainly due to the extinction coefficient term applied
to all WFCAM data.  This was derived from data with 0 < E'(B-V) < 2, where
E'(B-V) is the Bonifiacio et al. (2000) modified Schlegel et al. (1998)
extinction.  The error in the slopes (coefficients) becomes significant
by E(B-V)=10 leading to a systematic offset in the calibration in the 
J,H,K bands.  With the recent data in the Galactic Plane in the Bulge region
this problem has become more apparent.  Phil is carrying out his own
investigation for ways to improve the calibration in these regions and
STH and MJI are investigating a better approximation to the coefficients
for a much wider range of E(B-V) than was originally used.  The total no. 
of all WFCAM pointings affected is around 150, i.e. b'all, so there are 
several possible ways to proceed.  Watch this space      .                 <<<<

Narrow band observations of Jupiter tracked at the sidereal rate of Jupiter
were happily processed by the pipeline in its complete ignorance.  To, at 
least, MJI's surprise, even the stacked interleaved image products looked ok.

7. VISTA Update

We will be up to date with releases by the end of this week. Up to the
end of May v1.0 has already been released and up to the end of July will be
released by tomorrow (Friday 27th August). We have data up to 11th
August. There were a few bad nights with no data where the weather was
too bad for anyone to get to the telescope.

JRL and AKY are going to check that the raw data directories are correctly 
marked up when there is 'no data' or 'unprocessed data' i.e. no useful 
science data.

There needs to be a consistent way of treating junk data. Following
JRL's lead a new policy was converged upon. If the data is good enough
to process (e.g. 4+ good frames) then it gets processed. If there's
less than that (e.g. < 4 frames, or 4 low quality cloudy frames) then
they get deleted and an explanation is entered into a README file. As
with WFCAM, the individual images should be kept but the stack can be
chucked. When this happens it should be recorded so that PIs are aware;
though quite how this loop is best closed is a moot point.  AKY will
investigate producing a nightly log in the style of the WFCAM MSBTid.log
to summarise the QC info we generate and making it available via the web 
pages.                                                                     <<<<

It was noted that the OB status' are in the headers (since 15th October)
and are in a table on the web pages too.  As commented earlier, there are
still some problems with the values that ESO are supplying and these will
be resolved ....... eventchewally.

JPE and MJI noted that the next batch of ESO PR images will include the
Monoceros scarab beetle pic and following that an ambitious 40kx30k
colour image of a 3x3 tile region from the Orion SV data.  CASU will supply
the tiles and ESO will be doing the Orion gluing - best of luck.

JRL relayed from Marijn Franx the compliments of the UltraVISTA team on
the excellent quality of the v0.8 UltraVISTA data.  Let's hope that v1.0
maintains the standard.

8. CASUhelps

As a bit of light relief MJI relayed some examples of emails to, and replies 
from, Mme. CASUhelp.  The commonest reply was a variant on RTFWP or RTFOL, 
while "why do tile catalogues have missing headers for TV5_1 TV5_3 and 
TV5_5 ?" raised the biggest grin, thereby proving what a sad lot we are.

One of the more interesting questions related to "what happens if the 
extinction varies significantly during a tile OB ?".  The short answer
is nothing; slightly longer is that if the component pawprints of a tile 
have wildy varying extinction this isn't currently corrected for in the 
tiling; and the longer still answer is that the development tookit has
an option for dealing with this using the derived zpts for each pawprint.
This option would need to be automated along the lines of, if you believe
the variation in zpts apply the correction.  The issues of implementing
this fully hands off are still being thought through though it is a moot
point that, if the extinction really is varying significantly during a
tile OB whether or not the tile data is reliable enough to be of use, and/or
the best way to flag this in the tile headers.

9. AOB

SC has begun the process of trialling UBUNTU on one of the Blades.

A solution has been found to provide more storage space for VISTA in
the form of a 'Supermicro' system. Horrified at the price of purchasing 
Sun 'Thumpers' and the concomitant huge maintenance costs, we reversed the 
polarity for VISTA storage and are trialling a modern equivalent to what we 
originally designed in-house for WFCAM storage.  Basically a 4U PC disk 
server stuffed, in this case, with 36 lots of easily accessible 2TB disks 
and future JBOD expansion module(s) capable of similar 36 disk capacity 
each.  The server plus 1 expansion module will be configured to yield 4 
RAID6 disk systems totalling ~120TB of usable RAID6 storage and works out 
around 250 pounds/TB of realised storage. It has the usual in-built spare 
power supplies, fans etc...  We will start with the PC server unit (~64TB) 
and if ok get an extension unit (another ~64TB) which should cover VISTA
storage needs until the end of the current main 3 year grant period.

EMBS are in the loop regarding upgrading the motion detectors in each room
from binary, anything that moves ****** ..... to no its a moth you idiot.

Continuing Actions:

EGS  - finish adding catalogue software description to release pages
JRL    and add link to fortran example code

STH  - finish off last bits of calibration plan document updates

RGM  - finish off testing catalogue conversion software

JPE  - produce some text to go with the VISTA PR images

JPE  - contact ESO re: signing off pipeline deliverables
EGS  - SV time again still #2

JRL  - tie up loose ends with the sky subtraction.

New Actions:

JRL  - plonk NBSIZE background tracking parameter in FITS catalogue headers

MJI  - put in a link to the (WFCAM) illumination correct tables on the website

AKY  - investigate correlations of VISTA ellipticity measure with ??

JPE  - find out when VISTA PSPI PhaseIII meeting is likely to be

STH  - go through Chandra/Carina papers and identify those CASU should be on

JRL  - find the HAWKI data so we can check how it was done etc..

STH  - investigate gradual drift off spec of extinction corrections in high 
MJI    extinction regions in WFCAM data (and possibly also VISTA).

AKY  - investigate producing a nightly summary QC log for VISTA