minutes_100826.txt
minutes_100826.txt
—
Plain Text,
17 KB (17644 bytes)
File contents
Thursday 26th August 2010 APM Meeting tables 11:30 - 13:00
Present: JCR, MJI, STH, EGS, JRL, AKY, RGM, JPE
Apologies: MR, NAW
Agenda
======
1. Actions from last meeting
2. Comments on WFAU minutes
3. Meetings and Telecons
4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA
5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST
6. WFCAM update
7. VISTA update
8. CASUhelps
9. AOB
Minutes (powered by Hot Lava Java Extra Dark Roast - strength 6 YO!!)
=======
1. Actions from the last meeting
--------------------------------
JRL the first two lots of on-line notes are done and dusted, the rest is
EGS ongoing and will be completed soon for the CASUtools packages for
cataloguing, list-driving, stacking and wcs-fitting. MJI suggested
including the sample fortran code to read the catalogue files and
apply the assorted corrections - ongoing but getting there <<<<
RGM also reminded us that the NBSIZE parameter needs to be recorded
in the FITS headers, all other relevant cataloguing parameters are. <<<<
STH has computed the VEGA to AB conversions for VISTA and plonked them in
the technical section within the filter set web pages. As a check he
recomputed the WFCAM set and agreed with the Hewett et al. values to
1 mmag (using same Vega spectrum). Comparing with the appropriate
WFCAM and 2MASS values shows general agreement at the 0.01mag level
for these conversions. There was some discussion about including or
otherwise Ax/E(B-V) values but it was generally felt that this was
not a calibration issue per se since these values depend on which
extinction law is being used. RGM pointed out the table in Schlegel
et al. that is commonly used. Since we also provide the effective
central wavelength of the filters on the CASU web pages it was thought
that this was enough for users to make their own informed choice.
These extras still need adding to the calibration plan document. <<<<
MJI added an illumination table option. The illumination tables (WFCAM)
need to be linked somewhere on the web pages though. <<<<
RGM still testing <<<<
JPE contribute text to go with CASU versions of VISTA PR images - ongoing <<<<
JPE contact ESO re: signing off pipeline deliverables - still ongoing <<<<
EGS has made good progress with SV processing, it will soon be back to
where v0.8 was
MJI made zero progress after discovering the v1.0 SV NGC253 data was worse
than v0.8. To cut a long story short, this turned out to be due to a
new and little used sky estimation option (simplesky - only used for the
SV NGC253 project v1.0). This is being rectified even as we speak by
reprocessing these with the tilesky option used previously - ongoing <<<<
SC the ZFS issues have been bypassed by the Vogons .........
JRL has contacted Sybase regarding their hopeful demand for megabucks. CDS
were also paying �1000 per year for their licenses and are as shocked as
us. No answer from Sybase as yet. Decided best not to stick head above
parapet.
JRL has examined the reason for a set of peculiar stacked images in WFCAM
DR8 LAS data. This was due to an inconsistent set of interleave offsets
in a 2 dither 2x2 microstep sequecence. This problem has not been
noticed before (nor after) and seems to be a rare occurence. The files
have been left deprecated (SJW).
SC have comprehensively explored VISTA disk storage options. ZFS and
MJI thumpers, which now cost way too much, have been abandoned in favour of
a different networked disk server option (see AOB).
2. Comments on WFAU minutes
---------------------------
Fortnightly ! ...... way to go
Possible flagging of spurious objects around bright sources isn't going
to happen in the cataloguing software. Since most of these vanish when
band-merging (or similar) it seems more like a user-end problem to us.
WFAU have the latest CASUrtools software but this still does not include
the nebuliser and tiling software. There was some discussion regarding the
urgency of these final two bits of the puzzle, hereby boiled down to our
current best guess of requirements for this:
VHS doesn't need it urgently - though "ears" may later;
UltraVISTA are doing their own thing as are probably VIDEO ?;
the majority of this season's VVV gets tiled on-the-fly during processing
and its only the variablility aspect that might want it, or have we missed
something. But even here it would be a seriously bad idea to use tiles for
time series photometry ......;
so that leaves VMC and VIKING.
We have no idea what VIKING is up to but note that we keep being bugged by
Astrowise about processing details so we are guessing they are involved
somewhere - but please enlighten us. Likewise VMC, conflicting requests
from mr. are confusing us.
There ensued a discussion led by RGM on apermags, corrected, not corrected,
naming conventions ...... usw ...... that confused everybody even more.
3. Meetings and Telecons
------------------------
At the VISTA IOT on Tuesday 24th we learned that the camera has been
recentred and tilted with respect to the optical axis to correct for the
slight focus gradient. It is now hopefully re-aligned as the Prof ordered
and everything seems to be working well, bar the usual few minor glitches.
Prior to the IOT we generated an updated zpt trend plot which unambiguously
shows a gradual decline in the throughput of the system. Although only
a few % in the Ks band the effect is approaching 15-20% in the Y and Z-bands.
The primary suspect is .... the primary and further investigations are in
train to figure out the cause and ...... a solution. JPE noted also that
the secondary mirror has not been recoated in its three year lifetime and
its reflectivity (as a function of wavelength) also needs further monitoring.
At ESO Marina Rejkuba is still fixing some recalcitrant OB grades that got
mangled in their logging process. Apparently this problem will depart
for new data when the new OT tool is installed but meanwhile expect a
smallish proportion of the current OB grades in the FITS headers to need
revising. Due to the uncertain timescale for these to be fixed, we have
gone ahead with v1.0 data releases and the next monthly batches will be
making an appearance shortly. The OB grade updates plus other minor header
changes, updated photometric/astrometric calibration etc., will be saved
up for a v1.1 release.
There was the usual IOT digression on elliptical images and what, if any
the correlations were with ......... things. MJI kindly volunteered
AKY to investigate using our own database info. <<<<
JPE raised a few things that had come up during a VMC telecon and that
he had clarified during the telecon: first, the tiled images themselves are
not nebulosity filtered but the tile catalogues are derived from tiles that
are; second, CASU do not directly provide corrections for saturated star
magnitudes. This is partly because even for pawprints there is no single
correction that is optimal in all cases and for tiles it is of even more
dubious value. Should someone from CASU listen in on these telecons ?
About the only 1/2 decent use case so far for saturation corrections is mr.'s
desire to study large amplitude saturated Mira's in the MC tiles - fair doos.
In a related comment, MJI noted some meaningless (and misleading) plots of
VISTA data products that accompanied the IOT agenda. All of the apparent
problems in these plots (stripes, weird patterns etc.) were caused by
saturation effects in tiled data, and from using poor quality data with
extremely high counts in the sky background in at least one band. Plotting
distributions of parameters for saturated objects, particularly from tiles,
could obviously do with a health warning.
As noted in the previous minutes there is a VST Public Surveys and GTO
programs Review at ESO on 28/29th September. The agenda features various
fascinating science things and also includes some items on process and
processing viz talks on the ESO QC process, the Astrowise pipeline and
the VSTube Naples (sleeps with the fishes) pipeline.
JPE was tasked with attempting to find out when the VISTA PSPI PhaseIII
meeting at ESO is likely to be. <<<<
STH pointed out ('cos he's going) JENAM is in Portugal on 6 - 10 September,
where he'll be talking about the Vista pipeline, nebulosity filtering,
photometric calibration and lightcurves in the session:
SYMPOSIUM 5 Star Clusters in the Era of Large Surveys.
4. Data Archive
---------------
ING, AAO, UKIRT, VISTA all up to date and motoring
5. Optical / NIR Processing
---------------------------
Everything ticking along nicely. EGS is processing some IPHAS data from
October 2009 and MJI has taken up the cudgels on some Magellan IMACS
imaging data.
There was some discussion about HAWKI SV data of the Carina nebula that
was processed by CASU over a year ago. This was caused by the appearance
of draft papers on the Chandra study of Carina which need at the minimum
browsing to establish which, if any, should include CASU folk in the
author list. STH will investigate. <<<<
Plus an action on JRL to locate the HAWKI data so we can a.) figure out what
we did and b.) make some more colour pics <<<<
6. WFCAM Update
---------------
All 10A data is fully reduced and the final lot of June, July data was
flagged as ready for WFAU to copy on the 18th August. It had all finished
being transfered on August 22nd !
All the available 10B data has been received as expected and has been
ingested into the raw data archive. Processing of 10B is up to date with
the latest batch of 3 nights chugging away.
ESO transfers of UKIDSS and Calibration data are also up to date.
No requests for raw data or early access to processed data have been
made this month.
A video board for one of the quadrants of detector#4 was replaced recently
but no adverse effects have been noticed.
Around 3 weeks ago, Chris Davis (JAC) noticed some missing #4 data that he
had taken on 29th July 2009. About 2/3 of detector#4 data had gone walkabout
prior to transfer for that night. It looks like this is a very unusual
occurence and at a distance not clear what the underlying cause was.
JAC put the relevant data for 20090729 online for all cameras and it was
retransferred and reprocessed sans probleme.
Phil Lucas has been browsing in his extinction cupboard and spotted a small
problem with WFCAM calibration in regions of extremely high extinction,
E(B-V) > ~10. The photometric calibration is out of spec in areas with
high extinction. This is mainly due to the extinction coefficient term applied
to all WFCAM data. This was derived from data with 0 < E'(B-V) < 2, where
E'(B-V) is the Bonifiacio et al. (2000) modified Schlegel et al. (1998)
extinction. The error in the slopes (coefficients) becomes significant
by E(B-V)=10 leading to a systematic offset in the calibration in the
J,H,K bands. With the recent data in the Galactic Plane in the Bulge region
this problem has become more apparent. Phil is carrying out his own
investigation for ways to improve the calibration in these regions and
STH and MJI are investigating a better approximation to the coefficients
for a much wider range of E(B-V) than was originally used. The total no.
of all WFCAM pointings affected is around 150, i.e. b'all, so there are
several possible ways to proceed. Watch this space . <<<<
Narrow band observations of Jupiter tracked at the sidereal rate of Jupiter
were happily processed by the pipeline in its complete ignorance. To, at
least, MJI's surprise, even the stacked interleaved image products looked ok.
7. VISTA Update
---------------
We will be up to date with releases by the end of this week. Up to the
end of May v1.0 has already been released and up to the end of July will be
released by tomorrow (Friday 27th August). We have data up to 11th
August. There were a few bad nights with no data where the weather was
too bad for anyone to get to the telescope.
JRL and AKY are going to check that the raw data directories are correctly
marked up when there is 'no data' or 'unprocessed data' i.e. no useful
science data.
There needs to be a consistent way of treating junk data. Following
JRL's lead a new policy was converged upon. If the data is good enough
to process (e.g. 4+ good frames) then it gets processed. If there's
less than that (e.g. < 4 frames, or 4 low quality cloudy frames) then
they get deleted and an explanation is entered into a README file. As
with WFCAM, the individual images should be kept but the stack can be
chucked. When this happens it should be recorded so that PIs are aware;
though quite how this loop is best closed is a moot point. AKY will
investigate producing a nightly log in the style of the WFCAM MSBTid.log
to summarise the QC info we generate and making it available via the web
pages. <<<<
It was noted that the OB status' are in the headers (since 15th October)
and are in a table on the web pages too. As commented earlier, there are
still some problems with the values that ESO are supplying and these will
be resolved ....... eventchewally.
JPE and MJI noted that the next batch of ESO PR images will include the
Monoceros scarab beetle pic and following that an ambitious 40kx30k
colour image of a 3x3 tile region from the Orion SV data. CASU will supply
the tiles and ESO will be doing the Orion gluing - best of luck.
JRL relayed from Marijn Franx the compliments of the UltraVISTA team on
the excellent quality of the v0.8 UltraVISTA data. Let's hope that v1.0
maintains the standard.
8. CASUhelps
------------
As a bit of light relief MJI relayed some examples of emails to, and replies
from, Mme. CASUhelp. The commonest reply was a variant on RTFWP or RTFOL,
while "why do tile catalogues have missing headers for TV5_1 TV5_3 and
TV5_5 ?" raised the biggest grin, thereby proving what a sad lot we are.
One of the more interesting questions related to "what happens if the
extinction varies significantly during a tile OB ?". The short answer
is nothing; slightly longer is that if the component pawprints of a tile
have wildy varying extinction this isn't currently corrected for in the
tiling; and the longer still answer is that the development tookit has
an option for dealing with this using the derived zpts for each pawprint.
This option would need to be automated along the lines of, if you believe
the variation in zpts apply the correction. The issues of implementing
this fully hands off are still being thought through though it is a moot
point that, if the extinction really is varying significantly during a
tile OB whether or not the tile data is reliable enough to be of use, and/or
the best way to flag this in the tile headers.
9. AOB
--------
SC has begun the process of trialling UBUNTU on one of the Blades.
A solution has been found to provide more storage space for VISTA in
the form of a 'Supermicro' system. Horrified at the price of purchasing
Sun 'Thumpers' and the concomitant huge maintenance costs, we reversed the
polarity for VISTA storage and are trialling a modern equivalent to what we
originally designed in-house for WFCAM storage. Basically a 4U PC disk
server stuffed, in this case, with 36 lots of easily accessible 2TB disks
and future JBOD expansion module(s) capable of similar 36 disk capacity
each. The server plus 1 expansion module will be configured to yield 4
RAID6 disk systems totalling ~120TB of usable RAID6 storage and works out
around 250 pounds/TB of realised storage. It has the usual in-built spare
power supplies, fans etc... We will start with the PC server unit (~64TB)
and if ok get an extension unit (another ~64TB) which should cover VISTA
storage needs until the end of the current main 3 year grant period.
EMBS are in the loop regarding upgrading the motion detectors in each room
from binary, anything that moves ****** ..... to no its a moth you idiot.
Continuing Actions:
===================
EGS - finish adding catalogue software description to release pages
JRL and add link to fortran example code
STH - finish off last bits of calibration plan document updates
JRL
RGM - finish off testing catalogue conversion software
JPE - produce some text to go with the VISTA PR images
JPE - contact ESO re: signing off pipeline deliverables
EGS - SV time again still #2
STH
MJI
JRL - tie up loose ends with the sky subtraction.
New Actions:
============
JRL - plonk NBSIZE background tracking parameter in FITS catalogue headers
MJI - put in a link to the (WFCAM) illumination correct tables on the website
AKY - investigate correlations of VISTA ellipticity measure with ??
JPE - find out when VISTA PSPI PhaseIII meeting is likely to be
STH - go through Chandra/Carina papers and identify those CASU should be on
JRL - find the HAWKI data so we can check how it was done etc..
STH - investigate gradual drift off spec of extinction corrections in high
MJI extinction regions in WFCAM data (and possibly also VISTA).
AKY - investigate producing a nightly summary QC log for VISTA
