Jump to content


Photo

UCAC4/APASS SDSS u'g'r'i'z' differences

  • Please log in to reply
27 replies to this topic

#1 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 14 December 2013 - 06:43 AM

Hi-

I have dabbled in photometry for a while but am now trying to do it more seriously. I decided to go with Astrodon g',r',i' filters because it appears to be a growing standard, and I like the idea of well-defined cutoffs. I'm aware they may not be ideal for a number of reasons.

My question is mostly academic, but I'm interested in the actual absolute accuracy one can obtain in measuring, for example, g' and r' for a given unknown star.

The UCAC4 catalog conveniently includes APASS g',r' magnitudes - and they are claimed to be reliable fainter than mag 10, and provide sigmas for the values.

At the same time, there are SDSS standard stars, including values published by Smith in 2002 for the u'g'r'i'z' system - also with sigmas.

What puzzles me is that the values for the two systems don't agree all that well. As a specific example, for Hilt 31, Smith has r'=10.996 +/- 0.001, while UCAC4 has r'=10.972 +/- 0.001.

If the sigma's provided are simply standard deviations of the measurement then the discrepancy is expected - since it amounts to precision rather than accuracy. But for this standard star the two measurements are off by 0.024 - well outside the 0.001 sigmas.

So in terms of making an absolute measurement of a star, how can you calibrate your system accurately if the standards themselves have a fairly large variation?

If a nova appears somewhere and everyone chooses reference stars in the field and provides magnitudes everyone accepts - then the measurement from everyone will all be fairly consistent. But in absolute terms it will depend on the system used to define those reference stars.

I know that the ugriz system has been evolving and the Astrodon filters I have are already slightly out of date with the newest ones - but shouldn't there be more agreement in the measurement of standard stars - in absolute terms?

Thanks,
Frank

#2 brianb11213

brianb11213

    Cosmos

  • *****
  • Posts: 9047
  • Joined: 25 Feb 2009
  • Loc: 55.215N 6.554W

Posted 14 December 2013 - 07:25 AM

So in terms of making an absolute measurement of a star, how can you calibrate your system accurately if the standards themselves have a fairly large variation?

You measure loads of stars ... to identify those that do have significant variation ... and then take an average. The residual variation is a measure of transmission variations due to thin high cloud etc. which are not always obvious at the time the measurements were made.

If a nova appears somewhere and everyone chooses reference stars in the field and provides magnitudes everyone accepts - then the measurement from everyone will all be fairly consistent. But in absolute terms it will depend on the system used to define those reference stars.

Actually, "absolute" matters relatively little for many purposes, it's the light curve that's important. In any case the variations between stars in the same field should be a lot smaller than the "across the sky" measures necessary for calibrating absolute measures.

There's another point here - measures from an individual instrument with "standard" filters will be affected by the transmission / reflection variations in spectral response of the optical elements in the scope and also by the response curve of the sensor.

I'm quite sure that, whilst detecting variations between observed star & comparison star to an accuracy of a few millimagnitudes is not particularly difficult, generating tables of "absolute" magnitudes of objects spread across the sky with a whole bunch of different instruments & sensors which agree to the order of 0.1 mag is a far more difficult task.

#3 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 14 December 2013 - 07:57 AM

Well - if the standards themselves don't agree in an absolute sense, then the underlying scale is somewhat ill-defined.

I guess my main point is that the magnitudes in these studies are calculated with care - but there is no attempt to provide a standard error of the measurement - only a sigma standard deviation. This makes it impossible to say anything about the accuracy of the standard values.

To put it in practical terms, I see about 0.05 error in g' between Smith and APASS using only stars with gmag between 12 and 14. This is comparable to the absolute measurements I make with an 8" sct and 50 second total exposure time, after I calibrate based on UCAC4 values. It effectively means I am doing as well as anyone could possibly do in terms of providing absolute measurements, which surprises me.

I don't know if Smith's values are inherently better or if both sets of values are equally in error. I'm mainly surprised that the sigmas don't appear to include systematic error.

Frank

#4 Ed Wiley

Ed Wiley

    Apollo

  • -----
  • Posts: 1049
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 15 December 2013 - 12:56 PM

One thing you might do is dig into to sources of the data using the databases at Strasbourg.

http://cdsweb.u-strasbg.fr/

You frequently can run down the original source and see what kind of errors are reported. I suspect the errors of sigma = +/- 0.001 concern precision, not accuracy. I would guess statistically significant number of "independent" measures would be needed to access accuracy.

Ed

#5 StupendousMan

StupendousMan

    Vostok 1

  • -----
  • Posts: 180
  • Joined: 21 Aug 2005

Posted 15 December 2013 - 10:15 PM

(Disclaimer: I worked with Smith on SDSS photometry)

So, you've discovered a very important point: the difference between "accuracy" -- the degree to which a measurement agrees with an external value -- and "precision" -- which often means the difference between repeated measurements with the same instrument.

In this case, the Smith et al. measurements for stars in the ugriz system are correct; that is, Smith et al. used the standard SDSS ugriz system to measure a set of stars against a few fundamental standard -- white dwarfs with very carefully measured spectra. Smith et al. DEFINED the SDSS system.

The APASS catalog is based on measurements made by different telescopes with different filters. The APASS astronomers did their best to transform their magnitudes so that they agree with those measured by Smith et al., but small systematic error persist. You have noticed those systematic errors.

The quoted uncertainties in the APASS measurements do NOT take into account these systematic errors. It's a common practice in astronomy (and probably in most scientific fields) to leave systematic errors out of catalog listings.

If you want to do the best you can, I'd recommend that you spend a few hours observing a few fields which contain 3-5 stars from Smith et al. Make measurements in all your filters, and then compare carefully your raw magnitude values against the SDSS catalog values. Look for linear trends in the differences as a function of stellar color, and determine the color terms which convert your measurements to those of Smith et al., and vice versa. Then use those color terms to reduce your other measurements, too.

This will take a lot more work than simply using the APASS magnitudes of stars which are in the field of your target. Is it worth the extra effort to reduce the systematic errors in your magnitudes from, say, 0.03 to 0.02 or 0.015 mag? Well, that's up to you. For some purposes, it wouldn't matter, and for others, it would.

#6 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 16 December 2013 - 06:20 AM

Thanks for the reply. Yes - I'm aware of the difference between precision and accuracy - and I am specifically looking for estimates of accuracy.

The Smith values couldn't simply define the mags since they need to be quantitatively consistent with the underlying scale.

Overall my impression is that APASS may have inherent error of 0.05, while Smith may be around 0.01-0.02 - based on Ivesic 2007. So that would answer my main question that the APASS are inherently less accurate than the numbers from Smith - and that I could possibly improve my calibration by using only stars from Smith. But I shouldn't expect better than 0.01-0.02 - no matter how hard I try.

I already have a number of data sets that include the SDSS standard stars, so I'll try calibrating against only them and not APASS. But I would prefer to have actual estimates of errors so I can weight things appropriately and do error propagation. The sigmas provided seem only useful as proxies for data quality.

Frank

#7 StupendousMan

StupendousMan

    Vostok 1

  • -----
  • Posts: 180
  • Joined: 21 Aug 2005

Posted 16 December 2013 - 08:50 AM

I have to disagree slightly with you: the Smith et al. values do, in fact, define the SDSS magnitudes. There is no "underlying scale" beneath them. The Smith et al. magnitudes are based on the following

a) spectrophotometry of the sub-dwarf BD+17 4708
b) measurements made with the SDSS Photometric Telescope, in its natural system

That's it. I guess you could say that the first point should really be "spectrophotometry of Vega, indirectly referenced via BD +17 4708."

Since Smith et al. actually do define the SDSS magnitude system, there are few chances for systematic errors to creep into their values. The small quoted uncertainties in the Smith et al. magnitudes, which are based on repeated measurements, should be reasonable values to use for error propagation.

#8 robin_astro

robin_astro

    Mariner 2

  • -----
  • Posts: 247
  • Joined: 18 Dec 2005

Posted 16 December 2013 - 09:09 AM

Looking at this from the point of view of a spectroscopist where I and others have been trying to reconcile spectroscopic and photometric measurements, it seems to me that the photometric method has fundamental limitation which means that any absolute measurement is inevitably an approximation. This is because although the method describes the filter passband it does not define the spectral response of the detector or the rest of the optics. The results from different systems (with the inevitable differences in spectral response) are made to agree with each other by comparing the results of measurements of standards and making appropriate (colour dependent) corrections. This quality of the agreement though will depend on the specific detailed spectrum of the target. Not sure how much effect this has in practice but given this limitation a 2% absolute difference between measurements using different equipment perhaps does not seem unreasonable.

Cheers
Robin

#9 brianb11213

brianb11213

    Cosmos

  • *****
  • Posts: 9047
  • Joined: 25 Feb 2009
  • Loc: 55.215N 6.554W

Posted 16 December 2013 - 09:20 AM

Looking at this from the point of view of a spectroscopist where I and others have been trying to reconcile spectroscopic and photometric measurements, it seems to me that the photometric method has fundamental limitation which means that any absolute measurement is inevitably an approximation. This is because although the method describes the filter passband it does not define the spectral response of the detector or the rest of the optics. The results from different systems (with the inevitable differences in spectral response) are made to agree with each other by comparing the results of measurements of standards and making appropriate (colour dependent) corrections. This quality of the agreement though will depend on the specific detailed spectrum of the target. Not sure how much effect this has in practice but given this limitation a 2% absolute difference between measurements using different equipment perhaps does not seem unreasonable.

Yup. That's about it, in a nutshell.

#10 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 17 December 2013 - 04:38 AM

Well Smith in 2007 talks about the standard stars as on the AB system and zero-point referenced to a single star. But since it uses the AB system it is ultimately aiming to measure flux from the standards stars in an absolute sense - and there will be error in the result. Although the 2002 paper doesn't talk about the error in detail, it does allude to future work to "reduce the error in the mean magnitudes of each standard star..." That error is exactly what I'm talking about and trying to determine. That paper doesn't quantify it except in terms of the "the systemwide rms errors in the final SDSS photometric catalog will be less than 0.02 mag in r'..."

In 2007 Ivezic alludes to the SDSS catalog itself having errors up to 0.05m, but I don't see values for the error of the standard stars themselves.

Any photometric measurement will involve the compound effects of the source spectrum, the filter passband, and the detector response - but even if those things are kept constant or corrected, there will still be other systematic errors that affect the accuracy of the result.

So my conclusion is that - with care you can use the UCAC4 photometry to get to perhaps 0.05m, and the SDSS standard stars my get you to 0.01-0.02. I don't think it takes too much effort to have results limited in accuracy by the UCAC4 values - but my impression is that the SDSS values are better but I'm not sure by how much.

Frank

#11 StupendousMan

StupendousMan

    Vostok 1

  • -----
  • Posts: 180
  • Joined: 21 Aug 2005

Posted 17 December 2013 - 01:26 PM

You mention two separate issues.

First, the Smith et al. paper states: these are the relative magnitudes of a bunch of stars in the sky, measured with our particular telescope + filters + CCD. There are no systematic errors due to the spectral response functions of the telescope + filters + CCD (unless those responses changed with time). People who try to observe those same stars with their own equipment _will_ encounter systematic errors for those reasons, of course. Smith et al. then shift the zero-point of all their relative magnitudes so that one particular star (BD+17 4708) has some specific values. Fine. Still no contribution to systematic errors in the relative magnitudes.

Second -- someone tries to interpret the magnitudes in terms of fluxes on the AB system. NOW there will be all kinds of systematic errors, sure, due to imperfectly known properties of the filters and CCD and atmosphere and the spectrum of BD+17 4708, etc.

So, if all you want are to report magnitudes in the SDSS system by comparing your target to Smith et al.'s standard stars, you have to deal with systematics in matching your equipment to their equipment. But if you want to convert your measurements into fluxes, well, that's opening another can of worms.

What is it that you want to do?

#12 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 17 December 2013 - 04:39 PM

these are the relative magnitudes of a bunch of stars in the sky



Relative magnitudes of actual stars in the sky indicates an effort to measure actual physical properties, the relative flux of actual stars. Even if those values are relative to a single standard - they are still determined empirically - and will have some error associated. I am trying to determine what that error is so that I can relate my measurements to the fundamental errors in the standards on which they rely.

This is all basic experimental practice and is required for model fitting and error propagation. Simple standard deviations are useless for this sort of thing.

I wasn't sure just how well one could expect to do in determining a meaningful magnitude based on standard stars scattered around the sky. It looks like you can do around 0.02 mag - but not much better - and it is at least partly limited by the standards themselves. With APASS values it is around 0.05 - just based on how their measurements of the standard stars differ from the Smith values.

In short - what I want to do is quantitative photometry with meaningful error bars. I would use APASS values if their error was much less than my experimental error - but I don't think it is. Although you seem to insist that the SDSS values are somehow defined and have no error - obviously they couldn't be willy-nilly assigned arbitrary values, and there is a quantitative self-consistency involved to make them match the ground truth flux from the stars - with some error that would propagate into my calibration and affect my error bars.

Frank

#13 StupendousMan

StupendousMan

    Vostok 1

  • -----
  • Posts: 180
  • Joined: 21 Aug 2005

Posted 17 December 2013 - 06:49 PM

I'm sorry that I'm doing such a poor job of trying to explain things. Let me take a new approach. Let's count the ways that errors can creep into measurements of stars.

1. Photon noise. If you collect only N photons, poisson statistics suggests that the uncertainty in your count will be sqrt(N). Collect more photons to decrease your fractional uncertainty. Typically, this contribution to CCD photometry of standard stars is very small, of order a few millimags.

2. Correction to exoatmospheric values -- in other words, removing the effects of extinction. This requires extra work, such as making a number of measurements of stars at a range of airmasses. Even at very good sites, the atmospheric properties can vary over timescales of a few hours. This can lead to uncertainties of a percent or two on any particular night, though people who work hard at it and keep careful records can do better.

3. Transforming measurements to match some reference scale. If you compare your measurements to someone else's measurements, you need to perform this step. In some cases, the corrections can be done well and contribute only a small uncertainty -- for example, if one is studying stars in a narrow range of intermediate colors, within a rich field with many secondary standards. In other cases, such as the study of stars with unusual spectra (SNe, Miras) or use of unusual equipment (using a colored-glass filter instead of an interference filter), these effects can contribute many percent to the overall uncertainty.

4. Other sources of error. There are plenty, of course. My list isn't meant to be exhaustive.

What I've been trying to say here is that the catalog of SDSS magnitudes of Smith et al. contains no contribution from item number 3. Smith et al. were not trying to match their measurements to the measurements of any other group. They did not have to transform their measurements.

But the Smith et al. data _do_ suffer from effects 1 and 2 (as well as some others in category 4). Earlier in the conversation, you quoted from their paper a statement to the effect that they hoped to do

future work to "reduce the error in the mean magnitudes of each standard star..."

By that, they meant "if we observe these stars 10 more times each, on different nights, then we can start to beat down both the random errors due to photon noise (not so important) and due to correcting for atmospheric extinction (more important)."

If you want to do quantitative photometry with meaningful error bars, a good first step is to make measurements on 2 or 3 or 5 or 10 different nights. Do simple relative photometry of stars in the same field on each night, and compare the results. You'll see some small scatter around mean values. That sets the floor for the precision in your measurements. If you want to transform your measurements to one of the standard magnitude scales, then you'll need to figure out the typical uncertainty in _that_ procedure, and add it in quadrature to the uncertainty in your earlier results. That second step is where you'll need to incorporate the uncertainties from some catalog, whether it be APASS or SDSS. If you pick APASS, you'll likely have comparison stars in the same field as your targets: relatively big uncertainties in the catalog magnitudes, but no added uncertainties from the extinction corrections. If you pick SDSS, you'll very likely have to do the atmospheric corrections (adding uncertainties along the way), but the formal uncertainties in the catalog magnitudes will be smaller.

There is no obvious best procedure here, alas.

#14 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 17 December 2013 - 07:43 PM

I have already done measurements and calibrations based on the APASS values, and when I include known errors in my measurements I get scatter in the results that is larger than my error estimates, but on par with the discrepancy between the APASS values and SDSS values. That tells me that I could do better if I had better standard stars than APASS. But I don't know how much better I would do with SDSS - so my question from the start has been - how accurate can standard values realistically be. Ideally the standard stars would have good error estimates to go with them - so that I could use those error bars in a statistically meaningful way. I understand it is very hard to get good numbers for the systematic errors, and no attempt was made in 2002 - but there is at least talk of 0.01 as a target.

So I think that is the answer to my question in the first place. There are no good numbers for the errors in the standards, but around 0.01-0.02 is a ballpark.

It is harder for me to compare a calibration based only on SDSS standards since there are so few of them compared to APASS, but I will see what I can do with the imaging sessions I have already done.

Frank

#15 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 21 December 2013 - 04:37 PM

Here are some updates on this g'r' stuff if anyone is interested. I don't see many write ups of either photometry with Sloan filters, or their use in combination with UCAC4/APASS, so I'll just post some results here.

First is a plot of SDSS mags vs. UCAC4/APASS mags using the values in Smith 2002. This is straight from the two sets of data and involves no measurements by me.

In a wide view there is a big departure for stars brighter than mag 10, which isn't a surprise, but there is still scatter in the mag 11-13 range.

Frank

Attached Files



#16 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 21 December 2013 - 04:38 PM

Here is the same plot just using the central region.

Frank

Attached Files



#17 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 21 December 2013 - 04:47 PM

Here are some results based on my measurements of several fields over a range of Airmasses under fairly stable conditions, with 8" sct and 5x10s exposures with c, g', r' filters.

Many parts of the sky have a good number of UCAC4 stars with APASS magnitudes, but some of the SDSS standard stars away from the equator are in areas with no APASS comparison values. In addition, Nova Delphini 2013 has no APASS or SDSS standard stars in a field captured by my equipment. This makes it harder to do all sky photometry.

Nonetheless I was able to do a calibration based on first order extinction using the APASS values for calibration. I could then check the residuals relative to the calibration to judge self-consistency. I then used the Smith/SDSS values as a form of independent check on the quality obtained from an APASS calibration.

Here is the residual in the g' of the UCAC4 magnitudes when calibrated using a large number of UCAC4 stars.

Frank

Attached Files



#18 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 21 December 2013 - 04:50 PM

Here is the residual using the sdss standard stars relative to their calibrated values. This shows more error, and if the sdss values are inherently more accurate, it suggests there is error introduced using the less accurate APASS values as references. But I'm not sure about this conclusion and what the real errors are.

Frank

Attached Files



#19 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 21 December 2013 - 04:54 PM

Finally, here is an actual application of this stuff, which is a plot of g', r', g'-r' values for Nova Delphini 2013 in the later period after the peak. The error bars are not as rigorous as I would like them to be, but they are roughly based on the residuals seen in the many stars used for the calibration.

The g'-r' values are consistent with V-R plots I have seen, and show more of a trend than B-V, which stays near zero shortly after the peak.

I know that some of the R values were affected by leakage of emission lines in the filters, and different filters responded differently. That would be one benefit of g', r' here.

Frank

Attached Files



#20 StupendousMan

StupendousMan

    Vostok 1

  • -----
  • Posts: 180
  • Joined: 21 Aug 2005

Posted 22 December 2013 - 10:53 AM

Thank you for all that information.

Can you explain in a bit more detail the residual graphs you showed above? I'm not sure what "dM", "dG", etc., represent. Does "G" stand for "measurement in the SDSS g' band", for example?

#21 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 22 December 2013 - 11:54 AM

Hi-

Well there are many little details involved, but yes it is unfortunate to have a "primed" label on a filter. I dropped the prime in places, so G refers to the true g' magnitude of the star, and the residual compares the table value to the calculated one based on the calibration.

The sdss and apass values don't include a magnitude (M) estimate, so I use the aperture based magnitude from UCAC4 as the reference for my clear filter values.

All these calculations are happening in code I wrote, so there could be issues - but the fundamental thing is that I'm using a catalog that isn't intended for photometry. It aims to be "10% absolute and 5% differential" - and I'm just trying to find out how good it is in practice.

For the nova, there are AAVSO standard stars nearby in BVR and if everyone uses them the results will be matchable - except for filter differences and emission lines. But there is no similar standard for ugriz. I want to avoid empirical conversions from BVRI to ugriz, so I am trying to use a system strictly based on ugriz.

Frank

#22 WRAK

WRAK

    Apollo

  • -----
  • Posts: 1188
  • Joined: 18 Feb 2012
  • Loc: Vienna, Austria, Europe

Posted 22 December 2013 - 01:59 PM

Interesting thread - coming from the other world of double star observing I am often puzzled by inaccurate magnitudes for secondaries and a bit less often for primaries. I then look up APASS, UCAC4, Tycho II etc. to get a clue what is going on - for wide pairs most often but not always successful but for close pairs ...
The idea of doing ground based measurements of magnitudes within an error range of 0.05 or even less makes me a bit uneasy as conditions may change within minutes (not to speak of hours, days or months) to a degree far larger.
I would be happy to get magnitudes such precise from any calaogue as I am confronted rather often with estimated discoverer magnitudes off to one magnitude or even more.
Wilfried

#23 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 23 December 2013 - 04:27 AM

Anyone doing photometry over a period of time in the atmosphere will be limited by how stable the sky is, but you can look at the behavior of the residuals in all the fields to get an idea for that stability. I chose nights that looked very good and took data over about an hour. My key point is that the overall systematic error in my measurements appears to be less than the error introduced by using the APASS values - which again isn't too surprising. I'm just trying to determine what the errors are.

For double stars you should be able to get very accurate relative values without much effort.

Do you have a need for more accurate magnitudes of doubles that you look at?

Frank

#24 WRAK

WRAK

    Apollo

  • -----
  • Posts: 1188
  • Joined: 18 Feb 2012
  • Loc: Vienna, Austria, Europe

Posted 23 December 2013 - 07:02 AM

Frank, when preparing observation sessions with ~20 objects per session from the WDS catalog I have in average at least one double with questionable magnitudes. Most of the time I get useful data via Tycho II/UCAC4/APASS and often there are also photos giving a clue when comparing with other stars nearby with known magnitudes - but there always remain some pairs especially baffling as all these sources do not work.
Some examples are STF450 (companion assumed to be fainter than advertised +9.1mag but no data to be found in other sources), STF1881 (companion assumed to be fainter than advertised +8.81mag but no data to be found in other sources) and LEO55 (advertised magnitudes are Tycho II origin, but both values are for the position of the primary which seems to be wrong by ~2 arcseconds to make things even more unclear).
Wilfried

#25 freestar8n

freestar8n

    Skylab

  • *****
  • Posts: 4062
  • Joined: 12 Oct 2007

Posted 23 December 2013 - 07:41 AM

Hi-

Well doubles pose other challenges, but it's interesting if no one has tried to re-measure these guys. I might give it a try if conditions work out for me, but my sky time is extremely limited. Also - I am only working with Sloan filters - and clear filter - but they may be adequate for the needs here.

It may need a combination of video and ccd methods but it could be interesting. Basically I would calibrate the field using APASS values and ccd if there are references nearby - then use those as a reference for video measurements.

So I'll keep it in mind as a project to try.

Frank






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics