Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Standard deviation confusion

  • Please log in to reply
9 replies to this topic

#1 astrosatch

astrosatch

    Sputnik

  • -----
  • topic starter
  • Posts: 39
  • Joined: 24 Sep 2018

Posted 17 September 2019 - 04:03 AM

Hi!

I've been imaging for some time now and although

I'm familliar with all major concepts, standard deviation gives me a hard time to fully comprehend. I understand basic principal about distribution but real numbers are the ones I can't understand.

 

I've got theese numbers:

Max 65504

Min 16

Avg 4557

Std 2234

 

I understand all but std value. I thought value should be low, but it is large. What can I make out of it? Is it important information for acquiring images or can I just ignore it?

 

Thanks for reply.

Br,

Andrej



#2 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 17 September 2019 - 04:35 AM

It tells you about the range of the numbers, what fraction are in a certain range.

 

For a gaussian distribution (a bell curve) something like 66% of all data points are within one standard deviation of the average.

 

So if your average is 1000 and the std is 500, then 66% of your data will lie between 1000-500=500 and 1000+500=1500.

 

It can tell you about noise levels in your data etc. Its dead useful.

 

Google standard deviation.


  • pgs/sdg likes this

#3 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2092
  • Joined: 08 Apr 2015

Posted 17 September 2019 - 06:18 AM

To understand standard deviation, you first need to understand the mean - which is easy. It is simply the arithmetic average of all the data points. Some of the data points will be above the mean, and some below. Some will be close to the mean, and others farther away. What the standard deviation tells you is the average distance the data points are away from the mean. It is calculated by adding up the (absolute) distance to the mean for each data point and then taking the (RMS) average of those distances.

 

Understanding this, the standard deviation can tell you a lot. If it is small, relative to the mean, it tells you that the data points are fairly tightly grouped about the mean. If it is large, relative to the mean, it tells you that the data points are widely scattered about the mean.

 

Tim


  • pgs/sdg likes this

#4 astrosatch

astrosatch

    Sputnik

  • -----
  • topic starter
  • Posts: 39
  • Joined: 24 Sep 2018

Posted 17 September 2019 - 06:29 AM

Now I understand better. Thanks both for explanation.

 

 

 

"It can tell you about noise levels in your data etc. Its dead useful."

 

What exactly does this tell you?



#5 kathyastro

kathyastro

    Viking 1

  • *****
  • Posts: 765
  • Joined: 23 Dec 2016
  • Loc: Nova Scotia

Posted 17 September 2019 - 06:32 AM

Simplifying considerably, a large standard deviation means the histogram peak is wide.  A small standard deviation means that it is narrow.  The number tells a statistician just how wide or narrow.


  • pgs/sdg likes this

#6 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 17 September 2019 - 07:22 AM

"It can tell you about noise levels in your data etc. Its dead useful."

 

What exactly does this tell you?

 

If you look at the standard deviation of a part of your image which 'should' be of constant brightness, ie a bit of dark sky with no stars/galaxies, then the standard deviation tells you how noisy it is. A big number means lots of dakr and bright pixels, and generaly a low quality image, such as you'd get if you'd underexposed. A small SD means high quality data, with very little scatter.

 

Theres a related quantity which you often see, the signal/noise ratio or SNR. I would explain that but I've gotta get back to work!!! Worth a google too.


  • pgs/sdg likes this

#7 astrosatch

astrosatch

    Sputnik

  • -----
  • topic starter
  • Posts: 39
  • Joined: 24 Sep 2018

Posted 17 September 2019 - 07:57 AM

If you look at the standard deviation of a part of your image which 'should' be of constant brightness, ie a bit of dark sky with no stars/galaxies, then the standard deviation tells you how noisy it is. A big number means lots of dakr and bright pixels, and generaly a low quality image, such as you'd get if you'd underexposed. A small SD means high quality data, with very little scatter.

 

Theres a related quantity which you often see, the signal/noise ratio or SNR. I would explain that but I've gotta get back to work!!! Worth a google too.

Aha! I got it now. Yes that makes sense. Ok now to investigate my frames to see that more closely.

Thank you all.

 

Andrej


  • happylimpet and pgs/sdg like this

#8 Alex McConahay

Alex McConahay

    Cosmos

  • *****
  • Posts: 8182
  • Joined: 11 Aug 2008
  • Loc: Moreno Valley, CA

Posted 17 September 2019 - 09:09 AM

Take the example of two data sets of ten exposures each.

 

IN the first you left the lens cap on for the first nine shots, and got "0" exposure value, and then found out you were way overexposed, and got an ADU of 1,000,000.

 

In the second, you got reasonable exposure values of about 100,000 on each shot. There were some differences because of noise. There always is.

 

The mean average of both stacks is 100,000.  But the two stacks are very different in nature, as can be seen from the "Standard Deviation." In the first case the average difference is 301511. In the other just 1000. That means the shots in Stack B are much more alike. 

 

Alex

 

 

Attached Thumbnails

  • Capture.JPG

  • happylimpet and pgs/sdg like this

#9 LPA

LPA

    Explorer 1

  • -----
  • Posts: 52
  • Joined: 28 Jan 2014
  • Loc: Dallas, TX

Posted 17 September 2019 - 03:44 PM

These replies are mostly correct. Both the mean and sd are most useful when the image histogram is approximately bell-shaped. In that case essentially all values will be within 3 sd's of the mean, so higher sd corresponds to more spread out histogram, and the sd is 0 when all values are the same. All we can say in the case of asymmetry is that at least 8/9's of values will be within 3 sd's of the mean, not very useful for image interpretation or processing. In your case the max is 65504 and the mean is 4557 which is only about 2 times the sd. That tells me the image histogram is asymmetric, not bell-shaped, and/or there are some unmapped hot pixels. Most likely this image contains a few bright stars, a bunch of dimmer stars, and noise. In that case the image histogram itself will be more informative than these numerical summaries. This also is why image processing tools for stacking use robust scale estimators such as MAD (median of absolute deviations about the median) or IKSS (iterative k-sigma scale) as measures of variability for outlier detection during image integration since these measures are not tied to bell-shaped data and are not inflated by outliers, unlike the sd.



#10 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23721
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 17 September 2019 - 11:08 PM

It helps to visualize the data in a different way than a two dimensional plane of pixels of varying values, too. A histogram is a way of representing the information based on the frequency at which pixels of certain values are found. The histogram is, in effect, a plot of the distribution...such as the gaussian or poisson distribution...of the data:

 

3KBVwox.jpg

 

One standard deviation is represented by the dark blue band to one side of the mean...which in this case, of a normal (gaussian) distribution, is dead-center of the distribution. Two standard deviations extends into the lighter blue, and three standard deviations extends into the lightest blue. To one side, one standard deviation represents (in a normal distribution) ~34% of the samples (pixels). To both sides, it represents ~68%. Two standard deviations represents ~95%, and three standard deviations represents over 99%. Strong signals will tend towards a normal distribution in their nature. So, if you know what a standard deviation is, you can understand a lot about your signal.


Edited by Jon Rista, 17 September 2019 - 11:08 PM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics