Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

A Quick Look at Image Noise

  • Please log in to reply
80 replies to this topic

#1 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 08 November 2018 - 05:37 PM

I thought I would put together a quick visual comparison of what different forms of noise look like in a single frame of a deep sky image. My hope here is to give beginner and intermediate imagers who have never looked at a "single-row" or "single-column" noise plot of an image an understanding of what it does look like, and what it means when we say things like "swamp the read noise". These plots were produced with data acquired with an ASI183MM Pro.

 

Image Noise 2.jpg

 

(Full Size - Low Compression)

 

There are three plots here, sourced from data that has been calibrated (so no amp glow or hot pixels are represented here), sampled every pixel horizontally across the image (see image inlay for sample row):

 

DARK RED = Read Noise

DARK GRAY = Dark & Read Noise

BLUE = Photon Shot & Dark & Read Noise

 

The dark red plot represents the minimum noise of the camera, read noise. This is the baseline read noise, and represents the MINIMUM noise any part of the image can have. You cannot have less noise than read noise in any given sub-exposure no matter what you do. This is an interesting fact when it comes to doing narrow band imaging, where at a dark site, you may have some areas of the frame that have only read noise, even if you manage to expose for an hour or two per sub-exposure. That is actually just fine...what that means is, you are getting the maximum possible contrast! wink.gif Read Noise == Minimum Possible Noise.

 

* Note that the vertical median location of the plots here is representative of the total "offset" in the signal. For read noise, that offset represents the bias offset.

 

The dark gray plot represents the read noise combined with the dark current shot noise. The two are added in quadrature, so mathematically this means SQRT(Ndark^2 + Nread^2). Even with exposures of 5-10 minutes, the total dark signal is very tiny, and represents only a small increase in total noise. In fact, read noise is still the dominant noise term here, indicating that dark current, which with this particular camera is only about 0.002e-/s, so even with a 10 minute exposure, total dark current shot noise is actually less than read noise even at unity gain.

 

* Note that the offset is only marginally higher than read noise (dark red) in the dark noise (dark gray) plot. That represents the bias offset + about 1 electron of dark signal offset.

 

The blue plot represents the read and dark noise combined with the photon shot noise of the image signal. All three terms are again added in quadrature, so mathematically this means SQRT(Ssky + Sobj + Ndark^2 + Nread^2). With this particular exposure, read noise was swamped by the photon shot noise by about 5-7x. That is fairly middle ground, and an amount by which I think many beginners and intermediate imagers could achieve. It can be done with 5 to 10 minute exposures, depending on the gain used. You should note a few things about the blue plot. First, it has a noticable increase in even the minimum offset compared to the read and dark noise offsets. That is the combination of bias offset plus skyfog offset (the minimum average offset at the edges of the blue plot) plus the object signal offset (the higher average offsets within the middle areas of the blue plot). I have plotted a 54-period moving average, which is 1/100th the total number of samples, which better represents the actual signal level (offset).

 

* Note how the blue plot has significantly higher variation (noise) than the other two plots. This larger variation includes both the photon shot noise, as well as the read and dark noise. The read and dark noise are there, but they overall represent a tiny fraction of the unexpected deviations from the mean compared to the photon shot noise. DESPITE the noise here, though, you can actually fairly easily follow the changing height of the plot, and correlate that height with the brightness of the row marked in the inlaid image (from which the plot was sampled). Where the image gets brighter, the blue plot gets higher, and where the image gets darker, the blue plot gets lower.

 

You can also pick out some of the stars...there are quite a few, and they are usually represented by the large outlier spikes in the blue plot. Some of the stars are barely larger than one standard deviation of the noise in the blue plot, and are harder to pick out from the noise...their SNR is low. Other stars are much brighter than the standard deviation o the noise in the blue plot, and are easy to pick out from the noise...their SNR is high. The average SNR of the entire plot is high enough that you can pick out the relative brightness differences...it is easy to differentiate a bright area from a dark area in the plot, and also easy enough to pick out moderately brighter areas from medium brightness areas. The SNR is good...not ideal, it is a single sub-exposure, but it is good, and the signal is strong enough to clearly represent the structures of the object in the image.


Edited by Jon Rista, 08 November 2018 - 07:01 PM.

  • Stelios, mikefulb, pfile and 19 others like this

#2 Stelios

Stelios

    Fly Me to the Moon

  • *****
  • Moderators
  • Posts: 6749
  • Joined: 04 Oct 2003
  • Loc: West Hills, CA

Posted 09 November 2018 - 01:58 AM

Very interesting, John. 

 

Some questions, and please excuse them if too simplistic, I'm just trying to understand:

 

Is the distance between blue and the read/dark noise representing the sky fog? I would have expected the sky fog to be evenly distributed.

If the blue is signal, why do some of the blue lines go all the way down, cutting through the read and dark noise? 



#3 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 09 November 2018 - 02:53 AM

Very interesting, John. 

 

Some questions, and please excuse them if too simplistic, I'm just trying to understand:

 

Is the distance between blue and the read/dark noise representing the sky fog? I would have expected the sky fog to be evenly distributed.

If the blue is signal, why do some of the blue lines go all the way down, cutting through the read and dark noise? 

The blue offset represents skyfog + object signal + bias offset + dark offset. It is the total offset, and since the object signal is included, it is not flat, it varies with the signal. I've matched the image from which the blue plot was sampled with the plot itself. You can click the link to get a full size with lower compression to see it better, but as the blue plot increases, the pixels on the red line marked across the image will brighten, and as the blue plot decreases, the pixels on the red line marked in the image will darken. This is the full image signal...all offsets are represented.

 

The blue signal also represents all noise. Noisy pixels can deviate more than the average amount, and yes, you will notice that some of the pixels in the blue plot deviate enough to reach even a little lower than the read noise floor. Those would be very cold pixels, deviating due to read noise and dark current, and they just so happened to deviate more than the average amount.

 

Also note, the blue plot is a plot from an actual image. The gray plot is a plot from a dark frame. The red plot is from a bias frame. In reality, the various noise terms are not separated like this...I just plotted the three together to show how offset and noise distribution change. In the actual image, the blue plot represents ALL of the noise, including read and dark noise. All noise deviates around the offset, and the offset technically varies from pixel to pixel in an actual image. In colder pixels, where the sky and object signal is low (or even zero), all the noise would be read and dark. While most of the time read noise will vary within its standard deviation (~68% of the time), and the vast majority of the time it will deviate within two standard deviations (~95%), there are still times where read noise alone could result in deviations up to 3, 4, 5, or even more standard deviations. So some pixels could end up very dark, just because of read noise. Hence the reason the blue plot could sometimes reach below the read noise floor represented by the red plot.


Edited by Jon Rista, 09 November 2018 - 02:56 AM.

  • Dean J. likes this

#4 lakeorion

lakeorion

    Apollo

  • -----
  • Posts: 1236
  • Joined: 03 Aug 2010
  • Loc: Lake Orion MI

Posted 09 November 2018 - 11:17 AM

And for my clarification:

 

minimize dark red with Bias calibration,

minimize dark gray with calibrated darks,

minimize blue with integrating multiple calibrated lights?



#5 miwitte

miwitte

    Apollo

  • *****
  • Posts: 1027
  • Joined: 28 Dec 2016
  • Loc: Charlotte NC

Posted 09 November 2018 - 11:43 AM

id like to see that stacked, and with NR applied.



#6 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 09 November 2018 - 11:48 AM

And for my clarification:

 

minimize dark red with Bias calibration,

minimize dark gray with calibrated darks,

minimize blue with integrating multiple calibrated lights?

I am not sure what you mean when you say "minimize"...

 

These are from individual calibrated frames. A calibrated bias frame...single frame, with master bias subtracted, but with a pedestal added to preserve the offset. A calibrated dark frame...single frame, with master dark subtracted, but with a pedestal added to preserve the offset. And a calibrated light frame, with pedestal.

 

So the signals for each plot are just the shot noise and read noise, really.



#7 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 09 November 2018 - 12:36 PM

Here is a version of the image plot from a 6 hour integration, without any noise reduction:

 

Image Noise Stack CN.jpg

 

(Full Version...Low Compression)

 

Stacking does have a dramatic effect on reducing noise. The signal is much clearer, the fainter stars are now quite obvious.


Edited by Jon Rista, 09 November 2018 - 12:43 PM.

  • Dean J., FlankerOneTwo and TinySpeck like this

#8 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 09 November 2018 - 12:37 PM

Here is the same 6 hour integration, with moderately aggressive noise reduction with TGVDenoise in PI:

 

Image Noise NR.jpg

 

(Full Version...Low Compression)

 

There is an improvement here, the signal becomes smoother, however not as dramatic as the effect stacking has. NR is primarily an aesthetic smoothing of the data, which eliminates grain, however it is usually at the cost of accuracy.


  • mikefulb and Dean J. like this

#9 lakeorion

lakeorion

    Apollo

  • -----
  • Posts: 1236
  • Joined: 03 Aug 2010
  • Loc: Lake Orion MI

Posted 10 November 2018 - 11:08 PM

I am not sure what you mean when you say "minimize"...

 

These are from individual calibrated frames. A calibrated bias frame...single frame, with master bias subtracted, but with a pedestal added to preserve the offset. A calibrated dark frame...single frame, with master dark subtracted, but with a pedestal added to preserve the offset. And a calibrated light frame, with pedestal.

 

So the signals for each plot are just the shot noise and read noise, really.

OK I missed in the original post that the frame was already calibrated.

 

So the read noise is inherent to the camera, where does the dark noise come from with properly calibrated and integrated darks?



#10 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 11 November 2018 - 12:23 AM

OK I missed in the original post that the frame was already calibrated.

 

So the read noise is inherent to the camera, where does the dark noise come from with properly calibrated and integrated darks?

Dark current leads to DFPN as well as dark current shot noise. The remaining noise after dark calibration is the dark current shot noise...which, as you can see, is rather trivial with modern, and especially COOLED, cameras.



#11 miwitte

miwitte

    Apollo

  • *****
  • Posts: 1027
  • Joined: 28 Dec 2016
  • Loc: Charlotte NC

Posted 11 November 2018 - 01:24 AM

That is a pretty sweet way of showing noise I like it. Where did you find a method to show it like that it could come in handy.



#12 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 6599
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 11 November 2018 - 02:08 AM

Jon, thanks for the writeup.

 

It might be worth pointing out that any pixel rejection done during the integration can NOT improve your signal-to-noise ratio. In fact a simple average integration without pixel rejection will produce your best signal-to-noise ratio. Given this, pixel rejection should only be used to remove really gross outliers (random hot/cold pixels and cosmic ray strikes) and things like airplane/satellite tracks.

 

As for producing such plots, PixInsight's PixelMath can be used to sample and plot such data.

 

Here is a tutorial video on such a technique by Gerald Wechselberger:

 

  https://www.dropbox....nglish.wmv?dl=0

 

And a link to Gerald's other tutorials:

 

  http://www.werbeagen...PixInsight.html


  • Dean J. likes this

#13 Der_Pit

Der_Pit

    Messenger

  • -----
  • Posts: 474
  • Joined: 07 Jul 2018
  • Loc: La Palma

Posted 11 November 2018 - 07:10 AM

I am not sure what you mean when you say "minimize"...

 

These are from individual calibrated frames. A calibrated bias frame...single frame, with master bias subtracted, but with a pedestal added to preserve the offset. A calibrated dark frame...single frame, with master dark subtracted, but with a pedestal added to preserve the offset. And a calibrated light frame, with pedestal.

 

So the signals for each plot are just the shot noise and read noise, really.

OK, so this explains what I'm actually seeing in those plots.

Just, for the image plot - shouldn't you also have subtracted the 'calibrated' (i.e., 6h-averaged) intensity signal to only show the noise part of the image?  This way it's mixing noise and intensity, no?



#14 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 11 November 2018 - 03:35 PM

OK, so this explains what I'm actually seeing in those plots.

Just, for the image plot - shouldn't you also have subtracted the 'calibrated' (i.e., 6h-averaged) intensity signal to only show the noise part of the image?  This way it's mixing noise and intensity, no?

Noise and signal are two sides of the same coin. The goal was to show what a signal and its noise looks like when you look at it in a different way. The offset is as intrinsic a part of the signal as the noise is, so I don't think it should be removed. The variation in offset pixel to pixel is crucial to making an image anything other than a flat even gray.



#15 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 11 November 2018 - 03:46 PM

Jon, thanks for the writeup.

 

It might be worth pointing out that any pixel rejection done during the integration can NOT improve your signal-to-noise ratio. In fact a simple average integration without pixel rejection will produce your best signal-to-noise ratio. Given this, pixel rejection should only be used to remove really gross outliers (random hot/cold pixels and cosmic ray strikes) and things like airplane/satellite tracks.

 

As for producing such plots, PixInsight's PixelMath can be used to sample and plot such data.

 

Here is a tutorial video on such a technique by Gerald Wechselberger:

 

  https://www.dropbox....nglish.wmv?dl=0

 

And a link to Gerald's other tutorials:

 

  http://www.werbeagen...PixInsight.html

Hmm, I am not sure I fully agree that pixel rejection cannot improve SNR. I consider FPN, which would include hot pixels and other fixed structures that are often inherent in the data, as well as trails from airplanes, satellites and meteors, and even any remnant dust motes to all be "noise" in the sense that it interferes with an unobstructed view of the object signal. In that sense, pixel rejection does indeed improve SNR, because it is removing unwanted junk that obscures parts of the desired signal, and affects perception of the signal overall. That may not be a "mathematically pure" definition there, but I don't think it is an incorrect description of what noise does from a perceptual standpoint...noise obscures our observations of the signal of interest.

 

I think it also goes more deeply than that, and pixel rejection can improve SNR from a mathematical standpoint and the accuracy of the final results. Consider: Scaling and alignment is involved in the process of integrating an image, and how well aligned the data is will affect the statistical process. Outlier pixels due to noise or frame intrusions can  have an effect on those alignment processes, so rejecting them should improve the final result. Airplane trails in particular can affect large areas of the frame in a non-trivial fashion, and I've had frames with multiple meteor and sat trails. When done right (and it seems to be pretty difficult to really get it right, based on my experience as well as numerous threads on the topic), local normalization is supposed to do something similar, improve the alignment of the data in each frame before integration for optimal results.

 

I think the only time pure averaging would produce the best SNR would be if the data was pristine...pure shot noise, and nothing but.

 

What I do agree with you on is that pixel rejection should indeed only reject outliers. It should not be rejecting data en-mass, as that would mean you were rejecting data within the normal distribution of the signal, and yes, that would be detrimental to SNR. 


  • Dean J. likes this

#16 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 11 November 2018 - 04:07 PM



That is a pretty sweet way of showing noise I like it. Where did you find a method to show it like that it could come in handy.

I have a 2DPlot script in PI. I do not remember where I got it, but it looks like this:

 

FbVRTkh.jpg

 

I can right-click the plot and export as CSV, which I then import into Excel to produce the final charts.



#17 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2269
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 11 November 2018 - 05:13 PM

I do not remember where I got it

Herbert Walter


  • james7ca likes this

#18 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 6599
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 12 November 2018 - 07:02 AM

Hmm, I am not sure I fully agree that pixel rejection cannot improve SNR... 

Jon, I think if you review the integration tool documentation on PixInsight you will see that they state that it is mathematically impossible with any pixel rejection algorithm to improve the SNR over what you achieve with a simple average combination WITHOUT pixel rejection. That's why they recommend that you always do your initial integration with pixel rejection disabled and then you target your rejection method and parameters to get as close as you can to the SNR of that initial integration while still removing any obvious outliers.

 

Now, having said that it is obvious that in many cases you can produce a more perceptually pleasing image by using pixel rejection (by removing such things as airplane and satellite trails), but there is always some cost to the final SNR when you use rejection (although if done correctly that difference should be minimal, perhaps not even perceptible). Note, in particular, that I'm talking about SNR, not whether pixel rejection can result in a "better" looking image.

 

However, the point here is that your target SNR for any integration result with rejection is just the average integration done without pixel rejection (and you can't do any better than the latter). This is the key factor in trying to determine the best rejection method and the parameters used with same.

 

Actually, we've had a similar debate on this matter back in the year 2015 and here is a reference where I give further examples from the PixInsight documentation (I don't want to repeat that entire post here, so this is the link):

 

  https://www.cloudyni...-2#entry6539621


  • Dean J., Goofi and Der_Pit like this

#19 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 12 November 2018 - 12:40 PM

Jon, I think if you review the integration tool documentation on PixInsight you will see that they state that it is mathematically impossible with any pixel rejection algorithm to improve the SNR over what you achieve with a simple average combination WITHOUT pixel rejection. That's why they recommend that you always do your initial integration with pixel rejection disabled and then you target your rejection method and parameters to get as close as you can to the SNR of that initial integration while still removing any obvious outliers.

 

Now, having said that it is obvious that in many cases you can produce a more perceptually pleasing image by using pixel rejection (by removing such things as airplane and satellite trails), but there is always some cost to the final SNR when you use rejection (although if done correctly that difference should be minimal, perhaps not even perceptible). Note, in particular, that I'm talking about SNR, not whether pixel rejection can result in a "better" looking image.

 

However, the point here is that your target SNR for any integration result with rejection is just the average integration done without pixel rejection (and you can't do any better than the latter). This is the key factor in trying to determine the best rejection method and the parameters used with same.

 

Actually, we've had a similar debate on this matter back in the year 2015 and here is a reference where I give further examples from the PixInsight documentation (I don't want to repeat that entire post here, so this is the link):

 

  https://www.cloudyni...-2#entry6539621

From a purely mathematical standpoint, I agree.

 

From a perceptual standpoint, I've stuck with Winsorized Sigma rather than moving to Linear Fit clipping, as LF clipping integrations (at least with my CMOS data) never look as good as Winsorized integrations do. I consider all the other junk in the frames to be a form of noise (not gaussian or poisson, but "noise" in the sense that they affect the perception of the signal of interest), and without clipping outliers, that "noise" affects the averaging process.

 

Anyway, from a purely mathematical standpoint regarding only temporally random gaussian or poisson noise, a strait average will give you the best SNR.


  • Dean J. likes this

#20 TinySpeck

TinySpeck

    Vostok 1

  • *****
  • Posts: 191
  • Joined: 08 Oct 2017
  • Loc: Seattle area

Posted 27 November 2018 - 01:51 PM

This is really great, Jon, thank you.  Please check if I'm taking away the right stuff here.

 

The stacked (integrated) blue graph shows the signal part of the S + N for skyfog + object signal + bias offset + dark offset.  Noise has been greatly reduced.  Since it is above the red & gray there is a signal component to the skyfog + object.  I see in your image slice that there does seem to be some nebula (object) signal across the slice.

 

If your image slice included some sky with no objects (theoretically), would the blue signal line drop to the gray line level?  Or is there some "signal" associated with skyfog?


  • Dean J. likes this

#21 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 27 November 2018 - 03:53 PM

This is really great, Jon, thank you.  Please check if I'm taking away the right stuff here.

 

The stacked (integrated) blue graph shows the signal part of the S + N for skyfog + object signal + bias offset + dark offset.  Noise has been greatly reduced.  Since it is above the red & gray there is a signal component to the skyfog + object.  I see in your image slice that there does seem to be some nebula (object) signal across the slice.

 

If your image slice included some sky with no objects (theoretically), would the blue signal line drop to the gray line level?  Or is there some "signal" associated with skyfog?

The gray and red plots are only there for comparison. The blue plot contains ALL the noise...it is not just skyfog+object noise, the blue plot is skyfog+object+dark current+read noise. Noise and signal are the same thing, just different aspects of the same thing. You cannot separate noises out...it is all just "noise". The blue plot shows you how the signal varies, and demonstrates how noise makes determination of what the signal represents uncertain and difficult if the noise level is high.

 

The stacked image demonstrates how noise, the uncertain and random variations in the signal, is much lower than in a single frame. It shows how the signal and it's true nature is much more apparent when noise is lower. But, again, in the stacked image plot, that is again ALL noise...skyfog+object+dark current+read noise...it is all in there together. There is no natural separation of noises into different plots...all noise terms combine together and you cannot separate them.

 

The gray and red plots are just there for comparison. Those were generated from bias and dark frames, which contain no signal. They are distinct plots of distinct data sets, but them being separate is NOT how noise works in reality. They are just there so you can see what the minimum noise levels look like without any signal (and that signal's noise) added to them.


  • Dean J. likes this

#22 TinySpeck

TinySpeck

    Vostok 1

  • *****
  • Posts: 191
  • Joined: 08 Oct 2017
  • Loc: Seattle area

Posted 27 November 2018 - 04:59 PM

Thanks for the quick reply.  Bracken in "Deep Sky Imaging Primer" points out that there is a signal and a noise component to the dark (thermal) current, bias, skyfog, and object, though.  You can see that in your pre- and post-integration waveforms above too.  Pre- includes all signal + noise, post- has most of the noise removed leaving only signal.  

 

I agree that the noise from all sources merges together in your blue graph and can't be separated.  When you add several noisy signals you get the sum of the signals plus the quadrature sum of the noise, though, so removing that noise leaves you with the sum of the signals.  By that reasoning you'd see the noiseless blue line drop to the noiseless gray line with no object or skyfog signals present (assuming that you haven't removed the bias and thermal signals from the blue graph).

 

After reviewing Bracken I think I just answered my own question, too: yes, there is a skyfog "signal".  I guess that would have to be removed by something other than integration or noise reduction.  Maybe that's what "background equalization" does.


  • Dean J. likes this

#23 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 27 November 2018 - 05:23 PM

Thanks for the quick reply.  Bracken in "Deep Sky Imaging Primer" points out that there is a signal and a noise component to the dark (thermal) current, bias, skyfog, and object, though.  You can see that in your pre- and post-integration waveforms above too.  Pre- includes all signal + noise, post- has most of the noise removed leaving only signal.  

 

I agree that the noise from all sources merges together in your blue graph and can't be separated.  When you add several noisy signals you get the sum of the signals plus the quadrature sum of the noise, though, so removing that noise leaves you with the sum of the signals.  By that reasoning you'd see the noiseless blue line drop to the noiseless gray line with no object or skyfog signals present (assuming that you haven't removed the bias and thermal signals from the blue graph).

 

After reviewing Bracken I think I just answered my own question, too: yes, there is a skyfog "signal".  I guess that would have to be removed by something other than integration or noise reduction.  Maybe that's what "background equalization" does.

Well, technically speaking, yes...every noise comes from a signal. Every signal has noise, every noise is part of a signal.

 

The bias offset is actually from an applied voltage, and all voltages vary a bit with electronic voltage noise. Read noise would include that bit of variation, as well as input and output referred noises from the amplifier and ADC units, as well as other sources. The dark current is a signal...it is an additional accumulation of charge in the pixels. As charge, it is subject to shot noise, poisson noise.

 

So yes, they are all signals. That said, most of those signals pale in comparison to the signal in a single sub, assuming that sub has been sufficiently exposed. The dark current, for example, in my demonstration here is a couple of electrons. The signal is tens of electrons. So the "signal" from dark current doesn't really matter...it is "Swamped". Same goes for the noise from that dark current signal...the noise is barely an electron, and it is swamped by the read noise even. The read noise and dark current noise are swamped by the photon shot noise from the signal.

 

Now, regarding noise removal. That actually is really not what is happening. I think this is a subtle yet important distinction here. When you STACK, you are not removing noise. Stacking is just a way of integrating signal. No matter how it is done, any integration will only add noise. Signal grows with integration. As signal grows, so too does noise. Poisson noise is, after all, the square root of the signal. So, if you have S, then your shot noise is SQRT(S). If you have 10 * S, then your shot noise is SQRT(10 * S). Noise and signal both grow with integration...however, signal grows linearly, while noise grows in quadrature.

 

In the stacked example in post #7 in this thread, the noise was most definitely not "removed"...critical distinction here. The signal became much stronger, while noise becomes stronger, but not as fast. This is why the true nature of the signal is so much more apparent in the integrated 6 hours demonstrated in post #7...the signal is over 100x stronger than in the single sub. The noise, however, is only about 10x stronger. In the blue plot of post #7, it appears as though noise has been "reduced", simply because the scale of the chart is the same. Since the signal is so much stronger now than the noise, the noise only APPEARS to be less. If one were to not re-scale the stacked result, then the opposite would actually be true...the total noise of the stacked image would actually be about 10x stronger than that of a single sub, when viewed at the same scale. wink.gif I know, it seems un-intuitive, but that is actually the reality of things here.

 

Now, it is not possible to have "only signal". Noise is a part of signal...signals are noisy. That is just the way it is. You cannot have a noiseless signal, not at least if it is a real world signal. You will notice that even my green plot, which has had noise reduction applied, still has noise. I could even apply a gaussian blur, which would smooth out the plot even more...but it would STILL have noise, and it would also have artifacts from digitization (posterization). Also, stacking will not shift the signal down, since it is not removing noise...so the blue plot would never "drop" to the "noiseless" gray line...for one, the gray line is not noiseless. Stacking, if it was done just as a sum, would actually always shift the signal up. Now, subtracting an offset can shift the signal...however that actually does nothing with the temporally random noise at all. It simply adds or subtracts a constant to each pixel value. Again, noise is a part of signal. Shift a strong signal around, it is still a strong signal. Shift a noisy signal around, it is still a noisy signal. Shifts only affect offset. This is why you cannot "remove noise" by subtracting a dark. Subtracting a master dark removes FPN, fixed pattern noise, which is really just varying offsets in each pixel. But again, all that does is shift. Temporally random noise, however, is unaffected by said subtraction.

 

Signal and noise are inseparable but different aspects of the same thing. We only manage noise...we do not, and can not, actually "eliminate" or "totally remove" noise (temporally random noise, that is).


Edited by Jon Rista, 27 November 2018 - 05:33 PM.

  • Dean J. and cyber like this

#24 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22748
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 27 November 2018 - 05:40 PM

Another interesting distinction, again perhaps subtle, but important. The application of a noise reduction tool when processing an image is different than stacking. The two operate in different...domains. Stacking is an integration, it combines discrete bits of signal and their noises together to compound the signal and its noise, which strengthens the signal relative to the noise. Noise reduction, on the other hand, operates in a frequency or spatial domain and adjusts the data without further integrating.

 

Noise reduction in the more common forms will use spatial processing to balance out differences in neighboring pixels. This has the effect of reducing noise, but it is also usually at the cost of degradation in the accuracy of the signal. Spatial processing tends to spread information around, so information that belongs in just one pixel can end up being dispersed among it's neighbors. The signal does not get stronger with noise reduction processing...and in fact, it can become weaker or degraded in other ways, losing accuracy. For pretty pictures, the loss in accuracy is usually fine, although beware that faint details such as very distant galaxies may be lost or detrimentally altered. Noise reduction can and often will have an impact on color.

 

For scientific work, this kind of spatial processing can be quite detrimental, and the loss of accuracy unacceptable. If lower noise is required for any scientific purposes, then additional integration (more stacking of more distinct data) would be the preferred approach.


  • Dean J. likes this

#25 TinySpeck

TinySpeck

    Vostok 1

  • *****
  • Posts: 191
  • Joined: 08 Oct 2017
  • Loc: Seattle area

Posted 27 November 2018 - 06:03 PM

Yes, technically what stacking does is to increase the signal-to-noise ratio.  If you were to consider only the summed stack you would see the signal and the noise increasing as you describe.  But stacked images are always normalized (divided by the number of constituent images) which has the effect of reducing the noise, so I think it's reasonable to talk about noise reduction due to stacking.

 

You're right about noise never being totally eliminated.  By "noiseless" I meant "greatly reduced noise".

 

 

This is why you cannot "remove noise" by subtracting a dark. Subtracting a master dark removes FPN, fixed pattern noise, which is really just varying offsets in each pixel. But again, all that does is shift. Temporally random noise, however, is unaffected by said subtraction.

I'm not sure I understand or agree with this part.  Calibration with bias, darks, and flats is meant to reduce the signal portions, not the noise.  That's why we average together large numbers of each prior to using them for calibration, to reduce the noise and leave primarily signal.

 

That's a good distinction between stacking ("vertical" noise reduction) and image denoising ("horizontal").


  • Dean J. likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics