Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Increased noise with flats correction

  • Please log in to reply
21 replies to this topic

#1 dgpilot

dgpilot

    Messenger

  • *****
  • topic starter
  • Posts: 407
  • Joined: 25 Sep 2009
  • Loc: Northern California

Posted 12 May 2021 - 10:42 AM

Does anyone know why applying flats seems to increase noise?

 

In this example image I applied flats, dark flats and darks to the image on the right, the left image only has darks applied (no flats). The image on the right, overall is corrected quite well. The left one has the obvious issues in the corners such as vignetting.

Still.. I don't understand why the corrected image becomes so noisy? If I use something like Curves to match the tones.. the left uncorrected one still has substantially less noise.

 

Screenshot_144.jpg



#2 jdupton

jdupton

    Gemini

  • *****
  • Posts: 3,370
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 12 May 2021 - 10:50 AM

Mir (dgpilot),

 

   It doesn't increase as much as you think by looking at those two images. 

 

   In all likelihood, what you are seeing is the effect of an unequal stretch applied to the two images. That clouds your perception of the amount of noise present. As you calibrate an image, it gets smoother and flatter. When you apply an automatic stretch, the uncalibrated image cannot be stretched as much while the calibrated image can be stretched much farther. 

 

   The greater the stretch performed, the more the noise that was already there is revealed.

 

   If you are using PixInsight (I cannot tell for sure), you should stretch the calibrated image using the STF tool. Then drag the "New Instance icon" (little triangle in the lower left of the process window) onto the PI Workspace. Finally drag that newly created New Instance of the STF settings to the uncalibrated image. At this point, both images have been stretched by exactly the same amount. I think you will find that the calibrated image looks much better than the uncalibrated image.

 

 

John


Edited by jdupton, 12 May 2021 - 10:52 AM.

  • dswtan, Midnight Dan, dgpilot and 5 others like this

#3 jerahian

jerahian

    Apollo

  • *****
  • Moderators
  • Posts: 1,426
  • Joined: 02 Aug 2018
  • Loc: Maine

Posted 12 May 2021 - 10:56 AM

I suspect it's a difference in the screen transfer function applied to each, making the flat-corrected version seem noisier than the original.  As an experiment, see how the original looks when you apply STF to the flat-corrected version, and then drag and drop the STF "New Instance" for that one back to the original.

 

- Ara


  • dgpilot, bobzeq25, 2ghouls and 1 other like this

#4 dgpilot

dgpilot

    Messenger

  • *****
  • topic starter
  • Posts: 407
  • Joined: 25 Sep 2009
  • Loc: Northern California

Posted 12 May 2021 - 11:05 AM

I suspect it's a difference in the screen transfer function applied to each, making the flat-corrected version seem noisier than the original.  As an experiment, see how the original looks when you apply STF to the flat-corrected version, and then drag and drop the STF "New Instance" for that one back to the original.

 

- Ara

You guys are 100% correct, I failed to even consider STF being a factor. I applied the same STF stretch to both and they both show similar noise. Thank you.


  • John O'Grady and jerahian like this

#5 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 24,623
  • Joined: 27 Oct 2014

Posted 12 May 2021 - 11:09 AM

Corollary.  STF is a pretty crude stretch.   Also, usually too harsh.  I use other methods, reserve STF as a tool for seeing what's going on better.



#6 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 12 May 2021 - 11:55 AM

Just to be technical: Calibrating an image does not get rid of noise, it gets rid of unwanted signal.

 

Ed



#7 Narrowbandpaul

Narrowbandpaul

    Ranger 4

  • -----
  • Posts: 317
  • Joined: 24 Dec 2013
  • Loc: Constrained by the Heisenberg uncertainty principle

Posted 12 May 2021 - 01:15 PM

So flats don’t eliminate Fixed Pattern Noise? Maybe it should be fixed pattern unwanted signal but that doesn’t sound so catchy.

Regarding unequal STF stretches: yes, different stretches. But the real issue is you estimated this visually. Using the stats process you can determine the noise mathematically. To quote Dr Gregory House, numbers don’t lie, people do!

Use a preview on a bit of background sky..not gas/stars/galaxies and look at the standard deviation. Use the same preview each time so you compare identical pixels

#8 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 12 May 2021 - 03:18 PM

https://www.flicamer...Correction.html

http://isl.stanford....392b/lect07.pdf

 

I use CCD cameras and have never had a problem with Fixed Pattern noise, which seems to be random and well constrained in making calibration masters. The point I make to my AAVSO CCD students is that calibration frames have both signal and noise. The reason we take lots of calibration frames and either average or median combine them is  increase the signal-to-noise ratio (SNR) of the master calibration frame signal, mitigating the carry-over noise, that never disappears. Calibration actually adds noise to a science image, but the the cost is low compared to getting out the signal you want.

 

The concept of calibration frames having signal and that calibration eliminates these unwanted signals for our science frames (or image frames for APers) is, IMO, an important part of understanding how and why calibration works. 

 

Ed


Edited by Ed Wiley, 12 May 2021 - 03:19 PM.


#9 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,514
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 12 May 2021 - 07:01 PM

To me the critical thing to realize is that the whole point of calibration is to reduce noise terms from the sensor. Both darks and flats will reduce noise if used properly. The variation in dark current signal is itself a noise term.

In professional work the limiting noise may be due to the quality of the flats. But in the case of this thread I think things are ok.

If you don’t think the master dark reduces noise then just subtract a constant value from each pixel. You will find the result to have more noise than if you had used a good master dark.

Frank
  • Dean J. likes this

#10 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 12 May 2021 - 09:55 PM

Frank,

 

I make it a point to emphasize to my students the following concepts.

1. Signal is reproducible.

2. Noise is random

 

The reason we stress concepts such as a dark imaging having both a signal and noise is to help them understand what, exactly, is going on in the preparation and application of calibration frames. When, for example, you know that taking 20 flats and then making a master flat results in maximizing flat signal relative to the inherent noise (i.e. increases SNR)  then you understand more about the calibration process. And when you realize that you are actually adding a small bit of noise to your light frame when you calibrate because noise, being random, can never be eliminated, you have a better appreciation for why we strive to make good master calibration frames.

 

To stress, you never get rid of noise, you only mitigate its effects by striving for good calibration and high SNR. Proper calibration does not decrease noise, it tries to eliminate unwanted signal. Call it noise if you wish but IMO you miss some fine points that make calibration understandable.

 

Ed


  • elmiko and Dean J. like this

#11 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,514
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 12 May 2021 - 10:28 PM

Ok Ed. I certainly disagree and I think you are a rare hold out on viewing noise and signal as distinct things. I view them as defined by context and that is the usage I see in engineering and science - and certainly in the area of imaging sensor noise. That’s why terms like fpn exist.

But I don’t want to digress on this thread and I’m happy to disagree. If you want references on the topic I’m happy to provide them.

Frank
  • bobzeq25 likes this

#12 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 24,623
  • Joined: 27 Oct 2014

Posted 12 May 2021 - 10:56 PM

Ok Ed. I certainly disagree and I think you are a rare hold out on viewing noise and signal as distinct things. I view them as defined by context and that is the usage I see in engineering and science - and certainly in the area of imaging sensor noise. That’s why terms like fpn exist.

But I don’t want to digress on this thread and I’m happy to disagree. If you want references on the topic I’m happy to provide them.

Frank

You don't need references.  <smile>

 

Ed, if you have "unwanted signal" and it's not noise, where do you put it in the signal to noise ratio?  If on top, we can improve the signal to noise ratio significantly in our images.  Just shine a flashlight in the scope.  <grin>

 

Bottom line.  Noise means different things in different contexts, and it does _not_ have to be random.

 

But you're wrong about one thing.  Ed is not rare.  You see people on CN claiming "noise" is only the random component, all the time.


Edited by bobzeq25, 12 May 2021 - 10:58 PM.


#13 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 13 May 2021 - 12:07 AM

I am following Berry and Burnell (2005, The Handbook of Astronomical Image Processing). I have used bold to call attention to a couple of terms, no shouting intended. 

 

For example: p. 157: "The unwanted signals in a raw CCD images include two additive components and one multiplicative component. The additive components are a voltage offset, or bias, from zero volts; and a signal generated by thermal emission of electrons that grows linearly with exposure time. The multiplicative error arises because photosites have differing sensitivities to light. Calibration involves removing the bias, subtracting the dark current, and dividing the image by a map of photosite sensitivity.

 

And: p. 170: The dark frame contains the thermal electrons that accumulate during integration, of course, but it also contains thermal noise, a random variation in the number of thermal electrons that accumulate, plus all of the elements that make up a bias frame....Thermal noise, sigmaTE, is the random variation in the number of thermal electrons. It obeys a simple law that governs many random processes involving unlikely events over long intervals: the standard deviation in the number of thermal electrons is the square root of the number of electrons.

 

Just wanted all of you to know that I am not hanging out there alone; I'll go with Berry and Brunell simply because I have concluded that they have it right.

 

Seriously bobzep25: Calibration removes the unwanted signal, that is what calibration is all about. The noise in the SNR is the random noise that remains after calibration and it comes from the light frame, the dark master, the bias master (if used) and the flat master. Also, while I agree that noise means different things in different contexts, we are not taking about all contexts we are talking about calibrating CCD and CMOS astronomical images.

 

Ed



#14 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,514
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 13 May 2021 - 01:20 AM

That is a fairly basic book intended for amateurs. I don’t think it ever uses the term fpn but berry does in other writings. Anyone who uses the term fpn acknowledges the dual roles of noise and signals. And berry does.

In texts and journal articles they don’t follow or specify a rigid distinction. But it causes no confusion if you know the terms are used flexibly. Like big and small they are known from the context.

Frank

#15 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 13 May 2021 - 08:17 AM

My apologies for diverting the thread away from FPN with what I thought was an innocent comment about signal and noise. You will have to judge for yourselves whether the "fairly basic book for amateurs" is accurate or inaccurate in its treatment of signal and noise as it applies to CCD images and calibration. If you find it inaccurate(albeit incomplete as FPN is not covered) please let us all know how and why.

Ed



#16 spokeshave

spokeshave

    Mercury-Atlas

  • *****
  • Posts: 2,587
  • Joined: 08 Apr 2015

Posted 13 May 2021 - 08:54 AM

There are no established conventions in astrophotography for defining signal and noise. In other disciplines, it varies widely and it can also vary widely from one researcher to another within the same discipline. So contests of will over such definitions do not really serve the community. 

 

I was taught in grad school that signal is data that conveys information and noise is random and does not convey information - in one class. In another class, I was taught that signal is the desired portion of a dataset and everything else is noise. Both classes were taught by tenured professors who were also accomplished researchers and neither was right or wrong.

 

When no widely accepted conventions of terminology exist, all we really need to do is make sure that the terms are defined in the context of the conversation. Arguing about conventions of terminology when none exist is pointless.


  • N1ghtSc0p3 likes this

#17 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,514
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 13 May 2021 - 06:09 PM

My apologies for diverting the thread away from FPN with what I thought was an innocent comment about signal and noise. You will have to judge for yourselves whether the "fairly basic book for amateurs" is accurate or inaccurate in its treatment of signal and noise as it applies to CCD images and calibration. If you find it inaccurate(albeit incomplete as FPN is not covered) please let us all know how and why.

Ed

Two quick quotations:

 

 

Taken together, these artifacts are called pattern noise.  In some cameras the pattern is the same in every bias frame; in others the pattern is different in every bias frame.  Repeating pattern noise is better than changing pattern noise but all pattern noise is bad...  In some the pattern noise is larger than readout noise.

So he is referring to fixed variations in "signal" as pattern noise.  That is Richard Berry from https://manualzz.com...ews-the-qsi-532

 

But my favorite quotation on this stuff is from a pioneer in maximum entropy and Bayesian methods in image processing:

 

 

The concept of noise is always defined in a specific context.  As a consequence, what is considered noise in one case may be considered "signal" in another.  One man's weed is another man's wildflower.

That's a graduate level text on optics and imaging, "Probability, Statistical Optics, and Data Testing" by B. R. Frieden.  It's a rare text where the author actually takes the time to make this point.  Normally it is assumed and not stated.  As Bob alluded - it would be hard to write out any SNR expression if you had to fret about the distinction in terms of the causes of the terms.  In fact I can put most anything in a context where it is a "signal."

 

So if you want to go by Richard Berry, then clearly he is happy to refer to undesirable modulations of signal as a noise term.  It sounds like you feel that breaks Berry's own rules on this stuff - but he uses the term noise.

 

I think there is a fairly established convention in professional technical writings in experimental science and engineering - and that is to focus on the thing you are trying to measure as signal and anything that obfuscates it as noise.  Only in amateur writings do I see people trying to distinguish them - and it is a lost cause because most anything can be regarded as "signal."

 

But the way Berry Burnell talk about dark current pattern noise is ok in that they do describe it as ultimately due to variations in what we normally view as a "signal" -  i.e. dark current.  And on top of it there is Poisson noise.  That's fine - but the problem is when you say since it is inherently a signal it is always a signal - but obviously they don't really believe it because Berry himself calls it noise.  The real problem for beginners learning this stuff is that I have seen people try to wrestle in their minds that calibration only increases noise.  So why do it?  Well it actually decreases noise - when you allow your vocabulary to include terms like "pattern noise" - as Berry does.  It's a noise term that adds in quadrature with all the others - and it is reduced by master dark subtraction -  and applying flats.

 

And that book makes no mention of dithering at all.  That's when you would really get tongue tied trying to say you want to dither to reduce the signal.  Instead all those undesirable things are noise terms and there are ways to reduce them.  There is no need to think noise only grows when you do all this calibration stuff.

 

Frank



#18 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 13 May 2021 - 06:48 PM

Frank,

What is the goal of stacking and averaging or median combining 20 flats to produce a master flat? I am told it is to increase the SNR of the master. Perhaps I have been instructed incorrectly?

Ed



#19 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,514
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 13 May 2021 - 08:24 PM

Frank,

What is the goal of stacking and averaging or median combining 20 flats to produce a master flat? I am told it is to increase the SNR of the master. Perhaps I have been instructed incorrectly?

Ed

A sensor has different response to photons in each pixel - so the response needs to be normalized or the resulting image will have pattern noise in it.  So the master flat - and the master dark - need to capture that pattern noise well so it isn't imprinted into the calibrated result.  You can go ahead and say you want a very high SNR master flat - and in that sense you are talking about accurately capturing the signal in each pixel.  And you can then go ahead and apply that master flat in calibration - and say that the pattern noise in the image has been reduced.  It's all about context - and not about deciding what something "is" and sticking to it in all contexts.

 

The flat is different from the dark in that the dark pattern noise is additive and independent of the photon signal, while the flat is multiplicative and the resulting noise in the image does depend on the signal.  But ultimately they result in pattern noise in the image - and good calibration will reduce that pattern noise.  High SNR masters will reduce the pattern noise in the calibrated image.  That is what they do and that is what they are for.

 

The context of signal and noise changes in a very natural way.  If I am trying to listen the radio, cosmic radio sources will be a noise term.  If I am trying to listen to cosmic sources, radio stations will be a noise term.  They switch roles in the corresponding SNR expressions.

 

Frank



#20 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,854
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 13 May 2021 - 09:41 PM

Does anyone know why applying flats seems to increase noise?

 

In this example image I applied flats, dark flats and darks to the image on the right, the left image only has darks applied (no flats). The image on the right, overall is corrected quite well. The left one has the obvious issues in the corners such as vignetting.

Still.. I don't understand why the corrected image becomes so noisy? If I use something like Curves to match the tones.. the left uncorrected one still has substantially less noise.

 

attachicon.gifScreenshot_144.jpg

That is not actually an increase in noise. It is a revelation of the true signal that you have. If you manually adjusted the uncorrected image to have the same contrast as the corrected image, you would find that you have the same noise. It is not more noise, not really...its actually a change in contrast, thanks to the ability to properly stretch the data once the field is flat. 



#21 Ed Wiley

Ed Wiley

    Vanguard

  • *****
  • Posts: 2,112
  • Joined: 18 May 2005
  • Loc: Texas, USA

Posted 13 May 2021 - 10:31 PM

Frank,

I think we are ships passing in the night. Thanks for your comments, but unless we can settle on a common common subject and a common vocabulary I don't think we will make any progress.

 

dgpilot: I am so sorry I brought any of this up and sorrier that, like the old bass, I kept striking the lure. I will strive to do better the next thread.

 

Ed



#22 SeymoreStars

SeymoreStars

    Gemini

  • *****
  • Posts: 3,314
  • Joined: 08 May 2014
  • Loc: Pennsyltucky

Posted 14 May 2021 - 12:02 AM

Coming from a CCD background (FLI16803) into the CMOS(QHY600M) world, I came to a conclusion tonight. I will choose the Mode and Gain settings that provides the least amount of read noise (in electrons).

 

 

In the case of the QHY600M, I'll be using Mode-1 with Gain-56. This choice provides the most accurate reading of the ADC.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics