Jump to content


Photo

DSLR 'long exposure noise reduction'

  • Please log in to reply
120 replies to this topic

#1 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 21 April 2013 - 08:02 PM

It is my understanding that enabling the Canon (T3) 'long exposure noise reduction' is simply an automated dark frame (same exposure length) that follows the light frame (and then applied automatically to the light frame).

Somewhere (don't recall where) I read a recommendation that this function be disabled. Is there a problem here (would seem to me to be an automatic dark frame/bias frame) other than the fact that this will probably extend the time that the camera will spend in the imaging process? FWIW, I am a complete newbie here.

dave

#2 robininni

robininni

    Viking 1

  • -----
  • Posts: 777
  • Joined: 18 Apr 2011
  • Loc: Stephenville, TX

Posted 21 April 2013 - 08:27 PM

I believe your second paragraph is the answer--you take 'valuable' imaging time up doing the auto dark subtraction versus taking darks later (during the daytime for instance or on a cloudy or moonlight night). Other than that I believe it works well if I recall discussion about it correctly.

Rob

#3 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 21 April 2013 - 10:47 PM

the bonus is that your dark is going to be at pretty much the same temperature as your light, since it's taken immediately afterward. generally speaking DSLRs are uncooled and the camera firmware messes with the 'raw' data internally in an attempt to reduce the visibility of dark current. this *may* make scaling DSLR darks impossible. but i have done it anyway :)

anyway so much for the pluses. the drawbacks are: what rob states above (wasted sky time) and... noise. dark signal is signal just like any other signal, and there's quite a bit of uncertainty (noise) in that signal in a single frame.

calibrating CCD images *always* injects noise into the images. so we try to average lots of bias, darks and flats to come up with masters that have high SNR, so that the noise injection is minimized. subtracting a single in-camera dark is just about the worst thing from an SNR perspective of the dark signal.

there's a guy in the DSLR forum that swears up and down that stacking a bunch of in-camera dark subtracted lights is the same thing as subtracting a stack of separately taken darks. but i don't agree with this because we always have to register our light frames. this means the dark signal in the stacked lights no longer falls on top of itself from light frame to light frame. this is actually a good thing from a hot pixel rejection standpoint, but i think it destroys any correlation between the dark signal in each frame and therefore it is not the same thing mathematically...

#4 jgraham

jgraham

    Voyager 1

  • *****
  • Posts: 13751
  • Joined: 02 Dec 2004
  • Loc: Miami Valley Astronomical Society

Posted 21 April 2013 - 10:48 PM

I spent a lot of time last year exploring this and I decided to use Canon's automatic dark subtraction all the time. This runs against the grain of the prevailing wisdom, but if you dig into the details of the statistics of how noise reduction works it makes sense, to me at least. Using this routine does a couple of interesting things. For example, it provides nearly perfect frame-to-frame temperature matching. Also, the time it uses to record and apply a dark is not at all wasted. The purpose of stacking images is to reduce noise, it does nothing to improve signal. Although the dark does not record signal, it does record noise, so the noise that would have been recorded if you had taken a light was still recorded and applied to your image stack. The statistics of this approach is slightly different than the traditional method of using stand-alone darks. Statistically, this method is essentially a difference in means. Using shot-to-shot darks is a method called paired differences. When there is a strong blocking factor (in this case tempertaure) paired differences gives lower variances than a difference in means. If the blocking factor is weak (constant temperature) paired differences has the same variance as a difference in means. Soooooo, at worst using the automatic dark subtraction does no harm, but because the temperature often changes when using an uncooled camera the results of using the automatic darks usually gives better results. The traditional method certainly works fine, but so does Canon's noise reduction routines. The bottom line for me... I haven't taken a stand-alone dark in nearly 2 years and I don't miss them at all.

#5 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 22 April 2013 - 01:29 AM

yeah, that's the guy.

dark current is not noise. it's signal. unwanted signal, but not noise in the statistical sense. you should understand this since you seem to know something about statistics.

after you register your lights, the dark signal is shifted around all over the place. a given pixel stack of the registered lights has dark signal from different sensor pixels! therefore it does not matter if it's difference in means or paired differences or whatever, because you're integrating different dark signal in each pixel stack.

so the dark signal has been subtracted from each frame, but in a noisy fashion. this means that you've got some pixels where it was over-subtracted and some where it was under-subtracted. i don't see how you can hone in on the 'true' value of the dark signal at a given pixel location if you have shifted the dark signal all over the place before integration.

please address this, because it is the flaw in your reasoning.



#6 mmalik

mmalik

    Fly Me to the Moon

  • *****
  • Posts: 5362
  • Joined: 13 Jan 2012
  • Loc: USA

Posted 22 April 2013 - 02:01 AM

My take is that there is no harm trying long exposure noise reduction (NR) while one is learning the rigors of AP. I use it quite often; my imaging results of NR turned ON here.... Regards


Note: Some image processing instructions and Canon AP settings, including NR, here....

#7 jgraham

jgraham

    Voyager 1

  • *****
  • Posts: 13751
  • Joined: 02 Dec 2004
  • Loc: Miami Valley Astronomical Society

Posted 22 April 2013 - 06:30 AM

Yeppers, that's kinda the point. No harm in trying. I experimented with this enough with both actual images and bench-top data to convince me that it works and works very well. My background in statistical analysis of data just helped me to understand why it works as well as it does. I've also been working with uncooled cameras for about 10 years and I've fought the good fight on noise and hot pixels. Since I started using the internal darks I haven't seen a single hot pixel and I can push the gain higher with fewer artifacts. There are good reason to keep the gain low, but it is nice having that option when the need arises.

If you're at all curious there is no harm in taking the path less traveled. For me it has work very well and it is sooooo nice not having to fool with darks anymore. It did a while to shed the nagging thought that somehow I was loosing something by not taking more lights, but the results and convenience speak for them selves. Besides, I got over it real quick the first time I got to go straight to bed after taking my last image instead of staying up taking dark. (C'mon mom, 5 more minutes!) And yes, I used to have gigabytes of archived darks (long ago deleted).

Have fun folks, it is just a hobby.

#8 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 22 April 2013 - 11:28 AM

okay, well that's fine, but you don't have any mathematical or scientific data other than 'it works fine' and the appeal to 'i know statistics'.

unless your tracking is spectacularly bad there's never any reason to push the gain. you don't collect more photons that way and you lose dynamic range.

if you want to make radical claims like this, show us the math. otherwise what's in HAIP is the proper way to reduce CCD images.

#9 Tonk

Tonk

    Fly Me to the Moon

  • *****
  • Posts: 7089
  • Joined: 19 Aug 2004
  • Loc: Leeds, UK, 54N

Posted 22 April 2013 - 11:49 AM

Although the dark does not record signal, it does record noise


Considering that hot pixels are signal this statement is false. Hot pixels are not noise and can't be treated as such mathematically (its not random). However random noise from various sources is also recorded as well of course and that can never be avoided.

background in statistical analysis of data just helped me to understand why it works as well as it does


Good - but you need to brush up on what inputs are signal and which are noise. Get that wrong (as you did) and the following analysis is going to be very suspect

Pfile has it pretty well right.

#10 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 22 April 2013 - 01:46 PM

yeah, that's the guy.

dark current is not noise. it's signal. unwanted signal, but not noise in the statistical sense. you should understand this since you seem to know something about statistics.

after you register your lights, the dark signal is shifted around all over the place. a given pixel stack of the registered lights has dark signal from different sensor pixels! therefore it does not matter if it's difference in means or paired differences or whatever, because you're integrating different dark signal in each pixel stack.

so the dark signal has been subtracted from each frame, but in a noisy fashion. this means that you've got some pixels where it was over-subtracted and some where it was under-subtracted. i don't see how you can hone in on the 'true' value of the dark signal at a given pixel location if you have shifted the dark signal all over the place before integration.

please address this, because it is the flaw in your reasoning.


Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?

dave

#11 SKYGZR

SKYGZR

    Viking 1

  • *****
  • Posts: 845
  • Joined: 13 Aug 2009
  • Loc: Planet Earth, Milky Way Galaxy, Speeding towards the Virgo Supercluster

Posted 22 April 2013 - 01:59 PM

I find that in camera darks are the least hassle, and most reliable, yet that's just me.

#12 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 22 April 2013 - 04:49 PM


Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?

dave


as you say the uncertainty in the dark can not be totally eliminated, but it can be reduced dramatically...

...just like you do with your lights - you integrate a whole bunch of them so that you get down to the true value of the dark current at a given pixel location. you can also reject outlier pixels to clean up cosmic ray hits, etc.

the dark signal in the light is noisy, but that's no reason to increase the noise in your calibrated sub by subtracting a noisy dark.

#13 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 22 April 2013 - 06:16 PM


dave


Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?


as you say the uncertainty in the dark can not be totally eliminated, but it can be reduced dramatically...

...just like you do with your lights - you integrate a whole bunch of them so that you get down to the true value of the dark current at a given pixel location. you can also reject outlier pixels to clean up cosmic ray hits, etc.

the dark signal in the light is noisy, but that's no reason to increase the noise in your calibrated sub by subtracting a noisy dark.


Thanks for the reply. This is the heart of my question.

My simple minded view of this is that one would take a bunch of dark frames and average them to remove noise. Assume that conditions/duration are the same as the image subs.

Then you do the same for the set of image frames and average those. Subtract the averages and 'you are done' (am ignoring light flats and bias frames).

Now assume that these darks came from the Canon automatic dark frames that the camera takes, but that they are not subtracted from the image frame - just saved with the image frames.

Mathematically it does not matter if you do a frame by frame (light minus dark) subtract and average that result or just average the darks then average the image frames and do a single subtract. You'll get the same answer - not almost the same but exactly the same. Hence my question.

In further thinking about this maybe the biggest disadvantage beyond time in using the camera to take/apply auto darks is the fact that you then have no flexibility in how you apply your darks - pretty much stuck with averages.

dave

#14 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 22 April 2013 - 06:37 PM

well i think that is john's argument. but here is mine. right before you average your light frames, you register them. meaning you rotate and translate the images so that the stars all line up with one another.

that shifts the residual noise from the dark current all over the place. when you stack a column of pixels, those pixels no longer came from the location x,y on the sensor. so you're really combining the noise from the dark signal from a whole bunch of different pixel locations. because of this i just don't see how you can make any statistical arguments about the equivalence of the methods. in one method you honed in on the true value of the dark current at a pixel location and subtracted it from the same pixel in the light. in the other, you've got dark current error from a bunch of different pixels all averaged together.

when making your master dark, does it make sense to rotate and translate the dark subs first? cause that's what you end up doing in the in-camera case.

also, as you point out you are unable to do outlier rejection or other fancy stacking techniques on the darks... because you have no darks. for long exposures now you've got cosmic ray hits that occurred during the light exposure PLUS cosmic ray hits that occured during the dark exposure, all right there on your preprocessed light. of course you can still do outlier rejection when stacking those lights, so all is not lost. but you just put noise into your calibrated image that did not need to be there.

#15 jgraham

jgraham

    Voyager 1

  • *****
  • Posts: 13751
  • Joined: 02 Dec 2004
  • Loc: Miami Valley Astronomical Society

Posted 22 April 2013 - 06:54 PM

They are mathematically the same if there is no blocking factor. However, in an unregulated camera there is a blocking factor, temperature, and it can be very significant. I was puzzled for a long time why my dark libraries were hit and miss; sometimes doing okay, sometimes not. Then I took a close look at my source darks before I averaged them. They usually started out relatively cool, and then rapidly heated and the noise increased right along with it. If you want to see something interesting take a series of darks and look at the temperature and noise of each individual frame. No wonder my darks were hit'n miss, what ta' heck was I trying to average. Simple means and differences in means are built around uncertainty that is random and normally distributed. These distributions are far from normal, probably closer to an f-distribution. If you go crazy and average enough frames they'll migrate towards normal, but as long at the temperature is increasing there will be bias towards the high temperature side of the distribution. Now in the big scheme of things it is not that big of a deal as these methods are very simple and robust and they obviously work quite well. However, this is such a perfect real-world application for paired differences it is just beautifully simple and the way that Canon implements it is wonderful and smart. It also brings so many other statistical factors into ballance. For example, strictly speaking, when using a difference in means you really should use the same number of data points (source images) in each sample so that the variances match. The statistical work-around is to use pooled variances, but again the methods are sufficiently robust you can often away without it (and we do). All these little nuances go away when using paired differences since by definition you are always using the same number of darks and lights.

Very simple, very clean, very effective. I like it.

Enjoy.

#16 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 22 April 2013 - 07:15 PM

pf - thanks. I think I now understand what you are saying.

dave

#17 pfile

pfile

    Gemini

  • -----
  • Posts: 3164
  • Joined: 14 Jun 2009

Posted 22 April 2013 - 08:23 PM

a lot more jargon with no math. baffle them with BS.

i'm not real sure why canon deserves accolades for what they do. they are doing the best they can without making the time between shots be ridiculously large, but there's nothing deep or prescient about what they are doing. basically they are trying to get rid of hot pixels, which is what terrestrial photographers see and wonder what the heck is wrong with their camera.

the solution to your temperature problem is easy. sort the darks by EXIF temperature or ambient temperature. integrate only those darks which are within perhaps T+/- 2C of a given temperature and call that your master dark for T. the dark current in that master will be compatible with your lights with temperature T.

you must think that the dark current is equal at every pixel or something. but that's just not the case due to impurities in the silicon, etc. take a look at a high-quality master dark (40+ frames) which has been bias subtracted and tell me what you see. it's not a grey image.

#18 neutronman

neutronman

    Surveyor 1

  • -----
  • Posts: 1918
  • Joined: 10 Jan 2008
  • Loc: Dallas, Texas

Posted 22 April 2013 - 11:04 PM

Personally, I prefer NOT to spend the extra time under clear skies shooting auto dark frames (when I could shoot 10-12 darks after I go to bed).

However, I DO use auto darks when shooting night landscape shots, since my exposures are typically 30-45 sec and the short auto darks work great and save work later. But that's just me ;)

#19 microstar

microstar

    Apollo

  • -----
  • Posts: 1227
  • Joined: 05 Jan 2008
  • Loc: Canada

Posted 23 April 2013 - 12:43 AM

Makes sense John. Thanks for sharing your perspective and experience. I personally don't do dark subtraction at all -- just bad pixel mapping and sigma-rejection based stacking. That probably puts me beyond the pale vis a vis the calibration orthodoxy.
...Keith

They are mathematically the same if there is no blocking factor. However, in an unregulated camera there is a blocking factor, temperature, and it can be very significant. I was puzzled for a long time why my dark libraries were hit and miss; sometimes doing okay, sometimes not. Then I took a close look at my source darks before I averaged them. They usually started out relatively cool, and then rapidly heated and the noise increased right along with it. If you want to see something interesting take a series of darks and look at the temperature and noise of each individual frame. No wonder my darks were hit'n miss, what ta' heck was I trying to average. Simple means and differences in means are built around uncertainty that is random and normally distributed. These distributions are far from normal, probably closer to an f-distribution. If you go crazy and average enough frames they'll migrate towards normal, but as long at the temperature is increasing there will be bias towards the high temperature side of the distribution. Now in the big scheme of things it is not that big of a deal as these methods are very simple and robust and they obviously work quite well. However, this is such a perfect real-world application for paired differences it is just beautifully simple and the way that Canon implements it is wonderful and smart. It also brings so many other statistical factors into ballance. For example, strictly speaking, when using a difference in means you really should use the same number of data points (source images) in each sample so that the variances match. The statistical work-around is to use pooled variances, but again the methods are sufficiently robust you can often away without it (and we do). All these little nuances go away when using paired differences since by definition you are always using the same number of darks and lights.

Very simple, very clean, very effective. I like it.

Enjoy.



#20 Footbag

Footbag

    Fly Me to the Moon

  • *****
  • Posts: 5857
  • Joined: 13 Apr 2009
  • Loc: Scranton, PA

Posted 23 April 2013 - 07:26 AM

ICNR uses one single dark. I use 50 plus darks combined to create a master dark.

As well, if you use ICNR early in the night, you camera is still "warming up". Your temps may not be so close.

#21 NigelM

NigelM

    Explorer 1

  • -----
  • Posts: 53
  • Joined: 22 May 2009

Posted 23 April 2013 - 07:48 AM

If your lights are perfectly aligned, then mathematically there is no difference between taking a bunch of darks, subtracting one dark from each light then stacking the result, or stacking all the darks first to form a master dark, then subtracting this from each light and stacking the result.

The question is, what happens if you need to realign the lights to stack them (essentially dithering). Then the two methods are not mathematically the same. The argument that I have seen put forward is that in this case subtracting a master dark results in better signal-to-noise in the final image, due to the averaging effect of dithering the master dark.

I have not really seen believable maths to prove this - one problem is that people assume the noise is random and uniform, but if is was, then you would be better off subtracting one constant number from your frame to represent the dark. This is clearly not the case for real darks.

NigelM

#22 Footbag

Footbag

    Fly Me to the Moon

  • *****
  • Posts: 5857
  • Joined: 13 Apr 2009
  • Loc: Scranton, PA

Posted 23 April 2013 - 08:00 AM

So who's going to test this? Seems easy enough.

Take a 5m dark frame with ICNR on.

Then take a 5m dark with it turned off and calibrate it with multiple darks.

The image that is closest to 0 wins.

#23 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 23 April 2013 - 08:30 AM

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave

#24 Footbag

Footbag

    Fly Me to the Moon

  • *****
  • Posts: 5857
  • Joined: 13 Apr 2009
  • Loc: Scranton, PA

Posted 23 April 2013 - 08:37 AM

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave


The master dark is applied to the individual lights before stacking.

What do you guys mean about aligning darks? By temp or just ensuring orientation? Assuming both, there wouldn't be a way to misalign. The image is X pixels by Y pixels whether its a dark or light. Dithering should have no negative effect on darks.

#25 Dave Lee

Dave Lee

    Messenger

  • -----
  • Posts: 442
  • Joined: 14 Feb 2013
  • Loc: Pinehurst, NC USA

Posted 23 April 2013 - 09:11 AM

SNIP

The master dark is applied to the individual lights before stacking.

SNIP


That makes perfect sense. But then I don't understand what pfile is talking about regarding 'darks walking around' (or something like that).

dave






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics