You are not logged in. [Login] Entrance · Main Index · Search · New user · Who's Online FAQ · Calendar

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)
NigelM
member

Reged: 05/22/09

Re: DSLR 'long exposure noise reduction' [Re: microstar]
#5817013 - 04/23/13 08:48 AM

If your lights are perfectly aligned, then mathematically there is no difference between taking a bunch of darks, subtracting one dark from each light then stacking the result, or stacking all the darks first to form a master dark, then subtracting this from each light and stacking the result.

The question is, what happens if you need to realign the lights to stack them (essentially dithering). Then the two methods are not mathematically the same. The argument that I have seen put forward is that in this case subtracting a master dark results in better signal-to-noise in the final image, due to the averaging effect of dithering the master dark.

I have not really seen believable maths to prove this - one problem is that people assume the noise is random and uniform, but if is was, then you would be better off subtracting one constant number from your frame to represent the dark. This is clearly not the case for real darks.

NigelM

 Post Extras:
Footbag
Post Laureate

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' [Re: NigelM]
#5817026 - 04/23/13 09:00 AM

So who's going to test this? Seems easy enough.

Take a 5m dark frame with ICNR on.

Then take a 5m dark with it turned off and calibrate it with multiple darks.

The image that is closest to 0 wins.

 Post Extras:
Dave Lee
sage

Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' [Re: NigelM]
#5817056 - 04/23/13 09:30 AM

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave

 Post Extras:
Footbag
Post Laureate

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' [Re: Dave Lee]
#5817063 - 04/23/13 09:37 AM

Quote:

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave

The master dark is applied to the individual lights before stacking.

What do you guys mean about aligning darks? By temp or just ensuring orientation? Assuming both, there wouldn't be a way to misalign. The image is X pixels by Y pixels whether its a dark or light. Dithering should have no negative effect on darks.

Edited by Footbag (04/23/13 09:53 AM)

 Post Extras:
Dave Lee
sage

Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' [Re: Footbag]
#5817114 - 04/23/13 10:11 AM

Quote:

SNIP

The master dark is applied to the individual lights before stacking.

SNIP

That makes perfect sense. But then I don't understand what pfile is talking about regarding 'darks walking around' (or something like that).

dave

 Post Extras:
neutronman
Carpal Tunnel

Reged: 01/10/08

Loc: Dallas, Texas
Re: DSLR 'long exposure noise reduction' [Re: microstar]
#5817142 - 04/23/13 10:29 AM

Quote:

Makes sense John. Thanks for sharing your perspective and experience. I personally don't do dark subtraction at all -- just bad pixel mapping and sigma-rejection based stacking. That probably puts me beyond the pale vis a vis the calibration orthodoxy.
...Keith

Hey, that should work too. So many ways to skin the proverbial cat. Basically, whatever workflow one prefers that gets the job done!

 Post Extras:
pfile
Carpal Tunnel

Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: neutronman]
#5817337 - 04/23/13 12:17 PM

it's okay to use in-camera dark subtraction as long as you realize your subs will have more noise in them than if you subtracted a clean master dark.

the problem here is the claim that after stacking registered lights, the dark signal has also been averaged and that the in-camera dark subtraction is equivalent to making a master dark and subtracting that from the raw light.

imagine that before you register your lights, you draw a big cross in the same place on each image, like a danish flag, but one pixel wide. let that cross represent the dark signal in the sub. as you register your lights, in each sub the cross is rotated by different angles and where the two lines cross is shifted around. now you average the registered lights. do you have a strong horizontal/vertical cross pattern on your stack, or a smeared out version of the cross?

that's what i'm talking about. the dark signal, or the dark signal error no longer lies directly on top of itself after registration.

by 'aligning' darks i was trying to illustrate that if john's method really worked, then "dithering" your darks when making a master dark would be a good idea. but we don't do this. we stack the dark frames directly on top of one another in order to get the average value of the dark signal at each pixel location. if you shift your dark subs around before making the master dark, you're not really doing anything meaningful with the dark signal.

 Post Extras:
Footbag
Post Laureate

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5817379 - 04/23/13 12:43 PM

Quote:

imagine that before you register your lights, you draw a big cross in the same place on each image, like a danish flag, but one pixel wide. let that cross represent the dark signal in the sub. as you register your lights, in each sub the cross is rotated by different angles and where the two lines cross is shifted around. now you average the registered lights. do you have a strong horizontal/vertical cross pattern on your stack, or a smeared out version of the cross?

that's what i'm talking about. the dark signal, or the dark signal error no longer lies directly on top of itself after registration.

I'm pretty sure that the master dark and flat are applied to the lights before stacking and before rotation. In my further reading, it's not clear. I watched a DSS stacking process, and it still wasn't clear at which point the darks and flats are applied. But I don't see any reason they wouldn't be applied before the actual stacking.

I do my processing in DSS, and most of it happens behind the scenes. I think some programs allow you to do it manually. Someone who uses one of those may have to chime in.

 Post Extras:
pfile
Carpal Tunnel

Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: Footbag]
#5817386 - 04/23/13 12:47 PM

i use one of those programs. absolutely, positively, no doubt about it, darks, bias and flats are applied to the subs BEFORE registration.

i'm trying to show the flaw in John's reasoning here! my example about the cross is what happens to the dark signal in his method, not the method that we all use and is known to be correct, which is to calibrate images with master frames BEFORE registration.

 Post Extras:
Jerry Lodriguss
Vendor

Reged: 07/19/08

Loc: Voorhees, NJ
Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5817397 - 04/23/13 12:52 PM

I agree with pfile on this question.

The other thing that jgraham ignores is that the only way to increase the signal from the object, which is half of the signal-to-noise equation, is by gathering more photons from the object in a light frame.

If you use clear dark-sky time shooting in-camera darks, you waste the opportunity to improve the signal in your images.

This is a completely separate issue from what is happening with the thermal signal and its associated poisson noise in the darks.

There is no way you can argue that using dark-sky time to collect signal and shooting separate darks on a cloudy night is not better than in-camera dark-frame subtraction.

Saying that for you, personally, in-camera darks work better, just means you weren't doing your master cloudy-night darks correctly. :-)

Simply match the temps of the darks and the lights with Dark Library, or use automatic dark-frame matching in Images Plus.

In-camera dark-frame subtraction is fine for wide-angle scenics where you are only shooting a single frame or a couple of frames, but for serious long-exposure deep-sky work, it is a poor choice.

Jerry

 Post Extras:
jsines
sage

Reged: 09/06/11

Loc: Berkley. Michigan
Re: DSLR 'long exposure noise reduction' [Re: Dave Lee]
#5817400 - 04/23/13 12:52 PM

Quote:

It is my understanding that enabling the Canon (T3) 'long exposure noise reduction' is simply an automated dark frame (same exposure length) that follows the light frame (and then applied automatically to the light frame).

Somewhere (don't recall where) I read a recommendation that this function be disabled. Is there a problem here (would seem to me to be an automatic dark frame/bias frame) other than the fact that this will probably extend the time that the camera will spend in the imaging process? FWIW, I am a complete newbie here.

dave

The function needs to be disabled because you are taking bias frames in addition to the lights. The bias frame contains the readout signal which is just the signal from the camera reading the chip. It is present in the light frame, the dark frame, and the flat frame, and it needs to be subtracted from all of them. When you apply in-camera dark frame subtraction, I don't think you are able to subtract the bias frame from the dark frame.

My understanding of the correct process is -
* Calibrated Lights = (Light - Bias - k*(Dark-Bias))/(Flat - Bias), where k=dark scaling factor (in PixInsight)
* Register Calibrated Lights (Star Align)
* Stack Registered, Calibrated Lights

someone correct me if I'm wrong, please, I may be confused.

 Post Extras:
jsines
sage

Reged: 09/06/11

Loc: Berkley. Michigan
Re: DSLR 'long exposure noise reduction' [Re: Jerry Lodriguss]
#5817420 - 04/23/13 01:02 PM

Quote:

There is no way you can argue that using dark-sky time to collect signal and shooting separate darks on a cloudy night is not better than in-camera dark-frame subtraction.

Jerry

On a clear, cloudless night where I need to work the next morning, I may get 2 hours of imaging time before I need to go to bed. If I'm taking 3 minute lights, it's a matter of taking 10 lights per hour or 20 lights per hour, or 20 lights versus 40 lights in 2 hours.

Big difference in signal.

 Post Extras:
pfile
Carpal Tunnel

Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: jsines]
#5817424 - 04/23/13 01:02 PM

yes, if you are going to make your own master calibration frames you really need to turn off in-camera dark subtraction. otherwise your darks are not darks, they are just dark current noise.

jsines, your outline of the steps to calibrate and integrate images is correct.

i'm glad jerry has weighed in here because frankly every time the in-camera dark subtraction comes up, i have this one-sided argument with john about how registration is the flaw in his theory. he never addresses that but just comes back with more statistics jargon.

 Post Extras:
Footbag
Post Laureate

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5817486 - 04/23/13 01:28 PM

Quote:

i use one of those programs. absolutely, positively, no doubt about it, darks, bias and flats are applied to the subs BEFORE registration.

i'm trying to show the flaw in John's reasoning here! my example about the cross is what happens to the dark signal in his method, not the method that we all use and is known to be correct, which is to calibrate images with master frames BEFORE registration.

Gotcha. That explains why I agreed with you right up until then.

 Post Extras:
microstar
Pooh-Bah

Reged: 01/05/08

Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5817779 - 04/23/13 03:22 PM

Quote:

yes, if you are going to make your own master calibration frames you really need to turn off in-camera dark subtraction. otherwise your darks are not darks, they are just dark current noise.

jsines, your outline of the steps to calibrate and integrate images is correct.

i'm glad jerry has weighed in here because frankly every time the in-camera dark subtraction comes up, i have this one-sided argument with john about how registration is the flaw in his theory. he never addresses that but just comes back with more statistics jargon.

Just so this isn't one-sided, I'm afraid I agree with John's logic and I respectfully disagree that there is some unacknowledged flaw in his theory. For every pixel there is a true value of dark signal that is dependent on factors such as temperature. If you take a bunch of darks at the same temperature however you don't get the same value every time - the variation around the true value is the noise. The objective of calibration is to get the best estimate of the true value. One way to do this is to stack a bunch of frames and take an average of each pixel's value which should converge on the true value the more frames you stack. That is the method that you promote, which should result in very little noise being injected into your light frame when it's calibrated. I think your logic is that this is the only way to get a good estimate of the true dark currant, and therefore when you register your lights and lose pixel-to-pixel mapping, you are losing the value of stacking darks. That's not what John is arguing. He is saying that a single matched dark could be as good of an estimate of your true dark signal as a stacked set of darks taken over a range of temperatures. If you accept that, then it's a red herring that your pixels get shuffled during registration because you aren't trying to get your best estimate of the true dark signal by stacking darks, your are trying to get the best estimate by subtracting the best-matched dark from your light. I don't see a flaw in this logic, it's just another way to skin the cat. Or you could use my method with dithering, bad pixel mapping with bias subtraction only, and sigma-based stacking and dispense with dark subtraction altogether. They are just different ways of trying to get a nice picture for those of us not doing photometry.
...Keith

 Post Extras:
pfile
Carpal Tunnel

Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: microstar]
#5817856 - 04/23/13 03:53 PM

Quote:

Just so this isn't one-sided, I'm afraid I agree with John's logic and I respectfully disagree that there is some unacknowledged flaw in his theory. For every pixel there is a true value of dark signal that is dependent on factors such as temperature. If you take a bunch of darks at the same temperature however you don't get the same value every time - the variation around the true value is the noise.

yes, this is correct, and it is the problem with using a single dark frame as your master dark. the light frame already has dark current in it with some amount of dark noise. when you subtract a single dark, you are adding insult to injury by injecting more noise into the calibrated frame than need be. if you drive the SNR of the master dark up (by stacking the darks) you inject the least noise possible.

Quote:

actually, that's the objective of stacking or integrating images. one objective of calibration is to remove fixed patterns, which are CCD artifacts, from your lights. another objective is to compensate for different QE across pixels and remove optical train vignetting using flats.

Quote:

One way to do this is to stack a bunch of frames and take an average of each pixel's value which should converge on the true value the more frames you stack. That is the method that you promote, which should result in very little noise being injected into your light frame when it's calibrated. I think your logic is that this is the only way to get a good estimate of the true dark currant, and therefore when you register your lights and lose pixel-to-pixel mapping, you are losing the value of stacking darks. That's not what John is arguing. He is saying that a single matched dark could be as good of an estimate of your true dark signal as a stacked set of darks taken over a range of temperatures. If you accept that, then it's a red herring that your pixels get shuffled during registration because you aren't trying to get your best estimate of the true dark signal by stacking darks, your are trying to get the best estimate by subtracting the best-matched dark from your light. I don't see a flaw in this logic, it's just another way to skin the cat. Or you could use my method with dithering, bad pixel mapping with bias subtraction only, and sigma-based stacking and dispense with dark subtraction altogether. They are just different ways of trying to get a nice picture for those of us not doing photometry.
...Keith

i think you misunderstand what he is saying. there'd be no need to talk about paired differences and all that stuff if all he was trying to say is that a single dark at the exact same temperature as the light is better than a stack of darks in a narrow temperature range.

he's saying that the net effect of doing in-camera dark subtraction on N frames and then stacking those N frames is the same as stacking N dark frames and subtracting that from each light. that's a totally different thing. i think what he is saying would be true if you did not register the frames. but once you have done that the dark signal residual noise in pixel (X,Y) of one image is getting averaged with the residual noise in pixel (M,N) of the next. in order to make statistical arguments about what happens to the dark signal, you have to make sure that you are stacking the same pixels... and you're not once the frames have been rotated and translated.

again, i'm not saying that one should not do in-camera dark subtraction, i'm saying that fooling yourself into thinking both methods are mathematically equivalent is wrong.

 Post Extras:
microstar
Pooh-Bah

Reged: 01/05/08

Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5818150 - 04/23/13 05:56 PM

Hmmm, I see. In the first message about temperature as a blocking factor I don't see an argument that it's mathematically equivalent, but in John's second posting what you are objecting to is the argument that it is mathematically equivalent since you aren't averaging the same pixels once the frames are registered.

Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith

 Post Extras:
pfile
Carpal Tunnel

Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: microstar]
#5818321 - 04/23/13 06:38 PM

yes, i don't have any beef with the temperature argument. though i think it's known that dark current scales linearly with time and exponentially with temperature, so in theory one could properly scale a dark at a different temperature to match the lights. i think people don't do this because it would require too much characterization of the CCD - exponential terms increase very quickly and so small errors become large ones. probably each sensor is different. but linear scaling (for time) is very straightforward.

i'm willing to be proven wrong here, because what we're talking about is math. one should be able to prove that the two methods are the same. so far i have not seen that proof. your empirical measurement is a good idea.

in your experiments i guess an important thing is to figure out what the margin of error is in order to conclude that the results are truly the same or truly different. no idea how to compute that, though.

 Post Extras:
RedLionNJ
professor emeritus

Reged: 12/29/09

Loc: Red Lion, NJ, USA
Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5818440 - 04/23/13 07:10 PM

With you on most of your points, pfile. Two additional things worth pointing out:

1. Dark signal (at least as recorded in Canon CR2 files) does NOT increase linearly with time. Repeated tests done by myself and others show an interesting "curve" (for me, it's nearer two lines angled at around the 120s mark). This is why flats and matching dark flats are a better option than simply flats and bias - you can't really scale for the Canon CR2s.

2. (and this is in further defense of pfile and Jerry's logic) Imagine you have a light pixel with a value of 5 and you subtract a single dark which happens (through statistical noise) to have a value of 5 or higher in that pixel - that calibrated light now has zero in that pixel. It's never going to have anything else in there. You've "lost" information. But say you averaged 20 darks and the average value of that pixel was 3 - subtracting 3 from that value of 5 will still leave information. Entropy is lower.

Grant

 Post Extras:
microstar
Pooh-Bah

Reged: 01/05/08

Re: DSLR 'long exposure noise reduction' [Re: pfile]
#5818494 - 04/23/13 07:30 PM

Quote:

in your experiments i guess an important thing is to figure out what the margin of error is in order to conclude that the results are truly the same or truly different. no idea how to compute that, though.

I suppose you could do that with a resampling method (pull a random sample of a dozen of the lights multiple times to get a variance) but to be honest, if it is so close that you have to resample to test significance then it's probably a moot point anyway as it becomes a "so what if it's significant, can you see a difference" issue. I'll set this up for tonight and just see what the mean and variance is for the normalized stacks calibrated and processed in different ways and see where that goes.
...Keith

 Post Extras:
Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)

Extra information
13 registered and 16 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, tecmage

Forum Permissions
You cannot start new topics