Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
· Get a Cloudy Nights T-Shirt · Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu… uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)
terry59
Post Laureate
*****

Reged: 07/18/11

Loc: Colorado, USA
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818522 - 04/23/13 07:43 PM

Quote:

Hmmm, I see. In the first message about temperature as a blocking factor I don't see an argument that it's mathematically equivalent, but in John's second posting what you are objecting to is the argument that it is mathematically equivalent since you aren't averaging the same pixels once the frames are registered.

Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith




My kind of analysis...thank you!


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818534 - 04/23/13 07:52 PM

Quote:


but to be honest, if it is so close that you have to resample to test significance then it's probably a moot point anyway as it becomes a "so what if it's significant, can you see a difference" issue.
...Keith




this is true.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818547 - 04/23/13 08:00 PM

FWIW, I decided to do a simulation. It was pretty easy to think through the 'single pixel' problem (it doesn't matter when you apply the dark - frame by frame like a Canon camera or averaging darks before applying).

But when you apply registration (after dark compensations are applied) was more than my mind could figure out. So I set up a spreadsheet structured as follows.

1) Four Pixels, each with their own dark signal and dark noise (noise generated with the randbetween() function).

2) There were 4 signals. One was 60 (signal value) with noise randomly added while the others were 30 (signal value) with their own noise values.

3) I would move the 60 unit signal around between pixels (the other pixels receiving the 30 + noise signals)randomly - sounds a lot like my guiding :-)

4) I did '20 subs' on these 4 pixels. Each sub would have a light frame and a second dark frame (different noise values between the two). The light frame had a dark component and the second run was just a separate dark.

5) For each of the 20 subs (on each of the 4 pixels) I would then 'apply registration' by picking out the pixel (out of the 4) receiving the 60 unit (plus signal noise) signal.

6) I would them dark adjust #5 above two ways. I applied the Canon approach of subtracting the paired dark for that pixel in that sub. I also averaged all the darks for that pixel across all 20 subs and used that value for each sub in the run. Note that each pixel had its own dark frame set. I ended up with two different dark adjusted results, obviously.

I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

The Canon approach has the advantage of simplicity and (I would think) a clear advantage of better matching of temperatures in darks vs. lights.

FWIW.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818557 - 04/23/13 08:05 PM

Quote:

The Canon approach has the advantage of simplicity and (I would think) a clear advantage of better matching of temperatures in darks vs. lights.




Well put.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818664 - 04/23/13 09:22 PM

Quote:



I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

dave





I'm not sure I follow your methodology entirely, but I do believe you have one error in it. If I understood correctly, you did this for 20 lights for both methods. However, what you would need to do is 20 lights for in-camera dark subtraction, and 40 lights for using darks from a dark library. By not using in-camera darks, that imager captures twice as many subs in the same amount of time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818673 - 04/23/13 09:26 PM

Quote:


Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith




Very generous of you to do this test, Keith. But, I'll point out the same flaw in methodology. You'll want to capture 48x3 minutes without in-camera dark subtraction (or, change the in-camera dark subtract to 12 subs). In any event, you'll need to have half as many in-camera dark subtracted subs, as that is what happens in the real world - people using in-camera darks subtraction capture half as many subs in the same amount of dark time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08

Loc: Voorhees, NJ
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818698 - 04/23/13 09:44 PM

Quote:

I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.




Keith,

This isn't a valid comparison.

You are forgetting the signal portion of the equation.

If you don't shoot in-camera darks, you can shoot 48 x 3 min lights in the same amount of time as 24 x 3 min lights with ICDS, and double the amount of signal you collect.

Regardless of what is happening with the "noise", even if is the same with both methods, the ICDS method gathers only 1/2 the signal in the same amount of clear dark sky time.

Jerry


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Jerry Lodriguss]
      #5818708 - 04/23/13 09:49 PM

OK. I'll have time to run 40 lights and 20 ICDS frames. Should be able to get started in a few hours.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818743 - 04/23/13 10:08 PM

well i think based on the spreadsheet experiment plus of course doubling the number of subs, the SNR increase of doing the imaging with separate darks should be slightly more than sqrt(2).

i think that's a no-brainer - the difference in the # of subexposures is going to totally swamp the contribution of the dark current / dark current noise from an SNR perspective.

i was more interested in the equal subexposure case, even though in real life the 'separate darks' imager gets 2x the subs in the same time. from the spreadsheet simulation it sounds like the two cases are not the same but the difference is minute.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5819202 - 04/24/13 06:32 AM

Quote:

Quote:



I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

dave





I'm not sure I follow your methodology entirely, but I do believe you have one error in it. If I understood correctly, you did this for 20 lights for both methods. However, what you would need to do is 20 lights for in-camera dark subtraction, and 40 lights for using darks from a dark library. By not using in-camera darks, that imager captures twice as many subs in the same amount of time.




It was 20 lights (same light frames for both methods) and 20 darks. The only change was when and how the darks were applied. THen all this was repeated 20 times, of course.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5819307 - 04/24/13 08:54 AM

If you shoot darks separately then those darks can be used on separate objects. If you shoot the darks in sequence(Long Exposure Noise Reduction) with the object then you have to take as many darks as you take lights.

If I was going to shoot just as many darks as lights I really don't think there would be a big difference between shooting them in sequence as shooting them separately. However, I can't think of a reason why I would want to take as many darks as I take lights so I can't ever see a reason for taking the darks in sequence.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5819565 - 04/24/13 11:26 AM

Quote:


It was 20 lights (same light frames for both methods) and 20 darks. The only change was when and how the darks were applied. THen all this was repeated 20 times, of course.

dave





Yes, that's what I thought you were saying. What you're basically saying is that your model showed that applying 20 closely matched darks to their corresponding lights yielded you roughly the same SNR as applying 20 averaged darks to 20 lights.

If your spreadsheet model is accurate, then, as pfile noted, what you've shown is that not using in-camera noise reduction (using a dark library) yields a 1.4x improvement in SNR (since you would have twice as many lights in the same amount of time).


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5819604 - 04/24/13 11:50 AM






If your spreadsheet model is accurate, then, as pfile noted, what you've shown is that not using in-camera noise reduction (using a dark library) yields a 1.4x improvement in SNR (since you would have twice as many lights in the same amount of time).




You also need to consider that the benefit of doubling the number of lights decreases as you increase the number of lights.

If you are taking a very small number of short exposure lights and you plan on taking the exact same number of darks then you will probably get almost the same result with in camera darks as with combing them in post. However, I am not sure why you would ever choose to take a very small number of short exposure lights and the exact same number of darks.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: mpgxsvcd]
      #5819827 - 04/24/13 01:36 PM

I have never used in camera noise reduction so don't have any direct experience, but would advise against it for the following reason. Take a series of, say, five minute darks and then stretch them equally and examine them. You will inevitably see differences between them, mostly from cosmic ray hits. That is why we average together or combine using a statisical exclusion method a large number of darks to get a "master" dark frame for calibration...so that the frame we are going to use for calibration is a good representation of the dark current at that particular temperature and has little or no individualized frame contamination. When you let the camera do individual dark frame calibration, each light frame is being calibrated with a different single dark frame that probably has a number of cosmic ray hits in each. Subtracting these cosmic ray hits from the light frames results in dark tracks in each frame that are not collated frame to frame. Stacking the images using an averageing or statistical exclusion method will minimize the effect, but personally I'd rather not trash up the light frames by being calibrated with contaminated single dark frames.

Edited by Hap Griffin (04/24/13 01:39 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5819878 - 04/24/13 02:03 PM

Wouldn't the differences in temperature have much more of an effect on the darks than stray cosmic rays?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: mpgxsvcd]
      #5819902 - 04/24/13 02:18 PM

The temperture differences will be small regardless if the dark frame is taken at the same time as the light frame, or chosen from a library of master dfarks and various tempertures. I keep a library of master darks and biases for approximately every five degrees.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5819910 - 04/24/13 02:23 PM

Here is some information on cosmic ray hits in dark frames. I would encourage you to check for yourself by stretching a set of dark frames approximately to the extent that you would in processing a light frame and look at the differences. http://darkerview.com/CCDProblems/particlehit.php

Edited by Hap Griffin (04/24/13 02:23 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5820011 - 04/24/13 03:17 PM

wow, i want to look back thru my darks and see if i have any particle decay events. that's cool.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5820093 - 04/24/13 03:49 PM

I see them all the time in my CCD darks.

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' [Re: Hap Griffin]
      #5821161 - 04/25/13 12:45 AM

i thought i would experiment a little bit with some darks that i already have. i happen to have 20 1800s darks at -30C from my CCD camera. i paired the darks into even/odd sequence number pairs, and made a master dark from the 10 odd frames, then calibrated the 10 even frames with that master. i then separately calibrated the 10 even frames with the consecutive odd frame.

however, the experiment ended right there - the single-dark subtracted frames have a whole boatload of pixels with 0 value. this is because the noise in the dark signal is great enough that many pixels end up being negative after subtraction, and so are clamped to 0.

i will have to add a small pedestal to only the darks being calibrated to prevent this from happening.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)


Extra information
13 registered and 31 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 2919

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics