Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
· Get a Cloudy Nights T-Shirt · Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu… uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
DSLR 'long exposure noise reduction'
      #5814045 - 04/21/13 09:02 PM

It is my understanding that enabling the Canon (T3) 'long exposure noise reduction' is simply an automated dark frame (same exposure length) that follows the light frame (and then applied automatically to the light frame).

Somewhere (don't recall where) I read a recommendation that this function be disabled. Is there a problem here (would seem to me to be an automatic dark frame/bias frame) other than the fact that this will probably extend the time that the camera will spend in the imaging process? FWIW, I am a complete newbie here.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
robininni
scholastic sledgehammer


Reged: 04/18/11

Loc: Stephenville, TX
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5814101 - 04/21/13 09:27 PM

I believe your second paragraph is the answer--you take 'valuable' imaging time up doing the auto dark subtraction versus taking darks later (during the daytime for instance or on a cloudy or moonlight night). Other than that I believe it works well if I recall discussion about it correctly.

Rob


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: robininni]
      #5814424 - 04/21/13 11:47 PM

the bonus is that your dark is going to be at pretty much the same temperature as your light, since it's taken immediately afterward. generally speaking DSLRs are uncooled and the camera firmware messes with the 'raw' data internally in an attempt to reduce the visibility of dark current. this *may* make scaling DSLR darks impossible. but i have done it anyway

anyway so much for the pluses. the drawbacks are: what rob states above (wasted sky time) and... noise. dark signal is signal just like any other signal, and there's quite a bit of uncertainty (noise) in that signal in a single frame.

calibrating CCD images *always* injects noise into the images. so we try to average lots of bias, darks and flats to come up with masters that have high SNR, so that the noise injection is minimized. subtracting a single in-camera dark is just about the worst thing from an SNR perspective of the dark signal.

there's a guy in the DSLR forum that swears up and down that stacking a bunch of in-camera dark subtracted lights is the same thing as subtracting a stack of separately taken darks. but i don't agree with this because we always have to register our light frames. this means the dark signal in the stacked lights no longer falls on top of itself from light frame to light frame. this is actually a good thing from a hot pixel rejection standpoint, but i think it destroys any correlation between the dark signal in each frame and therefore it is not the same thing mathematically...


Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: robininni]
      #5814426 - 04/21/13 11:48 PM

I spent a lot of time last year exploring this and I decided to use Canon's automatic dark subtraction all the time. This runs against the grain of the prevailing wisdom, but if you dig into the details of the statistics of how noise reduction works it makes sense, to me at least. Using this routine does a couple of interesting things. For example, it provides nearly perfect frame-to-frame temperature matching. Also, the time it uses to record and apply a dark is not at all wasted. The purpose of stacking images is to reduce noise, it does nothing to improve signal. Although the dark does not record signal, it does record noise, so the noise that would have been recorded if you had taken a light was still recorded and applied to your image stack. The statistics of this approach is slightly different than the traditional method of using stand-alone darks. Statistically, this method is essentially a difference in means. Using shot-to-shot darks is a method called paired differences. When there is a strong blocking factor (in this case tempertaure) paired differences gives lower variances than a difference in means. If the blocking factor is weak (constant temperature) paired differences has the same variance as a difference in means. Soooooo, at worst using the automatic dark subtraction does no harm, but because the temperature often changes when using an uncooled camera the results of using the automatic darks usually gives better results. The traditional method certainly works fine, but so does Canon's noise reduction routines. The bottom line for me... I haven't taken a stand-alone dark in nearly 2 years and I don't miss them at all.

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5814571 - 04/22/13 02:29 AM

yeah, that's the guy.

dark current is not noise. it's signal. unwanted signal, but not noise in the statistical sense. you should understand this since you seem to know something about statistics.

after you register your lights, the dark signal is shifted around all over the place. a given pixel stack of the registered lights has dark signal from different sensor pixels! therefore it does not matter if it's difference in means or paired differences or whatever, because you're integrating different dark signal in each pixel stack.

so the dark signal has been subtracted from each frame, but in a noisy fashion. this means that you've got some pixels where it was over-subtracted and some where it was under-subtracted. i don't see how you can hone in on the 'true' value of the dark signal at a given pixel location if you have shifted the dark signal all over the place before integration.

please address this, because it is the flaw in your reasoning.



Edited by pfile (04/22/13 02:38 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5814594 - 04/22/13 03:01 AM

My take is that there is no harm trying long exposure noise reduction (NR) while one is learning the rigors of AP. I use it quite often; my imaging results of NR turned ON here.... Regards


Note: Some image processing instructions and Canon AP settings, including NR, here....


Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5814735 - 04/22/13 07:30 AM

Yeppers, that's kinda the point. No harm in trying. I experimented with this enough with both actual images and bench-top data to convince me that it works and works very well. My background in statistical analysis of data just helped me to understand why it works as well as it does. I've also been working with uncooled cameras for about 10 years and I've fought the good fight on noise and hot pixels. Since I started using the internal darks I haven't seen a single hot pixel and I can push the gain higher with fewer artifacts. There are good reason to keep the gain low, but it is nice having that option when the need arises.

If you're at all curious there is no harm in taking the path less traveled. For me it has work very well and it is sooooo nice not having to fool with darks anymore. It did a while to shed the nagging thought that somehow I was loosing something by not taking more lights, but the results and convenience speak for them selves. Besides, I got over it real quick the first time I got to go straight to bed after taking my last image instead of staying up taking dark. (C'mon mom, 5 more minutes!) And yes, I used to have gigabytes of archived darks (long ago deleted).

Have fun folks, it is just a hobby.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5815141 - 04/22/13 12:28 PM

okay, well that's fine, but you don't have any mathematical or scientific data other than 'it works fine' and the appeal to 'i know statistics'.

unless your tracking is spectacularly bad there's never any reason to push the gain. you don't collect more photons that way and you lose dynamic range.

if you want to make radical claims like this, show us the math. otherwise what's in HAIP is the proper way to reduce CCD images.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5815203 - 04/22/13 12:49 PM

Quote:

Although the dark does not record signal, it does record noise




Considering that hot pixels are signal this statement is false. Hot pixels are not noise and can't be treated as such mathematically (its not random). However random noise from various sources is also recorded as well of course and that can never be avoided.

Quote:

background in statistical analysis of data just helped me to understand why it works as well as it does




Good - but you need to brush up on what inputs are signal and which are noise. Get that wrong (as you did) and the following analysis is going to be very suspect

Pfile has it pretty well right.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5815470 - 04/22/13 02:46 PM

Quote:

yeah, that's the guy.

dark current is not noise. it's signal. unwanted signal, but not noise in the statistical sense. you should understand this since you seem to know something about statistics.

after you register your lights, the dark signal is shifted around all over the place. a given pixel stack of the registered lights has dark signal from different sensor pixels! therefore it does not matter if it's difference in means or paired differences or whatever, because you're integrating different dark signal in each pixel stack.

so the dark signal has been subtracted from each frame, but in a noisy fashion. this means that you've got some pixels where it was over-subtracted and some where it was under-subtracted. i don't see how you can hone in on the 'true' value of the dark signal at a given pixel location if you have shifted the dark signal all over the place before integration.

please address this, because it is the flaw in your reasoning.






Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
SKYGZR
scholastic sledgehammer
*****

Reged: 08/14/09

Loc: Planet Earth, Milky Way Galaxy...
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5815497 - 04/22/13 02:59 PM

I find that in camera darks are the least hassle, and most reliable, yet that's just me.

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5815854 - 04/22/13 05:49 PM

Quote:



Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?

dave




as you say the uncertainty in the dark can not be totally eliminated, but it can be reduced dramatically...

...just like you do with your lights - you integrate a whole bunch of them so that you get down to the true value of the dark current at a given pixel location. you can also reject outlier pixels to clean up cosmic ray hits, etc.

the dark signal in the light is noisy, but that's no reason to increase the noise in your calibrated sub by subtracting a noisy dark.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5816158 - 04/22/13 07:16 PM

Quote:

Quote:


dave




Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?


as you say the uncertainty in the dark can not be totally eliminated, but it can be reduced dramatically...

...just like you do with your lights - you integrate a whole bunch of them so that you get down to the true value of the dark current at a given pixel location. you can also reject outlier pixels to clean up cosmic ray hits, etc.

the dark signal in the light is noisy, but that's no reason to increase the noise in your calibrated sub by subtracting a noisy dark.




Thanks for the reply. This is the heart of my question.

My simple minded view of this is that one would take a bunch of dark frames and average them to remove noise. Assume that conditions/duration are the same as the image subs.

Then you do the same for the set of image frames and average those. Subtract the averages and 'you are done' (am ignoring light flats and bias frames).

Now assume that these darks came from the Canon automatic dark frames that the camera takes, but that they are not subtracted from the image frame - just saved with the image frames.

Mathematically it does not matter if you do a frame by frame (light minus dark) subtract and average that result or just average the darks then average the image frames and do a single subtract. You'll get the same answer - not almost the same but exactly the same. Hence my question.

In further thinking about this maybe the biggest disadvantage beyond time in using the camera to take/apply auto darks is the fact that you then have no flexibility in how you apply your darks - pretty much stuck with averages.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5816208 - 04/22/13 07:37 PM

well i think that is john's argument. but here is mine. right before you average your light frames, you register them. meaning you rotate and translate the images so that the stars all line up with one another.

that shifts the residual noise from the dark current all over the place. when you stack a column of pixels, those pixels no longer came from the location x,y on the sensor. so you're really combining the noise from the dark signal from a whole bunch of different pixel locations. because of this i just don't see how you can make any statistical arguments about the equivalence of the methods. in one method you honed in on the true value of the dark current at a pixel location and subtracted it from the same pixel in the light. in the other, you've got dark current error from a bunch of different pixels all averaged together.

when making your master dark, does it make sense to rotate and translate the dark subs first? cause that's what you end up doing in the in-camera case.

also, as you point out you are unable to do outlier rejection or other fancy stacking techniques on the darks... because you have no darks. for long exposures now you've got cosmic ray hits that occurred during the light exposure PLUS cosmic ray hits that occured during the dark exposure, all right there on your preprocessed light. of course you can still do outlier rejection when stacking those lights, so all is not lost. but you just put noise into your calibrated image that did not need to be there.


Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5816231 - 04/22/13 07:54 PM

They are mathematically the same if there is no blocking factor. However, in an unregulated camera there is a blocking factor, temperature, and it can be very significant. I was puzzled for a long time why my dark libraries were hit and miss; sometimes doing okay, sometimes not. Then I took a close look at my source darks before I averaged them. They usually started out relatively cool, and then rapidly heated and the noise increased right along with it. If you want to see something interesting take a series of darks and look at the temperature and noise of each individual frame. No wonder my darks were hit'n miss, what ta' heck was I trying to average. Simple means and differences in means are built around uncertainty that is random and normally distributed. These distributions are far from normal, probably closer to an f-distribution. If you go crazy and average enough frames they'll migrate towards normal, but as long at the temperature is increasing there will be bias towards the high temperature side of the distribution. Now in the big scheme of things it is not that big of a deal as these methods are very simple and robust and they obviously work quite well. However, this is such a perfect real-world application for paired differences it is just beautifully simple and the way that Canon implements it is wonderful and smart. It also brings so many other statistical factors into ballance. For example, strictly speaking, when using a difference in means you really should use the same number of data points (source images) in each sample so that the variances match. The statistical work-around is to use pooled variances, but again the methods are sufficiently robust you can often away without it (and we do). All these little nuances go away when using paired differences since by definition you are always using the same number of darks and lights.

Very simple, very clean, very effective. I like it.

Enjoy.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5816274 - 04/22/13 08:15 PM

pf - thanks. I think I now understand what you are saying.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5816399 - 04/22/13 09:23 PM

a lot more jargon with no math. baffle them with BS.

i'm not real sure why canon deserves accolades for what they do. they are doing the best they can without making the time between shots be ridiculously large, but there's nothing deep or prescient about what they are doing. basically they are trying to get rid of hot pixels, which is what terrestrial photographers see and wonder what the heck is wrong with their camera.

the solution to your temperature problem is easy. sort the darks by EXIF temperature or ambient temperature. integrate only those darks which are within perhaps T+/- 2C of a given temperature and call that your master dark for T. the dark current in that master will be compatible with your lights with temperature T.

you must think that the dark current is equal at every pixel or something. but that's just not the case due to impurities in the silicon, etc. take a look at a high-quality master dark (40+ frames) which has been bias subtracted and tell me what you see. it's not a grey image.


Post Extras: Print Post   Remind Me!   Notify Moderator  
neutronman
Carpal Tunnel


Reged: 01/10/08

Loc: Dallas, Texas
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5816615 - 04/23/13 12:04 AM

Personally, I prefer NOT to spend the extra time under clear skies shooting auto dark frames (when I could shoot 10-12 darks after I go to bed).

However, I DO use auto darks when shooting night landscape shots, since my exposures are typically 30-45 sec and the short auto darks work great and save work later. But that's just me


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5816729 - 04/23/13 01:43 AM

Makes sense John. Thanks for sharing your perspective and experience. I personally don't do dark subtraction at all -- just bad pixel mapping and sigma-rejection based stacking. That probably puts me beyond the pale vis a vis the calibration orthodoxy.
...Keith

Quote:

They are mathematically the same if there is no blocking factor. However, in an unregulated camera there is a blocking factor, temperature, and it can be very significant. I was puzzled for a long time why my dark libraries were hit and miss; sometimes doing okay, sometimes not. Then I took a close look at my source darks before I averaged them. They usually started out relatively cool, and then rapidly heated and the noise increased right along with it. If you want to see something interesting take a series of darks and look at the temperature and noise of each individual frame. No wonder my darks were hit'n miss, what ta' heck was I trying to average. Simple means and differences in means are built around uncertainty that is random and normally distributed. These distributions are far from normal, probably closer to an f-distribution. If you go crazy and average enough frames they'll migrate towards normal, but as long at the temperature is increasing there will be bias towards the high temperature side of the distribution. Now in the big scheme of things it is not that big of a deal as these methods are very simple and robust and they obviously work quite well. However, this is such a perfect real-world application for paired differences it is just beautifully simple and the way that Canon implements it is wonderful and smart. It also brings so many other statistical factors into ballance. For example, strictly speaking, when using a difference in means you really should use the same number of data points (source images) in each sample so that the variances match. The statistical work-around is to use pooled variances, but again the methods are sufficiently robust you can often away without it (and we do). All these little nuances go away when using paired differences since by definition you are always using the same number of darks and lights.

Very simple, very clean, very effective. I like it.

Enjoy.




Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5816982 - 04/23/13 08:26 AM

ICNR uses one single dark. I use 50 plus darks combined to create a master dark.

As well, if you use ICNR early in the night, you camera is still "warming up". Your temps may not be so close.


Post Extras: Print Post   Remind Me!   Notify Moderator  
NigelM
member


Reged: 05/22/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5817013 - 04/23/13 08:48 AM

If your lights are perfectly aligned, then mathematically there is no difference between taking a bunch of darks, subtracting one dark from each light then stacking the result, or stacking all the darks first to form a master dark, then subtracting this from each light and stacking the result.

The question is, what happens if you need to realign the lights to stack them (essentially dithering). Then the two methods are not mathematically the same. The argument that I have seen put forward is that in this case subtracting a master dark results in better signal-to-noise in the final image, due to the averaging effect of dithering the master dark.

I have not really seen believable maths to prove this - one problem is that people assume the noise is random and uniform, but if is was, then you would be better off subtracting one constant number from your frame to represent the dark. This is clearly not the case for real darks.

NigelM


Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: NigelM]
      #5817026 - 04/23/13 09:00 AM

So who's going to test this? Seems easy enough.

Take a 5m dark frame with ICNR on.

Then take a 5m dark with it turned off and calibrate it with multiple darks.

The image that is closest to 0 wins.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: NigelM]
      #5817056 - 04/23/13 09:30 AM

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5817063 - 04/23/13 09:37 AM

Quote:

It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).

dave




The master dark is applied to the individual lights before stacking.

What do you guys mean about aligning darks? By temp or just ensuring orientation? Assuming both, there wouldn't be a way to misalign. The image is X pixels by Y pixels whether its a dark or light. Dithering should have no negative effect on darks.

Edited by Footbag (04/23/13 09:53 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: Footbag]
      #5817114 - 04/23/13 10:11 AM

Quote:


SNIP

The master dark is applied to the individual lights before stacking.

SNIP





That makes perfect sense. But then I don't understand what pfile is talking about regarding 'darks walking around' (or something like that).

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
neutronman
Carpal Tunnel


Reged: 01/10/08

Loc: Dallas, Texas
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5817142 - 04/23/13 10:29 AM

Quote:

Makes sense John. Thanks for sharing your perspective and experience. I personally don't do dark subtraction at all -- just bad pixel mapping and sigma-rejection based stacking. That probably puts me beyond the pale vis a vis the calibration orthodoxy.
...Keith





Hey, that should work too. So many ways to skin the proverbial cat. Basically, whatever workflow one prefers that gets the job done!


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: neutronman]
      #5817337 - 04/23/13 12:17 PM

it's okay to use in-camera dark subtraction as long as you realize your subs will have more noise in them than if you subtracted a clean master dark.

the problem here is the claim that after stacking registered lights, the dark signal has also been averaged and that the in-camera dark subtraction is equivalent to making a master dark and subtracting that from the raw light.

imagine that before you register your lights, you draw a big cross in the same place on each image, like a danish flag, but one pixel wide. let that cross represent the dark signal in the sub. as you register your lights, in each sub the cross is rotated by different angles and where the two lines cross is shifted around. now you average the registered lights. do you have a strong horizontal/vertical cross pattern on your stack, or a smeared out version of the cross?

that's what i'm talking about. the dark signal, or the dark signal error no longer lies directly on top of itself after registration.

by 'aligning' darks i was trying to illustrate that if john's method really worked, then "dithering" your darks when making a master dark would be a good idea. but we don't do this. we stack the dark frames directly on top of one another in order to get the average value of the dark signal at each pixel location. if you shift your dark subs around before making the master dark, you're not really doing anything meaningful with the dark signal.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5817379 - 04/23/13 12:43 PM

Quote:

imagine that before you register your lights, you draw a big cross in the same place on each image, like a danish flag, but one pixel wide. let that cross represent the dark signal in the sub. as you register your lights, in each sub the cross is rotated by different angles and where the two lines cross is shifted around. now you average the registered lights. do you have a strong horizontal/vertical cross pattern on your stack, or a smeared out version of the cross?

that's what i'm talking about. the dark signal, or the dark signal error no longer lies directly on top of itself after registration.




I'm pretty sure that the master dark and flat are applied to the lights before stacking and before rotation. In my further reading, it's not clear. I watched a DSS stacking process, and it still wasn't clear at which point the darks and flats are applied. But I don't see any reason they wouldn't be applied before the actual stacking.

I do my processing in DSS, and most of it happens behind the scenes. I think some programs allow you to do it manually. Someone who uses one of those may have to chime in.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Footbag]
      #5817386 - 04/23/13 12:47 PM

i use one of those programs. absolutely, positively, no doubt about it, darks, bias and flats are applied to the subs BEFORE registration.

i'm trying to show the flaw in John's reasoning here! my example about the cross is what happens to the dark signal in his method, not the method that we all use and is known to be correct, which is to calibrate images with master frames BEFORE registration.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08

Loc: Voorhees, NJ
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5817397 - 04/23/13 12:52 PM

I agree with pfile on this question.

The other thing that jgraham ignores is that the only way to increase the signal from the object, which is half of the signal-to-noise equation, is by gathering more photons from the object in a light frame.

If you use clear dark-sky time shooting in-camera darks, you waste the opportunity to improve the signal in your images.

This is a completely separate issue from what is happening with the thermal signal and its associated poisson noise in the darks.

There is no way you can argue that using dark-sky time to collect signal and shooting separate darks on a cloudy night is not better than in-camera dark-frame subtraction.

Saying that for you, personally, in-camera darks work better, just means you weren't doing your master cloudy-night darks correctly. :-)

Simply match the temps of the darks and the lights with Dark Library, or use automatic dark-frame matching in Images Plus.

In-camera dark-frame subtraction is fine for wide-angle scenics where you are only shooting a single frame or a couple of frames, but for serious long-exposure deep-sky work, it is a poor choice.

Jerry


Post Extras: Print Post   Remind Me!   Notify Moderator  
jsines
sage
*****

Reged: 09/06/11

Loc: Berkley. Michigan
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5817400 - 04/23/13 12:52 PM

Quote:

It is my understanding that enabling the Canon (T3) 'long exposure noise reduction' is simply an automated dark frame (same exposure length) that follows the light frame (and then applied automatically to the light frame).

Somewhere (don't recall where) I read a recommendation that this function be disabled. Is there a problem here (would seem to me to be an automatic dark frame/bias frame) other than the fact that this will probably extend the time that the camera will spend in the imaging process? FWIW, I am a complete newbie here.

dave




The function needs to be disabled because you are taking bias frames in addition to the lights. The bias frame contains the readout signal which is just the signal from the camera reading the chip. It is present in the light frame, the dark frame, and the flat frame, and it needs to be subtracted from all of them. When you apply in-camera dark frame subtraction, I don't think you are able to subtract the bias frame from the dark frame.

My understanding of the correct process is -
* Calibrated Lights = (Light - Bias - k*(Dark-Bias))/(Flat - Bias), where k=dark scaling factor (in PixInsight)
* Register Calibrated Lights (Star Align)
* Stack Registered, Calibrated Lights

someone correct me if I'm wrong, please, I may be confused.


Post Extras: Print Post   Remind Me!   Notify Moderator  
jsines
sage
*****

Reged: 09/06/11

Loc: Berkley. Michigan
Re: DSLR 'long exposure noise reduction' new [Re: Jerry Lodriguss]
      #5817420 - 04/23/13 01:02 PM

Quote:


There is no way you can argue that using dark-sky time to collect signal and shooting separate darks on a cloudy night is not better than in-camera dark-frame subtraction.

Jerry





On a clear, cloudless night where I need to work the next morning, I may get 2 hours of imaging time before I need to go to bed. If I'm taking 3 minute lights, it's a matter of taking 10 lights per hour or 20 lights per hour, or 20 lights versus 40 lights in 2 hours.

Big difference in signal.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: jsines]
      #5817424 - 04/23/13 01:02 PM

yes, if you are going to make your own master calibration frames you really need to turn off in-camera dark subtraction. otherwise your darks are not darks, they are just dark current noise.

jsines, your outline of the steps to calibrate and integrate images is correct.

i'm glad jerry has weighed in here because frankly every time the in-camera dark subtraction comes up, i have this one-sided argument with john about how registration is the flaw in his theory. he never addresses that but just comes back with more statistics jargon.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5817486 - 04/23/13 01:28 PM

Quote:

i use one of those programs. absolutely, positively, no doubt about it, darks, bias and flats are applied to the subs BEFORE registration.

i'm trying to show the flaw in John's reasoning here! my example about the cross is what happens to the dark signal in his method, not the method that we all use and is known to be correct, which is to calibrate images with master frames BEFORE registration.




Gotcha. That explains why I agreed with you right up until then.


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5817779 - 04/23/13 03:22 PM

Quote:

yes, if you are going to make your own master calibration frames you really need to turn off in-camera dark subtraction. otherwise your darks are not darks, they are just dark current noise.

jsines, your outline of the steps to calibrate and integrate images is correct.

i'm glad jerry has weighed in here because frankly every time the in-camera dark subtraction comes up, i have this one-sided argument with john about how registration is the flaw in his theory. he never addresses that but just comes back with more statistics jargon.




Just so this isn't one-sided, I'm afraid I agree with John's logic and I respectfully disagree that there is some unacknowledged flaw in his theory. For every pixel there is a true value of dark signal that is dependent on factors such as temperature. If you take a bunch of darks at the same temperature however you don't get the same value every time - the variation around the true value is the noise. The objective of calibration is to get the best estimate of the true value. One way to do this is to stack a bunch of frames and take an average of each pixel's value which should converge on the true value the more frames you stack. That is the method that you promote, which should result in very little noise being injected into your light frame when it's calibrated. I think your logic is that this is the only way to get a good estimate of the true dark currant, and therefore when you register your lights and lose pixel-to-pixel mapping, you are losing the value of stacking darks. That's not what John is arguing. He is saying that a single matched dark could be as good of an estimate of your true dark signal as a stacked set of darks taken over a range of temperatures. If you accept that, then it's a red herring that your pixels get shuffled during registration because you aren't trying to get your best estimate of the true dark signal by stacking darks, your are trying to get the best estimate by subtracting the best-matched dark from your light. I don't see a flaw in this logic, it's just another way to skin the cat. Or you could use my method with dithering, bad pixel mapping with bias subtraction only, and sigma-based stacking and dispense with dark subtraction altogether. They are just different ways of trying to get a nice picture for those of us not doing photometry.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5817856 - 04/23/13 03:53 PM

Quote:


Just so this isn't one-sided, I'm afraid I agree with John's logic and I respectfully disagree that there is some unacknowledged flaw in his theory. For every pixel there is a true value of dark signal that is dependent on factors such as temperature. If you take a bunch of darks at the same temperature however you don't get the same value every time - the variation around the true value is the noise.





yes, this is correct, and it is the problem with using a single dark frame as your master dark. the light frame already has dark current in it with some amount of dark noise. when you subtract a single dark, you are adding insult to injury by injecting more noise into the calibrated frame than need be. if you drive the SNR of the master dark up (by stacking the darks) you inject the least noise possible.

Quote:


The objective of calibration is to get the best estimate of the true value.





actually, that's the objective of stacking or integrating images. one objective of calibration is to remove fixed patterns, which are CCD artifacts, from your lights. another objective is to compensate for different QE across pixels and remove optical train vignetting using flats.

Quote:


One way to do this is to stack a bunch of frames and take an average of each pixel's value which should converge on the true value the more frames you stack. That is the method that you promote, which should result in very little noise being injected into your light frame when it's calibrated. I think your logic is that this is the only way to get a good estimate of the true dark currant, and therefore when you register your lights and lose pixel-to-pixel mapping, you are losing the value of stacking darks. That's not what John is arguing. He is saying that a single matched dark could be as good of an estimate of your true dark signal as a stacked set of darks taken over a range of temperatures. If you accept that, then it's a red herring that your pixels get shuffled during registration because you aren't trying to get your best estimate of the true dark signal by stacking darks, your are trying to get the best estimate by subtracting the best-matched dark from your light. I don't see a flaw in this logic, it's just another way to skin the cat. Or you could use my method with dithering, bad pixel mapping with bias subtraction only, and sigma-based stacking and dispense with dark subtraction altogether. They are just different ways of trying to get a nice picture for those of us not doing photometry.
...Keith





i think you misunderstand what he is saying. there'd be no need to talk about paired differences and all that stuff if all he was trying to say is that a single dark at the exact same temperature as the light is better than a stack of darks in a narrow temperature range.

he's saying that the net effect of doing in-camera dark subtraction on N frames and then stacking those N frames is the same as stacking N dark frames and subtracting that from each light. that's a totally different thing. i think what he is saying would be true if you did not register the frames. but once you have done that the dark signal residual noise in pixel (X,Y) of one image is getting averaged with the residual noise in pixel (M,N) of the next. in order to make statistical arguments about what happens to the dark signal, you have to make sure that you are stacking the same pixels... and you're not once the frames have been rotated and translated.

again, i'm not saying that one should not do in-camera dark subtraction, i'm saying that fooling yourself into thinking both methods are mathematically equivalent is wrong.


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5818150 - 04/23/13 05:56 PM

Hmmm, I see. In the first message about temperature as a blocking factor I don't see an argument that it's mathematically equivalent, but in John's second posting what you are objecting to is the argument that it is mathematically equivalent since you aren't averaging the same pixels once the frames are registered.

Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818321 - 04/23/13 06:38 PM

yes, i don't have any beef with the temperature argument. though i think it's known that dark current scales linearly with time and exponentially with temperature, so in theory one could properly scale a dark at a different temperature to match the lights. i think people don't do this because it would require too much characterization of the CCD - exponential terms increase very quickly and so small errors become large ones. probably each sensor is different. but linear scaling (for time) is very straightforward.

i'm willing to be proven wrong here, because what we're talking about is math. one should be able to prove that the two methods are the same. so far i have not seen that proof. your empirical measurement is a good idea.

in your experiments i guess an important thing is to figure out what the margin of error is in order to conclude that the results are truly the same or truly different. no idea how to compute that, though.


Post Extras: Print Post   Remind Me!   Notify Moderator  
RedLionNJ
scholastic sledgehammer


Reged: 12/29/09

Loc: Red Lion, NJ, USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5818440 - 04/23/13 07:10 PM

With you on most of your points, pfile. Two additional things worth pointing out:

1. Dark signal (at least as recorded in Canon CR2 files) does NOT increase linearly with time. Repeated tests done by myself and others show an interesting "curve" (for me, it's nearer two lines angled at around the 120s mark). This is why flats and matching dark flats are a better option than simply flats and bias - you can't really scale for the Canon CR2s.

2. (and this is in further defense of pfile and Jerry's logic) Imagine you have a light pixel with a value of 5 and you subtract a single dark which happens (through statistical noise) to have a value of 5 or higher in that pixel - that calibrated light now has zero in that pixel. It's never going to have anything else in there. You've "lost" information. But say you averaged 20 darks and the average value of that pixel was 3 - subtracting 3 from that value of 5 will still leave information. Entropy is lower.

Grant


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5818494 - 04/23/13 07:30 PM

Quote:

in your experiments i guess an important thing is to figure out what the margin of error is in order to conclude that the results are truly the same or truly different. no idea how to compute that, though.




I suppose you could do that with a resampling method (pull a random sample of a dozen of the lights multiple times to get a variance) but to be honest, if it is so close that you have to resample to test significance then it's probably a moot point anyway as it becomes a "so what if it's significant, can you see a difference" issue. I'll set this up for tonight and just see what the mean and variance is for the normalized stacks calibrated and processed in different ways and see where that goes.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
terry59
Post Laureate
*****

Reged: 07/18/11

Loc: Colorado, USA
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818522 - 04/23/13 07:43 PM

Quote:

Hmmm, I see. In the first message about temperature as a blocking factor I don't see an argument that it's mathematically equivalent, but in John's second posting what you are objecting to is the argument that it is mathematically equivalent since you aren't averaging the same pixels once the frames are registered.

Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith




My kind of analysis...thank you!


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818534 - 04/23/13 07:52 PM

Quote:


but to be honest, if it is so close that you have to resample to test significance then it's probably a moot point anyway as it becomes a "so what if it's significant, can you see a difference" issue.
...Keith




this is true.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818547 - 04/23/13 08:00 PM

FWIW, I decided to do a simulation. It was pretty easy to think through the 'single pixel' problem (it doesn't matter when you apply the dark - frame by frame like a Canon camera or averaging darks before applying).

But when you apply registration (after dark compensations are applied) was more than my mind could figure out. So I set up a spreadsheet structured as follows.

1) Four Pixels, each with their own dark signal and dark noise (noise generated with the randbetween() function).

2) There were 4 signals. One was 60 (signal value) with noise randomly added while the others were 30 (signal value) with their own noise values.

3) I would move the 60 unit signal around between pixels (the other pixels receiving the 30 + noise signals)randomly - sounds a lot like my guiding :-)

4) I did '20 subs' on these 4 pixels. Each sub would have a light frame and a second dark frame (different noise values between the two). The light frame had a dark component and the second run was just a separate dark.

5) For each of the 20 subs (on each of the 4 pixels) I would then 'apply registration' by picking out the pixel (out of the 4) receiving the 60 unit (plus signal noise) signal.

6) I would them dark adjust #5 above two ways. I applied the Canon approach of subtracting the paired dark for that pixel in that sub. I also averaged all the darks for that pixel across all 20 subs and used that value for each sub in the run. Note that each pixel had its own dark frame set. I ended up with two different dark adjusted results, obviously.

I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

The Canon approach has the advantage of simplicity and (I would think) a clear advantage of better matching of temperatures in darks vs. lights.

FWIW.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818557 - 04/23/13 08:05 PM

Quote:

The Canon approach has the advantage of simplicity and (I would think) a clear advantage of better matching of temperatures in darks vs. lights.




Well put.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5818664 - 04/23/13 09:22 PM

Quote:



I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

dave





I'm not sure I follow your methodology entirely, but I do believe you have one error in it. If I understood correctly, you did this for 20 lights for both methods. However, what you would need to do is 20 lights for in-camera dark subtraction, and 40 lights for using darks from a dark library. By not using in-camera darks, that imager captures twice as many subs in the same amount of time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818673 - 04/23/13 09:26 PM

Quote:


Now you have piqued my interest. I'm imaging a narrowband target tonight because of the moon and I wasn't planning to image with my DSLR, but I'll piggyback my second scope and DSLR and run an experiment while the mono CCD is collecting frames. I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.
...Keith




Very generous of you to do this test, Keith. But, I'll point out the same flaw in methodology. You'll want to capture 48x3 minutes without in-camera dark subtraction (or, change the in-camera dark subtract to 12 subs). In any event, you'll need to have half as many in-camera dark subtracted subs, as that is what happens in the real world - people using in-camera darks subtraction capture half as many subs in the same amount of dark time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08

Loc: Voorhees, NJ
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818698 - 04/23/13 09:44 PM

Quote:

I'll run 24 x 3min light frames with in-camera dark subtraction on, then 24 x 3min light frames with the in-camera dark subtraction off, then I'll finish off with 24 x 3min dark frames. I'll do the math in imageJ for the lights calibrated with the stacked darks, the lights with in-camera darks, and the lights just dithered with BPM and sigma-stacking and see what comes up. I'll post the results here once I get a chance to work them up.




Keith,

This isn't a valid comparison.

You are forgetting the signal portion of the equation.

If you don't shoot in-camera darks, you can shoot 48 x 3 min lights in the same amount of time as 24 x 3 min lights with ICDS, and double the amount of signal you collect.

Regardless of what is happening with the "noise", even if is the same with both methods, the ICDS method gathers only 1/2 the signal in the same amount of clear dark sky time.

Jerry


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Jerry Lodriguss]
      #5818708 - 04/23/13 09:49 PM

OK. I'll have time to run 40 lights and 20 ICDS frames. Should be able to get started in a few hours.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5818743 - 04/23/13 10:08 PM

well i think based on the spreadsheet experiment plus of course doubling the number of subs, the SNR increase of doing the imaging with separate darks should be slightly more than sqrt(2).

i think that's a no-brainer - the difference in the # of subexposures is going to totally swamp the contribution of the dark current / dark current noise from an SNR perspective.

i was more interested in the equal subexposure case, even though in real life the 'separate darks' imager gets 2x the subs in the same time. from the spreadsheet simulation it sounds like the two cases are not the same but the difference is minute.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Dave Lee
sage


Reged: 02/14/13

Loc: Pinehurst, NC USA
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5819202 - 04/24/13 06:32 AM

Quote:

Quote:



I then repeated this whole thing 20 times (20 runs of 20 subs across 4 pixels). Obviously the answer we are looking for is 60. For the Canon camera approach the result (average of the 20 runs of 20 subs after registration) was 70.7 For the traditional post processing approach the result was 70.5. I view these as the same.

Based on this I would judge that (mathematically) the approaches are roughly equivalent. The more traditional approach has the advantages of more flexibility, the opportunity to generate better darks than you might get from just your subs and potentially more imaging time.

dave





I'm not sure I follow your methodology entirely, but I do believe you have one error in it. If I understood correctly, you did this for 20 lights for both methods. However, what you would need to do is 20 lights for in-camera dark subtraction, and 40 lights for using darks from a dark library. By not using in-camera darks, that imager captures twice as many subs in the same amount of time.




It was 20 lights (same light frames for both methods) and 20 darks. The only change was when and how the darks were applied. THen all this was repeated 20 times, of course.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5819307 - 04/24/13 08:54 AM

If you shoot darks separately then those darks can be used on separate objects. If you shoot the darks in sequence(Long Exposure Noise Reduction) with the object then you have to take as many darks as you take lights.

If I was going to shoot just as many darks as lights I really don't think there would be a big difference between shooting them in sequence as shooting them separately. However, I can't think of a reason why I would want to take as many darks as I take lights so I can't ever see a reason for taking the darks in sequence.


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: Dave Lee]
      #5819565 - 04/24/13 11:26 AM

Quote:


It was 20 lights (same light frames for both methods) and 20 darks. The only change was when and how the darks were applied. THen all this was repeated 20 times, of course.

dave





Yes, that's what I thought you were saying. What you're basically saying is that your model showed that applying 20 closely matched darks to their corresponding lights yielded you roughly the same SNR as applying 20 averaged darks to 20 lights.

If your spreadsheet model is accurate, then, as pfile noted, what you've shown is that not using in-camera noise reduction (using a dark library) yields a 1.4x improvement in SNR (since you would have twice as many lights in the same amount of time).


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5819604 - 04/24/13 11:50 AM






If your spreadsheet model is accurate, then, as pfile noted, what you've shown is that not using in-camera noise reduction (using a dark library) yields a 1.4x improvement in SNR (since you would have twice as many lights in the same amount of time).




You also need to consider that the benefit of doubling the number of lights decreases as you increase the number of lights.

If you are taking a very small number of short exposure lights and you plan on taking the exact same number of darks then you will probably get almost the same result with in camera darks as with combing them in post. However, I am not sure why you would ever choose to take a very small number of short exposure lights and the exact same number of darks.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: mpgxsvcd]
      #5819827 - 04/24/13 01:36 PM

I have never used in camera noise reduction so don't have any direct experience, but would advise against it for the following reason. Take a series of, say, five minute darks and then stretch them equally and examine them. You will inevitably see differences between them, mostly from cosmic ray hits. That is why we average together or combine using a statisical exclusion method a large number of darks to get a "master" dark frame for calibration...so that the frame we are going to use for calibration is a good representation of the dark current at that particular temperature and has little or no individualized frame contamination. When you let the camera do individual dark frame calibration, each light frame is being calibrated with a different single dark frame that probably has a number of cosmic ray hits in each. Subtracting these cosmic ray hits from the light frames results in dark tracks in each frame that are not collated frame to frame. Stacking the images using an averageing or statistical exclusion method will minimize the effect, but personally I'd rather not trash up the light frames by being calibrated with contaminated single dark frames.

Edited by Hap Griffin (04/24/13 01:39 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
mpgxsvcd
Carpal Tunnel


Reged: 12/21/11

Loc: Raleigh, North Carolina
Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5819878 - 04/24/13 02:03 PM

Wouldn't the differences in temperature have much more of an effect on the darks than stray cosmic rays?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: mpgxsvcd]
      #5819902 - 04/24/13 02:18 PM

The temperture differences will be small regardless if the dark frame is taken at the same time as the light frame, or chosen from a library of master dfarks and various tempertures. I keep a library of master darks and biases for approximately every five degrees.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5819910 - 04/24/13 02:23 PM

Here is some information on cosmic ray hits in dark frames. I would encourage you to check for yourself by stretching a set of dark frames approximately to the extent that you would in processing a light frame and look at the differences. http://darkerview.com/CCDProblems/particlehit.php

Edited by Hap Griffin (04/24/13 02:23 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5820011 - 04/24/13 03:17 PM

wow, i want to look back thru my darks and see if i have any particle decay events. that's cool.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hap Griffin
Vendor (Imaging Infinity)
*****

Reged: 04/15/08

Loc: Sumter, SC
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5820093 - 04/24/13 03:49 PM

I see them all the time in my CCD darks.

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Hap Griffin]
      #5821161 - 04/25/13 12:45 AM

i thought i would experiment a little bit with some darks that i already have. i happen to have 20 1800s darks at -30C from my CCD camera. i paired the darks into even/odd sequence number pairs, and made a master dark from the 10 odd frames, then calibrated the 10 even frames with that master. i then separately calibrated the 10 even frames with the consecutive odd frame.

however, the experiment ended right there - the single-dark subtracted frames have a whole boatload of pixels with 0 value. this is because the noise in the dark signal is great enough that many pixels end up being negative after subtraction, and so are clamped to 0.

i will have to add a small pedestal to only the darks being calibrated to prevent this from happening.


Post Extras: Print Post   Remind Me!   Notify Moderator  
RedLionNJ
scholastic sledgehammer


Reged: 12/29/09

Loc: Red Lion, NJ, USA
Re: DSLR 'long exposure noise reduction' new [Re: RedLionNJ]
      #5821762 - 04/25/13 10:43 AM

Point immediately above already covered...

Quote:

With you on most of your points, pfile. Additional thing worth pointing out:

Imagine you have a light pixel with a value of 5 and you subtract a single dark which happens (through statistical noise) to have a value of 5 or higher in that pixel - that calibrated light now has zero in that pixel. It's never going to have anything else in there. You've "lost" information. But say you averaged 20 darks and the average value of that pixel was 3 - subtracting 3 from that value of 5 will still leave information. Entropy is lower.






Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: RedLionNJ]
      #5821837 - 04/25/13 11:29 AM

well, i missed it in all the action in this thread. it is never a good thing to destroy data. however, i assume that in normal usage you'd never zero out much data as you'll probably have some signal, DSO or skyglow, that gets the average ADU way above the dark signal.

Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5822947 - 04/25/13 08:06 PM Attachment (16 downloads)

Here is my empirical test of the theory. I ran some lights the night before last with and without In-camera Noise Reduction turned on and followed up with dark frames. First, I planned to run more lights without ICNR, but due to a glitch in Sequence Generator Pro (who knew that if you disconnect the DSLR, as I did in order to access the Menu to turn off ICNR in the camera, then reconnect it resets the ISO to 100) I collected a bunch of frames at 100 ISO before I realized it had reset. Anyway, I was able to collect 16 x 5min frames at ISO800 with my DSLR with ICNR turned on and turned off. I then stacked these in different ways and analyzed them to see the effects on the stacked frames. The 4 treatments were:

1. ICNRx8: ICNR x 8 frames stacked in DeepSkyStacker with Average combine (to simulate the reduced number frames that can be collected due to the longer exposure time of ICNR)
2. ICNRx16: ICNR x 16 frames stacked in DSS with an Average combine (to compare with the Dark-subtracted and Bad Pixel Map methods).
3. DSx16: No ICNR x16 frames calibrated with x16 dark frames and Avg combine in DSS
4. BPM-Bx16: No ICNR x16 frames calibrated with a Bad Pixel Map generated from a single Dark and a Master BIAS frame created from 100 bias frames.

All frames were collected with Sequence Generator Pro and output as CR2 files with dithering turned on. All calibration (for the DS and BPM frames) was done with Nebulosity. Stacking was done with DSS.

Here are the results of analysis of the final stack produced by DSS in ImageJ:

Mean StdDev Min Max
ICNRx8 2077 648 109 65273
ICNRx16 2075 624 54 65239
DSx16 1627 399 96 65267
BPM-Bx16 1439 399 86 65267

The first thing you notice is that the Standard Deviation (Sq Rt of the variance) is considerably lower in the Dark-subtracted and the Bad Pixel Map stacks. However, the mean is also considerably lower in these frames. So I set out to see if I could normalize these in Nebulosity to reach an equivalent mean. I used the Match Histogram tool in Nebulosity with the BPM stack (lowest mean value) as the reference. Here is the result:

ICNRx8 1741 508 199 51259
ICNRx16 1711 504 77 52793
DSx16 1443 399 0 65217
BPM-Bx16 1439 399 86 65267

The Match Histogram tool brought the means closer together, and the SD of the ICNR stacks was lower, but the mean values were still considerably higher in the ICNR stacks. However, look at what happens to the Max Value as a result of the histogram matching, the ICNR frames have a Max about 14000 ADU lower than the DS or BPM stacks after trying to match the histograms to the BPM stack.

So what's happening? I can think of two possibilities, one of which hasn't been discussed in this thread. First, maybe the Canon ICNR isn't just subtracting the dark, it is also stretching it (this has been mentioned before and seems quite likely given other analyses of Canon frames, it is also supported by the fact that the ICNR frames were color balanced better - I shot with an IDAS LPS-V4 filter - and I think this could have only happened during the in-camera processing) and/or second, maybe this is an artifact of the Canon ICNR doing its calculations on 14bit files while the calibration in Nebulosity and DSS are doing the calculations in 32bit (this I'm not sure would be a factor, but I put it out there for discussion).

My take on these numbers is that the ICNR has higher variance among the pixels than BPM or DS calibration. If that's the case, this higher variance would be seen as higher noise in the images. Below is a jpg of the above stacked frames that have been color-balanced then linearly stretched in PhotoShop. I chose a region centered on the Cat's Eye Planetary Nebula -- the nebula itself is highly overexposed because of the length of exposure and the stretching, but a part of the very faint outer shell is visible and comparable between the images. I've tried to make the linear stretching as consistent as possible across frames. As predicted by the ImageJ analysis, the ICNR frames look noisier than the BPM and DS calibrated frames, and the stack of 8 ICNR frames is noticably noisier than the stack of 16 ICNR frames. So you definitely lose by shooting only half as many light frames as many on this thread predicted. However, based upon the discussion I would have expected the ICNRx16 frame to be comparable to the BPM-Bx16 and DSx16 stacks. But the ICNRx16 looks noticealy noisier to me than either the BPM or the DS calibration. Although not really reflected in the numbers so much, the BPM image looks a little noiser to me than the DS frame, but the SNR of these two is quite comparable. Given what the two difference calibration techniques do, perhaps that isn't all that surprising. The BPM made from the single dark frame only corrected 97 pixels by using the data from surrounding pixels, but these were the hottest pixels in the image. Once the Bias was subtracted only the variability in the rate of dark signal accumulation among the neutral to warm pixels remains. Dark subtraction based upon 16 dark frames removes more of the dark signal and seems slightly smoother to me.

My take on this (I'm sure others will chime in) is that if you like collecting lots of Darks, then this seems to be the best method to calibrate your images. If you are lazy (like me) then it can be quite effective (at least at the temperatures I shoot at: the range over the evening was +5C to 0C) to take as few as 1 Dark and make a Bad Pixel Map to remove the outlier hot pixels then rely upon dithering to average out the noise without a big cost in SNR. In my experiment at least, the ICNR was not a good option because you lose SNR by only gathering half as many light frames and (I suspect) because Canon doesn't just do a Dark subtraction when it processes the ICNR images -- it does some sort of stretch that also stretches the noise.

Now I leave it to those much more knowledgeable than me to tell me where my analysis went wrong!
...Keith

Edited by microstar (04/25/13 08:11 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Footbag
Post Laureate
*****

Reged: 04/13/09

Loc: Scranton, PA
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823017 - 04/25/13 08:33 PM

Nice experiment. The only image I'd be happy with is the DSx16. That is what I would've expected.

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823045 - 04/25/13 08:41 PM

i would not be surprised if canon plays tricks when subtracting that in-camera dark. i think craig stark determined that they are already doing stuff to the raw frame even when ICNR is turned off to try to squash the dark signal.

who knows, it may have something to do with the problem i encountered in my experiment. perhaps they want to make sure somehow that they will never underflow a pixel on the dark subtraction.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5823056 - 04/25/13 08:45 PM

also, can i ask what your pixel offsets and rotation angles are in DSS? in DSS is 'black point correction' turned off in the DCRAW control? actually i can't even remember if DSS shows you that DCRAW control.

Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5823127 - 04/25/13 09:02 PM

Quote:

also, can i ask what your pixel offsets and rotation angles are in DSS? in DSS is 'black point correction' turned off in the DCRAW control? actually i can't even remember if DSS shows you that DCRAW control.




Aren't the pixel offsets and rotation angles unique to each subframe? So are you asking for all of these for each subframe in each stack?

I can't find any black point correction in DSS.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823197 - 04/25/13 09:36 PM

Quote:

Quote:

also, can i ask what your pixel offsets and rotation angles are in DSS? in DSS is 'black point correction' turned off in the DCRAW control? actually i can't even remember if DSS shows you that DCRAW control.




Aren't the pixel offsets and rotation angles unique to each subframe? So are you asking for all of these for each subframe in each stack?

I can't find any black point correction in DSS.
...Keith




I did have per channel background calibration turned on in DSS however.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823247 - 04/25/13 10:08 PM

yes, it's per subframe. but since you were dithering the x/y offsets should be relatively predictable, except for the effects of field rotation and/or differential flexure.

i'm just curious if the frame-to-frame shift is 10 pixels or 1 pixel, or what.


Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823251 - 04/25/13 10:10 PM

Wonderful experiment! Thanks for taking the time to do this. Very neat stuff.

Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823339 - 04/25/13 11:02 PM

Quote:

the ICNR frames look noisier than the BPM and DS calibrated frames, and the stack of 8 ICNR frames is noticeably noisier than the stack of 16 ICNR frames. So you definitely lose by shooting only half as many light frames as many on this thread predicted. However, based upon the discussion I would have expected the ICNRx16 frame to be comparable to the BPM-Bx16 and DSx16 stacks. But the ICNRx16 looks noticeably noisier to me than either the BPM or the DS calibration.




One thing I would like to point out that first couple of frames taken at the start of an imaging session will always be noisier than the rest. I have done extensive ICNR imaging, I always discard first, sometimes first and second images of the ICNR session. In case you did your experiment with ICNR being the first couple of images of the session, that would explain bit more noise in them.

If you can repeat the experiment and discard first couple of images of the session, that may fix the "start of an imaging session noise phenomena". I don't know what causes this but ambient temp miss-equilibrium is the best explanation I could come up with. Although I have seen this phenomena in ICNR sessions, I am pretty sure it is true for out-camera NR sessions as well. In short, discard first few images of a session, regardless of the NR method to rule out this phenomena as a variable in the results. Regards


Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823361 - 04/25/13 11:20 PM

Good point. The temperature of the sensor is changing the fastest during the first couple of images to the point where even the internal darks can struggle to keep up with it.

Neat stuff.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5823395 - 04/25/13 11:50 PM

Regarding the different colour balances between the ICNR and BP/DS images: You said you used Nebulosity to do the BadPixel map and Dark Subtraction. I am guessing that means you exported FITS or TIFFs for DSS to stack in that run while DSS had CR2 files directly for the ICNR runs.

This would I suspect explain the variance in colour balance (especially if Nebulosity did the demosaic/debayer instead of DSS).

If you use nebulosity's Batch Conversion to convert the ICNR images to FITS files and then stack those with DSS we will likely see the same colour balance as the BPM/DS images. Would be interesting to see if that changes the Mean values as well.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5823397 - 04/25/13 11:53 PM

Quote:

One thing I would like to point out that first couple of frames taken at the start of an imaging session will always be noisier than the rest. I have done extensive ICNR imaging, I always discard first, sometimes first and second images of the ICNR session.




Really? Given that sensor temperature tends to start low and go up over a couple frames before stabilizing I would expect those initial lower temperature frames to have *less* noise!

Do you perhaps tend to run a lengthy live-view focus session immediately before the imaging starts so that the camera starts out very warm?


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823401 - 04/25/13 11:55 PM

Quote:

Quote:

the ICNR frames look noisier than the BPM and DS calibrated frames, and the stack of 8 ICNR frames is noticeably noisier than the stack of 16 ICNR frames. So you definitely lose by shooting only half as many light frames as many on this thread predicted. However, based upon the discussion I would have expected the ICNRx16 frame to be comparable to the BPM-Bx16 and DSx16 stacks. But the ICNRx16 looks noticeably noisier to me than either the BPM or the DS calibration.




One thing I would like to point out that first couple of frames taken at the start of an imaging session will always be noisier than the rest. I have done extensive ICNR imaging, I always discard first, sometimes first and second images of the ICNR session. In case you did your experiment with ICNR being the first couple of images of the session, that would explain bit more noise in them.

If you can repeat the experiment and discard first couple of images of the session, that may fix the "start of an imaging session noise phenomena". I don't know what causes this but ambient temp miss-equilibrium is the best explanation I could come up with. Although I have seen this phenomena in ICNR sessions, I am pretty sure it is true for out-camera NR sessions as well. In short, discard first few images of a session, regardless of the NR method to rule out this phenomena as a variable in the results. Regards




Here are the ImageJ stats dropping the first 2 frames, so ICNRx14:

Mean StdDev Min Max
1774.147 589.470 108 65235

That is somewhat lower than ICNRx16:
2074.523 623.793 54 65239

but still quite a bit higher than the BPM and DS stacks.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5823407 - 04/25/13 11:58 PM

Hmm... thinking about that variance in Mean value between the ICNR and BPM/DS images... I wonder if Canon is doing a subtraction and then normalizing (adding the mean value of the dark frame to all pixels equally) to prevent the single-dark-subtract from decreasing image brightness/creating zero-value pixels.

Keith: Would you be able to run one of your dark frames and perhaps one of the Bias frames through ImageJ to get their Mean value? I would be quite interested to know if either turns out to have a mean of something close to 400...

Also - thanks for running that test for us all!


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5823415 - 04/26/13 12:03 AM

Quote:

also, can i ask what your pixel offsets and rotation angles are in DSS? in DSS is 'black point correction' turned off in the DCRAW control? actually i can't even remember if DSS shows you that DCRAW control.




Here are the pixel offset and rotation for the DS stack:

dx dY Angle
0.00 0.00 0.00 °
-5.04 3.36 -0.08 °
1.96 0.93 0.02 °
-2.27 0.75 -0.02 °
-2.98 1.10 -0.05 °
-3.79 2.08 -0.06 °
-8.80 2.63 -0.11°
-4.87 2.83 -0.07 °
-6.87 3.03 -0.10 °
1.20 0.85 0.01 °
-3.30 1.60 -0.04°
-6.75 2.56 -0.09°
-8.98 3.38 -0.12°
-3.16 1.18 -0.02°
-9.73 3.56 -0.13°
-9.91 3.39 -0.14°

...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5823418 - 04/26/13 12:06 AM

Quote:

Regarding the different colour balances between the ICNR and BP/DS images: You said you used Nebulosity to do the BadPixel map and Dark Subtraction. I am guessing that means you exported FITS or TIFFs for DSS to stack in that run while DSS had CR2 files directly for the ICNR runs.

This would I suspect explain the variance in colour balance (especially if Nebulosity did the demosaic/debayer instead of DSS).

If you use nebulosity's Batch Conversion to convert the ICNR images to FITS files and then stack those with DSS we will likely see the same colour balance as the BPM/DS images. Would be interesting to see if that changes the Mean values as well.




Good point, but I let DSS do all of the debayering. Still, you are correct, the ICNR files came into DSS directly as CR2 files and the BPM and DS files came in as FITS files from Nebulosity. I didn't test to see if that made a difference.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5823470 - 04/26/13 01:10 AM

Keith - thank you so much for taking the time to do this experiment and for doing such a thorough and thoughtful analysis.

@Mike - even if any given sub(s) is somewhat noisier than the norm, I'm not sure it generally makes sense to toss them. Keep in mind that those subs are going to give you an improvement in SNR by virtue of increasing your signal. The subs would have to be awfully noisy for the increase in signal to be more than offset by a greater increase in noise. Modern cameras such as your 60Da have very low noise anyway, so a less than perfect dark subtracted light is going to still net improve your SNR.


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5823493 - 04/26/13 01:43 AM

Quote:

Do you perhaps tend to run a lengthy live-view focus session immediately before the imaging starts so that the camera starts out very warm?




1. As most folks know, live-view focusing at times can take some time; it is possible as you point out could be causing session startup noise in first few images.

2. Other cause would be temp stabilization as you and I have surmised.

3. One other cause I could think of, a permutation of the previous, would be that generally my camera temp at the start of a session is warmer than the ambient before it cools down (i.e., ambient mostly cooler than the camera at the start of my session). Thx


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823514 - 04/26/13 02:14 AM

mike - that's like the worst of all worlds then, you only get 1/2 the sky time due to ICNR and then you have to throw away 2-3 subs?

from the analysis above, it's sounding like if you don't like making master darks then a better strategy is to turn off ICNR, make a single dark and use it to do CosmeticCorrection on your lights. then rely on dithering and the fact that you have 2x the frames now to boost your SNR.

in my synthetic tests i'm also having problems with the single-dark subtraction. my calibrated frames have lots of 0'd pixels even after adding an offset to the target dark frames. just like you are seeing, the statistics of the integrated stacks are different making the noise comparison impossible.

i wonder if instead of calibrating the frames traditionally if i took the absolute value of the difference between the target frame and it's dark if statistically that would be the same thing as adding the perfect offset so that no pixels clip...?

or... i think rather than trying to analyze darks directly, i could just emulate this experiment with old lights.

Edited by pfile (04/26/13 02:20 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5823537 - 04/26/13 03:01 AM

Quote:

that's like the worst of all worlds then, you only get 1/2 the sky time due to ICNR and then you have to throw away 2-3 subs?




I am big fan of ICNR but I think I'll give OCNR (out-camera NR) a try this summer (with master darks I mean); don't know if I'll be able to completely wean myself from ICNR though. Thx


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5823539 - 04/26/13 03:05 AM

Quote:

even if any given sub(s) is somewhat noisier than the norm, I'm not sure it generally makes sense to toss them. Keep in mind that those subs are going to give you an improvement in SNR by virtue of increasing your signal. The subs would have to be awfully noisy for the increase in signal to be more than offset by a greater increase in noise. Modern cameras such as your 60Da have very low noise anyway, so a less than perfect dark subtracted light is going to still net improve your SNR.




Good advice Scott, will try keeping them; generally tossed ones are not too bad, just not as good. Thx


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823583 - 04/26/13 04:35 AM

Quote:

I have done extensive ICNR imaging, I always discard first, sometimes first and second images of the ICNR session.




I see others have also picked up on the fact the you loose 1/2 your imaging time AND have to chuck the outriders out.

You realy sold ICNR for what it is - a lazy and inefficient option .

Na I'll stick to the trad method. In a country that has afforded me 0 (yeah ZERO) clear nights since 1st Jan this year I simply cannot afford the luxury of loosing 1/2 my imaging time. Especially when I have had 4 months of nights I can shoot darks to my hearts content at any practical temp I like.

Perfect matching temps of darks is not the overbearing issue that is being touted as that is the only apparent benefit of ICNR - a closer than actually needed temp match from ICNR.

Good master darks/dark libraries/dithering are always going to be superior and more efficient methods IMO.


Post Extras: Print Post   Remind Me!   Notify Moderator  
JMW
Pooh-Bah


Reged: 02/11/07

Loc: Nevada
Re: DSLR 'long exposure noise reduction' new [Re: Tonk]
      #5823647 - 04/26/13 06:16 AM

I am a bit curious about building up your dark libraries for your DSLR. How many darks do you consider sufficient for each time/temp pair? How large of range of times do your build up. For example time in seconds: 15, 30, 60, 90, 120 ,180 ,300. Do you do temp in 5 degree increments, C or F? Do you pick up the sensor temp from EXIF data or the ambient temp. How often do you redo your darks, every 3 months as the season change or longer. I have been doing in camera noise reduction because I am more impulsive about my DSLR imaging and I haven't figured out how to go about preparing my darks with time/temp variables.

With my CCD I just pick a temp for that season the I know my cooling can maintain and stick with that. I then only have to worry about master darks for the times I prefer. I usually do 30 seconds, 3, 5, 10, 15, 20 minutes. I redo them a couple of times a year.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: JMW]
      #5823710 - 04/26/13 07:36 AM

I don't set out to build a massive dark library for all occasions. I just aim to reuse what I already have if possible or go ahead and shoot darks for that session if I need to (which afterwards get added to the library). My ISO and exposure times hardly vary so its largely reuse of matched temperature. I live in a moderated climate so temp ranges are not wide either. Hot nights just don't exist (but neither do clear ones this year )

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Tonk]
      #5824217 - 04/26/13 11:46 AM

it may be an artifact of the 40D, or just certain canon cameras, but check out this thread:

http://pixinsight.com/forum/index.php?topic=5107.0

as you can see the horizontal pattern "noise" in the master bias is not really gone until 160 bias frames have been integrated.

if you are making master darks only, depending on your camera, you may need a *lot* of them to avoid injecting this pattern "noise" into your lights when you calibrate.

of course if you have a lot of dithered subs you may be able to reject the pixels comprising the pattern during integration. or, if you're not trying to dig down into the stack for faint details it might not matter.


Post Extras: Print Post   Remind Me!   Notify Moderator  
terry59
Post Laureate
*****

Reged: 07/18/11

Loc: Colorado, USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824322 - 04/26/13 12:42 PM

Quote:

it may be an artifact of the 40D, or just certain canon cameras, but check out this thread:

http://pixinsight.com/forum/index.php?topic=5107.0

as you can see the horizontal pattern "noise" in the master bias is not really gone until 160 bias frames have been integrated.

if you are making master darks only, depending on your camera, you may need a *lot* of them to avoid injecting this pattern "noise" into your lights when you calibrate.

of course if you have a lot of dithered subs you may be able to reject the pixels comprising the pattern during integration. or, if you're not trying to dig down into the stack for faint details it might not matter.




Rob - An Astrodon 50D?


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: terry59]
      #5824403 - 04/26/13 01:17 PM

Quote:



Rob - An Astrodon 50D?





meaning, a canon 50D with the IR cut removed and replaced with an Astrodon L filter. maybe it should be called a "hap griffin 50d" instead...


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824464 - 04/26/13 01:42 PM Attachment (4 downloads)

You may not want to throw out ICNR just yet. "Falcon-" made a great point: is it really comparable to import CR2 files into DSS for the ICNR frames but FITS files output by Nebulosity for the calibrated (BPM and DS) frames? So, I ran my CR2 files through the batch converter in Nebulosity to convert DSLR RAW files to FITS. I then stacked the FITS files using DSS and the same parameters as the previous stacks. Well, much to my surprise, it makes quite a difference. First, as Falcon- predicted, the colors came out the same as the other FITS stacks (do the CR2 files carry the color balance setting with them when processed in DSS?). Secondly, according to the ImageJ measurement, the standard deviation drops to the same range as the the other FITS stacks, BUT the means go even higher than they were when the stack is made from CR2 files.

Mean StdDev Min Max
4911.520 393.811 517 65163
4910.932 390.375 281 65172
1438.862 399.203 86 65267
1626.873 398.592 96 65267

I'm thinking that ImageJ is a rather crude way to try to measure the noise and I'm not sure how well it actually measures noise on a light frame when means vary as much as they have with the different methods. But from the image below it looks like the frames are less noisy than the stacked image derived from the CR2 files (it could be that if the CR2 stack is color-balanced to increase the red, that what we are seeing is noise from stretching the red channel more than the green and blue). But again, the stack of 8 frames is noticably noisier than the stack of 16 frames.

I did not expect that software can have such an impact. Since the stack made from the FITS files don't appear to be as noisy as the stack made from the CR2 files in DSS, I would have to conclude that some processing of the CR2 (DCRAW?) increases the noise in the frames. That would seem to put the ICNR frames in the same general noise range as the BPM and DS frames. However, I think the biggest strike against using ICNR is that you only collect half the frames that you would be able to otherwise by doing your calibration frame(s) separately. The other disadvantage of ICNR from my perspective is that you don't know what the Canon processing is doing. I suspect "pfile" may be correct and Canon "adjusts" things to make sure there are no black pixels. Clearly the means are always much higher in the ICNR frames, whether they are imported as CR2 or FITS files. As has been pointed out, this fits with Craig Stark's article which found that things just aren't linear when scaling frames that come out of Canon DSLRs. But what exactly Canon does is a bit of a mystery, which I personally find disconcerting.

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!

...Keith

Edited by microstar (04/26/13 01:46 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5824514 - 04/26/13 02:12 PM

well, you have to be careful with the DCRAW settings. DCRAW in DSS can be configured to take the white balance info from the CR2 file, or it can try to automatically determine the white balance... or it can just ignore the white balance. at the very least you should make sure the DCRAW settings in Neb and DSS are the same. of course calibrating and debayering everything in one program is better, so your 2nd experiment is probably more correct.

why not just stack the images in nebulosity as well? the FITS standard allows a whole lot of different but legal representations of pixel data. when fits files are generated by one application, other applications may misinterpret them...


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824553 - 04/26/13 02:31 PM

Hmm... interesting. *Visually* the CR2 sourced and FITS source stacks look to have similar noise levels yet the imagej StdDev values differ greatly. I wonder if we are seeing the impact of debayer/colour balance methods swamping the effect we are actually discussing.

Quote:

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!




Keith: Yes, this info is useful! I am curious though to try and tease out a bit more.... Would you be able to put your source CR2 files online? I would like to try looking at these in a per-colour-channel basis.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824564 - 04/26/13 02:38 PM

that's a good point, i was about to say that perhaps superpixel debayering should be used, but once those frames are registered there's pixel interpolation going on anyway. i don't think there's any way to avoid that...

Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824569 - 04/26/13 02:41 PM

Really neat stuff. I did all of my testing on my benchtop. If'n I recall right I found that the internal noise reduction became approximately equivalent to using separate lights and darks at about 8 light/dark pairs, that is 8 light/dark paris with noise reduction on versus 16 lights and 16 darks, the idea being that the darks would come from library and hence have no time penalty associated with them. Noise aside, the bigger issue for me was the elimination of hot pixels.

Thanks for posting the resuts of you analysis, this is really interesting stuff.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5824593 - 04/26/13 02:55 PM

i think the sky time drawback of ICNR is serious enough that if it's really just hot pixels you are concerned with, then a cosmetic correction procedure (bad pixel mapping) is the way to go.

short of that, dither the subs and reject the hot pixels during integration.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824616 - 04/26/13 03:04 PM

by the way, as far as i can tell, when not registering frames your assertion that average of differences and subtraction of the averaged dark is the same thing is true, empirically speaking. possibly due to oversubtracted pixels left in the "in camera" darks the statistical properties of the two results are very slightly different, but the noise in both results is very much the same.

however, once i 'dither' the single-dark calibrated darks and/or rotate them and then integrate them, the mean, median, and noise values are different. i'm having trouble normalizing the results, but i think that the fact that the results need normalization in the first place means that after registration the result is something different from a mathematical perspective.


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824637 - 04/26/13 03:12 PM

Quote:

Hmm... interesting. *Visually* the CR2 sourced and FITS source stacks look to have similar noise levels yet the imagej StdDev values differ greatly. I wonder if we are seeing the impact of debayer/colour balance methods swamping the effect we are actually discussing.

Quote:

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!




Keith: Yes, this info is useful! I am curious though to try and tease out a bit more.... Would you be able to put your source CR2 files online? I would like to try looking at these in a per-colour-channel basis.




That's what I was suggesting in my last post, I think in color balancing the CR2 files the red is being stretched to match the G & B channels, and that may be why the noise is higher.

I could put the source files in my Dropbox as zipped files containing ICNRon lights, ICNRoff lights, and Darks/Bias. Is that what you would like?
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5824727 - 04/26/13 03:45 PM

Yes - that would be perfect.

Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5825247 - 04/26/13 08:11 PM

Quote:

Yes - that would be perfect.




It took forever for these to upload -- they are big files (about 220MB each) and I won't keep them up long because they are using half of my Dropbox space. So if you want to download the raw files here they are:

https://dl.dropboxusercontent.com/u/59005817/ICNRon.zip
https://dl.dropboxusercontent.com/u/59005817/ICNRoff.zip
https://dl.dropboxusercontent.com/u/59005817/Dark.zip

...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
JMW
Pooh-Bah


Reged: 02/11/07

Loc: Nevada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5825295 - 04/26/13 08:50 PM

We haven't hit hot weather yet but living at 4800 feet altitude the temp drops throughout the night. The forecast tonight is for 73F at 8PM and 53 at 5AM. 40 degree changes are not untypical in the summer months. I have done in camera noise reduction so I don't to deal with the temp swings. Without regulated cooling I don't know how to manage darks.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: JMW]
      #5825310 - 04/26/13 08:55 PM

Jeff: all modern Canon DSLRs have an internal temperature sensor - that info is saved in the EXIF data and you can sort files based on temperature using that. Apps like BackyardEOS will even stick that info into the file name for even easier sorting. That is how I deal with/identify temperature swings.

Post Extras: Print Post   Remind Me!   Notify Moderator  
JMW
Pooh-Bah


Reged: 02/11/07

Loc: Nevada
Re: DSLR 'long exposure noise reduction' new [Re: JMW]
      #5825355 - 04/26/13 09:33 PM

I just downloaded exiftool and was amazed at the amount of EXIF data my photo programs don't display. Camera Temp in Celsius was in there. Recently purchased BackyardEOS but haven't had a chance to dive in yet.

So I guess the idea is pick a time interval, automatically shoot darks and classify the darks by temp as the night time temps dip each hour.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: JMW]
      #5825556 - 04/27/13 12:47 AM

if you have a cloudy night, or maybe a full moon night which has the same temperature profile as a good night, just put your camera outside and run darks. after a few nights like this you'll have a pretty good library.

Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5825899 - 04/27/13 09:01 AM

That's a good idea. I control all my gear from inside my house so on cloudy nights I'd often set my camera outside with its lens cover on and let it take darks. Pretty simple to do. It is just nice not having to fool with darks anymore. One less thing to fiddle with. I am starting to experiment with dithering and I am encouraged by the results. An easy way to dither short exposures (60 seconds or less, depeding on you mount and focal length) is to just disable guiding. With my LXD75 I called this natural dithering.

Have fun!


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5826115 - 04/27/13 11:39 AM

Quote:

well, you have to be careful with the DCRAW settings. DCRAW in DSS can be configured to take the white balance info from the CR2 file, or it can try to automatically determine the white balance... or it can just ignore the white balance.




Found the settings you were talking about in DSS. Neither the "Use Auto White Balance" or the Use Camera White Balance" checkboxes were checked so DSS shouldn't have been adjusting the color channels of the RAW frames. Also, the "Set the black point to zero" box was also unchecked. So I'm still not sure why DSS treats the CR2 files differently from the FITS files. Both CR2 and FITS files used the same AHD debayering method. Still looks like something that the Canon software is doing? Unless there are other DCRAW setting that DSS doesn't expose.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5826583 - 04/27/13 03:43 PM

well, it could be that nebulosity's DCRAW settings happen to be different. that's where you created the FITS files, right?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5841968 - 05/05/13 04:20 PM Attachment (6 downloads)

Ok - took me a while but I *did* take a look at the data.

Before anything else I found something that DOES effect the analysis up till now. The ICNR images had some transient drifting high cloud/haze!! I think this means the stacked results from the ICNR runs up till now have not been properly illustrative of the method's results. Thankfully though it turns out that 8 of the frames are clear of the clouds so we can still the comparisons just by picking those 8 best ICNR frames and reducing the number of the no-ICNR frames to match.

So here is what I did:

- Did all calibration and stacking in Nebulosity for consistency

- did a uniform batch-crop on all frames after align to entirely remove alignment artefacts on the edges prior to stacking

- Used "pixel binning" demosaic (aka superpixel debayer) so that no colour data is mixed in from neighbouring pixels (or at least this is true for Blue and Red pixels, for Green channel each pixel is combined data for two actual sensor sites)

- Used pure Average Stack method as we want to test the math/data here rather then test various stacking methods

- Also produced one stack with Standard Deviation 1.5 method to compair against more likely real world stacking methods (this not really relavent to the question at hand though)

- Used PixInsight's NoiseEvaluation script on each stacked file to pull our stats *before* any modification what so ever.

- Extracted the colour channels into separate files to produce forum-post images.

- Used PixInsight's HistrogramTransformation process to align the histogram peak to exactly the same place for each image by adjusting the black point only (no pixels clipped). After that applied an identical mid-point transform to each image. I think this should have given us an apples to apples comparison when visually appraising noise levels.


Stacks I created/evaluated:

Stack A) 8 exposures, Long Exposure Noise Reduction active, no darks, Average stacking

Stack B) 8 exposures, Long Exposure Noise Reduction off, 16 darks used, Average stacking

Stack C) 16 exposures, Long Exposure Noise Reduction off, 16 darks used, Average stacking

Stack D) 16 exposures, Long Exposure Noise Reduction off, 16 darks used, Standard Deviation 1.5 pixel rejection stacking


So before getting into my interpretation of the data, here is the raw data data itself as well as the center 200 pixels of the Blue channel of each frame. The data is a direct copy from PixInsight's ProcessConsole. The first value is the noise evaluation, the second two values are information about how the MRS noise evaluation method was working and can likely be ignored for our needs.

ICNRon_8x300s_pureavg
R = 4.510e-04, N = 1755244 (59.45%), J = 4
G = 3.920e-04, N = 1308019 (44.30%), J = 4
B = 6.509e-04, N = 1577837 (53.44%), J = 4


ICNRoff_8x300s_pureavg
R = 3.940e-04, N = 1483531 (52.26%), J = 4
G = 3.539e-04, N = 1111487 (39.15%), J = 4
B = 5.815e-04, N = 1357521 (47.82%), J = 4


ICNRoff_16x300s_pureavg
R = 2.879e-04, N = 1388363 (47.34%), J = 4
G = 2.526e-04, N = 972472 (33.16%), J = 4
B = 4.157e-04, N = 1235743 (42.14%), J = 4


ICNRoff_16x300s_StDev1_5
R = 3.338e-04, N = 1536284 (52.38%), J = 4
G = 3.486e-04, N = 1244584 (42.44%), J = 4
B = 5.537e-04, N = 1502592 (51.23%), J = 4


(edited to indicate number of Darks used for each stack)

Edited by Falcon- (05/05/13 04:35 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5841991 - 05/05/13 04:33 PM Attachment (4 downloads)

I was going to say that the first thing to consider is a test of the method on an even-footing basis as regards data quantity. Unfortunately I now realize that i neglected to reduce my number of darks from 16 to 8 when I reduced the number of frames to 8.



So alas this is not a true method-only test.

However I feel it is still worth doing this comparison given that one of the differences between ICNR and out of camera dark calibration is that you can use a different number of darks vs lights.

Stack A vs Stack B

aRed = 4.510e-04
bRed = 3.940e-04

aGreen = 3.920e-04
bGreen = 3.539e-04

aBlue = 6.509e-04
bBlue = 5.815e-04


So the out-of-camera darks frame is giving consistently lower noise. Alas it does not mean much other then "more darks are good!" I will try and re-do the A and B stacks but with equal numbers of darks soon(ish) so we can do a value noise level comparison.

One interesting thing to note is that the stack A seems to have done a poor job correcting for the dead pixel in the top-left quadrant of the image. Dithering and stddev/sigma/k-sigma stacking methods would help here but I do find it interesting to see that dead pixel is much less prominent in stack B.


Edited by Falcon- (05/05/13 04:39 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842012 - 05/05/13 04:49 PM Attachment (5 downloads)

Stack A vs Stack B

This is the comparison that brings into focus one of the main arguments against in camera darks: You loose 1/2 time you could have been collecting photons.

aRed = 4.510e-04
cRed = 2.879e-04

aGreen = 3.920e-04
cGreen = 2.526e-04

aBlue = 6.509e-04
cBlue = 4.157e-04

Here there is no contest. Both statistically and visually the stack C wins by a large margin.

Once again we see that the cold pixel is less visually prominent in the dark-subtracted image.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842031 - 05/05/13 04:59 PM Attachment (5 downloads)

Stack C vs Stack D

As I indicated before, this is not *really* part of the topic, but I was curious and thought others might be as well.

For Stack D I used StdDev 1.5 for both creation of the MasterDark and for stacking of the lights themselves. Likely this was too strong of a factor and I am sure I could get some cleaner results paying with some variations (especially if I used PixInsight rather then nebulosity) but this will do for now.

cRed = 2.879e-04
dRed = 3.338e-04

cGreen = 2.526e-04
dGreen = 3.486e-04

cBlue = 4.157e-04
dBlue = 5.537e-04


Noise has gone back up - still better then both Stack A and Stack B and sure enough the background looks slightly crunchier.

That said though, the stars do seem slightly shaper and the cold pixel is MUCH reduced. Stack D *MAY* be a slightly more visually appealing image....

The conclusion I draw from this part of the exercise is that it is indeed important to be careful in selection of pixel rejection criteria. Better options may have helped to keep the extra sharpness and defect correction while not introducing quite as much noise.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842037 - 05/05/13 05:05 PM

What next?

Unfortunately due to my mistake while processing the 8-frame external-darks stack we have not addressed one of the key points of the pro-ICNR argument - that the noise removal is equally effective with equal numbers of ICNR frames vs external darks.

I also got to wondering... the 16 frame stack OBVIOUSLY had a better S/N ratio - but it got me curious. What if you had ICNR off and then where unable to take darks at all? As such in addition to the redo of stack B i am also going to do a stack of all 16 no-ICNR frames *without* applying the darks.


Thanks again for sharing the test data Keith!


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842212 - 05/05/13 06:36 PM Attachment (4 downloads)

Ok - Try #2!

Calibration/stacking/post-processing all done as before. Three stacks produced:

Stack X) 8 exposures, Long Exposure Noise Reduction on, no darks, Average stacking (same as previous Stack A)
Stack Y) 8 exposures, Long Exposure Noise Reduction off, 8 darks, Average stacking
Stack Z) 16 exposures, Long Exposure Noise Reduction off, no darks, Average stacking

Stats (per channel):

xR = 4.512e-04
yR = 4.230e-04
zR = 2.820e-04

xG = 3.924e-04
yG = 3.763e-04
zG = 2.467e-04

xB = 6.503e-04
yB = 6.191e-04
zB = 4.024e-04

Note that Stack X has essentially identical results to Stack A. This is exactly as it should be and a good sanity check against this run vs my previous run.

Stack X vs Stack Y:

Here we have the theory test case at last. Interestingly the out-of-camera darks *DOES* produce a slightly cleaner image. It is a fairly subtle difference visually but there now we do have numbers. With this dataset and this camera Long Exposure Noise Reduction (in-camera darks) does not produce an identical result to an equal number of darks taken separately. It is close to the same result though so as my previous Stack A vs Stack C comparison shows the shorter time under the sky is the dominant factor.

Stack X vs Stack Z vs Stack C:

This was mostly for my own curiosity. Partially the hot blue pixel here is totally uncorrected and any large scale pattern noise (banding, amp glow, etc) will *not* be corrected for. The noise levels in Stack Z and C are surprisingly close - this would seem to indicate to me that the 16-frame dark subtraction did a good job of removing pattern noise(unwanted signal) without introducing much additional random noise. That leaves the X vs Z image. Again, quite clearly the 16frame stack beats out the ICNR stack easily on a S/N basis. The question of if a Bias or bad-pixel-map image would be sufficient instead darks (be it in-camera or external) would likely be a per-camera-model question and not really have a generalized answer.

That "mean value" question from earlier:

It was noted earlier in the thread that the ICNR images seemed to have a higher mean value. I can confirm that Stack Z (where darks where not subtracted) had a similar mean to the ICNR stack, while Stack Y had the same lower-mean noted previously. From this I conclude that Canon's Long Exposure Noise Reduction calibration process is likely two step:

1) Subtrack dark from image
2) Add mean value entire dark frame to each pixel of the image to restore pixel brightness values

In other words the differing means is entirely a cosmetic difference that does not effect the image data.

My Conclusion:

I will continue to run without ICNR for my imaging sessions (but will keep ICNR in the back mind should I come across an odd edge case where darks are somehow otherwise impossible).


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842254 - 05/05/13 06:59 PM

Fine work Sir.

Confirms the basic theory nicely. A single dark leaves more random noise in the calibrated image than a master dark made from a much larger number of darks.

The other down side of ICNR is you can't capture the darks to build a dark library. Once captured a dark frame is potentially reuseable. With ICNR - poof! gone.

I will continue to use my 50+ frame master darks . I now know its totally worth it. Thanks


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: DSLR 'long exposure noise reduction' new [Re: Tonk]
      #5842656 - 05/05/13 11:13 PM

Thank you for all your efforts and thorough analysis, Sean. Very well done.

Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5842711 - 05/06/13 12:03 AM

Quote:

Thanks again for sharing the test data Keith!




Thank YOU for doing such a thorough analysis and for cleaning up my initial muddled attempt. Nice job! Goes to show what a little group collaboration can achieve. An excellent reference for anyone interested in this question.

...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
jsines
sage
*****

Reged: 09/06/11

Loc: Berkley. Michigan
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5843841 - 05/06/13 03:40 PM

Nice analysis, a lot of work involved, but a question - since (I assume) most people stack with lights, darks, bias, and flats, why didn't you include bias and flat frames?

The bias signal is subtracted 3 times during calibration (light, dark, flat). However, those who do ICNR are not able to subtract the bias signal from the ICNR dark, and are only subtracting the bias signal 2 times.

I suspect you'll find an even larger difference in SNR if you use bias and flats, but that's an even larger experiment.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: jsines]
      #5844041 - 05/06/13 05:23 PM

jsines: the main reason I used only darks as this was *specifically* a test of the effectiveness of the Canon firmware-built-in Long Exposure Noise Reduction.

There is another reason though. You talk about the Bias being subtracted 3 times but in reality the bias is only subtracted from the Lights ONCE and any other use of Bias frames is to make sure nothing re-adds the bias back in. Dark frames do contain the same Bias signal as separate Bias frames so doing dark subtraction does the job of removing the Bias signal from the Lights - you do not *need* to have separate bias frames.

There is of course an exception. Some programs (PixInsight for example) can do Dark Scaling - this attempts to automatically adjust the master dark to best match the dark-signal in the light for best possible correction. This is great for solving problems temperature variations BUT since Bias signal does not scale with temperature of exposure time you need to first remove the Bias from the Dark before doing the Dark Scaling. At this point you need to also separately subtract the bias from the light.

Flats also need the bias removed from them to avoid reintroducing it into the Lights, Flat-Darks *or* Bias frames do get used there. In this case we are not testing anything related to flats so there is no need to complicate the task at hand by using them.


So as relates to the actual original intention of the thread the lack of Bias frames is not a weakness of the ICNR method as that Bias data is included in the in-camera darks.

Quote:

I suspect you'll find an even larger difference in SNR if you use bias and flats, but that's an even larger experiment




Actually use of flats would slightly decrease the S/N ratio as unless you are using a stupendously large number of Flats and Flat-Darks the flats will introduce a small amount of random noise. However the correction of vignetting and dust-shadows is VERY much worth the very small reduction in S/N ratio. Perhaps one could say you trade a bit of statistical S/N for a lot of *useable* S.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5844568 - 05/06/13 10:25 PM

thanks for doing all that work. for my part i failed miserably to normalize my experiments on darks alone.

Quote:

With this dataset and this camera Long Exposure Noise Reduction (in-camera darks) does not produce an identical result to an equal number of darks taken separately. It is close to the same result though so as my previous Stack A vs Stack C comparison shows the shorter time under the sky is the dominant factor.




unfortunately i think it's still impossible to prove or disprove the original assertion that sum of in-camera dark subtracted lights == sum of master-subtracted lights, only because you just can't use the same dataset for both stacks in the experiment. if the camera also handed you the dark it used to subtract the light frame we might be able to do something with that. the closest you can come is to make lights interleaved with darks with ICNR turned off and then manually do the calibration both ways. but this dataset was not constructed that way (right?)

i still think it's just intuitively false though.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5844608 - 05/06/13 10:43 PM

pfile: notice I was a careful in my wording - the whole "this dataset" thing, etc.

So ya, truly conclusive results are not here, but I still take these results as a strong indicator.

Someone COULD set up a test rig, perhaps indoors in a very dark room so conditions can be controlled (ambient temp, no transient clouds, etc), to capture multiple data sets and REALLY prove it scientifically.... but I am not *THAT* motivated


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5844802 - 05/07/13 12:35 AM

Quote:

the closest you can come is to make lights interleaved with darks with ICNR turned off and then manually do the calibration both ways. but this dataset was not constructed that way (right?)




Nope, but you also wouldn't get Canon's in-camera processing that way. This may not be conclusive but it's close enough for people to make their own decision on ICNR.

I think I'll continue to use bad pixel mapping as it should be at least as good as your stack without darks which turned out pretty good. I'll also continue to dither and use Std Dev based stacking (but based on this I'll probably set my rejection higher). As was mentioned very early on in the thread, there are many ways to skin the cat.
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5844820 - 05/07/13 01:24 AM

Ya, as we can see from that conveniently central hot pixel in the test data *all* the methods would have benefited from Dither. As for the sigma value, that is dependent on the number of frames used, 1.5 was perhaps too tight here, but if it was 32 or 64 frames it would probably have been about right.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)


Extra information
11 registered and 15 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 2913

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics