Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
· Get a Cloudy Nights T-Shirt · Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu… uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823514 - 04/26/13 02:14 AM

mike - that's like the worst of all worlds then, you only get 1/2 the sky time due to ICNR and then you have to throw away 2-3 subs?

from the analysis above, it's sounding like if you don't like making master darks then a better strategy is to turn off ICNR, make a single dark and use it to do CosmeticCorrection on your lights. then rely on dithering and the fact that you have 2x the frames now to boost your SNR.

in my synthetic tests i'm also having problems with the single-dark subtraction. my calibrated frames have lots of 0'd pixels even after adding an offset to the target dark frames. just like you are seeing, the statistics of the integrated stacks are different making the noise comparison impossible.

i wonder if instead of calibrating the frames traditionally if i took the absolute value of the difference between the target frame and it's dark if statistically that would be the same thing as adding the perfect offset so that no pixels clip...?

or... i think rather than trying to analyze darks directly, i could just emulate this experiment with old lights.

Edited by pfile (04/26/13 02:20 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5823537 - 04/26/13 03:01 AM

Quote:

that's like the worst of all worlds then, you only get 1/2 the sky time due to ICNR and then you have to throw away 2-3 subs?




I am big fan of ICNR but I think I'll give OCNR (out-camera NR) a try this summer (with master darks I mean); don't know if I'll be able to completely wean myself from ICNR though. Thx


Post Extras: Print Post   Remind Me!   Notify Moderator  
mmalik
Post Laureate
*****

Reged: 01/13/12

Loc: USA
Re: DSLR 'long exposure noise reduction' new [Re: srosenfraz]
      #5823539 - 04/26/13 03:05 AM

Quote:

even if any given sub(s) is somewhat noisier than the norm, I'm not sure it generally makes sense to toss them. Keep in mind that those subs are going to give you an improvement in SNR by virtue of increasing your signal. The subs would have to be awfully noisy for the increase in signal to be more than offset by a greater increase in noise. Modern cameras such as your 60Da have very low noise anyway, so a less than perfect dark subtracted light is going to still net improve your SNR.




Good advice Scott, will try keeping them; generally tossed ones are not too bad, just not as good. Thx


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: mmalik]
      #5823583 - 04/26/13 04:35 AM

Quote:

I have done extensive ICNR imaging, I always discard first, sometimes first and second images of the ICNR session.




I see others have also picked up on the fact the you loose 1/2 your imaging time AND have to chuck the outriders out.

You realy sold ICNR for what it is - a lazy and inefficient option .

Na I'll stick to the trad method. In a country that has afforded me 0 (yeah ZERO) clear nights since 1st Jan this year I simply cannot afford the luxury of loosing 1/2 my imaging time. Especially when I have had 4 months of nights I can shoot darks to my hearts content at any practical temp I like.

Perfect matching temps of darks is not the overbearing issue that is being touted as that is the only apparent benefit of ICNR - a closer than actually needed temp match from ICNR.

Good master darks/dark libraries/dithering are always going to be superior and more efficient methods IMO.


Post Extras: Print Post   Remind Me!   Notify Moderator  
JMW
Pooh-Bah


Reged: 02/11/07

Loc: Nevada
Re: DSLR 'long exposure noise reduction' new [Re: Tonk]
      #5823647 - 04/26/13 06:16 AM

I am a bit curious about building up your dark libraries for your DSLR. How many darks do you consider sufficient for each time/temp pair? How large of range of times do your build up. For example time in seconds: 15, 30, 60, 90, 120 ,180 ,300. Do you do temp in 5 degree increments, C or F? Do you pick up the sensor temp from EXIF data or the ambient temp. How often do you redo your darks, every 3 months as the season change or longer. I have been doing in camera noise reduction because I am more impulsive about my DSLR imaging and I haven't figured out how to go about preparing my darks with time/temp variables.

With my CCD I just pick a temp for that season the I know my cooling can maintain and stick with that. I then only have to worry about master darks for the times I prefer. I usually do 30 seconds, 3, 5, 10, 15, 20 minutes. I redo them a couple of times a year.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tonk
Postmaster
*****

Reged: 08/19/04

Loc: Leeds, UK, 54N
Re: DSLR 'long exposure noise reduction' new [Re: JMW]
      #5823710 - 04/26/13 07:36 AM

I don't set out to build a massive dark library for all occasions. I just aim to reuse what I already have if possible or go ahead and shoot darks for that session if I need to (which afterwards get added to the library). My ISO and exposure times hardly vary so its largely reuse of matched temperature. I live in a moderated climate so temp ranges are not wide either. Hot nights just don't exist (but neither do clear ones this year )

Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Tonk]
      #5824217 - 04/26/13 11:46 AM

it may be an artifact of the 40D, or just certain canon cameras, but check out this thread:

http://pixinsight.com/forum/index.php?topic=5107.0

as you can see the horizontal pattern "noise" in the master bias is not really gone until 160 bias frames have been integrated.

if you are making master darks only, depending on your camera, you may need a *lot* of them to avoid injecting this pattern "noise" into your lights when you calibrate.

of course if you have a lot of dithered subs you may be able to reject the pixels comprising the pattern during integration. or, if you're not trying to dig down into the stack for faint details it might not matter.


Post Extras: Print Post   Remind Me!   Notify Moderator  
terry59
Post Laureate
*****

Reged: 07/18/11

Loc: Colorado, USA
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824322 - 04/26/13 12:42 PM

Quote:

it may be an artifact of the 40D, or just certain canon cameras, but check out this thread:

http://pixinsight.com/forum/index.php?topic=5107.0

as you can see the horizontal pattern "noise" in the master bias is not really gone until 160 bias frames have been integrated.

if you are making master darks only, depending on your camera, you may need a *lot* of them to avoid injecting this pattern "noise" into your lights when you calibrate.

of course if you have a lot of dithered subs you may be able to reject the pixels comprising the pattern during integration. or, if you're not trying to dig down into the stack for faint details it might not matter.




Rob - An Astrodon 50D?


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: terry59]
      #5824403 - 04/26/13 01:17 PM

Quote:



Rob - An Astrodon 50D?





meaning, a canon 50D with the IR cut removed and replaced with an Astrodon L filter. maybe it should be called a "hap griffin 50d" instead...


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824464 - 04/26/13 01:42 PM Attachment (4 downloads)

You may not want to throw out ICNR just yet. "Falcon-" made a great point: is it really comparable to import CR2 files into DSS for the ICNR frames but FITS files output by Nebulosity for the calibrated (BPM and DS) frames? So, I ran my CR2 files through the batch converter in Nebulosity to convert DSLR RAW files to FITS. I then stacked the FITS files using DSS and the same parameters as the previous stacks. Well, much to my surprise, it makes quite a difference. First, as Falcon- predicted, the colors came out the same as the other FITS stacks (do the CR2 files carry the color balance setting with them when processed in DSS?). Secondly, according to the ImageJ measurement, the standard deviation drops to the same range as the the other FITS stacks, BUT the means go even higher than they were when the stack is made from CR2 files.

Mean StdDev Min Max
4911.520 393.811 517 65163
4910.932 390.375 281 65172
1438.862 399.203 86 65267
1626.873 398.592 96 65267

I'm thinking that ImageJ is a rather crude way to try to measure the noise and I'm not sure how well it actually measures noise on a light frame when means vary as much as they have with the different methods. But from the image below it looks like the frames are less noisy than the stacked image derived from the CR2 files (it could be that if the CR2 stack is color-balanced to increase the red, that what we are seeing is noise from stretching the red channel more than the green and blue). But again, the stack of 8 frames is noticably noisier than the stack of 16 frames.

I did not expect that software can have such an impact. Since the stack made from the FITS files don't appear to be as noisy as the stack made from the CR2 files in DSS, I would have to conclude that some processing of the CR2 (DCRAW?) increases the noise in the frames. That would seem to put the ICNR frames in the same general noise range as the BPM and DS frames. However, I think the biggest strike against using ICNR is that you only collect half the frames that you would be able to otherwise by doing your calibration frame(s) separately. The other disadvantage of ICNR from my perspective is that you don't know what the Canon processing is doing. I suspect "pfile" may be correct and Canon "adjusts" things to make sure there are no black pixels. Clearly the means are always much higher in the ICNR frames, whether they are imported as CR2 or FITS files. As has been pointed out, this fits with Craig Stark's article which found that things just aren't linear when scaling frames that come out of Canon DSLRs. But what exactly Canon does is a bit of a mystery, which I personally find disconcerting.

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!

...Keith

Edited by microstar (04/26/13 01:46 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5824514 - 04/26/13 02:12 PM

well, you have to be careful with the DCRAW settings. DCRAW in DSS can be configured to take the white balance info from the CR2 file, or it can try to automatically determine the white balance... or it can just ignore the white balance. at the very least you should make sure the DCRAW settings in Neb and DSS are the same. of course calibrating and debayering everything in one program is better, so your 2nd experiment is probably more correct.

why not just stack the images in nebulosity as well? the FITS standard allows a whole lot of different but legal representations of pixel data. when fits files are generated by one application, other applications may misinterpret them...


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824553 - 04/26/13 02:31 PM

Hmm... interesting. *Visually* the CR2 sourced and FITS source stacks look to have similar noise levels yet the imagej StdDev values differ greatly. I wonder if we are seeing the impact of debayer/colour balance methods swamping the effect we are actually discussing.

Quote:

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!




Keith: Yes, this info is useful! I am curious though to try and tease out a bit more.... Would you be able to put your source CR2 files online? I would like to try looking at these in a per-colour-channel basis.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824564 - 04/26/13 02:38 PM

that's a good point, i was about to say that perhaps superpixel debayering should be used, but once those frames are registered there's pixel interpolation going on anyway. i don't think there's any way to avoid that...

Post Extras: Print Post   Remind Me!   Notify Moderator  
jgraham
Postmaster
*****

Reged: 12/02/04

Loc: Miami Valley Astronomical Soci...
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824569 - 04/26/13 02:41 PM

Really neat stuff. I did all of my testing on my benchtop. If'n I recall right I found that the internal noise reduction became approximately equivalent to using separate lights and darks at about 8 light/dark pairs, that is 8 light/dark paris with noise reduction on versus 16 lights and 16 darks, the idea being that the darks would come from library and hence have no time penalty associated with them. Noise aside, the bigger issue for me was the elimination of hot pixels.

Thanks for posting the resuts of you analysis, this is really interesting stuff.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: jgraham]
      #5824593 - 04/26/13 02:55 PM

i think the sky time drawback of ICNR is serious enough that if it's really just hot pixels you are concerned with, then a cosmetic correction procedure (bad pixel mapping) is the way to go.

short of that, dither the subs and reject the hot pixels during integration.


Post Extras: Print Post   Remind Me!   Notify Moderator  
pfile
Post Laureate


Reged: 06/14/09

Re: DSLR 'long exposure noise reduction' new [Re: pfile]
      #5824616 - 04/26/13 03:04 PM

by the way, as far as i can tell, when not registering frames your assertion that average of differences and subtraction of the averaged dark is the same thing is true, empirically speaking. possibly due to oversubtracted pixels left in the "in camera" darks the statistical properties of the two results are very slightly different, but the noise in both results is very much the same.

however, once i 'dither' the single-dark calibrated darks and/or rotate them and then integrate them, the mean, median, and noise values are different. i'm having trouble normalizing the results, but i think that the fact that the results need normalization in the first place means that after registration the result is something different from a mathematical perspective.


Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5824637 - 04/26/13 03:12 PM

Quote:

Hmm... interesting. *Visually* the CR2 sourced and FITS source stacks look to have similar noise levels yet the imagej StdDev values differ greatly. I wonder if we are seeing the impact of debayer/colour balance methods swamping the effect we are actually discussing.

Quote:

I think I've gone about as far as I can with this experiment. I'll let others decide whether the information is useful to them!




Keith: Yes, this info is useful! I am curious though to try and tease out a bit more.... Would you be able to put your source CR2 files online? I would like to try looking at these in a per-colour-channel basis.




That's what I was suggesting in my last post, I think in color balancing the CR2 files the red is being stretched to match the G & B channels, and that may be why the noise is higher.

I could put the source files in my Dropbox as zipped files containing ICNRon lights, ICNRoff lights, and Darks/Bias. Is that what you would like?
...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
Falcon-
Post Laureate
*****

Reged: 09/11/09

Loc: Gambier Island, BC, Canada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5824727 - 04/26/13 03:45 PM

Yes - that would be perfect.

Post Extras: Print Post   Remind Me!   Notify Moderator  
microstar
Pooh-Bah


Reged: 01/05/08

Loc: Canada
Re: DSLR 'long exposure noise reduction' new [Re: Falcon-]
      #5825247 - 04/26/13 08:11 PM

Quote:

Yes - that would be perfect.




It took forever for these to upload -- they are big files (about 220MB each) and I won't keep them up long because they are using half of my Dropbox space. So if you want to download the raw files here they are:

https://dl.dropboxusercontent.com/u/59005817/ICNRon.zip
https://dl.dropboxusercontent.com/u/59005817/ICNRoff.zip
https://dl.dropboxusercontent.com/u/59005817/Dark.zip

...Keith


Post Extras: Print Post   Remind Me!   Notify Moderator  
JMW
Pooh-Bah


Reged: 02/11/07

Loc: Nevada
Re: DSLR 'long exposure noise reduction' new [Re: microstar]
      #5825295 - 04/26/13 08:50 PM

We haven't hit hot weather yet but living at 4800 feet altitude the temp drops throughout the night. The forecast tonight is for 73F at 8PM and 53 at 5AM. 40 degree changes are not untypical in the summer months. I have done in camera noise reduction so I don't to deal with the temp swings. Without regulated cooling I don't know how to manage darks.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | (show all)


Extra information
3 registered and 18 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 2912

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics