Return to the Cloudy Nights Telescope Reviews home page
   · Get a Cloudy Nights T-Shirt · Submit a Review / Article   

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu.... uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | (show all)
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Creating a "dark" library
      #3279789 - 08/18/09 01:42 PM

After reading a few books, forum posts, blogs and web sites, it looks like creating a "dark" library is something that would save me a lot of prime shooting hours at night.

There are some that say that you can shoot them at any time as long as the temp is the same, but others say that it should be done as close to the "light" image as possible so you can get the "internal thermal current" of the camera to match.

What I am thinking of doing (and should have done already) is to start taking dark exposures outside at night (or in the garage, or even in the house). My plan is to take about 30 darks for each ISO and 30 second increment, for each 5 deg of temperature (I'll start with the most common ISO and exposure times).

Does this sound about right, or is it over aggressive for darks?

Also, is there any way to embed the temp in the .RAW image that applications can read? I know my camera does not have that function, but perhaps some 3rd party app does it for me.

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Mike Clemens
Post Laureate


Reged: 11/26/05
Posts: 4670
Loc: Wasilla, Alaska 61N
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3279875 - 08/18/09 02:20 PM

I've always got best results by shooting afterwards. I'm mistrustful of dark libs on unregulated cams.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Adam E
sage
*****

Reged: 06/29/09
Posts: 363
Loc: Orange Park, FL
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3279921 - 08/18/09 02:40 PM

The kind of library you're talking about may become outdated as your CMOS sensor ages. I think that temperature is important to think about here, but the individual characteristics of your CMOS sensor are just as important. Thise characteristics will, in all likelyhood, change with age. It would suck to put together a dark collection only to find out that it doesn't work so well in a year or two.

--------------------
Astro-Tech AT106, Vixen ED80sf, Orion ST80 (Guidescope), Sky Watcher 8" Dobsonian, Celestron CGEM, Canon 350D (Modified), Celestron NexImage, Orion StarShoot AutoGuider, Astro-Tech Field Flattener, IDAS LPS-P2


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: Adam E]
      #3279944 - 08/18/09 02:50 PM

A year or two seems like a good length of time.. and the darks can always be rotated out over time. However, I did not think of the CMOS sensor aging.

Another thing that I thought of as I was trying out some sample shots. During the corse of 15 min, the temp changed by [+/-] 2 deg. That got me to think. Would it not be best to take a series of darks "during" the lights. What I mean, take 5 lights, 1 dark, 5 lights 1 dark .. That way the darks had the same range of temp change as the lights. I know that cameras do not have a way to take a dark frame without the lens cap, so it would have to be manual.. Just an idea.

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08
Posts: 887
Loc: Voorhees, NJ
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3280019 - 08/18/09 03:27 PM

Hi Roy,

If you are shooting a series of exposures, the internal camera temperature will continue to increase until it stabilizes. My camera taken 2-3 hours for this to happen. Other cameras may be different, and it probably also depends on ambient temperature conditions, like if it is dropping a lot.

So, the bottom line is, if you are going to shoot an hour or two worth of lights, the internal temp in the camera is not going to be the same for each frame.

Likewise for the darks, unless you spend a couple of hours pre-warming up the camera. But this is a moot point anyway because the thermal signal is going to be different in each light frame anyway.

So, what to do?

Well, the answer is simple... create a dark frame library, as well as a bias frame library. Then, and here is the key part, use a program like Images Plus with automatic dark frame matching in the calibration step. It will scale the thermal signal in the master dark to match each individual light frame.

It works like magic. Other programs may do this as well, but I am not familiar enough with them to know for sure. I kind of stick with what works for me once I figure out what that is, and Images Plus works perfectly.

For my dark frame library, I create a master dark for every 10 degrees Farenheit. That way, any lights I shoot will be within 5 degrees, and this works just fine in Images Plus.

Using bias frames will allow you to scale your darks. This means if you shoot 10 minute darks, you can use them to calibrate shorter exposures taken at the same nominal temperature and the same ISO. This saves a lot of work. You certainly don't need one for every 30 second increment.

I standardize on ISO 1600 for deeps-sky objects.

I created my dark frame library when I first got my 20Da in 2005, and four years later, I'm still using the same master darks with excellent results, so I'm not finding much if any change in the CMOS sensor over time.

Jerry

--------------------
A Beginner's Guide to DSLR Astrophotography
http://www.astropix.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
David Rosenthal
Carpal Tunnel
*****

Reged: 06/23/06
Posts: 1776
Loc: NJ
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280044 - 08/18/09 03:40 PM

I have done the same thing as Jerry except I use Maxim DL and it does the same thing as ImagesPlus. Sacles or Optomizes the Darks, using Bias, to match the lights. Works like a champ. My Dark Library is coming up on 2 years old and I have not noticed any decrease in effictiveness.

--------------------
CS, David Rosenthal | Northern NJ


My Image Gallery
http://www.EastCoastAstro.com

Atlas EQ-G EQMOD USBDIR [Self-Hypertuned]
8" SCT | AT8IN-SC | Borg 76ED | Takahashi FS-60C
Self Modded 400D | DSI Pro | SPC900NC


Post Extras: Print Post   Remind Me!   Notify Moderator  
David Rosenthal
Carpal Tunnel
*****

Reged: 06/23/06
Posts: 1776
Loc: NJ
Re: Creating a "dark" library new [Re: David Rosenthal]
      #3280046 - 08/18/09 03:41 PM

One other thing, if you store your Darks in a raw format called FITS, you can store the ISO, temperature and exposure length in the image header and software like Maxim DL can use this info.

--------------------
CS, David Rosenthal | Northern NJ


My Image Gallery
http://www.EastCoastAstro.com

Atlas EQ-G EQMOD USBDIR [Self-Hypertuned]
8" SCT | AT8IN-SC | Borg 76ED | Takahashi FS-60C
Self Modded 400D | DSI Pro | SPC900NC


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280098 - 08/18/09 04:12 PM

Jerry,

I have you book (err..CD).. And that is one thing that made me want to create my own dark frame library. However i am still confused with the whole bias/dark/flat thing, so have just been sticking with darks for now.

What would be nice is if some would write a "dark frames for dummys" step by step guide .. (hint, hint)..

I'll try to do as you suggest, but I am sure I will get a lot of it wrong .. then again, all I am wasting is cloudy days and nights.. so not anything important.

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: David Rosenthal]
      #3280110 - 08/18/09 04:16 PM

Quote:

One other thing, if you store your Darks in a raw format called FITS, you can store the ISO, temperature and exposure length in the image header and software like Maxim DL can use this info.




Right now I am just using the Canon software to take pictures (MaxIM DL still does not support the 500D) and everything is in canons RAW format. But if I did use MaxIM, how do you get the temp information into the header?

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pierre Stromberg
sage
*****

Reged: 02/07/08
Posts: 213
Loc: Woodinville, WA
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280137 - 08/18/09 04:29 PM

I tried doing the same thing, creating a set of darks for every ten degrees or so. Even so, I've still had a lot of challenges getting darks to eliminate amp glow from my Nikon D50. It's very frustrating.

When I'm really desparate, I turn on long exposure noise reduction on my camera which is the equivalent of taking a dark immediately after the exposure and automatically subtracting. That really works the best for me. But of course it takes twice as long for each exposure. At the end of the night, I suppose I could take a bunch of darks but then that time would have been spent doing more enjoyable activities, like sleeping.

P

--------------------
------------
1978 orange tube Celestron C8
1990's Celestron C90
Astro-Tech AT66
Atlas EQG with EQMOD
Nikon D50
DMK31AU03.AS

Homepage: Explore the outer limits of human belief
http://www.paranormalamerika.blogspot.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08
Posts: 887
Loc: Voorhees, NJ
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3280241 - 08/18/09 05:30 PM

Hi Roy,

What exactly are you confused about?

A dark simply removes the thermal signal... it's not that complicated...

Jerry

--------------------
A Beginner's Guide to DSLR Astrophotography
http://www.astropix.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280289 - 08/18/09 05:54 PM

I understand what a dark does. Whats confusing for us new folks is the whole dark/bias/flat mixture. Your book helps, but its still a lot to learn.

Whats most confusing is the "creation of a master" from a collection of dark images (and bias images). My understanding is as follows:

Same ISO/Temp/Time
------------------
Image = LIGHT - DARK
Image = (LIGHT - BIAS) - (DARK - BIAS)
Image = ((LIGHT - BIAS) - (DARK - BIAS)) - (FLAT - BIAS)

And you are also saying that you can also take a DARK and let the software calculate the time of the dark based on the bias and some fancy computations ... That is where my head explodes.

Thats why I wanted to create a dark library with so many variations. Your book talks about the dark-bias stuff, but I got lost.

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Mike Clemens
Post Laureate


Reged: 11/26/05
Posts: 4670
Loc: Wasilla, Alaska 61N
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3280396 - 08/18/09 06:50 PM

I think...

image =
(light - (scaled dark) - bias)
divided by
(flat - (scaled flat dark) - bias)


Post Extras: Print Post   Remind Me!   Notify Moderator  
gatsbyiv
super member


Reged: 03/29/09
Posts: 157
Loc: Whitehouse Station, NJ
Re: Creating a "dark" library new [Re: Mike Clemens]
      #3280444 - 08/18/09 07:15 PM

A tip: if you don't want to stay up and monitor the temperature while your camera is out there taking library darks, you can get a chart of the actual temperature by searching on wolframalpha.com. (Try "temperature August 15, 2009 [ZIP code]") I've used this to figure out an average temperature for a set of darks or exclude a few shots when the temperature was changing quickly.

--------------------
-----------------------
CGEM + Orion EON 120
LXD 75 + Astro Telescopes 102ED
Canon Rebel XSi (no mod... yet)
Just enough knowledge to be dangerous


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tim C
sage
*****

Reged: 11/11/07
Posts: 340
Loc: Marietta, GA
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280511 - 08/18/09 07:47 PM

Quote:


Well, the answer is simple... create a dark frame library, as well as a bias frame library. Then, and here is the key part, use a program like Images Plus with automatic dark frame matching in the calibration step. It will scale the thermal signal in the master dark to match each individual light frame.

It works like magic. Other programs may do this as well, but I am not familiar enough with them to know for sure. I kind of stick with what works for me once I figure out what that is, and Images Plus works perfectly.

For my dark frame library, I create a master dark for every 10 degrees Farenheit. That way, any lights I shoot will be within 5 degrees, and this works just fine in Images Plus.

Using bias frames will allow you to scale your darks. This means if you shoot 10 minute darks, you can use them to calibrate shorter exposures taken at the same nominal temperature and the same ISO. This saves a lot of work. You certainly don't need one for every 30 second increment.

I standardize on ISO 1600 for deeps-sky objects.

Jerry




Hi Jerry,

Creating a dark frame library sounds easier than I thought it would be. I just wanted to confirm a few things:

1) You don't have to create different dark frame exposure lengths to match the light exposure length? So I could take all of the darks at 5 minutes and use them if my light exposures are say 3 minute or 8 minutes?

2) You take all of your deep sky shots at ISO 1600 (that one is not related to the dark frame library necessarily but just curious?

3) The bias frames are just the shortest exposure your camera is capable of (using the same ISO as the darks and lights)?

I use deep sky stacker so will have to research to see if DSS performs the same was as Images Plus. I have your CD as well but it's been a while since I've read it so will go back through it. I hate shooting darks in the field.

Tim

--------------------
Tim
Pictures of Space: http://tcardin.zenfolio.com/f749202565


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08
Posts: 887
Loc: Voorhees, NJ
Re: Creating a "dark" library new [Re: Roy Salisbury]
      #3280520 - 08/18/09 07:52 PM

Hi Roy,

You are making this un-necessarily complicated. (He said just as he proceeded to make it REALLY complicated...)

BIAS

Bias is a low-level charge that is applied to a sensor in the form of a fixed offset voltage value. It is present in every frame, light and dark.

So if you subtract a dark from a light, you have also subtracted the bias.

In this simple case, where the thermal signal matches exactly, and the exposure for the dark is the same as for the light, then you don't need to subtract a separate bias frame.

SCALABLE DARKS

Now... thermal signal scales linearly with exposure time. That means if you have a 2 min dark, and a 1 min light, you can still subtract the dark from the light... you simply let the software divide the dark by two.

The problem is, you're also dividing the bias by two. So you don't correctly subtract the bias then.

The way around this problem for scaled darks is to subtract the bias separately from both, and get it out of the equation at the start.

Then only the thermal signal is left in the darks, which technically at this point are called "thermal frames" and not "darks".

So, once you subtract the bias from both, you can scale a thermal frame down to a shorter exposure light frame, and remove the thermal signal.

Pretty cool deal... shoot 10 minute darks, and you can use them for any exposure that is less than 10 minutes long taken at the same ISO and temperature.

MASTER FRAMES

Now, moving on to master frames... the reason you create a master frame is to reduce the noise... let me explain... There is a difference between signal and noise. But everyone confuses this. What we remove with dark frames is thermal signal, not thermal noise.

However, every signal has noise associated with it. It's called photon, or shot, or Poisson noise. It's the square root of the signal.

So when you subtract a single dark frame, you are actually doing two things... removing the thermal signal, but you are also adding the thermal noise. Yes, this sounds confusing and counter-intuitive.

To reduce the amount of this real noise that is added to the lights when you subtract a dark, you create a master dark. A master dark has less noise than a single dark, because the noise is random and it averages out.

So, to get really complicated... a

light frame = photon signal + photon noise + thermal signal + thermal signal noise + bias signal + bias signal noise

If you subtract a dark frame, you have

Image frame = light frame - dark frame. This removes the thermal signal and the bias signal (because bias is also contained in the dark). The original light frame photon noise and bias noise remain, but you have added the thermal signal noise and bias signal noise from the dark frame now. (The bias signal noise should be insignificant).

So, even though we have vastly improved our image frame by removing the hot pixels and color blotches that come from the thermal signal, we have really increased the noise. To reduce this, use a Master dark and master bias, which add less noise when the light is calibrated.

Even still, the light frame technically has more noise now after calibration.

But the thermal signal is usually much more objectionable if there is a lot of it.

DEALING WITH NOISE

NOW, the way to deal with this noise is by gathering more signal from the photons that you really care about... those from the deep-sky object.

The only real way to deal with true noise is by increasing the signal in the signal-to-noise ratio.

FLATS

Flats are a completely separate subject. They correct for things like uneven pixel response, vignetting, and dust spots.

BUT... you have the equation wrong here... flats are not subtracted, they are divided. You can only do this with an astronomical image processing program.

Personally, I never use flats... I believe that uneven pixel response is small, or it is probably being dealt with already in the sophisticated image processing taking place by the image processor in the camera before the raw file is written.

I don't have much vignetting, and the little that I have I can correct with software. If your vignetting is severe, then flats are really a poor solution. You will never recover the lost signal-to-noise ratio in the corners where the vignetting is. If the vignetting is bad, you need to fix it at the source. Flats are not going to fix the kind of vignetting you get on a full-frame sensor from using an f/6 telecompressor on an f/10 SCT.

I find it is easier to keep the sensor clean than to shoot flats.

Jerry

--------------------
A Beginner's Guide to DSLR Astrophotography
http://www.astropix.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08
Posts: 887
Loc: Voorhees, NJ
Re: Creating a "dark" library new [Re: Tim C]
      #3280535 - 08/18/09 08:01 PM

Quote:

1) You don't have to create different dark frame exposure lengths to match the light exposure length? So I could take all of the darks at 5 minutes and use them if my light exposures are say 3 minute or 8 minutes?




No. You do not have to shoot the dark exposures to exactly match the lights, if you also shoot bias frames and use a good program.

You will get the best results if you scale them down. So shoot the darks at the longest exposure you use for your lights, and then you can use them with shorter exposures. I use 10 minutes.

Quote:

2) You take all of your deep sky shots at ISO 1600 (that one is not related to the dark frame library necessarily but just curious?




Yes, I usually shoot all of my deep-sky shots at ISO 1600. I might use ISO 400 or 800 for a globular cluster.

Quote:

3) The bias frames are just the shortest exposure your camera is capable of (using the same ISO as the darks and lights)?




Yes. Shortest shutter speed the camera can do. At the same ISO and temperature as the darks and lights.

Quote:

I hate shooting darks in the field.




Shooting darks in the field is a terrible waste of clear dark sky time that can be much better utilized in gathering photons to improve the signal in the signal-to-noise ratio.

There is no substitute for this, and you can't do it on cloudy nights. But you can shoot darks on cloudy nights and get great results.

Jerry

--------------------
A Beginner's Guide to DSLR Astrophotography
http://www.astropix.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280578 - 08/18/09 08:24 PM

OK.. To try to make this as un-technical as possible (and I am a real technical person normally) ...

Darks - Take a bunch of long exposures as each ISO and temp range. Perhaps ISO 400, 800, 1600 and each 5-10 deg of shooting. So if my normal imaging is between 70 and 90 deg (hey, I live in the desert), I would do maybe 70/75/80/85/90. I'm thinking 30 subs @ 10 min.

Bias - Same as darks but @ 1/40000 sec.

Flats - Forget about them unless I have a real good reason and know exactly why I would use them.

Then let the software create the masters that results in a single dark and bias "master" for each ISO and Temp.

Is this right?

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Roy Salisbury
sage


Reged: 05/26/09
Posts: 413
Loc: Las Vegas, NV
Re: Creating a "dark" library new [Re: Jerry Lodriguss]
      #3280589 - 08/18/09 08:30 PM

Off-Topic: In reference to your "New Book on CD-ROM Coming September 2009!" for beginners .. If you need someone to pre-review it just to make sure it really is for beginners like myself, then I'm your man! You can't get any more beginner than me!

Roy

--------------------
Scope: CPC 800 / AT66ED
Cameras: Canon 500D / QHY9M / SSAG

Hualapai Valley Observatory


Post Extras: Print Post   Remind Me!   Notify Moderator  
Tim C
sage
*****

Reged: 11/11/07
Posts: 340
Loc: Marietta, GA
Re: Creating a "dark" library [Re: gatsbyiv]
      #3280672 - 08/18/09 09:15 PM

Quote:

A tip: if you don't want to stay up and monitor the temperature while your camera is out there taking library darks, you can get a chart of the actual temperature by searching on wolframalpha.com. (Try "temperature August 15, 2009 [ZIP code]") I've used this to figure out an average temperature for a set of darks or exclude a few shots when the temperature was changing quickly.




I could be dreaming but isn't there a gadget that you can plug into your camera to record the temperature when the picture was taken along with the other data like ISO, exposure length, etc. I thought I saw something like this somewhere.

--------------------
Tim
Pictures of Space: http://tcardin.zenfolio.com/f749202565


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | (show all)


Extra information
7 registered and 7 anonymous users are browsing this forum.

Moderator:  Charlie Hein, knuklhdastnmr 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 621

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics