Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Trouble with Darks

  • Please log in to reply
26 replies to this topic

#1 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 19 May 2019 - 11:07 PM

I need some help with my darks.  I've searched the forums but I can seem to find anyone with a similar problem.  When I stack dark frames with my lights using DSS, I lose almost all data from my lights.  I still have stars on a dark background, but all nebulosity is gone.  The histogram is just a thin line on the left side.  The blacks are totally clipped.  

I'm attaching two Ha images.  The first is a stretched stacked image of the Sadr region with no darks.  The second is a stretched stacked image with lights and darks.

 

I'm also attaching the master dark.  

 

I'm hoping somebody has run across this before and it's a simple fix.  As you can see, I desperately need those darks to help get rid of that wicked amp glow :-)

In case it helps, I'm using a ZWO 183 MM Pro. 32 X 300 seconds subs and 20 Darks.

 

Thanks for the help.

Attached Thumbnails

  • Autosave001 no calibration stretched.jpg
  • lights and darks.jpg
  • MasterDark_ISO0_300s.jpg

Edited by ftur62, 19 May 2019 - 11:08 PM.


#2 ngc7319_20

ngc7319_20

    Apollo

  • *****
  • Posts: 1015
  • Joined: 25 Oct 2015
  • Loc: MD

Posted 20 May 2019 - 12:24 AM

I need some help with my darks.  I've searched the forums but I can seem to find anyone with a similar problem.  When I stack dark frames with my lights using DSS, I lose almost all data from my lights.  I still have stars on a dark background, but all nebulosity is gone.  The histogram is just a thin line on the left side.  The blacks are totally clipped.  

I'm attaching two Ha images.  The first is a stretched stacked image of the Sadr region with no darks.  The second is a stretched stacked image with lights and darks.

 

Ummm... Maybe I'm missing something here, but why are you stacking the dark frames WITH the light frames? Does not make sense to me.  Dark frames should be subtracted from the light frames before stacking (if there is rapid variation), or possibly subtract a master dark if things are more stable?



#3 TelescopeGreg

TelescopeGreg

    Apollo

  • -----
  • Posts: 1067
  • Joined: 16 Jul 2018
  • Loc: Auburn, California, USA

Posted 20 May 2019 - 12:41 AM

In DSS you don't direct it to do specific pixel math.  It has that stuff built-in, driven by how the images are loaded.  I'm assuming he fed the images into DSS as Lights and Darks (separate file fetching steps).  Besides, if they were all fed in as lights, DSS would reject the Darks anyway, as they wouldn't be able to get registered.

 

To the OP's question, no, I haven't seen this before.  But, my first thought is to check how the stack with the darks was stretched.  It's expected that the resulting stack will be pretty dark overall, and need to be stretched a bunch to bring out the details.  But the details should be there. 

 

As a check, do you see the amp glow in the master dark?  If it's not there, then it is not going to be effective in removing the glow from the lights.  And I don't see it in what you have posted. 

 

Odd ball question...  Are you sure that's amp glow (not something bright just outside the field of view leaking in)?

 

If you want to experiment with different tools, StarTools has an "amp glow" mode in the Wipe tool, which might work better.



#4 Stelios

Stelios

    Cosmos

  • *****
  • Moderators
  • Posts: 7719
  • Joined: 04 Oct 2003
  • Loc: West Hills, CA

Posted 20 May 2019 - 01:33 AM

It's normal for the resulting image to be almost completely black. That's because it's not stretched. Stretching the histogram will reveal the detail. 



#5 pyrasanth

pyrasanth

    Surveyor 1

  • *****
  • Posts: 1851
  • Joined: 08 Jan 2016

Posted 20 May 2019 - 02:12 AM

If I'm reading your post correctly your doing your calibration incorrectly.

 

A dark master is produced by first stacking several hundred bias frames to produce a bias master. You then take 20-30 dark frames of an exposure that matches the exposure for your light frames. You then calibrate the single dark frames with the bias master then integrate the dark frames to produce a dark master.

 

The light frames are then calibrated with the dark master & bias master along with a master flat which will produce your fully calibrated light frame of which multiples can be stacked to produce your final image.



#6 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 20 May 2019 - 04:30 AM

The OP clearly states that the blacks are clipped in the stack so much of the above is missing the target.

 

Sounds like for some reason the darks have too high a pedestal, and so subtracting them from the lights is giving negative numbers and losing data.

 

First thing I would try is loading the dark into some software (like fitswork) and subtracting a constant from it, if possible. ie if the lowest value in the dark is 321, subtract 321 and save it, and use that as your dark. Trouble is this is a bit of a fudge and will make the flats not work properly, unless you're lucky enough to exactly correct the pedestal issue.

 

i wonder if you could have used a different offset/brightness setting for the lights and darks?


  • RossW likes this

#7 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 20 May 2019 - 04:32 AM

Also - whats the mean background value for your lights and darks? Just look at an example of each. Hopefully the light will be higher than the dark, but i suspect it may not be.


  • RossW likes this

#8 D_talley

D_talley

    Gemini

  • *****
  • Posts: 3292
  • Joined: 07 Jul 2005
  • Loc: Albuquerque, New Mexico

Posted 20 May 2019 - 07:29 AM

If I'm reading your post correctly your doing your calibration incorrectly.

 

A dark master is produced by first stacking several hundred bias frames to produce a bias master. You then take 20-30 dark frames of an exposure that matches the exposure for your light frames. You then calibrate the single dark frames with the bias master then integrate the dark frames to produce a dark master.

 

The light frames are then calibrated with the dark master & bias master along with a master flat which will produce your fully calibrated light frame of which multiples can be stacked to produce your final image.

The OP is using DSS which does all that for you. You just load the files and it produces the calibrated light.



#9 pyrasanth

pyrasanth

    Surveyor 1

  • *****
  • Posts: 1851
  • Joined: 08 Jan 2016

Posted 20 May 2019 - 08:34 AM

The OP is using DSS which does all that for you. You just load the files and it produces the calibrated light.

DSS is not doing it correctly or the data being input is not correct. I work to the old adage of garbage in & garbage out. Sometimes it is better to understand the manual process instead of relying on packages to do the work for you. There is nothing wrong with using automated tools but if the process is understood it can be easier to fix issues- does the op understand what is going on beneath the hood of DSS- I suspect not.



#10 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 20 May 2019 - 08:41 AM

DSS is not doing it correctly or the data being input is not correct. I work to the old adage of garbage in & garbage out. Sometimes it is better to understand the manual process instead of relying on packages to do the work for you. There is nothing wrong with using automated tools but if the process is understood it can be easier to fix issues- does the op understand what is going on beneath the hood of DSS- I suspect not.

Well lets be nice and not suggest this is down to a lack of understanding of the OP. It doesnt really matter, and there's no reason to accuse him of putting garbage in! Ive had peculiarities like this before, including one image set which i never got to the bottom of. 

 

Im sure with our communal mighty brains we can work this out.


  • pyrasanth likes this

#11 pyrasanth

pyrasanth

    Surveyor 1

  • *****
  • Posts: 1851
  • Joined: 08 Jan 2016

Posted 20 May 2019 - 09:10 AM

Well lets be nice and not suggest this is down to a lack of understanding of the OP. It doesnt really matter, and there's no reason to accuse him of putting garbage in! Ive had peculiarities like this before, including one image set which i never got to the bottom of. 

 

Im sure with our communal mighty brains we can work this out.

point taken. It may not be intentional garbage but it can give the same resultsmile.gif I have also seen a set of flats that would not correct and yet they were taken with the same settings as ones that worked later. This is a challenging hobby to say the least- so apologies to the OP if that post came out wrong.


  • happylimpet and B 26354 like this

#12 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 20 May 2019 - 10:38 AM

point taken. It may not be intentional garbage but it can give the same resultsmile.gif I have also seen a set of flats that would not correct and yet they were taken with the same settings as ones that worked later. This is a challenging hobby to say the least- so apologies to the OP if that post came out wrong.

And I shouldnt come on high and mighty as Im forever being snarky on here and then regretting it! Its so easily done with text only.

 

Im pretty sure its a change in the brightness/offset settings that are causing this. I reckon its fixable.



#13 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 20 May 2019 - 12:05 PM

Thanks everyone for the input.  No offense taken related to the "snarkyness."  If it is a case of garbage in, I just want to know.  I'm a quick learner smile.gif.  

 

When you say it's probably a change in the brightness/offset settings. Do you mean the gain/offset settings?  I was using APT and I'm pretty sure the only way to set the gain is to go into the ZWO/ASI driver.  I didn't change it between my lights and darks, but I will double check that.  

 

It does makes sense since the amp glow in the darks is extremely faint. To see it in the master dark, I have to stretch the heck out of it.  The amp glow in the lights are definitely more pronounced.  I would have thought that the amp glow in the darks would be just as visible as the light frames since I was using the same exposure settings. 


Edited by ftur62, 20 May 2019 - 01:51 PM.

  • sink45ny likes this

#14 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 20 May 2019 - 04:41 PM

I was thinking just of offset, which ZWO call 'brightness'. Though a change in gain could also have this effect, and cause a mismatch. Better check both! Did you check your background levels on the darks and light?

 

Presumably theres a settings file stored with the data, or in a FITS header, or somewhere, so you could check if gain and brightness(offset) are the same for lights and darks.



#15 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 21 May 2019 - 11:48 AM

Thanks for the clarification.  I'm a bit of a newbie and just recently made the switch from a DSLR to a CMOS astronomical camera. These shots are first light on the new camera.   I'm away from my computer and rig for a couple of days.  I'll check the background levels when I get back home. 

 

Here's another strange piece of the puzzle.  I did a series of 20 darks using Astrophotography tools.  I let it run and went to bed.  The other day, I was looking at the darks and three of them are exposed differently.  Darks 1 - 9 and 13 - 20 look like the first image below.  Darks 10 - 12 look like the second image. 

 

I'm assuming the lighter (more exposed) versions are how the darks should look. I think I'll try to run another set of darks when I get back home and make sure I'm using the correct offset and gain.  Hopefully that will solve it. 

 

Thanks again for your help. 

Attached Thumbnails

  • D_2019-05-18_08-01-06_Bin1x1_300s__-10C.png
  • D_2019-05-18_08-21-51_Bin1x1_300s__-10C.png

Edited by ftur62, 21 May 2019 - 11:51 AM.


#16 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 21 May 2019 - 06:37 PM

Thats a bit weird.  If that had happened during the generation of the darks you used above it would explain the problem, as the master dark will then have a too-high mean value.

 

Something amiss with the software acquiring the images.



#17 pyrasanth

pyrasanth

    Surveyor 1

  • *****
  • Posts: 1851
  • Joined: 08 Jan 2016

Posted 22 May 2019 - 03:58 AM

How are your darks being taken-on or off the telescope?

 

I find my darks are best created with the camera off the telescope and put into a box- even with the cap on you can still get light leaks through the camera chassis. It is also possible for temperature regulation to affect the result. I make sure I'm at the designated temperature (-20 C in my case) and the camera capped & in a dark box.

 

I would look to see if APT is working as expected as the inconsistent results are odd. If you can try & use another package to control the camera.



#18 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 22 May 2019 - 11:34 AM

I should be able to troubleshoot the issue tomorrow and this thread has given me some ideas on how to get to the root of the problem.  I'll double check the gain/offset and I'll try shooting the darks off of the scope in a dark room. I'll also try using Sharpcap or the ZWO capture software that came with the camera.  Thanks for all the info.



#19 Der_Pit

Der_Pit

    Viking 1

  • -----
  • Posts: 857
  • Joined: 07 Jul 2018
  • Loc: La Palma

Posted 22 May 2019 - 12:11 PM

Just to be sure - is this really a completely different intensity level, or do all other (black) images have some high intensity / hot pixel that the autoscale display jumps on?  My guess would be that all the darks show more or less identical histograms...


  • happylimpet likes this

#20 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 23 May 2019 - 12:44 PM

Also - whats the mean background value for your lights and darks? Just look at an example of each. Hopefully the light will be higher than the dark, but i suspect it may not be.

You are correct.  The darks are higher than the lights.  Darks are 3140 and the Lights are 2050.  Any idea what would cause that?  I checked the gain and offset for the darks and the lights and they are the same.  


  • happylimpet likes this

#21 jdupton

jdupton

    Surveyor 1

  • *****
  • Posts: 1950
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 23 May 2019 - 01:18 PM

ftur62,

 

I should be able to troubleshoot the issue tomorrow and this thread has given me some ideas on how to get to the root of the problem.  I'll double check the gain/offset and I'll try shooting the darks off of the scope in a dark room. I'll also try using Sharpcap or the ZWO capture software that came with the camera.  Thanks for all the info.

 

You are correct.  The darks are higher than the lights.  Darks are 3140 and the Lights are 2050.  Any idea what would cause that?  I checked the gain and offset for the darks and the lights and they are the same.  

 

   Did you by any chance capture the target light frames using a different program than was used to capture the calibration frames? It seems that can cause problems such as yours. There was another thread just like this a couple of weeks ago. The poster captured light frames using ASIAir and a set of calibration frames using SGP. He had problems in calibration. He then also tried using SharpCap for the calibration frames. The problems became worse. It turned out all combinations of mixed software capture were producing different file characteristics and caused a failed calibration.

 

   Read through this thread and see if it applied to your situation.

https://www.cloudynights.com/topic/660479-calibration-i-dont-get-it/

 

   Since you have mentioned having APT, SharpCap, and ZWO Capture software in this thread, make sure you use only one of the above to capture all frames -- Target, Dark, Flat, and Flat-Dark (if you use them). Mixing the software can cause problems if any are inadvertently set for different usage patterns (such as some using the native driver and other using the ASCOM driver). (Gain and offset may also be set differently by default amongst the capture programs.) If you suspect different gain or offset may be being used, you can verify that theory by reading the FITs Header information in a sample of each file type.

 

   Good luck tracking down the calibration problem. Let us know what you find to finally fix it.

 

 

John



#22 ftur62

ftur62

    Lift Off

  • -----
  • topic starter
  • Posts: 20
  • Joined: 17 Sep 2018
  • Loc: Colorado

Posted 23 May 2019 - 04:03 PM

ftur62,

 

   Read through this thread and see if it applied to your situation.

https://www.cloudynights.com/topic/660479-calibration-i-dont-get-it/

 

 

John

Thanks so much.  That thread really helps.  I did all of my captures using APT, so I don't think that's the issue.  However I did see this in the thread:  

"...so i can't really posit why the dark signal in the darks would be higher than the lights... this does happen from time to time with CCD cameras, usually in the context of narrowband filters against a very dark sky. in that case the background signal is almost nonexistent and just the noise in the dark signal (in the light) is enough to cause the dark master to oversubtract some pixels (the pixels where the dark noise caused the pixel value in the light to be lower than the corresponding pixel in the master dark, which of course has less pixel value variation from noise.)...in that situation what you can do is add an output pedestal to the calibrated lights so that you don't get any clipping"

 

I was shooting narrowband (Ha) in a Bortle 4 sky,  But's its still hard to believe that the signal in the darks would be higher than the lights.  

 

Boosting the overall signal in the light frame doesn't sound like the optimal solution, but if I wanted to give it a try how would I go about adding a constant value to the light frames? 



#23 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 23 May 2019 - 07:39 PM

Thanks so much.  That thread really helps.  I did all of my captures using APT, so I don't think that's the issue.  However I did see this in the thread:  

"...so i can't really posit why the dark signal in the darks would be higher than the lights... this does happen from time to time with CCD cameras, usually in the context of narrowband filters against a very dark sky. in that case the background signal is almost nonexistent and just the noise in the dark signal (in the light) is enough to cause the dark master to oversubtract some pixels (the pixels where the dark noise caused the pixel value in the light to be lower than the corresponding pixel in the master dark, which of course has less pixel value variation from noise.)...in that situation what you can do is add an output pedestal to the calibrated lights so that you don't get any clipping"

 

I was shooting narrowband (Ha) in a Bortle 4 sky,  But's its still hard to believe that the signal in the darks would be higher than the lights.  

 

Boosting the overall signal in the light frame doesn't sound like the optimal solution, but if I wanted to give it a try how would I go about adding a constant value to the light frames? 

I'd do it the other way round - when your software (DSS or whtever) has made the master dark, load that into fitswork and subtract a constant from it. Then you only have to adjust one image, not a whole load.

 

You gave to use processing>pixel arithmetic>'add value' or similar and enter a negative number.

 

https://www.fitswork...re/softw_en.php


Edited by happylimpet, 23 May 2019 - 07:40 PM.


#24 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2299
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 23 May 2019 - 08:29 PM

I'd do it the other way round - when your software (DSS or whtever) has made the master dark, load that into fitswork and subtract a constant from it. Then you only have to adjust one image, not a whole load.

 

You gave to use processing>pixel arithmetic>'add value' or similar and enter a negative number.

 

https://www.fitswork...re/softw_en.php

Wouldn't that potentially dark clip your dark?



#25 happylimpet

happylimpet

    Soyuz

  • *****
  • Posts: 3995
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 24 May 2019 - 03:17 AM

Wouldn't that potentially dark clip your dark?

Hopefully not. As its a master dark with less noise than individual frames there should be a smaller range of values than the in the latter. And if it does clip, then the individual lights must also be clipped (and even more so due to their greater noise), so its probably the least of your worries.

 

But its a fair point. I would subtract the minimum possible to leave the histogram of the calibrated light all just marginally positive.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics