48 replies to this topic

[han.k]

With modern cameras bias and flats are all you need.  Bias is trivial, flats are relatively easy.

@WadeH237 has already indicated that darks will help. You can skip them but the result will be less good.  My cooled IMX571 sensor camera (one of best modern sensors on the market) is clearly suffering from hot pixels so skipping darks in the processing  would be ill advised strategy.

 

Secondly frames will suffer from unequal dark current so the measured pedestal will differ for each pixel. The only way to get it removed from a light is applying a master-dark. The temperature of the dark should be the same as the light. If the dark temperature is lower or higher then the light you get worse results. I have measured the effect and reported it. Link will follow.

 

Thirdly without applying darks the flats will overcompensate the flux drop in the corners of the lights. So the flat will not work properly as desired.

 

Replacing flat darks by bias frames for no reason would be a risky strategy.  It doesn't take more time to make flat-darks then bias frames. But flat-darks guarantees you will compensate for a possibly non-linear behavior of CMOS sensors in the first second(s). So my advice for calibration is to use :

 

Darks, plenty at the same exposure time and temperature as the light frames.

 

Flats, using a  light source which colour is reasonable white.

Flat-darks at same exposure time and temperature as the flat frames

 

Han

[han.k]

In August I did several experiments with darks and one experiment was to test if a dark is required. Below one visual result. To get a measurable result two series of stacked darks where subtracted from each other. One represents the lights, the other the darks:

 

 

First image, no darks results in a high noise value. The second image based on the subtraction of two dark series at 26°C and 23°C,  ∑(12x60sec_+26°C) -  ∑(12x60sec_23°C) gives the lowest noise value. So conclusion is that 1) darks help and 2) darks and lights temperature should match.

 

It doesn't matter if the experiment is done with a mono or colour sensor. The colour sensor is the same as the mono except it has a Bayer filter array in front of the pixels.

 

Full report here

 

Han

Edited by han.k, 02 October 2023 - 03:05 AM.

[Oort Cloud]

In August I did several experiments with darks and one experiment was to test if a dark is required. Below one visual result. To get a measurable result two series of stacked darks where subtracted from each other. One represents the lights, the other the darks:

combined level 1000 v3 stretched.png

First image, no darks results in a high noise value. The second image based on the subtraction of two dark series at 26°C and 23°C, ∑(12x60sec_+26°C) - ∑(12x60sec_23°C) gives the lowest noise value. So conclusion is that 1) darks help and 2) darks and lights temperature should match.

It doesn't matter if the experiment is done with a mono or colour sensor. The colour sensor is the same as the mono except it has a Bayer filter array in front of the pixels.

Full report here

Han

Han, as we've mentioned earlier, on these newer cameras, there are more ways to remove hot pixels than dark subtraction. That's like saying the only way to remove light pollution is with a light pollution filter.

[Tapio]

That experiment was made with ASI1600MM camera.
I believe result could be different with 533 or 2600 cameras.

[han.k]

I see no reason it will be different for other CMOS sensor cameras. But we can easily test it. Share 24 or more darks of your 533 camera taken at a typical operating temperature and I will repeat the measurements. If you have darks at other operating temperatures please share them as well.  I have a IMX571 camera from Touptek but lets first test it for the 533 camera.

[WadeH237]

Han, as we've mentioned earlier, on these newer cameras, there are more ways to remove hot pixels than dark subtraction. That's like saying the only way to remove light pollution is with a light pollution filter.

Right.

 

But remember that all pixels have dark current, not just the ones that have so much that they show up as "hot pixels".  When you use the methods that work by detecting outlier pixels, there are a couple of things to consider.  First, methods that work by detecting outlier pixels and hiding them (such as CosmeticCorrection) are putting fake data into those pixels.  Doing an aggressive dither is better, since a proper integration will reject outlier pixels outright, instead of putting fake values into them.  But none of these methods will do anything to correct small amounts of dark current that varies from pixel to pixel - and every pixel has some.

 

The dark current that is not addressed by shortcuts will be faint, and it may not be objectionable for bright objects, which is the entire reason that the idea of skipping darks comes from.  But it will affect your ability to dig really deep into the faint stuff.

 

If you want to get the very most out of your data, then skipping darks is not a great idea.  As far as I am concerned, building a dark library is maybe a twice a year task.  But once you have the library, a proper dark calibration takes virtually zero effort, versus skipping darks.  For me, the conclusion is obvious, and I will always use darks.

[Alan_]

People differ on this, and it depends on the camera.  I don't use them with my 2600s, they were absolutely necessary to eliminate amp glow on my 183s.

 

I also only cool my 2600s to 0C.  This explains why.

 

https://www.youtube....h?v=3RH93UvP358

I suspect this is the graph that drove the choice of 0C:

 

This is the broadband light polution noise case for an f6 scope. As explained later on by Robin, for narrow band imaging the light polution noise is reduced by a factor of 25 to 100. So the graph is only for the "broadband imaging" case.

 

To get the narrowband case the polution noise must be reduced by a factor of 25-100, so then the thermal noise will dominate over light polution noise at much lower temperatures.

 

Therefore for narrow band images (and for higher f-number scopes), it is worth using lower camera temperatures.

Edited by Alan_, 02 October 2023 - 11:13 AM.

[Oort Cloud]

Right.

But remember that all pixels have dark current, not just the ones that have so much that they show up as "hot pixels". When you use the methods that work by detecting outlier pixels, there are a couple of things to consider. First, methods that work by detecting outlier pixels and hiding them (such as CosmeticCorrection) are putting fake data into those pixels. Doing an aggressive dither is better, since a proper integration will reject outlier pixels outright, instead of putting fake values into them. But none of these methods will do anything to correct small amounts of dark current that varies from pixel to pixel - and every pixel has some.

The dark current that is not addressed by shortcuts will be faint, and it may not be objectionable for bright objects, which is the entire reason that the idea of skipping darks comes from. But it will affect your ability to dig really deep into the faint stuff.

If you want to get the very most out of your data, then skipping darks is not a great idea. As far as I am concerned, building a dark library is maybe a twice a year task. But once you have the library, a proper dark calibration takes virtually zero effort, versus skipping darks. For me, the conclusion is obvious, and I will always use darks.

I have mentioned before, and perhaps forgot in this thread, that dithering and having at least 50-100 or so lights, is required in order to take care of the hot pixels without dark subtraction.

When we're talking about the dark current on the 533/2600, the dark current is so low that my opinion is that it is negligible. For me, this all started when I picked up a 533mc a few years back. Of course, it was cloudy, so I spent the first couple of days building a dark library. To my chagrin, when I used the darks I'd created, they did not successfully remove the hot pixels. So since I needed to dither anyway, I did a few test stacks (bias subtraction from lights vs dark subtraction from lights.

For that particular camera, with no amp glow, and very low thermal noise (cooled to either 0 or -10), the two stacks were indistinguishable by eye. I applied the same stretch to both, and pixel peeped like crazy, and I was unable to determine any difference in the noise levels between the two. Since my only interest is pretty pictures, bias subtractions works just as well, and gives me the freedom to change gain, exp & temp as needed without having to worry if I have compatible darks.

For me, it's a no-brainer, but for others it may be worth the extra effort.

[WadeH237]

When we're talking about the dark current on the 533/2600, the dark current is so low that my opinion is that it is negligible.

Why do you say that?

 

The dark current on these cameras is actually quite high.  As I mentioned in my story, I didn't use darks when I got my ASI2600MC Pro - because I believed statements like this.  When I did my own testing, I learned the truth.

[bobzeq25]

 

 

Replacing flat darks by bias frames for no reason would be a risky strategy.  It doesn't take more time to make flat-darks then bias frames. But flat-darks guarantees you will compensate for a possibly non-linear behavior of CMOS sensors in the first second(s). So my advice for calibration is to use :

 

With cameras other than 1600s and 294s, the risk is ZERO.  Those are the ONLY cameras with that non-linear behavior, people have wrongly extended their behavior to CMOS cameras in general.  It's just not true.
 

[bobzeq25]

Why do you say that?

 

The dark current on these cameras is actually quite high.  As I mentioned in my story, I didn't use darks when I got my ASI2600MC Pro - because I believed statements like this.  When I did my own testing, I learned the truth.

He says that because that's what the data says.  The dark current of a 533MC at -10C is .0005 electrons/pixel/second.  With a 300 s sub (which would only be needed for narrowband). that's 0.15 electrons.  The read noise is 1.5 electrons, 10X more.  Since noise adds as the square, the read noise is actually 100X greater.  So, looking at overall noise, the thermal noise component is insignificant.

 

Here's the data.

 

https://astronomy-im...33mc-pro-color/

 

The 2600 is similar.

[Mike in Rancho]

Han, as we've mentioned earlier, on these newer cameras, there are more ways to remove hot pixels than dark subtraction. That's like saying the only way to remove light pollution is with a light pollution filter.

But as you noted in your later post, it can take a lot of subs and plenty of times people run short of that.  I've seen quite a few images with remnant hot pixels (the end results tends to be scattered blobs of red, green and blue) lately, and they seem to be from using bias on both sides.  I used bias for lights and flats to start out with my 2600, but changed over to darks thereafter due to hot pixels not rejecting, despite dithering.  Yes seemed more prone to it with fewer subs and/or multiple session integration.  And it can be more likely in narrowband where much longer but fewer subs are acquired.  Though we've also been seeing lately that people are also going quite long (overly so, IMHO, but I care about blown stars) in broadband as well.

 

For me, a pre-made and reusable master darks library is a very easy solution, and my hot pixel issues went away.  Dark flats not so much, as with a Newt it would likely take more than just popping the OTA dustcap on, like maybe a big tarp.  So I use bias for the flats side.

 

I would be curious to see Han's experiment again, but with a 571, and at a normal capture temperature like 0C.  26C does not seem to be a real world test situation.  I'd also like to see those calculations run in a bias vs dark flats scenario, where the exposures are typically going to be under 10s, probably by quite a bit.  My L flats can frequently be sub-1s.  I'd be surprised if there's any noticeable difference in flats calibration or remnant image noise between a bias and a 0.5s dark flat with a 571 at 0C.

 

But if there is I'd consider getting a big dark tarp and seeing if NINA can run off matching dark flats for me. 

[Oort Cloud]

That's interesting to me, Mike, as I had the opposite experience with my 533; dark subtraction left me with hot pixels all clustered together, and it was only dithering that finally allowed me to get rid of them.

I never revisited it, as I only cared about getting rid of hot pixels at the time, but maybe I'll give darks another try. When I did my original comparisons though, I didn't see any difference between 2 stacks of the same data, where one set of lights had the master bias subtracted instead versus the other having the master dark subtracted.

Also, I agree, testing should be done at 0C. 26C doesn't really tell us much.

Edited by Oort Cloud, 02 October 2023 - 12:38 PM.

[han.k]

He says that because that's what the data says.  The dark current of a 533MC at -10C is .0005 electrons/pixel/second.  With a 300 s sub (which would only be needed for narrowband). that's 0.15 electrons.  The read noise is 1.5 electrons, 10X more.  Since noise adds as the square, the read noise is actually 100X greater.  So, looking at overall noise, the thermal noise component is insignificant.

 

 

The 0.0005 electrons/pixel/second dark current should be negligible. The noise (=sqrt(dark current[e-]) caused by such a dark current is so low it is difficult to measure and will be flooded by sky noise. The dark current nor the read noise are not a problem for imaging  The problem in most cases is that pixels have an unequal pedestal value. Due to this unequal pedestal values it is normally difficult to measure read- or dark current noise using a single dark. The unequal pedestal values could generate noise in the image a few magnitudes larger then the read noise or dark current. The only way to measure the sensor noise is to subtract one dark from an other. Read noise is then measured and calculated by (bias1 - bias2)/sqrt(2).  These unequal pedestal values are then nullified by the subtraction.This principle is applied in sensor test program available in SharpCap or CCDciel and any other sensor test.

 

I just measured my  IMX571. The previous effect is not very visible in this modern sensor. That could be an argument for not using a dark. But there is also random telegraph noise and hot pixels.

 

Han

Edited by han.k, 02 October 2023 - 01:48 PM.

[han.k]

Here the difference visualized using my IMX571 camera in LCG mode. Left a stack of 20 darks of 200 sec exposure. Right the difference of 20 darks stacked minus 20 darks stacked (representing the lights calibrated with darks). The difference is minimal. The hot pixels are more visible in the single stack then in the difference of two stacks  The noise in the difference has increased from 0.9 e-  to about 1.2 e-.  If you take maybe 100 darks then likely the  noise of the difference can likely be be brought back to 0.9 e-. So an argument to make many darks. But I can understand the argument to skip darks for modern sensors if you 1) accept more hot pixels and 2) A little less then perfect flat correction. For most OSC cameras users the sky noise will be very dominant and above effects will not be visible.

 

Han

 

[psandelle]

No one has touched upon one important point: I take a dark library because there are so many days/nights I can't image and I have to do SOMETHING astro related, so if I take darks at least I feel like I'm getting something accomplished.

 

Paul

[WadeH237]

He says that because that's what the data says.  The dark current of a 533MC at -10C is .0005 electrons/pixel/second.  With a 300 s sub (which would only be needed for narrowband). that's 0.15 electrons.  The read noise is 1.5 electrons, 10X more.  Since noise adds as the square, the read noise is actually 100X greater.  So, looking at overall noise, the thermal noise component is insignificant.

Averaged over all the pixels, that is correct.

 

But not all pixels are affected equally.  There is a (relatively) small number of pixels that are more strongly affected, and the effect on those pixels is quite dramatic.  They are easily as obvious when I blink uncalibrated frames as what I used to see in my old ST-10XME - a CCD design that is multiple decades old.

 

If I remember correctly, there was something on the order of 100,000 to 200,000 pixels on each of my ASI2600 cameras with obvious dark current.  The reason that the overall dark current number is so low, is that there are over 26,000,000 pixels on the sensor.

 

To reiterate, these sensors are good enough that you can get away without darks.  That is not the same thing as saying that there is no reason to do darks.  If you really want the most out of the camera, skipping darks is not the way to get there.

 

But if you are happy with the results that you get by skipping them, that's really all that matters to your workflow.  There was a day when I would have been happy with the "no dark" results that I can get from these cameras.  But the longer I do this, the more that I realize the the best results come from paying attention to tiny details.  I just retired a $5000 CCD camera (not the ST-10) because it developed a faint pattern in the bias that would not calibrate out, and only shows up in the faintest areas.

 

YMMV.

[Oort Cloud]

No one has touched upon one important point: I take a dark library because there are so many days/nights I can't image and I have to do SOMETHING astro related, so if I take darks at least I feel like I'm getting something accomplished.

Paul

Paul,

Might I interest you in my "Process my data thread" thread, where several users in addition to myself have provided oodles of data to practice with on those Cloudy Nights...some of it (in fact most at this point) is even from Bortle 1, and captured with very good technique (thanks, Gilmour!)

Dave

[Oort Cloud]

Averaged over all the pixels, that is correct.

But not all pixels are affected equally. There is a (relatively) small number of pixels that are more strongly affected, and the effect on those pixels is quite dramatic. They are easily as obvious when I blink uncalibrated frames as what I used to see in my old ST-10XME - a CCD design that is multiple decades old.

If I remember correctly, there was something on the order of 100,000 to 200,000 pixels on each of my ASI2600 cameras with obvious dark current. The reason that the overall dark current number is so low, is that there are over 26,000,000 pixels on the sensor.

To reiterate, these sensors are good enough that you can get away without darks. That is not the same thing as saying that there is no reason to do darks. If you really want the most out of the camera, skipping darks is not the way to get there.

But if you are happy with the results that you get by skipping them, that's really all that matters to your workflow. There was a day when I would have been happy with the "no dark" results that I can get from these cameras. But the longer I do this, the more that I realize the the best results come from paying attention to tiny details. I just retired a $5000 CCD camera (not the ST-10) because it developed a faint pattern in the bias that would not calibrate out, and only shows up in the faintest areas.

YMMV.

Thanks for digging into the weeds a bit more here, Wade. This is really what I was getting at: not that darks should be omitted, but simply that dithering and rejection stacking can get you like 90% or so of the way there, without having to take darks, which is time-consuming.

If the camera has amp glow, all bets are off. If it has high dark current and/or is uncooled, all bets are off. If you want the absolute best image quality, all bets are off. But if you're like me, work full time, and have to have your rig be fully automated in order to even participate in the hobby, then with the right camera and technique, skipping darks is a viable option, and still produces images that get lots of oohs and aahs from family and friends.

[bobzeq25]

My cure for hot pixels (and even pretty warm ones).

 

CosmeticCorrection on each sub in PI.  My standard method is Use Auto Detect with the default sigma of 3.0.  There are options.

 

Stack a good number of subs (20 is minimal) with a good data rejection algorithm.  My standard method is linear fit clipping.  Default high sigma of 3.0.

 

And they're gone (with both my 2600s).  So I don't use darks.  Bias and flats.

Edited by bobzeq25, 02 October 2023 - 06:14 PM.

[F.Meiresonne]

No one has touched upon one important point: I take a dark library because there are so many days/nights I can't image and I have to do SOMETHING astro related, so if I take darks at least I feel like I'm getting something accomplished.

 

Paul

Mmm, seems like conditional behaviour

 

Why build a dark library?  With a cooled camera you have to take darks only once  , at least for a while. You can put the camera on the exact spot  of temp. Unless you set youre cooling on every session different i don't see really the point to make a library.

[bobzeq25]

I see no reason it will be different for other CMOS sensor cameras.

Of course it will be different.  Newer CMOS cameras have _much_ lower thermal noise than old ones.  That totally changes the game.

 

People lumping "CMOS cameras" together as a group are making a mistake.  Leads to silliness like "CMOS needs flat darks".  No, only a few do (essentially only 1600s and 294s), for others bias is just as good.

 

Look at the data for thermal noise on cameras on an individual basis.  Look at the data, not your intuition.

 

For some, it's negligible.  Individual hot pixels can be removed in processing a number of ways (the rejection algorithms used in stacking are a major help), you needn't take darks for those.

 

Nothing wrong with using darks.  In many situations, nothing wrong with omitting them.  Just a personal choice, not right/wrong.

 

Data can help you figure out which situations, but it needs to be equipment and site specific.
 

Edited by bobzeq25, 07 October 2023 - 02:32 PM.

[psandelle]

Mmm, seems like conditional behaviour

 

Why build a dark library?  With a cooled camera you have to take darks only once  , at least for a while. You can put the camera on the exact spot  of temp. Unless you set youre cooling on every session different i don't see really the point to make a library.

I like to have my darks and lights the same length, and the same mode/gain/offset. I tend to take 180sec lights for RGB, but do some 60sec, and they are at one mode/gain/offset on my QHY camera, while I do NB at 300sec and sometimes 180, at a different mode/gain/offset, so, a library of four "books".

Paul

[han.k]

 

Look at the data for thermal noise on cameras on an individual basis.  Look at the data, not your intuition.

 

For some, it's negligible.  Individual hot pixels can be removed in processing a number of ways (the rejection algorithms used in stacking are a major help), you needn't take darks for those.

I have looked to the data and by using darks you get a little smoother image for an IMX571 camera. If you try to remove hotpixels by an algorithm you will also remove faint stars and the depth of the image will be lost. For me there are only two ways to process. Either you use darks or your don't use darks. An rejection algorithm is not a replacement for using darks and it will likely take more time then applying a master dark.

 

Han