587 replies to this topic

[Astrojedi]

The issue I am having with this thread is that the thread is about specific sensors and their performance yet in every post many posters keep generalizing. The thing is I don’t disagree with the generalizations but the gap between calibration and no calibration in real world conditions (e.g. LP) for these sensors is much smaller.

Edited by Astrojedi, 28 December 2020 - 10:58 AM.

[Astrojedi]

I find it very difficult to believe that your master dark contains no useful information.

 

Below is a crop of a typical master dark - is this the kind of thing you are seeing?

 

MasterDarkCrop.png

 

This does not contain "just randomness" but it shows the effect of dark current non-uniformity (DCNU) i.e. different pixels have different levels of dark current.  In dark frames, some pixels are consistently brighter than others whilst others are consistently dimmer than others. It creates a fixed pattern.  Sure, this pattern might appear random but it is consistent from one dark frame to another and therefore appears both in dark frames and in light frames.  This fixed pattern is what we are seeing in the above master dark and it's the reason we create a master dark.  

 

Mark

Mark,

 

I don’t disagree with you but it would be more useful if the data you were presenting was specific to the sensors in question. Also let us address real world practicalities as your application of the theory needs to consider all variables not a narrow (idealized conditions) problem statement you have decided on - actually this holds true for others on this thread as well not just you.

 

From my light polluted backyard the value of dark calibration with the 2600 is pretty minimal - actually close to zero. The sensor has extremely low dark current and has small amount of FPN. CDS eliminates quite a lot of the FPN. 
 

Having said that I still use darks as a dark library is easy to build for a cooled camera but benefit is quite limited.

Edited by Astrojedi, 28 December 2020 - 10:26 AM.

[Astrojedi]

Personally, I always use DBE, even after calibrating with flats. Flats are used for vignetting, dust motes, everything that is a "signature" of the imaging train.

 

But light pollution is also a gradient. Something definitely unwanted in your image and that you need to get rid of (or diminish as much as you can). There's no way I can see the object in its full glory unless I do one pass of DBE to remove these gradients.

 

Flats cannot compensate for the light pollution. Unfortunately, I would add.

This is correct. DBE or gradient removal addresses light pollution which cannot be addressed by flats alone. But good flats allow the gradient removal algorithms to do a better job if your optical system has meaningful vignetting or dust motes. If your optical system is fairly uniform over the area of the sensor and your sensor has a relatively uniform response across pixels you can get away without using flats.

 

For me that is not an option from my heavily light polluted backyard and larger sensors.

[sn2006gy]

 

 

Anyway, I think that I've reached about the limit of what I can contribute here.  Remember that most of us do this for fun.  The only way to do it "wrong" is to not gain enjoyment or personal satisfaction out of it.

 

I kind of have to chuckle, because my passion and enjoyment about what I get without having to dark calibrate was exactly what was being contested here

 

I share my data hoping to help others and because I know its the best way for me to learn myself.  Any notion of right or wrong way to do anything is never in my vocabulary...  I just look at what i'm experimenting with as another path of enjoyment with a bit fewer hassles, mostly because of modern technology and advancements in processing techniques.

 

It is a luxury i paid for, but it's something I share... for what good is wisdom and my experiences if i just take it to my grave.

[bobzeq25]

The "issue" with this sensor is there is really no consistency between dark  frames, and there is no FPN when stretched.

That you can see.  Others can see it.
 

[Astrojedi]

The "issue" with this sensor is there is really no consistency between dark  frames, and there is no FPN when stretched. If there was of coarse they would have value in the calibration process.  

There is likely still some residual FPN as CDS with current implementations cannot eliminate all of it but what is left should not have much of an impact on the final ‘pretty picture’. It is pretty much irrelevant.

[bobzeq25]

 If your optical system is fairly uniform over the area of the sensor and your sensor has a relatively uniform response across pixels you can get away without using flats.

 

Here's the thing about what "you" can get away with.  It depends on who "you" is, and what "you's" standards are.  It's quite similar to "We".

 

The basic techniques of astrophotography are quite powerful.  Gather a lot of data.  Process it intensively on a computer. 

 

You can do it badly and still get something.  People can "get away" with a lot, and still get something that is satisfactory to them.

 

But, if you want the best images you're capable of doing, you will not omit the calibration frames.  They're fundamental.

 

The minor point is that they make images better.  The major one is that they make better processing possible.

 

That's what some very experienced and very talented people have been trying to explain here.  It appears to have been futile in some ways, although the higher level discussions have been quite interesting.
 

Edited by bobzeq25, 28 December 2020 - 11:17 AM.

[sharkmelley]

Mark,

 

I don’t disagree with you but it would be more useful if the data you were presenting was specific to the sensors in question. Also let us address real world practicalities as your application of the theory needs to consider all variables not a narrow (idealized conditions) problem statement you have decided on - actually this holds true for others on this thread as well not just you.

 

From my light polluted backyard the value of dark calibration with the 2600 is pretty minimal - actually close to zero. The sensor has extremely low dark current and has small amount of FPN. CDS eliminates quite a lot of the FPN. 
 

Having said that I still use darks as a dark library is easy to build for a cooled camera but benefit is quite limited.

Interestingly, the master dark I posted was from the ASI2600MC PRO.  Also I don't see the relevance of CDS to DCNU.

 

Mark

Edited by sharkmelley, 28 December 2020 - 11:29 AM.

[bobzeq25]

The issue I am having with this thread is that the thread is about specific sensors and their performance yet in every post many posters keep generalizing. The thing is I don’t disagree with the generalizations but the gap between calibration and no calibration in real world conditions (e.g. LP) for these sensors is much smaller.

The gap itself depends on the specific situation and the specific calibration frames.  But the important thing is....

 

Whether or not the gap is significant depends on "to who?".

 

To me, the only time the gap is small is sometimes, with darks.  But usually it's not small for me with darks.  With bias (or dark flats) and flats, never.

 

The only generalization that has been wrong is when people have said, for me and this camera the gap is small, so everyone should follow my example.

 

I say do what you like.  But don't say calibration frames are useless (as in the title).  That's silly, regardless of the camera.  As is saying there's no imformation in bias.  Or darks.  Or flats, for that matter.  That's beyond "silly".

 

It's not silly to talk about the _nature_ of the gap, which is what the experienced and talented folks have been doing.

Edited by bobzeq25, 28 December 2020 - 11:34 AM.

[Astrojedi]

Here's the thing about what "you" can get away with.  It depends on who "you" is, and what "you's" standards are.  It's quite similar to "We".

 

The basic techniques of astrophotography are quite powerful.  Gather a lot of data.  Process it intensively on a computer. 

 

You can do it badly and still get something.  People can "get away" with a lot, and still get something that is satisfactory to them.

 

But, if you want the best images you're capable of doing, you will not omit the calibration frames.  They're fundamental.

 

The minor point is that they make images better.  The major one is that they make better processing possible.

 

That's what some very experienced and very talented people have been trying to explain here.  It appears to have been futile in some ways, although the higher level discussions have been quite interesting.
 

Obviously you did not read my posts. Neither did I say calibration frames are useless neither did I say they are not applicable in every situation. This is the issue... you are ‘attacking’ not discussing when you put words in my mouth and create arguments I am not making. 

 

90% of the time most of us are imaging from light polluted locations. Even my club’s ‘dark site’ has enough LP to completely overwhelm any dark current terms from the 2600 even without calibration. 

And this is not about ‘talent’ it is about science and its application. I don’t have to be a ‘talented’ astrophotographer (whatever that means) to understand this.

Edited by Astrojedi, 28 December 2020 - 11:46 AM.

[Astrojedi]

Interestingly, the master dark I posted was from the ASI2600MC PRO.  Also I don't see the relevance of CDS to DCNU.

 

Mark

Relevance of CDS is to FPN as I mentioned above.
 

 

Edited by Astrojedi, 28 December 2020 - 11:44 AM.

[sn2006gy]

 

 

I say do what you like.  But don't say calibration frames are useless (as in the title).  That's silly.  As is saying there's no imformation in bias.  Or darks.  Or flats, for that matter.

 

I don't think anyone is claiming there was never any use for calibration.

 

Just that on his 533, he found calibration was useless... and not in a pejorative sense of useless.

[Astrojedi]

Quite obvious this is not really a nuanced debate... it is more about us vs. them. Which is quite ridiculous. The theory is all fine. I understand it just as well as anyone here... why don’t we apply it to the real world.

 

Let me ask the question to the experts in a different way.

 

Let’s say I am imaging from my backyard SQM 18.50 - 19.00 with the 2600MC. How much total contribution will the dark current have relative to LP and other unwanted signals? Noise contribution vs. other sources? Do you think it is material?

 

Lets try the above from a SQM 21.4 site. How much does the 2600 dark current matter?

[sharkmelley]

Relevance of CDS is to FPN as I mentioned above.

So you are making a distinction between FPN and DCNU (not everyone in this thread does).

 

BTW, if you want something interesting (but totally off topic) to investigate, why does the FFT of the ASI2600MC PRO master dark look like this:

 

 

Mark

[bobzeq25]

And this is not about ‘talent’ it is about science and its application.

And that's where we disagree.  In judging whether or not a gap is important, experience and talent (surrogates for standards) matter a great deal.

 

There is no agreed on scientific standard for when a gap is small here.  No agreed on scientific standard on what level of noise is significant in astrophotography.

 

This is an excellent read.  It discusses the nature of noise in amateur astrophotography.  And gives some excellent methods (I've never seen better) for reducing noise, both in data acquisition and data processing. 

 

Thing is, they're not for everyone.  Some people will find them too much work for too little benefit.  _For them_.  Among other things, they absolutely require the sophisticated processing methods found in PixInsight.  So step one in applying the processing techniques is to learn PixInsight.  <smile>

 

I've talked about doing astrophotography badly.  This is what doing it well looks like.  Again, it's not for everyone.

 

https://jonrista.com...duction-part-1/

Edited by bobzeq25, 28 December 2020 - 12:04 PM.

[sn2006gy]

 

 

90% of the time most of us are imaging from light polluted locations. Even my club’s ‘dark site’ has enough LP to completely overwhelm any dark current terms from the 2600 even without calibration. 

 

On the 533/2600/6200 "swamping the sensor" happens in seconds with broadband and even much quicker than most people realize with NB filters. My 2600 is in B1/B2 skies and to target dynamic range my exposures are shorter than most. To me, the "noise" from over exposing is exponentially more detrimental and absolutely visible in my master lights than any dark current/FPN so it fascinates me to no end to focus on dark calibration...

 

obviously i'm biased towards my experience on OSC but i think that's a good metric because "things are only better with mono" as they say around here

[sn2006gy]

So you are making a distinction between FPN and DCNU (not everyone in this thread does).

 

BTW, if you want something interesting (but totally off topic) to investigate, why does the FFT of the ASI2600MC PRO master dark look like this:

 

ASI2600MCPRO.png

 

Mark

 

Good question - how did you generate this? I'm curious.

 

AND.. that looks a lot more like the pattern I see "smashed all over the place" in my rejection subs.

 

Also, does anyone have this from other cameras?  And how would dark calibration even help here? 

[bobzeq25]

I don't think anyone is claiming there was never any use for calibration.

 

Just that on his 533, he found calibration was useless... and not in a pejorative sense of useless.

And then suggested "we" should only do lights.  That's incorrect.

[Astrojedi]

We need to stop being so dogmatic about calibration.

 

While yes I agree it makes sense for most sensors and in most situations we should have the intellectual flexibility to discuss instances when it is acceptable not to use a particular calibration frame.

 

I almost always use calibration frames. But I see sufficient evidence that darks may not be very useful with the 2600MC when imaging from typical skies.

 

There is nothing heretical here or anything inconsistent with the imaging theory being regurgitated ad nauseam.

[sn2006gy]

And then suggested "we" should only do lights.  That's incorrect.

 

Nah... Put on someone else's thinking camp for a minute rather than how your path to success was shaped.

 

The 533 is a beautiful modern sensor - very easy, very forgiving, very new person friendly. It's square, it's small.  It frames easily, It doesn't show field curvature much at all because its so tiny and it fits many scopes/optics because of these characteristics.  It's also cooled, It has no amp glow to speak of, it has very little (negligible) dark current.

 

If we want to espouse the fun, utility and enjoyment of a hobby then why not welcome people to use the 533 and not overly concern them about calibration to get started- with examples to boot?  Most people buying the 533 would be thrilled to just start imaging. It yields an image without calibration that is beautiful and easy to work with and can be fine tuned with proper calibration as experience builds...

 

The number one cited reason of failure in every new person I've helped to this hobby is improper calibration...

 

A bad flat looks much worse than wondering what a dust most looks like... and if someone learns to dither motes may not survive. Ditto with donuts.. Similarly, a light leaked dark because someone said to take darks will spoil worse than uncalibrated - on these particular sensors.

 

This isn't an excuse to NOT calibrate and IGNORE SNR or any of that - but mostly to remind us that we have new tech and new people to the hobby may not have to do things we did and that's not a bad thing.

 

Many people starting with the 533 won't have as much complexity to worry about either... so simplicity through and through has its merits for enjoyment and optimization/tweaking can come with experience rather than mandate.

 

Now, if you chose to run a 183 or even a 1600 - you absolutely have to dark calibrate period or you will have massive amp glow and pattern noise in your master light images no matter what.

Edited by sn2006gy, 28 December 2020 - 12:26 PM.

[Astrojedi]

And that's where we disagree.  In judging whether or not a gap is important, experience and talent (surrogates for standards) matter a great deal.

 

There is no agreed on scientific standard for when a gap is small here.  No agreed on scientific standard on what level of noise is significant in astrophotography.

 

This is an excellent read.  It discusses the nature of noise in amateur astrophotography.  And gives some excellent methods (I've never seen better) for reducing noise, both in data acquisition and data processing. 

 

Thing is, they're not for everyone.  Some people will find them too much work for too little benefit.  _For them_.  Among other things, they absolutely require the sophisticated processing methods found in PixInsight.  So step one in applying the processing techniques is to learn PixInsight.  <smile>

 

I've talked about doing astrophotography badly.  This is what doing it well looks like.  Again, it's not for everyone.

 

https://jonrista.com...duction-part-1/

Well you cannot keep making ‘scientific arguments’ and then suddenly flip to subjective criteria.

[endless-sky]

While yes I agree it makes sense for most sensors and in most situations we should have the intellectual flexibility to discuss instances when it is acceptable not to use a particular calibration frame.

Acceptable to who? Under whose standards?

 

We have been arguing for pages whether the calibration steps are useful, harmful or don't make any difference.

 

What baffles me is that whether those calibration frames gain you 50%, 20%, 5% or 1% better quality on your final picture, when it costs basically zero to do them (a cloudy afternoon/night, if you have a thermally controlled camera), why not doing them?

 

If we skip all the steps that are negligible (1-3% here, 2-5% there, etc.), by the end of all the skips they add up.

 

Are we after the best quality our skills / equipment can deliver us? I know I am.

 

Cyclists go to the point of paying hundreds/thousands of $ to get rid of a few grams on their bycicles. What are a few grams, when the whole bycicle weighs kilograms? Yet, they do it, if that gains them even a minimum advantage.

 

Calibrations are basically free, yet we are discussing if we should do them or not.

[Jon Rista]

I see it more as a logic problem.

 

And I originally wrote a huge diagram explaining the logic but in the end I just ended up asking myself the same question over and over.

 

My question for you, is at what point is something good enough?

 

I can't explain the logic problem until I know what you're implying.

 

No one said my 40 dark calibrated one was good. In fact, we found more issues with everything BUT my master lights

 

The rabbit hole went deeper.

 

So *IF* the 40 darks was enough, why did it then become about dithering more? Especially when the master lights show 0 symptoms of walking noise or not being calibrated. (if you can answer this question, then my logic question may become apparent)

 

I understand the math of SNR

 

I understand the reasons for dithering.

 

Please keep the context only based on images/subs/rejections I shared. I can't honestly answer this if experiences of non 533,2600,6200 cameras are thrown in.

I am not "implying" anything...I've been extremely explicit, with exact math, in my posts. There is no implication here. There is simple mathematical fact and clear, exact visual demonstrations.

Edited by Jon Rista, 28 December 2020 - 12:34 PM.

[Jon Rista]

We need to stop being so dogmatic about calibration.

 

While yes I agree it makes sense for most sensors and in most situations we should have the intellectual flexibility to discuss instances when it is acceptable not to use a particular calibration frame.

 

I almost always use calibration frames. But I see sufficient evidence that darks may not be very useful with the 2600MC when imaging from typical skies.

 

There is nothing heretical here or anything inconsistent with the imaging theory being regurgitated ad nauseam.

Whether you calibrate only with biases, or calibrate with darks...and of course calibrate with flats in either case...you MUST calibrate. This is not an option. If you do not calibrate, then the ESSENTIAL and critical normalization process that occurs during image integration to make all the frames optimally compatible with each other from a signal alignment and dispersion standpoint WILL NOT WORK PROPERLY!! 

 

Calibration is not an option. Period. Doesn't matter how good the technology is. 

 

Now, if the debate is just about whether for some particular sensor A, whether you "can" calibrate with "just" a master bias rather than a master dark...well, my you need to determine that on a sensor by sensor (and I'd say even camera by camera) basis. In some cases, you may need to characterize a sensor more than once over its lifetime to answer that question. If there is no notable difference in DFPN between a dark and a bias, then you may be able to calibrate with only a master bias. You will, however, you MUST, however, CALIBRATE! Otherwise, the rest of the math for integration will not work properly. Dithering alone is not going to remove the offsets added to each pixel by the camera circuitry. 

Edited by Jon Rista, 28 December 2020 - 12:35 PM.

[Jon Rista]

Interestingly, the master dark I posted was from the ASI2600MC PRO.  Also I don't see the relevance of CDS to DCNU.

 

Mark

As far as I know CDS only removes reset noise. I do not believe it has any effect on DCNU (or DSNU, dark signal non-uniformity, the term I've encountered more often).