49 replies to this topic

[sharkmelley]

Again, RTFM.

So to summarise this in a single sentence, your contention that bias is the least critical of all calibration frames was based on your misunderstanding of how your processing software works.

[galacticinsomnia]

Okay, I want to interrupt this program for just a moment.
sharkmelley, bq octantis, I respect both of your insights, but personally cannot correlate your points in a manner that is easy to understand.  Which is why I call it calibration voodoo, it is also why I do not promote calibration frames for new comers. It doesn't just automagically fix image issues.  Some people just want to capture images of space without all the added work for something that may or may not help.  When teaching others, the "hello world" approach is appropriate and applicable IMO. 

You have vinetting, then how about looking at the image train and eliminating the Vinetting instead of using software to remove the problem, if possible, that would be the best course of action.  For dust and noise, then dithering is quite applicable also, as well as having plenty of time on the target. 

With all the math, the numbers, etc etc and the explaination of what these frames do, adding subtracting, dividing, etc etc.  is enough to make a newcomers head spin. 

Perhaps, you MAY find an improvement by trying calibration frames is appropriate. 

Thank you both for your contributions, I enjoy reading and seeing your posts, as well as trying to learn what the heck you are talking about.. lol ! 

Clear Skies !!

 

 

[galacticinsomnia]

Stacking is not required either, or a dSLR, just  point a cell phone into an eyepiece and take one click......

The point is that calibration frames can improve your photos for an extremely minimal time cost.........

Exactly, they May or May Not, improve your image.

I love your work, respectfully.

Clear Skies !!

 

[sharkmelley]

You have vinetting, then how about looking at the image train and eliminating the Vinetting instead of using software to remove the problem, if possible, that would be the best course of action.  For dust and noise, then dithering is quite applicable also, as well as having plenty of time on the target. 

With all the math, the numbers, etc etc and the explaination of what these frames do, adding subtracting, dividing, etc etc.  is enough to make a newcomers head spin. 

An optical train will almost invariably exhibit vignetting so "removing the problem" is rarely possible.

 

No-one needs to understand the math because the calibration/stacking software takes care of the technicalities. Newcomers easily understand that the corners of an image tend to be darker than the centre and that dust near the sensor casts shadows.  It's then not difficult to understand that flats can compensate for this.

 

Mark

[galacticinsomnia]

An optical train will almost invariably exhibit vignetting so "removing the problem" is rarely possible.

 

No-one needs to understand the math because the calibration/stacking software takes care of the technicalities. Newcomers easily understand that the corners of an image tend to be darker than the centre and that dust near the sensor casts shadows.  It's then not difficult to understand that flats can compensate for this.

 

Mark

Saying NO ONE needs, is intellectually dishonest and kind of shows your bias (no punn intended).

How can you tell if the calibration and stacking software is doing things correctly?  Unless you understand what it is meaning to do, then you don't know what DATA it has eaten, and what data is preserved or modified with its estimations or appropriations. 

It is also not difficult to understand that there are other ways to remove gradients that do not require calibration, and are far easier to implement in post. 

Plenty of examples right here, and it is pretty insulting to say what I am doing is wrong, or that NO ONE needs. 
I'm here to learn, to help, motivate and encourage.  For newcomers having issues with thier images, calibration frames are not the first go to answer for me. 

https://www.cloudyni...-edph-ii-again/
https://www.cloudyni...wo-frame-panel/
https://www.cloudyni...ield-flattener/
https://www.cloudyni...rica-61edph-ii/
https://www.cloudyni...fid-61-edph-ii/
https://www.cloudyni...ity-in-cepheus/
https://www.cloudyni...822-in-cepheus/
https://www.cloudyni...bula-61-edphii/
https://www.cloudyni...bula-61-edphii/
https://www.cloudyni...bula-happy-4th/
https://www.cloudyni...ebula-ngc-6992/
https://www.cloudyni...us-first-light/
https://www.cloudyni...lephant-nebula/
https://www.cloudyni...ata-5hrs-total/
https://www.cloudyni...vbony-503-80ed/
https://www.cloudyni...1-bodes-galaxy/
https://www.cloudyni...ting-for-bodes/
https://www.cloudyni...nus-star-cloud/
https://www.cloudyni...acked-raw-data/
https://www.cloudyni...d-image-source/



Clear Skies !!

[sharkmelley]

Saying NO ONE needs, is intellectually dishonest and kind of shows your bias (no punn intended).

How can you tell if the calibration and stacking software is doing things correctly?  Unless you understand what it is meaning to do, then you don't know what DATA it has eaten, and what data is preserved or modified with its estimations or appropriations. 

I stand by what I said.  No-one needs to understand the math i.e. both newcomers and experienced imagers can use calibration/stacking software without needing to understand the mathematical foundations.  I have friends who are clueless about math but they but they understand intuitively that flats are able to remove the effects of vignetting and dust shadows.

 

Sure, the more a user understands the mathematical concepts, the better it is.  You call it "calibration voodoo" but I can assure you it is based on a solid mathematical principles and you shouldn't let a lack of understanding of those principles prevent you using it with confidence.  You will find that gradient removal becomes a whole lot easier once you start calibrating with flats.

 

Mark

Edited by sharkmelley, 30 December 2021 - 07:37 PM.

[galacticinsomnia]

I stand by what I said.  No-one needs to understand the math i.e. both newcomers and experienced imagers can use calibration/stacking software without needing to understand the mathematical foundations.  I have friends who are clueless about math but they but they understand intuitively that flats are able to remove the effects of vignetting and dust shadows.

 

Sure, the more a user understands the mathematical concepts, the better it is.  You call it "calibration voodoo" but I can assure you it is based on a solid mathematical principles and you shouldn't let a lack of understanding of those principles prevent you using it with confidence.  You will find that gradient removal becomes a whole lot easier once you start calibrating with flats.

 

Mark

I call it calibration voodoo because regardless of mathmatic principles in which it is based, the algorythms employed in various software produce different outcomes, obviously.  IE a stack from Siril is different from a stack DSS, as is different from Sequator, APP, Pixinsight, Affinity Photo, etc. 
I call it voodoo, because regardless of how things are supposed to work, it hasn't been so in my experiences.

As I have said previously, I do take flats and bias, and rarely do I find they make any difference in my final output, I'm not saying they can't, I'm not saying they are invalid, I'm saying it's not my go to response.

Clear Skies !!

[sharkmelley]

I call it calibration voodoo because regardless of mathmatic principles in which it is based, the algorythms employed in various software produce different outcomes, obviously.  IE a stack from Siril is different from a stack DSS, as is different from Sequator, APP, Pixinsight, Affinity Photo, etc. 

If you are avoiding calibration because you don't fully understand what's going on in the software then you definitely want to avoid stacking because it does additional "voodoo" things such as demosaicing, star alignment and integration.

[BQ Octantis]

So to summarise this in a single sentence, your contention that bias is the least critical of all calibration frames was based on your misunderstanding of how your processing software works.

Sure Mark. We are all victims of our own successes and failures—it's called experiential learning. And my success has been workflows—without bias frames or flat darks—that produce eye-watering images that are limited by the quality of my lenses and the sensitivity of my sensors. But you are correct that I shouldn't extrapolate that to all lenses and all sensors. (And certainly not all processing software—RTFM!)

 

Which really begs something more fundamental about the processing "voodoo": is my experience of omitting bias frames unique to Canon raw files? I've processed RAW files from the 10D (CRW), 450D (CR2), 600D (CR2), 650D (CR2), and 850D (CR3) without issue. But Canon has included the bias value for the camera in the raw file from the beginning. Is that simply not an option for non-Canon DSLRs?

 

[sharkmelley]

Which really begs something more fundamental about the processing "voodoo": is my experience of omitting bias frames unique to Canon raw files? I've processed RAW files from the 10D (CRW), 450D (CR2), 600D (CR2), 650D (CR2), and 850D (CR3) without issue. But Canon has included the bias value for the camera in the raw file from the beginning. Is that simply not an option for non-Canon DSLRs?

It depends on what you use to decode your raw files.  You said you are currently using RawTherapee.  Now RawTherapee, LightRoom and Photoshop's ACR all subtract the bias level as part of their processing and the bias level is picked up from the EXIF header.  So if you are using TIFFS or JPEGs created by such applications then the bias level has already been subtracted.  This very fact demonstrates the importance of the bias level.

[BQ Octantis]

This very fact demonstrates the importance of the bias level.

 

Agreed. But it does not demonstrate the importance of bias frames, which was the OP's question and the source of the debate.

 

Indeed, the calculus (and the measured values from my 600D) shows that subtracting the average bias level is sufficient to <1% luminosity error when applying the flat.

 

Which again begs, are bias frames required for cameras other than Canons? And—possibly more importantly to the OP—for which AP software are they required?

Edited by BQ Octantis, 31 December 2021 - 10:52 AM.

[sharkmelley]

Which again begs, are bias frames required for cameras other than Canons?

There is no difference for Canons because, in general, all DSLR/Mirrorless cameras have non-zero bias levels, although some early Nikons had a bias level of zero.  So apart from the cameras with zero bias, all cameras need bias.  The bias can be provided using bias frames but in some cases a constant level is sufficient (as suggested here for Siril). 

As already mentioned, RawTherapee and Adobe's ACR also use a constant level but it's automatically obtained from the EXIF of the raw file and not provided by the user.  But a constant level does not provide an adequate calibration in all cases.  For instance the bias frames of my Sony A7S have amp glow at the edges of the frame and so using a constant level is really not a good substitute.

 

The bottom line is that the bias is critical but sometimes you can get away with using a constant instead of actual bias frames.

[Michael Covington]

In order to do the arithmetic correctly to apply flats, there must be either bias frames or flat darks.  They supply an important number for the calculation.  This applies to all cameras.  

[spazmore]

All of this back and forth about bias frames can really confuse and put off a beginner.

 

Here's what I'll say - it takes me around one minute to take almost 100 bias frames (no shutter wear either if I switch to electronic shutter only) on my Sony A6300, then around an extra 30 seconds for the bias frames to transfer to my computer and to copy them to the "biases" folder for SiriL to use. So we're talking about 2 extra minutes at the end of an imaging session here, instead of half an hour or more of reading and math and frustration, to determine whether or not they're useful to you.

[BQ Octantis]

The bottom line is that the bias is critical but sometimes you can get away with using a constant instead of actual bias frames.

 

Possibly the most insightful point in this thread so far.

 

So bias frames may be critical; or they may be entirely unnecessary.

 

No wonder they're controversial!

Edited by BQ Octantis, 31 December 2021 - 01:34 PM.

[galacticinsomnia]

Possibly the most insightful point in this thread so far.

 

So bias frames may be critical; or they may be entirely unnecessary.

 

No wonder they're controversial!

Truth ! 

There is nothing controversial in uncertainty, it just is...like bias frames lol

Clear Skies !! 

[galacticinsomnia]

If you are avoiding calibration because you don't fully understand what's going on in the software then you definitely want to avoid stacking because it does additional "voodoo" things such as demosaicing, star alignment and integration.

I'm really hoping more people read this thread.  The more I read about the "it depends" scenarios.  Software vs's camera sensor, vs aquistion details, vs temp. etc etc etc,  and then have the audacity to try and belittle others and blame them when methods that are dogmatic in their promotion don't work, welll...it's not appropriate and not helpful.

So for you newcomers, feel free to push back.  While the idea that if it doesn't work for you, then you are doing it wrong, can be a correct assessment, it can also be an incorrect one, and chasing unicorns will leave you unsatisfied.

Calibration frames....
They may or may not work for you, don't get brow beat into using them, they are not a solution to all of your problems.

As you evolve your process you will discover when, where, and how to maybe have them work for your benefit and possibly improve your images.

Personally, I'm currently imaging about 80% of the time with NO calibration frames, and about 20% with bias and flats, and found the best time to use them is when my enviroment is having varying light pollution, but YMMV.

You can image without them.  !

Clear Skies !!

[rgsalinger]

What I would have said is: "If you don't think that calibration frames will give you better data, run a simple experiment and see what happens".

 

I did this a while back and all of my CMOS cameras benefitted from them. With an uncooled DSLR the outcome might be different for a number of reasons.

 

I remember hearing Tony Halas say a long time ago that with DSLR's he found that dithering eliminated the need to take darks.

 

So, if you asked me I would say definitely yes take them, but if it's difficult to do so (I use sky flats) experiment and see what you find with your equipment. 

 

Rgrds-Ross

[galacticinsomnia]

What I would have said is: "If you don't think that calibration frames will give you better data, run a simple experiment and see what happens".

 

I did this a while back and all of my CMOS cameras benefitted from them. With an uncooled DSLR the outcome might be different for a number of reasons.

 

I remember hearing Tony Halas say a long time ago that with DSLR's he found that dithering eliminated the need to take darks.

 

So, if you asked me I would say definitely yes take them, but if it's difficult to do so (I use sky flats) experiment and see what you find with your equipment. 

 

Rgrds-Ross

As you stated previously, it can be off putting.  Keeping new comers engaged and encouraging the climb to improvement in aquisition and post process work, is why I suggest what I do.  There is nothing like a try it and see approach, but as you can tell in the many threads regarding, DFB frames, there are so many variables, it can indeed discourage many a user, and it seems like it would be wise to consider alternatives that will encourage future growth, excitement and possibly heaven forbid some joy in the results of the work, even if it goes against the grain of what some users promote.  Calibration frames to me, are just another method, they are not a necessity to imaging, YMMV. 
 
Clear Skies !!



 

[17.5Dob]

In my experience, using Nikon's, and a plethora of OTA's, calibration frames are necessary to get the "best" results. You could certainly skip them, but the final results will suffer moderately to seriously. This hobby is built on trying to control myriad small problems, and usually, when someone is trying to shortcut one thing, that mindset carries on to others.

[rgsalinger]

Well, if I had it to do "all over again" the first thing I would do is buy a book - the Deep Sky Imaging Primer - which I think has a very clear exposition about pretty much all the basics of astro-photography and join a local club if you can find one that's active in your area. At my club's dark sky site there's always someone out there with a DSLR, pretty much all the common mounts, etc when we have a clear members only Saturday night. 

 

Rgrds-Ross

[Mike in Rancho]

Interesting discussion.  If a middle ground can be found, I think it might be that what each calibration frame does should be explained to newcomers, rather than simply declared as either "mandatory" or "not needed."

 

The Bracken Primer is indeed a good source for this, but it can get more complicated and custom to each particular setup and/or software suite.

 

When I started a bit over year ago, I took every type of calibration frame and threw them all into the stacker -- because I thought I had to.  Only later did I learn to not use bias and dark flats at the same time, and darks themselves are another story, particularly with DSLR.

 

That said, I have also downloaded countless uncalibrated beginner stacks from CN in order to evaluate "what's there," and most of them are train wrecks.  Artificial flatting and other techniques in post can only do so much, and can also cause detail to be lost.

 

And while their purpose should be explained, including that they should be used only if needed to achieve that purpose, most beginners probably aren't going to be using the type of software BQ has with default bias subtraction.  They are more likely to be using DSS or other astro software, and probably have optical systems that show significant vignetting and/or sensor dust.  You can't beat actual flats for that.  And actual flats then (in most real world cases) require bias or dark flats.  I have tried lights/flats only in DSS, and the results are not pretty.

 

To me, darks are probably what would be considered the most "maybe/maybe not" calibration frame for DSLR.  I have D5300s, and Mark of course has an entire thread on the partial futility of D5300 darks.  So, I dither, and bias is used for both sides of the calibration formula.  Rarely, and though I have to get lucky for them to work right, I will still attempt darks when the ambient temps are hot enough such that I get some sensor edge glow.

 

The Elf was also doing an experiment on whether and how much darks helped with his Canons, but I'm not sure he ever finished it?

[BQ Octantis]

Well I, for one, am grateful for well-resourced geniuses like Mark who have sacrificed their time and money to test and publish their results with a wide range of sensors. While each of us has a completely valid perspective—optimized for our unique experience with our respective use cases—when those geniuses apply their experience, the value of these forums is increased exponentially by their much wider aperture of observations.

 

Thanks, Mark. And Happy New Year!

 

BQ

Edited by BQ Octantis, 31 December 2021 - 10:46 PM.

[galacticinsomnia]

Well I, for one, am grateful for well-resourced geniuses like Mark who have sacrificed their time and money to test and publish their results with a wide range of sensors. While each of us has a completely valid perspective—optimized for our unique experience with our respective use cases—when those geniuses apply their experience, the value of these forums is increased exponentially by their much wider aperture of observations.

 

Thanks, Mark. And Happy New Year!

 

BQ

I couldn't agree more. 

Here here..  Cheers !!

Clear Skies !!

[Feldhusen]

What I would have said is: "If you don't think that calibration frames will give you better data, run a simple experiment and see what happens".

 

I did this a while back and all of my CMOS cameras benefitted from them. With an uncooled DSLR the outcome might be different for a number of reasons.

 

I remember hearing Tony Halas say a long time ago that with DSLR's he found that dithering eliminated the need to take darks.

 

So, if you asked me I would say definitely yes take them, but if it's difficult to do so (I use sky flats) experiment and see what you find with your equipment. 

 

Rgrds-Ross

Guess none here deny that calibration frames have merit. Try for yourself is a good strategy.

 

My personal experience (living in a cold climate) is that calibration frames are not needed if one 1) use a lens or telescope with very moderate vignetting, and 2) have no dust motes to fight, and 3) find software correction after the fact sufficient.

 

I do not say that calibration frames are unneccessary. I just say that calibration frames can be skipped if...

 

To correct all but moderate vignetting flats seems to be the obvious way to go, accompanied by bias frames.

 

Have now skipped dedicated astro cameras and am into modded (RP) and unmodded (P) Canon cameras and even turn some of my Sony cameras skywards just to figure out. Lousy conditions and easy access to astrophotography is my drive now. Am happy to have lenses and telescopes with very moderate vignetting.

 

My stacking is done in AstroArt and post processing is done in Photoshop so my routines obviously deviate from what is common practice here - ordinary photography is my main area of interest.

 

Guess it it time to try out flat + bias frames.

 

Will most likely skip them anyway, but that is me, not you! 

 

Find this thread interesting. Treasures are hidden in the noise between 'calibrated' and 'non calibrated' ones.

Edited by Feldhusen, 01 January 2022 - 09:05 AM.