Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Flat Calibration

  • Please log in to reply
45 replies to this topic

#26 bulrichl

bulrichl

    Mariner 2

  • -----
  • Posts: 233
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 20 March 2020 - 07:48 AM

Hi Larry,

 

Actually everything was explained in this thread. In summary:

 

Normally you would expose bias frames as short as possible. However, if I understand it correctly, a fundamental problem of bias frames of the Panasonic MN 34230 sensor is: frames that are captured with an exposure time of < about 0.3 s have a median that does not match the median of shortly exposed dark frames. This could pose a problem in the image calibration. IIRC Jon Rista recommended an exposure time of 0.3 s for bias frames of this sensor.

 

Independent of the above mentioned "bias problem" of the MN 34230, one should not use scaling of the MasterDark (in PixInsight: dark frame optimization) with sensors that exhibit amp glow. The MN 34230 is part of this group of sensors. Not using scaling of the MasterDark means: you don't need a MasterBias for the light frame calibration. However, you definitely need a master calibration file for the calibration of the flat frames: either a MasterBias or a MasterFlatDark. In your case, in which the exposure time for the flat frames is < 1 s, the difference between these two will be negligible. So it will make no difference whether you use a MasterBias or a MasterFlatDark for the calibration of the flat frames.

 

Besides, one can always try both approaches of flat frame calibration and compare the results...

 

Bernd

 


  • bobzeq25 and Hawkdl2 like this

#27 Hawkdl2

Hawkdl2

    Viking 1

  • *****
  • Posts: 705
  • Joined: 22 Feb 2015
  • Loc: Claremont, CA

Posted 20 March 2020 - 08:55 AM

Thanks.  Yes most of this I gleaned form the thread.  The uncertainty is what constitutes "short duration flats" vs. "longer duration flats".   While I am likely to move to flat darks in the future,  I'm working on a lot of data form last year with flats that I no longer have the original flat frames - just the integrated flat master, for each night.  Those were all from less than 1 sec exposures (some as short as 0.12 sec) and calibrated "old school" with bias frames.  If "less than 1 sec", and was suggested above as recommended by some CMOS camera manufactures is "short enough", then I should be fine (even though in the future I might instead use a dark flat.  

 

A lot of people report needed NB flat exposures of >5 seconds or more.  I don't see that.  With my Amazon flat panel on bright, my flat exposures are nearly always between 0.1 and 0.8 sec., and seem to work very well.  

 

I have always bias subtracted flats and never scale my darks (I've always used a temp/gain/time-matched dark library), but I have always included the bias frame in the light calibration step.  I suspect I might have been double subtracting the bias since my flats were previously bias-substracted.  

 

I don't know if I can do this, but I'm going to try dark-flat subtracting my existing integrated bias-calibrated flat frames.  In theory, if the bias frames used at the time were under representing the bias.  If that over-calibrates the flat masters, I might try adding the original bias back.  I don't know what that will do but it might be an interesting experiment.



#28 bobzeq25

bobzeq25

    Hubble

  • *****
  • Posts: 19,040
  • Joined: 27 Oct 2014

Posted 20 March 2020 - 11:32 AM

Thanks.  Yes most of this I gleaned form the thread.  The uncertainty is what constitutes "short duration flats" vs. "longer duration flats".   While I am likely to move to flat darks in the future,  I'm working on a lot of data form last year with flats that I no longer have the original flat frames - just the integrated flat master, for each night.  Those were all from less than 1 sec exposures (some as short as 0.12 sec) and calibrated "old school" with bias frames.  If "less than 1 sec", and was suggested above as recommended by some CMOS camera manufactures is "short enough", then I should be fine (even though in the future I might instead use a dark flat.  

 

A lot of people report needed NB flat exposures of >5 seconds or more.  I don't see that.  With my Amazon flat panel on bright, my flat exposures are nearly always between 0.1 and 0.8 sec., and seem to work very well.  

 

I have always bias subtracted flats and never scale my darks (I've always used a temp/gain/time-matched dark library), but I have always included the bias frame in the light calibration step.  I suspect I might have been double subtracting the bias since my flats were previously bias-substracted.  

 

I don't know if I can do this, but I'm going to try dark-flat subtracting my existing integrated bias-calibrated flat frames.  In theory, if the bias frames used at the time were under representing the bias.  If that over-calibrates the flat masters, I might try adding the original bias back.  I don't know what that will do but it might be an interesting experiment.

Instead of floundering around.

 

Get "Inside PixInsight", he explains how to avoid double subtracting bias.

 

Look at these workflows for DSS, very educational.

 

http://deepskystacke...lish/theory.htm

 

Sometimes in AP it's good to do some experimenting, see what works.  Sometimes it's better to understand the theory.  In my opinion, image calibration definitely falls into the latter category.  <smile>

 

 



#29 tleroy1

tleroy1

    Vostok 1

  • -----
  • Posts: 174
  • Joined: 04 May 2018
  • Loc: Georgetown, Ontario, Canada

Posted 20 March 2020 - 12:25 PM

I must admit, this entire thread is pretty much greek to me, but it's great to get some exposure to the topic and see some things to research and learn.

 

I've read http://www.elf-of-lo...ingStarted.html and will read the books referenced there too.

 

Thanks to all who contribute and build our knowledge.



#30 Hawkdl2

Hawkdl2

    Viking 1

  • *****
  • Posts: 705
  • Joined: 22 Feb 2015
  • Loc: Claremont, CA

Posted 20 March 2020 - 12:44 PM

Bob, Thanks for all you help, but I reserve the right to flounder at will!  

 

Thanks for all your help.  However, the fundamental question is when is a flat "short enough", for CMOS camera like the 1600 that a standard bias is sufficient.  



#31 ekallgren

ekallgren

    Viking 1

  • *****
  • Posts: 618
  • Joined: 23 Dec 2011
  • Loc: Northern Nevada, Reno

Posted 20 March 2020 - 03:52 PM

Hi Larry.

I'm jumping into this late. I think that you just put your finger on the issue with the 1600 is there is no "Standard" exposure for a bias frame. they can and do vary. 

Just mt 2 cents.



#32 zer0

zer0

    Lift Off

  • *****
  • Posts: 13
  • Joined: 03 Mar 2015
  • Loc: Tokyo

Posted 20 March 2020 - 11:55 PM

I have always bias subtracted flats and never scale my darks (I've always used a temp/gain/time-matched dark library), but I have always included the bias frame in the light calibration step.  I suspect I might have been double subtracting the bias since my flats were previously bias-substracted.  

Since you divide the lights by the calibrated flats and not subtract them, it is actually important to subtract bias from the flats, and it does not contribute to the double bias subtraction problem. Instead, this danger is with the darks: if you subtract the bias from the lights but not from the darks before subtracting those, the bias gets double subtracted from the original lights.

 

That's why you either

  • subtract bias from both the lights and darks separately, and then subtract bias-free darks from bias-free lights:
    ([light + bias + dark] - bias) - ([bias + dark] - bias) = (light + dark) - (dark) = light (before dividing by the calibrated flat master)
  • or don't subtract bias from either lights or darks, and then it gets subtracted from the lights as part of darks:
    (light + bias + dark) - (bias + dark) = light (again, before dividing by the calibrated flat master)

Note that in both cases above we still need the bias (or flat dark) frames for flat calibration: so that in the end we divide by bias-free flats.

 

I'm just repeating what spokeshave's written in his post #15 of this thread (https://www.cloudyni...ion/?p=10050922), it's very clear.

 

Bob, Thanks for all you help, but I reserve the right to flounder at will!  

 

Thanks for all your help.  However, the fundamental question is when is a flat "short enough", for CMOS camera like the 1600 that a standard bias is sufficient.  

I assume this can be answered for every sensor by taking darks at various short exposures and statistically comparing them with bias. Although it seems to me this is where the mentioned Poisson noise may come into play obscuring the results. I need to read on it somewhere.


  • bobzeq25 likes this

#33 Hawkdl2

Hawkdl2

    Viking 1

  • *****
  • Posts: 705
  • Joined: 22 Feb 2015
  • Loc: Claremont, CA

Posted 21 March 2020 - 10:30 AM

Hi Larry.

I'm jumping into this late. I think that you just put your finger on the issue with the 1600 is there is no "Standard" exposure for a bias frame. they can and do vary. 

Just mt 2 cents.

I agree that this is the issue and for me, I just became aware of it a few weeks ago - previously blissfully thinking a simple "standard" bias was the way to go.  It's possible Jon Rista or someone else has posted on this, but I haven't seen an evaluation of the 1600 bias characteristics at very short exposures from essentially 0 to 1 sec and if and where its purported "odd" bias characteristics manifest.  I plan to work on this today.  If I can make the comparison comparison competently, I'll post the results.  If someone has already done this, I would be grateful for a link. 

 

Since you divide the lights by the calibrated flats and not subtract them, it is actually important to subtract bias from the flats, and it does not contribute to the double bias subtraction problem. Instead, this danger is with the darks: if you subtract the bias from the lights but not from the darks before subtracting those, the bias gets double subtracted from the original lights.

 

That's why you either

  • subtract bias from both the lights and darks separately, and then subtract bias-free darks from bias-free lights:
    ([light + bias + dark] - bias) - ([bias + dark] - bias) = (light + dark) - (dark) = light (before dividing by the calibrated flat master)
  • or don't subtract bias from either lights or darks, and then it gets subtracted from the lights as part of darks:
    (light + bias + dark) - (bias + dark) = light (again, before dividing by the calibrated flat master)

Note that in both cases above we still need the bias (or flat dark) frames for flat calibration: so that in the end we divide by bias-free flats.

 

I'm just repeating what spokeshave's written in his post #15 of this thread (https://www.cloudyni...ion/?p=10050922), it's very clear.

 

I assume this can be answered for every sensor by taking darks at various short exposures and statistically comparing them with bias. Although it seems to me this is where the mentioned Poisson noise may come into play obscuring the results. I need to read on it somewhere.

I agree, spokeshave's write-up is probably the best and most clear I've seen.  I've have pretty much always used the approach that he describes as:

 

(Light-Dark)/(flat-bias)   At least that what I've thought I was doing!

 

My lights and darks are not bias-subtracted, but flats are bias-subtracted, so I've had that right notwithstanding the issue of bias vs. flat dark for the asi1600 sensor.  However, I've always added a bias file in PI's ImageCalibration dialogue box process when I calibrate lights.  I got this from Light Vortex, or one of the other web sites that were very popular a couple of years ago (maybe still are?).  But I have no idea what PI is doing with that bias frame. 

 

With these discussion in mind, I tried re-processing the data I'm currently working on without adding the bias frame: adding only raw light frames and an uncalibrated dark master and a bias-subtracted master flat.   There was virtually no difference visually or in measured noise using NoiseEvaluation. Is there another way to assess possible differences in the two approaches?   Having previously added a bias master in that dialogue box is why I've begin to wonder if I was double subtracting bias somewhere along the line. 

 

As for the difference between a bias and a ~0.5sec flat-dark on the ASI1600mm, your suggestion is spot on and exactly what I have planned to experiment with today.  I've not seen anyone directly address this.  Jon Rista has posted a fair amount on the subject, but I haven't seen a direct evaluation of the bias characteristics of this sensor.    



#34 rgsalinger

rgsalinger

    Fly Me to the Moon

  • *****
  • Moderators
  • Posts: 5,788
  • Joined: 19 Feb 2007
  • Loc: Carlsbad Ca

Posted 21 March 2020 - 11:13 AM

My own view is that when my flats get down to around 1 second, I'm not bothering with dark flats. Someday, my processing skills might be sufficient to tell the difference but I'm not there yet. I just load up my darks, biases and flats in PI and let it handle all of the issues for me. Seems to work just fine. Now, I do get that if my flats were 1 minute long, then there's a problem. So, I'd eat my hat if there was much of a difference if you can keep your flats at under 2-3 seconds. 

Rgrds-Ross



#35 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2,203
  • Joined: 08 Apr 2015

Posted 21 March 2020 - 12:54 PM

Bob, Thanks for all you help, but I reserve the right to flounder at will!  

 

Thanks for all your help.  However, the fundamental question is when is a flat "short enough", for CMOS camera like the 1600 that a standard bias is sufficient.  

For the QHY163M (the camera I use) I believe that QHY advises to take flat and bias frames longer than 0.3s. I generally consider anything shorter than 1s to be a "bias" (i.e. accumulated thermal fixed pattern noise can be ignored) and anything longer to be a "flat dark" and I generally try to make my flats no shorter than 1s - which can be a challenge for LRGB flats. I would probably be OK with 0.3s or 0.5s flats. 

 

For a cooled camera, this is probably fine since thermal fixed pattern noise is smaller the lower the temperature. 

 

Tim


  • bobzeq25 likes this

#36 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 21 March 2020 - 02:48 PM

I'm not ashamed to admit that a large port of this thread went straight over my head due to a lack experience.

 

So may i ask a few questions in layman terms based on my camera being a CCD

 

 

Q1) Before making my master Dark - Do i calibrate the darks by subtracting the master bias  or do i just stack all the darks without subtracting the Bias ?

 

 

Q2)  Now i guess this question depends on the answer to Q1, but when calibrating my flats - Do i subtract the Bias from the flats ?

 

Would be grateful if the replies could be kept simple as possible without lots of jargon or a lesson in Physics lol.gif

 

Cheers,

Rich.,



#37 ekallgren

ekallgren

    Viking 1

  • *****
  • Posts: 618
  • Joined: 23 Dec 2011
  • Loc: Northern Nevada, Reno

Posted 21 March 2020 - 03:02 PM

Q1)  Yes subtract the bias from all the dark frames then stack them.

 

Q2)  The flat frames need to calibrated  individually with the master bias and the master dark. 

 

I hope this is plane enough lol.gif  



#38 bulrichl

bulrichl

    Mariner 2

  • -----
  • Posts: 233
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 21 March 2020 - 05:53 PM

Q1)  Yes subtract the bias from all the dark frames then stack them.

Argh! No, NEVER calibrate your dark frames. Stack them as they come out of the camera.

 

I explained the reason for this recommendation as well in my guide, https://pixinsight.c...ibration.11547/

 

This will be my last contribution to this apparently never ending thread. Have fun,

 

Bernd



#39 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 21 March 2020 - 06:40 PM

Bend,

 

Thank you for you contribution this far. In reference to your guide is a little confusing as the guide assumes that calibration of darks and flats is done as per [1] yet your guide doesn't explain that is reference to sources at the end of the documen, so for me I found it very confusing as I reread section 1 general settings and could find no reference to calibration and it was only when I scrolled to the end of the document that I noticed the link.

 

Please not that this is not a criticism, merely an observation to help beginners use your guide more easily..

 

Rich.



#40 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 21 March 2020 - 07:00 PM

Bernd,

 

I was just reading through source [1] link from your guide and in the opening paragraph from section on bias and dark integration written by Vincent peris it states the following:-

 

"So we'll run the integration twice: first for the bias frames and then again for the dark frames (bias subtraction of dark frames will take place with the IC tool)"

 

 

Argh! No, NEVER calibrate your dark frames. Stack them as they come out of the camera.

 

This seems to suggest that darks should be calibrated and have bias subtracted which seems to counter your advice on not calibrating dark files.



#41 ekallgren

ekallgren

    Viking 1

  • *****
  • Posts: 618
  • Joined: 23 Dec 2011
  • Loc: Northern Nevada, Reno

Posted 21 March 2020 - 10:40 PM

The way I understand it it can do it 1 of 2 ways in Pixinsight.

 

1. You can Stack the Darks as they come out of the camera and add the Master Dark to Image Calibration and check the Calibrate option (in Master Dark) when calibrating Flats and Lights and   leave Optimize unchecked.

Or 

2. You can Subtract the Master Bias from the Individual Darks using Image Calibration and not check the Calibrate option When Calibrating the Flats and Lights in Image Calibration.

With my results it makes no difference. YMMV.



#42 ekallgren

ekallgren

    Viking 1

  • *****
  • Posts: 618
  • Joined: 23 Dec 2011
  • Loc: Northern Nevada, Reno

Posted 21 March 2020 - 10:48 PM

This is my Pixinsight workflow:

 

1. Stack bias frames to produce a master bias. in Image Integration
2. Calibrate dark frames with master bias. Stack these dark frames to produce a master dark. in Image Integration
3. Calibrate flat frames with master bias and master dark flat. Stack these flat frames to produce a master flat. in Image Calibration
4. Calibrate the light frames with the master bias, and master flat. in Image Calibration
5. Apply cosmetic correction to the calibrated light frames to stack. in CC
6. Select the best light frames and optimize their weightings.
7. Register the light frames to stack with each other. in Star Alignment
8. Stack the light frames. in Image Integration
9. Apply the Drizzle algorithm (Optional)



#43 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 22 March 2020 - 02:56 AM

The way I understand it it can do it 1 of 2 ways in Pixinsight.

 

1. You can Stack the Darks as they come out of the camera and add the Master Dark to Image Calibration and check the Calibrate option (in Master Dark) when calibrating Flats and Lights and   leave Optimize unchecked.

Or 

2. You can Subtract the Master Bias from the Individual Darks using Image Calibration and not check the Calibrate option When Calibrating the Flats and Lights in Image Calibration.

With my results it makes no difference. YMMV.

 

Thanks Eric,

That helps to clears things up for me and helps me to understand the process and i think the bit i was getting hung up on was the difference between the steps where using the optimize option.

 

Cheers,

Rich



#44 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 26 March 2020 - 09:39 AM

Hi All,

 

After going round in circles and getting tied up in knots worrying about manual calibration, i decided to give BPP tocreate my calibration masters and to calibrate my lights.  In the past i had always been warned to stay away from BPP, but now i wish i had been using it earlier. IT makes life so much easier - you just feed it your raw bias, darks & flats and let it worry about the calibration process.

 

So thanks for everyone's help and for people suggesting that i should give BPP a try

 

So i no longer worry about calibration and now just spend my time actually processing images rather than going round in circles worrying about calibration.

 

Here is a HA test result of BPP calibration - Thie image has had no post processing (i.e. no DBE etc) and has only been Drizzle integrated & stretched to export it to a jpeg file.  I'm pretty happy with the results as the image seems pretty clean even before processing.

 

Cheers,

Rich.

 

49701193967_3e74e5ad8d_h.jpg



#45 Madratter

Madratter

    Voyager 1

  • *****
  • Posts: 12,009
  • Joined: 14 Jan 2013

Posted 26 March 2020 - 10:19 AM

I often use WBPP with good results. They do warn you that the integration phase is likely to be suboptimal and I have found that to be the case. I can often do better than the result that WBPP gives.



#46 ribuck

ribuck

    Viking 1

  • -----
  • topic starter
  • Posts: 633
  • Joined: 01 Nov 2005
  • Loc: Newcastle, England

Posted 26 March 2020 - 12:35 PM

Hey Madratter,

 

It's only the calibration i'm doing with the batch script - I'm doing integration manually as i was always warned not to use BPP for integration as it gives less than optimal results as you have mentioned. But for the creation of calibration masters as well as calibration of lights it seem to do a cracking job.

 

Rich




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics