Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Flat frames rarely work in Nebulosity

  • Please log in to reply
28 replies to this topic

#1 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 09 February 2018 - 06:22 PM

This happens nearly all the time:

I try so hard to get a good set of flat frames, immediately after imaging my light frames, without disturbing anything.

The dust motes are clearly seen in the master flat, yet when applied, it does nothing or makes it worse.

The vignetting is still there too.

 

I had 30 dark frames

I had 30 flat frames, about 1/2 histogram, done immediately after imaging

I had a stack of bias frames, done at the same ISO as the flats, where they are applied.

 

I applied these to 32 light frames.

 

Any help is appreciated

Attached Thumbnails

  • dust_mote_flat.JPG
  • dust_mote_final.JPG
  • NEB4_Preproc_set.JPG

Edited by StevenBellavia, 09 February 2018 - 06:24 PM.


#2 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 09 February 2018 - 06:36 PM

I just used Nebulosity 4 to register and edit a picture. I followed a Nebulosity video tutorial posted by Astrononyshed. Maybe give that a look.


https://youtu.be/OMhLYg-xPuc

#3 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 09 February 2018 - 07:09 PM

I just used Nebulosity 4 to register and edit a picture. I followed a Nebulosity video tutorial posted by Astrononyshed. Maybe give that a look.


https://youtu.be/OMhLYg-xPuc

See all that, did all that.  I have been battling this flat frame issue for years.  I wish Craig Stark would reply to my emails, but he rarely does.

Even the vignetting is still there.  What's the point of flat frames?  I am trying Deep Sky Stacker now to see if it will do it correctly, just to see if it is the frames themselves, or the stacking method.



#4 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 09 February 2018 - 07:15 PM

If I find anything I'll get back to you.

#5 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 09 February 2018 - 07:32 PM

If I find anything I'll get back to you.

Thanks!

 

And its NOT Nebulosity (so my apologies to Craig Stark, wherever you are).

 

Dust motes and vignetting still right there, using Deep Sky Stacker too.

 

So something is wrong with the flat frames themselves.  What?  The range is a little low (11,304 mean out of the full 65K ).  Could that do it?

Attached Thumbnails

  • dust_mote-vignette_DSS.JPG
  • single_flat_frame.JPG


#6 MCovington

MCovington

    Soyuz

  • *****
  • Posts: 3994
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 09 February 2018 - 07:52 PM

The symptom is that the flats are having some effect, but not the full effect that they ought to, right?

 

That is because you need flat darks.

 

You can use your bias frames as flat darks, since with a DSLR, bias frames and flat darks are essentially indistinguishable.

 

http://www.covington...slr/#Arithmetic


Edited by MCovington, 09 February 2018 - 07:55 PM.


#7 MCovington

MCovington

    Soyuz

  • *****
  • Posts: 3994
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 09 February 2018 - 07:54 PM

I should add that some software will use bias frames as flat darks.  But that is not common.  If the software asks for flat darks and you don't give it any, your flats will not be applied correctly and will have less than the intended effect.



#8 Charlie B

Charlie B

    Apollo

  • -----
  • Posts: 1308
  • Joined: 22 Mar 2008
  • Loc: Sterling, Virginia

Posted 09 February 2018 - 08:02 PM

I should add that some software will use bias frames as flat darks.  But that is not common.  If the software asks for flat darks and you don't give it any, your flats will not be applied correctly and will have less than the intended effect.

+1 I notice you did not use flat darks.

 

Charlie B



#9 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 10 February 2018 - 11:50 AM

 

I should add that some software will use bias frames as flat darks.  But that is not common.  If the software asks for flat darks and you don't give it any, your flats will not be applied correctly and will have less than the intended effect.

+1 I notice you did not use flat darks.

 

Charlie B

 

I am adding the bias frames to the flats, per Craig Stark's instructions.

 

You don't need dark flats if the flat exposures are short,  (which they are - 1/30th second, ISO 100).  In this case, the bias frame is essentially the dark flat (1/4000th sec, also at ISO 100).

 

When doing narrow-band work, the flat frames become much longer, so then you need dark flats.  

 

I have tried many combinations so far to see what might be happening.  I thought it might be setting (or not setting) the black point to zero, based on the bias frames.  But that wasn't it either.

 

Attached are 4 cases of stacking:

1. lights, darks, flats and bias

2. lights, darks flats (no bias)

3. lights, flats, bias (no darks)

4. lights, flats (no dark, no bias)

5. lights only (no darks, flats or bias)

 

6. And the most interesting case is when I took the lights-only stacked file, and the master flat, brought it into GIMP (2.9 beta version, 16 bit), and inverted the master flat and made it an addition layer (divide would not work).  That is the best so far.  But I shouldn't have to so that.

Attached Thumbnails

  • dust_mote-vignette_DSS-lights-flats-darks-bias.JPG
  • dust_mote-vignette_DSS-no-bias2.JPG
  • dust_mote-vignette_DSS-no-darks.JPG
  • dust_mote-vignette_DSS-flats-only.JPG
  • dust_mote-vignette_DSS-lights-only.JPG
  • dust_mote-vignette_GIMP+layer-invert-addition.JPG

Edited by StevenBellavia, 10 February 2018 - 12:13 PM.


#10 Ron359

Ron359

    Viking 1

  • *****
  • Posts: 876
  • Joined: 21 Apr 2008
  • Loc: -105 +39

Posted 10 February 2018 - 12:38 PM

Since you say your only at 11K out of 65K, your exposures are too short.  Nebulosity (Craig) recommends about 2/3 of the max which is about 40K and thats what I usually shoot them at.  Try that out and see if it helps.  Also, use the AV setting on your DSLR as that should get you close.  Can't take long.     


Edited by Ron359, 10 February 2018 - 12:40 PM.


#11 Charlie B

Charlie B

    Apollo

  • -----
  • Posts: 1308
  • Joined: 22 Mar 2008
  • Loc: Sterling, Virginia

Posted 10 February 2018 - 01:49 PM

 

 

I should add that some software will use bias frames as flat darks.  But that is not common.  If the software asks for flat darks and you don't give it any, your flats will not be applied correctly and will have less than the intended effect.

+1 I notice you did not use flat darks.

 

Charlie B

 

I am adding the bias frames to the flats, per Craig Stark's instructions.

 

You don't need dark flats if the flat exposures are short,  (which they are - 1/30th second, ISO 100).  In this case, the bias frame is essentially the dark flat (1/4000th sec, also at ISO 100).

 

When doing narrow-band work, the flat frames become much longer, so then you need dark flats.  

 

I have tried many combinations so far to see what might be happening.  I thought it might be setting (or not setting) the black point to zero, based on the bias frames.  But that wasn't it either.

 

Attached are 4 cases of stacking:

1. lights, darks, flats and bias

2. lights, darks flats (no bias)

3. lights, flats, bias (no darks)

4. lights, flats (no dark, no bias)

5. lights only (no darks, flats or bias)

 

6. And the most interesting case is when I took the lights-only stacked file, and the master flat, brought it into GIMP (2.9 beta version, 16 bit), and inverted the master flat and made it an addition layer (divide would not work).  That is the best so far.  But I shouldn't have to so that.

 

My flats are on the order of 0.5 S or greater.  I'm surprised your flats are so short.

 

Regards,

 

Charlie B



#12 MCovington

MCovington

    Soyuz

  • *****
  • Posts: 3994
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 10 February 2018 - 02:52 PM

Bias frames and dark flats are likely to be indistinguishable, but you must still tell the software to dark-subtract the flats (using flat darks or bias frames).  Otherwise the numerical values are wrong and the flats are underapplied because of the rather large bias added by the DSLR.  (This was a DSLR, wasn't it?  I lost track of which camera was being used.)



#13 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 10 February 2018 - 03:59 PM

Bias frames and dark flats are likely to be indistinguishable, but you must still tell the software to dark-subtract the flats (using flat darks or bias frames).  Otherwise the numerical values are wrong and the flats are underapplied because of the rather large bias added by the DSLR.  (This was a DSLR, wasn't it?  I lost track of which camera was being used.)

Thanks Michael!

 

I re-processed everything, and then tried using my bias frames as the flat darks, but still no change, I am afraid.  I still have the dust motes and the vignetting.

 

My latest theory:  I think the flat frames and light frames are too dark.  The mean on the Master Flat is 11,307 (out of 16bit, 65K)  The mean on my dark frames is 8188.  An individual, light frame is 9874.

 

I think these are all too close to do the stacking.  Not much room left to separate out the noise from the signal.  I guess I didn't swamp my read noise enough with the 120 second expsoure.

Attached Thumbnails

  • NEB4_Preproc_set_alt1.JPG
  • Final_stack_stretched_using_flat_darks.JPG


#14 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 10 February 2018 - 04:14 PM

Aha!  A clue!

 

 

1. processed a single light frame with a single flat frame.  The flat frame over-compensated (which I think is good news!)

 

2. processed a single Light, Flat, Bias (applied to Flat), and Dark frame.  And THAT worked!  Why?  There is something about the full 30 that is messing up.  Is it the Histogram matching?

 

So now I have to start over.  But progress is being made!

 

Attached Thumbnails

  • Single-Light+Flat.JPG
  • Single-Light+Flat+Bias+Dark.JPG


#15 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 10 February 2018 - 04:32 PM

Hmm. Just a thought. Review each flat again (with auto stretching in Nebulosity turned off). Maybe you'll find one off. I use Canon's DPP or Adobe Camera RAW to view each frame. And discard any that are bad.

Edited by AbPho, 11 February 2018 - 02:29 AM.


#16 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 10 February 2018 - 06:55 PM

Hmm. Just a thought. Review each flat again (with auto stretching in Nebulosity turned off). Maybe you'll find one off. I user Canon's DPP or Adobe Camera RAW to view each frame. And discard any that are bad.

Thanks!
 

I always do that with every frame - light, dark, bias and flats, before processing, because there can always be a bad one. (Like once in a while it is only half an image when the save has a glitch).

 

I checked more carefully, and the mean goes from 11102 to 11354.  Not a big range, and a standard deviation of each stack at 1.75 should throw out outliers.

 

This is my last attempt, using average, instead of SDEV 2.0 for the preprocessing sets.  Still no improvement.

Attached Thumbnails

  • Final_stack_stretched_using_avg_proc.JPG


#17 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 11 February 2018 - 02:45 AM

At least you are heading in the right direction.

 

I spent a lot of time this weekend capturing the darks, flats, and bias frames for my Andromeda image.  But the stacked image is still highly monochromatic and has a bad sepia colour cast. It must mean my light frame data is garbage.  It was bad even with just the lights stacked.  :(  And those were taken from a Bortle 1 dark sky site too. Talk about a wasted trip.

 

I wonder if that's a result of me using a higher ISO resulting in a lower dynamic range image. Hence the monochromatic look. 

 

Anyways.  Good luck Stephen. I'm throwing in the towel for now.  No more editing for a few weeks. 



#18 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 12 February 2018 - 09:19 AM

At least you are heading in the right direction.

 

I spent a lot of time this weekend capturing the darks, flats, and bias frames for my Andromeda image.  But the stacked image is still highly monochromatic and has a bad sepia colour cast. It must mean my light frame data is garbage.  It was bad even with just the lights stacked.  frown.gif  And those were taken from a Bortle 1 dark sky site too. Talk about a wasted trip.

 

I wonder if that's a result of me using a higher ISO resulting in a lower dynamic range image. Hence the monochromatic look. 

 

Anyways.  Good luck Stephen. I'm throwing in the towel for now.  No more editing for a few weeks. 

Don't get frustrated (as if I should talk).

 

I did make more progress, partly (mainly) because of your suggestion to check each frame.  So I re-checked every single one, and one of my Bias frames was just a tad bit suspicious.

Look at the histogram.   It has a spread.  None of the others did. (and if you can see, in the reduced-size jpeg, it is much noisier too)  So I tossed it, one of 30, and now the final stack is better.  But still not perfect.

 

I wish there was a slider bar that you could adjust how much of the flat frame you want to apply.  Seems I either get too much or too little.  And I can never get the Synthetic Flat Fielder to work in Nebulosity, no matter how much I crop or how many points I pick.  it always complains and fails.

 

But I salvaged a not terrible end result (and it seems, for me, it is always a salvage operation, and never smooth):

 

https://www.astrobin...333104/?nc=user

Attached Thumbnails

  • Bias_0044.JPG
  • Bias_0047.JPG


#19 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 12 February 2018 - 12:10 PM

Since you say your only at 11K out of 65K, your exposures are too short.  Nebulosity (Craig) recommends about 2/3 of the max which is about 40K and thats what I usually shoot them at.  Try that out and see if it helps.  Also, use the AV setting on your DSLR as that should get you close.  Can't take long.     

Yes.  I think that IS it.  But my flats are generated automatically, using APT, where you put the camera into Av mode, and it sets the exposure.

 

Thanks!  Will try it as soon as it stops raining here.

 

I just found how to override the automatic exposure control in the users manual, and using CloudyNights, how to do it in APT.

 

https://www.cloudyni...tool-apt/page-8

Attached Thumbnails

  • APT-flat-advanced.JPG

Edited by StevenBellavia, 12 February 2018 - 12:25 PM.


#20 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 12 February 2018 - 01:44 PM

Aha!  I took my Master flat, and stretched it.

I then re-processed my single light frame, but with the new, stretched flat.

And now I finally get it to overcompensate.

 

So this is good news and bad news.  it means that the exposure for the flat is more critical then I thought.  A little too dark, and it will be under-compensated.  A little too bright and over-compensated.

Attached Thumbnails

  • Master_Flat_stretched2.JPG
  • Single_light+MD+MF-MD2-F_stretched2.JPG

Edited by StevenBellavia, 12 February 2018 - 01:55 PM.


#21 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 12 February 2018 - 02:00 PM

I read you want your flat expoaure to be such that the spike in the histogram is around the 50% mark.

#22 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 12 February 2018 - 02:06 PM

I read you want your flat expoaure to be such that the spike in the histogram is around the 50% mark.

Yes, I did too.  And I had hoped that using AV mode on the camera would do that.  But it doesn't.



#23 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 12 February 2018 - 02:12 PM

Av, Tv, etc suck. You are better off shooting with manual exposure mode. Cameras are rather stupid.

I used a studio strobe's modelling lamp through a big softbox as my light source. Kept the iso low, set my aperture the same as when I took the lights, and played with the shutter until my histogram looked good. I want sure about the white balance so I set that manually too. Otherwise the red channel was going to blow out (Ha modded camera and all).

#24 StevenBellavia

StevenBellavia

    Mariner 2

  • *****
  • topic starter
  • Posts: 224
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 12 February 2018 - 02:37 PM

So does this mean that I have to start shooting 3 sets of flats?

 

1. 1/3 histogram
2. 1/2 histogram
3. 2/3 histogram.

 

Then create a master for each, as well as the Darks and Bias (or Dark Flats, which perhaps are better), and then try them out on at least a single light frame to see which flat works, before diving in and doing the histogram matching, de-bayering, aligning and stacking?

 

That is tedious.  But maybe better than losing a night out in the cold?



#25 AbPho

AbPho

    Vostok 1

  • -----
  • Posts: 113
  • Joined: 22 Jul 2013

Posted 12 February 2018 - 02:55 PM

I would start with 50%. They go fast. Stack and test.


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics