Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Testing flats without "star" image? (Pixinsight)

  • Please log in to reply
25 replies to this topic

#1 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 23 September 2022 - 04:20 PM

Hi,

 

I want to try to create as good flats as possible at home.

I wonder if this would be possible to do somehow without having to have lights with stars in them and running WBPP?

 

Thanks!



#2 D_talley

D_talley

    Aurora

  • *****
  • Posts: 4,535
  • Joined: 07 Jul 2005
  • Loc: Albuquerque, New Mexico

Posted 23 September 2022 - 05:00 PM

You don't use lights with stars in them to make flats. 



#3 rodmichael

rodmichael

    Ranger 4

  • *****
  • Posts: 305
  • Joined: 16 Nov 2016
  • Loc: South Dakota, USA

Posted 23 September 2022 - 05:13 PM

I may not understand your question, but stars are not an imaging component of flat frames and are not something with which your light frames are calibrated when using flat fames. To my understanding, which could be faulty, flats are used only to account for imperfections in your optical/imaging train. Flat frames are used to subtract optical imperfections, eg. vignetting, dust motes, from your light frames. Flat frames should have no impact on the stars in your light frames other than to subtract optical imperfections that may exist between your observing target and your light sensor.

 

CS, Rod Michael


Edited by rodmichael, 23 September 2022 - 05:14 PM.

  • DanMiller likes this

#4 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 29,850
  • Joined: 27 Oct 2014

Posted 23 September 2022 - 05:29 PM

Take a flat.  Run Image Statistics normalized real.  The value should be 0.4-0.6.  Precise gets you nothing at all.  Run script Image Analysis FlatContourPlot, see if the plot "makes sense".


  • psandelle and Whereisclearsky like this

#5 psandelle

psandelle

    Skylab

  • *****
  • Posts: 4,377
  • Joined: 18 Jun 2008
  • Loc: Los Angeles

Posted 23 September 2022 - 06:39 PM

Take a flat.  Run Image Statistics normalized real.  The value should be 0.4-0.6.  Precise gets you nothing at all.  Run script Image Analysis FlatContourPlot, see if the plot "makes sense".

You know, my flats have always just worked (when I did them manually, I'd keep track of where the histogram fell, mean, etc; now I have Voyager do them), but it was cool trying this and seeing they actually DO work!

 

Paul
 


  • bobzeq25 likes this

#6 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 24 September 2022 - 04:55 AM

I think people misunderstood my question.

 

My flats don't work no matter what I do, I have had problems with previous camera Sony A6400, and I have problems now with Lumix S5.

I have tried LED panels, iPads and iPhones, diffusers and stuff.

 

It never gets perfect, lately it's negative vignette which isn't very bad, but it's still there.

 

It doesn't matter what brightness my flats are, always bad results.

 

So I would like to somehow try taking flats differently, and see if there is still negative vignette.

The issue is that I would need to collect new lights for each test, and it's cloudy here 95% percent of the time.

 

So I came up with the idea that perhaps I could test it differently, like take a photo of a wall and then extract the flats or something, no idea...


Edited by Whereisclearsky, 24 September 2022 - 04:55 AM.


#7 matt_baker

matt_baker

    Explorer 1

  • -----
  • Posts: 78
  • Joined: 07 Feb 2018

Posted 24 September 2022 - 06:59 AM

Depending what program you use, in this case Pixinsight, you're able to tune your flats by adding or subtracting a pedestal. When you say negative vignette, do you mean it's brighter in the corners? If so it's over correcting.

 

If you have PixInsight, try this: https://www.cloudyni...elmath-process/



#8 lambermo

lambermo

    Apollo

  • -----
  • Posts: 1,210
  • Joined: 16 Jul 2007
  • Loc: .nl

Posted 24 September 2022 - 09:04 AM

Have a look at Fabian Neyer's presentation at CEDIC on correcting the standard flat with night flats, slides 12 - 20 http://www.cedic.at/...abian_Neyer.pdf

-- Hans


  • jdupton and halflife like this

#9 psandelle

psandelle

    Skylab

  • *****
  • Posts: 4,377
  • Joined: 18 Jun 2008
  • Loc: Los Angeles

Posted 24 September 2022 - 09:46 AM

First check the flats in PI as Bob suggested to see if the flats themselves are the problem. If not, it could be a number of other things, but first check the flats to see if they are the culprit. If they are at fault, then it's a matter of working with your flat-acquisition method. If they're good, we can go from there.

 

Paul



#10 badgie

badgie

    Messenger

  • *****
  • Posts: 430
  • Joined: 13 Jan 2021
  • Loc: SF Area, California

Posted 24 September 2022 - 10:39 AM

Two suggestions as diagnostics:

1. Try cross calibrating flats. Calibrate a t shirt flat with a panel flat or a clear sky flat. This will show how flats are changing.

2. calibrate your individual flat subs with your masterflat (and an darks/bias etc) if they aren't perfectly corrected then it is your data pipeline that is the problem
  • jdupton likes this

#11 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 29,850
  • Joined: 27 Oct 2014

Posted 24 September 2022 - 11:05 AM

I think people misunderstood my question.

 

My flats don't work no matter what I do, I have had problems with previous camera Sony A6400, and I have problems now with Lumix S5.

I have tried LED panels, iPads and iPhones, diffusers and stuff.

 

It never gets perfect, lately it's negative vignette which isn't very bad, but it's still there.

 

It doesn't matter what brightness my flats are, always bad results.

 

So I would like to somehow try taking flats differently, and see if there is still negative vignette.

The issue is that I would need to collect new lights for each test, and it's cloudy here 95% percent of the time.

 

So I came up with the idea that perhaps I could test it differently, like take a photo of a wall and then extract the flats or something, no idea...

Are you also taking bias?  They're necessary for flats to work well.  Because of the nature of flat correction.

 

Generally the problem with flats is not the flat itself.  It would be hard to take a flat that overcorrects.  Get the exposure _roughly_ in the center of the histogram.  Exact does not matter, all you're trying to do is stay away from the edges. 

 

It's trivial to omit bias, or get the calibration process wrong.  People do that all the time.  I recommend Astro Pixel Processor.


Edited by bobzeq25, 24 September 2022 - 11:07 AM.

  • psandelle and kathyastro like this

#12 badgie

badgie

    Messenger

  • *****
  • Posts: 430
  • Joined: 13 Jan 2021
  • Loc: SF Area, California

Posted 24 September 2022 - 11:16 AM

I'll add to this, I had been happy calibrating in SIRIL for some time before switching to PI.  When I first switched, I couldn't get a flat to work for the life of me.  I checked the flats in SIRIL and there was no issue.  I honestly never completely solved the problem because as soon as I switched to using WBPP for generating flats my problems went away. Using those master flats, I could then use both the automatic and calibration tools successfully.  I suspect that there is a setting that has to be changed from default value that is missed in the tutorials, or that the options changed since the tutorials were made.  There were many other issues to learn in PI and I haven't even thought about this until thinking it through now.  I tried the steps in my last post as well as trying SIRIL, concluded I had a workflow issue and a suitable fix and moved on.  Looking through this perhaps there was a pedestal value set to non-zero when calibrating the flats initially.  Mathematically, they should be calibrated with no pedestal.


  • psandelle likes this

#13 sharkmelley

sharkmelley

    Fly Me to the Moon

  • *****
  • Posts: 5,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 24 September 2022 - 11:30 AM

So I would like to somehow try taking flats differently, and see if there is still negative vignette.

The issue is that I would need to collect new lights for each test, and it's cloudy here 95% percent of the time.

Why do you think you need a new set of lights each time?  Why can't you try the different sets of flats on the same set of lights?

 

Mark


  • psandelle likes this

#14 iwannawon

iwannawon

    Mariner 2

  • *****
  • Posts: 256
  • Joined: 22 Dec 2014
  • Loc: Red zone, SW Houston, TX

Posted 24 September 2022 - 12:07 PM

similar to what bob said about flat issues not necessarily being due to flats. most times i had issues, it ended up being my darks. it looked like my flats but was always a light leak or something if i got sloppy with my darks (or flat darks, but same concept). i couldn't see the issue but looking at adu values showed very slightly higher than expected compared to bias. when i started taking darks in a dark closet or with a blanket over to ensure it was truly dark, my overall calibration worked better. that being said, when i had the 294mm it worked better with dark flats (rather than bias) and longer exposures for flats than some other cameras. i'm not familiar with your cameras though. i didn't rewatch it... maybe it's one of the other parts, but i think this goes over it (adam block). if haven't seen already. showing an example of different darks fixing an issue that visually appeared flat related. might not be your issue but if you've tried different things with flats that didn't fix it, might be worth trying something different like this.
https://www.youtube....h?v=uVx07KIdDMU


  • psandelle likes this

#15 jdupton

jdupton

    Aurora

  • *****
  • Posts: 4,572
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 24 September 2022 - 01:00 PM

Whereisclearsky,
 

I want to try to create as good flats as possible at home.
I wonder if this would be possible to do somehow without having to have lights with stars in them and running WBPP?

   Yes, you can check your Flats without needing to use a Light frame of any kind. I always highly recommend that users do this at least once just to see how it works and how flat their Master Flat really is. I outlined the method in another post not too long back. Here it is.
 

[snip]

To test the flatness of the source, you want to try to flatten one image from the [flat light source] with a second (different) image from the [same flat light source]. This test will reveal any linear gradients present in the [flat] panel light source and will also reveal any off-centered radial gradients that could be present. (Many [light sources may] use an edge-lit [panel] methodology with LEDs placed on one or more sides of the panel. That can sometimes result in slightly brighter light at those edges than at the center [or vice versa] -- sort of a square version of inverse vignetting.)
 
   Here is the testing procedure that can be used to check any flat fielding light [source].

  • Set up your system to take Flats as you normally would.
     
  • Take at least 1 Flat-Dark [or Bias frame] -- as many as five is better.
  • Take 1 to 5 Flat frames.
     
  • Rotate the Flat Light [source] by 90°. Rotate only the Flat Light [source] -- not the camera.
  • Take 1 to 5 additional Flat frames.

    Now, reduce the data using PixInsight.
     
  • Integrate the Flat-Dark [or Bias] frames (if more than 1 was taken).
  • Calibrate both sets of Flats using the Master Flat-Dark [or Master Bias].
     
  • Integrate the first set of 5 calibrated Flats into a Master_Flat_00.
  • Integrate the second set of 5 calibrated Flats into a Master_Flat_90.
     
  • Open PixelMath and calibrate one of the Master Flat sets using the other.
    Use the following equation in the RGB/K pane of PM. Set PM to create a new image for the result.
    Master_Flat_00 / Master_Flat_90 / 2.0
     
  • Run the Script "FlatContourPlot" against the PixelMath result.
    [Use parameters: Resolution = 50%, Sigma = 3.0, Contours = 15 and Gradient = 50%]
   Ideally, the result of step [11] will show up as quite flat since we are flat fielding the Flat. If the Flat light source really is perfectly flat, then the two rotated flats should be identical. The result of dividing one Master Flat by a rotated second Master Flat should be 1.00 (exactly the same everywhere). When divided by 2.0, you then get a Flat image with a uniform brightness of 0.5 (normalized) (or 32767 ADU in 16 bit format).
 
   You can estimate the "flatness error" of the two Master Flats by reading out the numbers at the right of the FlatContourPlot. The total flatness error is given by  (max - min) / (average(max, min)) * 100  which is the percentage flat differences across the whole flat light source. For most imaging we do, anything better than about 5% should be good enough. For scientific imaging, the criteria is down around 0.5% to 1% deviation from complete flatness.
 
   Any unevenness of the FlatContourPlot result shows where the two Flats differed after rotation. The unevenness helps to show show that there is a potential issue with the flatness of the light source. If you get a flatness of greater than 5% then you may have to work on the diffusing the light source even more.
 
   This test does not check for perfectly centered vignetting in the light source. For testing that, you need another Flat generated by a different method to compare against. The independent method can be a diffused Sky Flat or perhaps a "Wall Flat". To take a Sky Flat, you can put a[n option] translucent polycarbonate plate over the telescope pointing up at the sky at early dusk. The comparison of [your normal Master Flat frame] and a Master Sky Flats would be very similar -- calibrate one of your Master Light [Source] Flats using the [Master] Sky Flat. The same PixelMath equation given above can be used for this test. If your light box shows no inverse vignetting from the tracing panel, dividing it by a Sky Flat should give a very flat result in the FlatContourPlot.

   Posts #8 and #10 above mention these methods. In regard to your Flats over or under correcting your images, the fault may lie in the Master Dark not matching the Lights or the Master Flat-Dark (or Master Bias) not matching the Flats in terms of Gain, Offset, Exposure, Temperature, and capture software w/ driver type. If the dark calibration frames don't match what they are calibrating, then the flat fielding by the Master Flat can fail to work properly.

 

   How you calibrate your Flats for making a Master Flat can depend on which camera you are using. Some cameras may require special attention to some details (like exposure time needed to get to 50% of the histogram) when making the Master Flat. Let us know which camera you are using so we can eliminate that possibility.
 

 

John


Edited by jdupton, 24 September 2022 - 01:03 PM.

  • psandelle and bobzeq25 like this

#16 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 25 September 2022 - 06:31 AM

I meant I want to check flats against "fake lights" or something to see if the white vignette is gone.

 

Take a flat.  Run Image Statistics normalized real.  The value should be 0.4-0.6.  Precise gets you nothing at all.  Run script Image Analysis FlatContourPlot, see if the plot "makes sense".

Could you explain in more details how do I do that?

 

Depending what program you use, in this case Pixinsight, you're able to tune your flats by adding or subtracting a pedestal. When you say negative vignette, do you mean it's brighter in the corners? If so it's over correcting.

 

If you have PixInsight, try this: https://www.cloudyni...elmath-process/

Yes it's brighter quite often.

 

Are you also taking bias?  They're necessary for flats to work well.  Because of the nature of flat correction.

 

Generally the problem with flats is not the flat itself.  It would be hard to take a flat that overcorrects.  Get the exposure _roughly_ in the center of the histogram.  Exact does not matter, all you're trying to do is stay away from the edges. 

 

It's trivial to omit bias, or get the calibration process wrong.  People do that all the time.  I recommend Astro Pixel Processor.

There is no point in taking bias if you take darks and flat darks.

 

Why do you think you need a new set of lights each time?  Why can't you try the different sets of flats on the same set of lights?

 

Mark

Because you need to have same focus in your lights and flats.

----

 

 

I have tried taking test flats last night, used LED white panel but no diffuser of any kind.

I have tried using t-shirt and white paper before but was getting much worse results.

 

Result:

 

https://i.imgur.com/C4xd5Q5.jpg

C4xd5Q5.jpg

 

Left: Lights + Darks + Flats + Flat darks

Right: Lights

 

Stacked in Affinity Photo to save time.

 

There is also some light pollution from nearby street lamp, but I was getting similar results before away from lights.

 

I have also uploaded all frames here in case anyone want to take a look.

Stars are out of focus since MPB.com sent me a crap lens lol.

 

https://mega.nz/file...YE7H2boZP1mjdvs

 

Thanks all.



#17 Churmey

Churmey

    Mariner 2

  • *****
  • Posts: 237
  • Joined: 05 Feb 2017

Posted 25 September 2022 - 07:15 AM

I use to have issues with my flats, especially when I was learning the hobby. It truly is hard to believe, just how easy it is, to create a condition, throughput the night, such that the flats will not work. Did you move your mount for any reason? Did you manually focus your scope? Did you adjust the rotation of your camera, to the millionth degree, at all during the night? Touching your system, at all, increases the likelyhood that 'something' will happen such that the sensor-to-optics conditions will change to the slightest such that flats will no longer work. 

 

Again, Flats use to be my largest issue, simply because I was learning. I was continually re-focusing during the night (touching scope), maybe I was re-slewing during the night (touching scope), maybe I was re-doing polar alighnment (touching scope), adjusting cords (touching system) etc.....you get my drift.  

 

There was a point in time, when I said to myself, that I was going to create a scenerio to where after polar alignment, I will not have to touch my system until after flats are completed. This required that I put focus motors on my systems.   After I approached this hobby with "do not touch' until after flats are completed, my issues with flats went away! At the end of my light aquisition, I let the mount slowly slew to position for flats, and I gently lay my tracing light pad over the scope to do my flats.  Once flats are done, only then do I put my hands on the mount. 

 

The most common issue with flats, is that something moved ever so slightly (as it relates to the sensor position to optics) from the time you began acquisition of lights, until you completed the flats at the end of acquisition. 


Edited by Churmey, 25 September 2022 - 07:25 AM.


#18 sharkmelley

sharkmelley

    Fly Me to the Moon

  • *****
  • Posts: 5,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 25 September 2022 - 07:25 AM

 

Why do you think you need a new set of lights each time?  Why can't you try the different sets of flats on the same set of lights?

 

 

Because you need to have same focus in your lights and flats.

 

Can't you just set the lens to infinity for your flats?



#19 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 25 September 2022 - 08:42 AM

Can't you just set the lens to infinity for your flats?

In camera lens you can't just fully turn the focus ring to infinity and have it focused right.



#20 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 29,850
  • Joined: 27 Oct 2014

Posted 25 September 2022 - 10:15 AM

Dark flats work as well as bias.  In most cases, there's no difference between them.

 

I take my flats immediately after I take my lights, and simply don't touch the focus.  That works.

 

My last and best suggestion.  Since you have PixInsight.

 

Take bias, flats, darks.

 

Integrate the bias and darks into a master bias and master dark.

 

Calibrate the flats with the master bias and master dark.  Integrate the flats into a master flat.

 

Calibrate the lights with the master bias, master flat, master dark.

 

Debayer the calibrated lights.

 

Staralign the debayered lights.

 

Integrate the aligned lights.

 

The process is described in detail in the book Inside PixInsight by Warren Keller.

 

Still have the problem?

 

Could this be a problem with the flats themselves?  Possible, but unlikely.  It's substantially more likely to be a processing problem.  I'm unfamiliar with Affinity, but the fact that you said you stacked in Affinity "to save time" is troublesome.  When I see flat problems here, they're most often caused by taking shortcuts in processing.


Edited by bobzeq25, 25 September 2022 - 10:16 AM.


#21 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 25 September 2022 - 10:48 AM

Dark flats work as well as bias.  In most cases, there's no difference between them.

 

I take my flats immediately after I take my lights, and simply don't touch the focus.  That works.

 

My last and best suggestion.  Since you have PixInsight.

 

Take bias, flats, darks.

 

Integrate the bias and darks into a master bias and master dark.

 

Calibrate the flats with the master bias and master dark.  Integrate the flats into a master flat.

 

Calibrate the lights with the master bias, master flat, master dark.

 

Debayer the calibrated lights.

 

Staralign the debayered lights.

 

Integrate the aligned lights.

 

The process is described in detail in the book Inside PixInsight by Warren Keller.

 

Still have the problem?

 

Could this be a problem with the flats themselves?  Possible, but unlikely.  It's substantially more likely to be a processing problem.  I'm unfamiliar with Affinity, but the fact that you said you stacked in Affinity "to save time" is troublesome.  When I see flat problems here, they're most often caused by taking shortcuts in processing.

I was suggested by everyone to not take bias, including Pixinsight developers.

I just use WBPP script so there is no issue with the processing I suspect.

 

Affinity has nothing to do with the issue, same results as Pixinsight basically.



#22 badgie

badgie

    Messenger

  • *****
  • Posts: 430
  • Joined: 13 Jan 2021
  • Loc: SF Area, California

Posted 25 September 2022 - 02:20 PM

Just to clarify (and reading your post you may well be doing this, but it isn't clear) You need either an appropriate bias or dark for flat frames to make sense mathematically.  This is because flats are used multiplicatively; any offset on the masterflat will be an error.

 

Also, your vignetting looks very ugly! Is there a chance your filter wheel has a problem or dew shield impingement?  I'm not used to seeing that degree of asymmetry.  There are some known issues with CW vs CCW moves:

 

A simple test, suppose you have filters 1,2,3.

 

Goto Filter 1

Goto Filter 2

Take a flat

Goto Filter 3

Goto filter 2

Take a flat.

 

this will compare CW vs CCW moves.  Also maybe compare the flats for different filters and see if the vignetting is similar for all. 



#23 choward94002

choward94002

    Apollo

  • *****
  • Posts: 1,294
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 26 September 2022 - 12:20 AM

Just to clarify (and reading your post you may well be doing this, but it isn't clear) You need either an appropriate bias or dark for flat frames to make sense mathematically.  This is because flats are used multiplicatively; any offset on the masterflat will be an error.

 

Also, your vignetting looks very ugly! Is there a chance your filter wheel has a problem or dew shield impingement?  I'm not used to seeing that degree of asymmetry.  There are some known issues with CW vs CCW moves:

 

A simple test, suppose you have filters 1,2,3.

 

Goto Filter 1

Goto Filter 2

Take a flat

Goto Filter 3

Goto filter 2

Take a flat.

 

this will compare CW vs CCW moves.  Also maybe compare the flats for different filters and see if the vignetting is similar for all. 

Hmm ... I was not aware of any issues regarding the rotation direction of the EFW, what are you referring to?  Is this a general problem, or for a specific vendor?



#24 badgie

badgie

    Messenger

  • *****
  • Posts: 430
  • Joined: 13 Jan 2021
  • Loc: SF Area, California

Posted 26 September 2022 - 12:21 AM

It's known on some qhy models. I don't know about efw.

#25 Whereisclearsky

Whereisclearsky

    Viking 1

  • -----
  • topic starter
  • Posts: 542
  • Joined: 18 Dec 2020
  • Loc: Weston-super-Mare

Posted 26 September 2022 - 01:33 PM

Just to clarify (and reading your post you may well be doing this, but it isn't clear) You need either an appropriate bias or dark for flat frames to make sense mathematically.  This is because flats are used multiplicatively; any offset on the masterflat will be an error.

 

Also, your vignetting looks very ugly! Is there a chance your filter wheel has a problem or dew shield impingement?  I'm not used to seeing that degree of asymmetry.  There are some known issues with CW vs CCW moves:

 

A simple test, suppose you have filters 1,2,3.

 

Goto Filter 1

Goto Filter 2

Take a flat

Goto Filter 3

Goto filter 2

Take a flat.

 

this will compare CW vs CCW moves.  Also maybe compare the flats for different filters and see if the vignetting is similar for all. 

As I said Im taking darks and dark flats. 
Also no filters and mirrorless camera. 




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics