Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Serious Flat Frame issues - ASI1600MM-C - What am I doing wrong here?!

  • Please log in to reply
59 replies to this topic

#51 Peter in Reno

Peter in Reno

    Voyager 1

  • *****
  • Posts: 12,310
  • Joined: 15 Jul 2008
  • Loc: Reno, NV

Posted 16 December 2017 - 01:46 AM

Yes, I used T-shirt flats with non-EdgeHD and EdgeHD C-8 SCTs as well without issues. My CCDs were always Sony interlines. One was SXVR-M25C OSC and the other is the current ICX-694.

 

Peter



#52 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,990
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 16 December 2017 - 02:22 AM

I explored some of my flats in detail and one thing I realized is that if the flats don't have much vignetting then you can stretch them pretty hard - at which point the rings are visible.  But you need to measure the amplitude of the ring modulations to know how much impact they really have.

 

For my Ha flat the ADU value goes from 32909 +/- 180 to 32726 +/- 177 - going from peak to valley in one of the rings.  That's a change of only 183 or 0.5% - which is very small.  

 

A flat is very flat except for vignetting and dust motes - so you can stretch it very hard to reveal slight variations in the flat that take on visible shapes - like circular arcs.

 

But an actual calibrated light of an image normally has detail in it that doesn't work well when stretched.

 

This explains why the flats "work" for me - even if they have "bad" reflections in them.  The rings have such a small amplitude I would only be able to see them in a scene that itself was very flat.

 

So they may indeed be due to reflections - and it may be possible to cure them.  But I don't have much motivation to do so if I am just imaging rather than doing photometry.  And even then the effect is very small.

 

So on that basis - the rings may in fact be undesirable and due to reflections.  But in practice they will never be noticed as long as the amplitude is small enough.  And 0.5% is pretty small.  If anyone has rings in their flats - I recommend measuring the peak to valley change in ADU value to see just what impact it might have.

 

Frank

 

[addendum]  The other thing is that the rings are likely present in the lights - and are a reasonable match to what is in the flats.  I think that it's similar to a pupil ghost that the flat will be affected in a similar way to a light - but not exactly - because the ghost depends on the illumination in the object.  That's why they are so insidious.

 

But as long as the effect is small, and as long as the light captures some of the ring effect - the flat will have an overall benefit and make the small effect even smaller after calibration - but maybe not remove it completely.


Edited by freestar8n, 16 December 2017 - 02:29 AM.


#53 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 02:59 AM

 

I have used triple layer T-shirt flats using sunlight for at least five years without issues. It even works under cloudy skies.

 

Peter

There are definitely people who have no problems with flats and they are easy to take and work fine - but for others it can be more delicate.  

 

Peter - I forget, but did you ever work with sct's?  Refractors may make things easier.

 

And Jon Rista is mostly DSLR lenses?

 

The OP appears to have and SCT like John and me.

 

Frank

 

I don't have problems with the Canon lens. It's well blackened and flocked on the inside. Amazingly so.

 

I had tons of problems with the 8" RC. 



#54 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 03:11 AM

 

I generally go by the SNR equation:

 

SNR = S/N

 

Anything in the denominator there is noise. The noise in a single uncalibrated sub would be:

 

N = SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

That leaves your final object SNR formula as:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

The only thing in the numerator here is the object signal. Everything else, including the FPN and DFPN terms, are noise. The signal from the sky, which sure is a signal, has an IMPACT on the final SNR that leaves it only behaving as a noise. Same goes for dark current. It, too, is a signal, but the IMPACT on the final SNR leaves it only behaving as a noise.

 

When we calibrate, we remove the FPN components, so:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2)

 

We end up back at the more familiar SNR formula. Technically speaking, calibration doesn't eliminate the FPN. Even if it perfectly removes the fixed sensor patterns, the random noise pattern of the master dark itself becomes a new FPN term. Stack enough undithered subs, and spatially random remainder of the noise in the dark (subtract one master dark from another, and you'll get this) will become the limiting factor on your SNR, as it will be in every calibrated sub. Thing is, this remnant FPN is totaly spatially random, it is otherwise imperceptible from temporally random noise...because it was derived from temporally random noise. Is it a signal? Or a noise? It's definitely FPN! tongue2.gif

 

The total signal that you get is the object signal plus the unwanted signal from the sky.  The uncertainty in the sum of those two signals decreases the resulting SNR.  In this case, you cannot remove the unwanted signal--all you can do is to increase SNR to reduce the effect of the unwanted signal.  Regardless, sky fog is not noise.  It is unwanted signal (with it's own contribution to the total noise.)

 

As for your calculation, this is where conflating spatial noise with temporal noise creates problems.  Right up front, your definition of noise isn't right.  You are adding the spatial rms variation of FPN into the noise term and in this case, that's not right.  Those are fixed values (hence the 'F' in the name.)  Flat calibration does indeed remove FPN and that's part of the reason that you calibrate the image.  So...I agree with your final SNR equation.  BTW, both FPN and DFPN are removed during calibration.  Subtracting both introduces a shot noise contribution, but that's orders of magnitude smaller than what FPN variations introduce across the sensor.

 

John

 

John, you need to finish reading Janesick's books. My SNR formula is strait out of Janesick's work. The SNR formula is per-pixel, and the FPN contributions are linear with the signal (hence the reason they are the DSNU and PRNU times the dark and object signals, then squared, which keeps them linear). The definition is valid, it's just not what you've accepted as valid. ;) When you finish reading Janesick's work, you'll either disagree with him as well, which is certainly your prerogative...or finally understand. shrug.gif

 

As for sky signal. In an unprocessed, uncalibrated sub, sky signal is there, yes. However it does not remain there. When you remove the sky gradient and adjust black point, you remove the skyfog signal, it's offset. Modeling the gradient models the signal that came from the sky, allowing it to be removed through subtraction. Thing is, when you do that, you don't remove the noise it added to the total noise in the image. The standard deviation remains the same, the offset shifts towards black. 

 

In the end, the sky only contributes noise, it contributes no meaningful information to the signal you keep around, hence the reason it is only in the denominator as a noise term, and not in the numerator as part of the signal. 



#55 Peter in Reno

Peter in Reno

    Voyager 1

  • *****
  • Posts: 12,310
  • Joined: 15 Jul 2008
  • Loc: Reno, NV

Posted 16 December 2017 - 11:30 AM

FYI. This is the T-shirt flat I am using:

 

http://astrogizmos.c...T-Shirt OTA.htm

 

I bought three of them so I can stack them to help dim the light. I point the scope toward the sky and away from the sun (sun is behind the scope).

 

At time, they were selling them for smaller scopes (from 3" to 18") but according to the current web site, they only make them for 18" to 24" scopes. I think they can work on smaller scopes but you may need large rubber bands to hold T-shirts to the scope.

 

Peter



#56 cfosterstars

cfosterstars

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,893
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 16 December 2017 - 12:49 PM

I'm happy to have a discussion about this noise terminology elsewhere - but this is definitely a side topic from the OP's - and I don't have a sense he is interested.  So I recommend another thread.  I have spawned threads on pattern noise numerous times so I'm happy for someone else to give it a shot.

 

One reason to get back to the flats issue is that I'm still not sure what the OP's problem is with his flats.  I'm also not sure what role reflection has played in the rings - or if they are now gone now after darkening things.

 

[Edit]  I just saw his note above among the other notes about noise - and he says that darkening things has reduced the rings, but they still remain and he hopes to get rid of them.  So if he can in fact get rid of the rings, and if they look a lot like mine, then that will be interesting to me.  But it doesn't explain why flats work for me despite the rings.

 

The issue of the checkerboardish pattern applies to his flats and mine and my main point is - I had never seen it before and I don't see it at all in my flats when I view them at 100%.  I also don't see it in uncalibrated lights - whereas I do see vignetting and dust - etc.  It's a very subtle thing if it is present - but certainly flats are good for more than just vignetting and dust donuts - because they correct for prnu also - whatever form it takes.

 

Frank

Frank,

 

As I have made improvements, my flat are "working" but not completely. There are still subtle defects remaining in the image corners that are only visible with serious processing. You dont see them in the flats, but you do see them remaining in the processed images. At this point in correcting the issue, the degree of artifacts is probably low enough for me to process them out of the image, but that is not work I want to have to do every time with every image. If you look at my other post:

 

https://www.cloudyni...ring-artifacts/

 

You will get a sense of how the imaging has improved. My current flats show subtle rings across the image that seem to correct out, but the corners are still an issue. I still have more testing and work to do and I think I know what is wrong. I am waiting for masks for my filter and I have some Astrodon replacement filters on order. I feel that the combination of what I am planning should completely eliminate the issue. It will take more time to prove and I will keep updating the post as I drive to conclusive data.



#57 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,990
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 16 December 2017 - 05:03 PM

I don't do any local manipulation of images - and I don't even like to remove gradients except possibly a linear gradient that is clear across the images.  So it's important to me that flats work well - and for me they seem to even though there are things happening with rings and corner issues.

 

Here is one example that is completely uncropped, using EdgeHD11 at f/7 with Atik 8300:

 

http://www.astrogeek.../images/m8.html

 

Here is another that is highly stretched with EdgeHD11 at f/10 with ASI1600:

 

http://www.astrogeek...88/ngc6188.html

 

So I'm not sure what problems remain with your setup - but if you need to manually correct for flat problems in the corners or something - then I agree it is worth trying to fix.

 

I have accumulated many types and brands of filters over the years - but I never worked with filters that weren't mounted.  They are all 1.25" or 2" and threaded.  I guess that means my filter wheel needs to be a bit thicker - but it does avoid all the problems caused by mounting the filters in the wheel instead of just screwing them in.

 

Frank



#58 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 17 December 2017 - 01:19 AM

John, you need to finish reading Janesick's books. My SNR formula is strait out of Janesick's work. The SNR formula is per-pixel, and the FPN contributions are linear with the signal (hence the reason they are the DSNU and PRNU times the dark and object signals, then squared, which keeps them linear). The definition is valid, it's just not what you've accepted as valid. wink.gif When you finish reading Janesick's work, you'll either disagree with him as well, which is certainly your prerogative...or finally understand. shrug.gif

 

 

As for sky signal. In an unprocessed, uncalibrated sub, sky signal is there, yes. However it does not remain there. When you remove the sky gradient and adjust black point, you remove the skyfog signal, it's offset. Modeling the gradient models the signal that came from the sky, allowing it to be removed through subtraction. Thing is, when you do that, you don't remove the noise it added to the total noise in the image. The standard deviation remains the same, the offset shifts towards black. 

 

In the end, the sky only contributes noise, it contributes no meaningful information to the signal you keep around, hence the reason it is only in the denominator as a noise term, and not in the numerator as part of the signal. 

 

 

 

Jon,
Let’s first talk about signals.  Consider the configuration shown in figure 1 below.  In it, we have two flat panels that illuminate two binary masks.  A dimmer controls each flat panel so that the brightness can be set anywhere from 0% to 100%.  A perfect 50:50 beam splitter allows a camera to view both panels simultaneously.  If you turn up panel #1 to full brightness and panel #2 to be dark, the camera will see only light from the first binary mask as shown in image A in Figure 2.  By taking many frames, an average signal can be computed and this is defined as the signal for the first channel in our setup.  The rms statistical variations at each point in the image represent the noise—at each point.

 

We can then turn down the brightness of panel #1 to be dark and turn up the brightness of panel #2 to be bright.  We will see a clear image of the second mask as shown in image B of Figure 2.  This is also a signal and we can do the same thing to measure the average signal and the rms noise in the second channel of our setup.

 

If we then set the dimmers for both flat panels to be at 50%, we will mix the two signals as shown in image C of figure 2.  In this case we can make a measurement of the total signal, to find the average result and the rms noise, which will vary with position within the image.  In this case, we can mathematically combine the two signals by adding them and treating the noise by adding the two noise components in quadrature.  The problem here is that with only a single measurement of the combined signal, we can’t easily separate the result into the two component signals.

 

Now, if we leave everything as it is with both panels set at 50% and simply remove binary mask #2, we will flood the signal from mask #1 with uniform light to produce image #4 in Figure 2.  The light flooding the signal from channel #1 is still the signal from channel #2.  It simply reduces the contrast of the signal in channel #1.  Since the second signal now has a constant value, we can take the resulting image and adjust it’s contrast to simulate the situation that we started with (looking only at channel #1) but by doing that we will alter the resulting SNR.  The fact that you can now compute the signal and the noise of the result does not in any way change the fact that the image (as shown in Figure 2, #4) is the result of combining two signals—one just happens to have a constant value. 

 

This is exactly analogous to sky fog.  The light from sky fog is a signal and if you could put a big black card up above the atmosphere to block the view of the sky, you could measure its average value and the noise that it contributes directly.  Whenever you take an image of the sky it always contains the desired signal along with the unwanted signal that comes from sky fog.  The fact that the absolute value of the strength of the sky fog doesn’t enter into the SNR equation does mean in any way that the unwanted signal merely contributes noise—even though it acts that way mathematically after you renormalize the result.

 

Now as far as Janesick is concerned, your comments are a bit presumptuous and frankly, it sounds like you might be missing the main point of the book.  In “Photon Transfer,” Janesick discusses the parameters, statistics, and methods to produce photon transfer data and how it can be used to evaluate sensor performance.  When you produce PT data, you work with an ensemble of pixels that are evenly illuminated to produce the data.  Janesick would have indeed included PRNU and DSNU as noises sources because he is analyzing the spatial statistics of the uniformly illuminated patch to characterize overall sensor performance factors. 

 

Noise is not unwanted signal.  Noise is the variation that we see when we measure something over and over and signal is the average of what we measure.  When you take PT data, you don’t make measurements over and over to get the statistics.  You take a single measurement and use the statistics of what you get from all of those evenly illuminated pixels to derive what you might get if you were to measure something over and over.  Since you are working in the spatial domain, you also get information about things that change spatially across the sensor.  Some of those things are fixed in time and randomly distributed in space and that’s why you can add them in quadrature to the variations that come from temporal variation across the sensor.  If your goal is to work out the spatial SNR across the detector without calibration, you can compute this kind of number but it make no sense for analyzing a real image.  Signals and gain effects that are spatially “fixed” (meaning that they have a temporal average that remains constant in position) should be calibrated out of real image data using flats.  Optical vingetting and FPN (as Janesick defines it) both fall into that category and produce the same result in that they multiply the signal by a spatially varying factor so that they can be flat calibrated in exactly the same way.   Once you do that and subtract unwanted signals, you are back to the point where you can simply look the repeatability of the measurement as a reliable measure of the (perfect) pixel-to-pixel spatial noise variation.

 

 

John

Attached Thumbnails

  • Slide1.jpg
  • Slide2.jpg

Edited by jhayes_tucson, 17 December 2017 - 01:45 AM.

  • Terry White likes this

#59 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,990
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 17 December 2017 - 02:19 AM

 

Noise is not unwanted signal.  Noise is the variation that we see when we measure something over and over and signal is the average of what we measure.

Well - I guess the ship has sailed on my suggestion not to hijack this thread and instead talk about this elsewhere.

 

So if the OP is ok with this I will just mention briefly - John your definition of noise is very limiting and is not consistent with literature usage in many domains.  There are many types of noise, but the common usage - which provides maximum clarity - is to regard noise as anything that obfuscates the signal you are trying to measure.  Noise and signal are based entirely on context - and usually the context and usage are clear.  But if in your mind you are constantly putting anything that looks like "signal" in the signal category - you will indeed be confused - and most discussions of noise models won't make sense.

 

I have provided numerous sources to back my more general usage of the term "noise" - but although many people on the web say things like, "noise must be random" - I have never once been given a citation to back that claim - or a claim that anything that looks like a signal in some context is by definition signal and cannot be noise.

 

So - it's clear the main reason you object to terms like FPN is that you think of it as inherently a signal - when to me and everyone else who uses the term - with no confusion or misunderstanding at all - it is a noise term - and it's behavior is well understood.

 

But we don't really need to discuss things in more detail if you are stuck on a particular definition of noise.  I find it so restricting that I would be tongue tied talking about any kind of measurement or noise model.  Is it a signal somehow?  Oh - then I guess it isn't noise - but it is unwanted signal.  Where does it go in the SNR calculation?

 

So I don't see any way to make progress on this topic.  You have a definition of noise that is incompatible with the literature and makes it difficult to contribute to discussions of sensor noise and noise models in CN, or to be comfortable referring to the literature and its equations, definitions, and terminology.

 

Frank


  • Jon Rista likes this

#60 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 17 December 2017 - 02:22 AM

Sorry, Frank. I read your previous post requesting the conversation on noise be moved to another thread only after I'd posted my last response. I agree, though. It might be worth starting a new thread and asking the mods to move some of the posts there. It's a good debate, but we shouldn't mess with the OP's topic. 

 

@John: Janesick's PTC characterizes the sensor. Thee sensor's characteristics are the sensor's characteristics. They don't change just because you move from a test context to a non-test context. The SNR formula is the SNR formula...it doesn't change depending on how you use the sensor. Anyway. It might be worth starting a new thread on the subject and either having our posts moved there, or copy them there and delete them here. 


Edited by Jon Rista, 17 December 2017 - 02:24 AM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics