Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Serious Flat Frame issues - ASI1600MM-C - What am I doing wrong here?!

  • Please log in to reply
59 replies to this topic

#26 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 15 December 2017 - 04:17 AM

John - I am quite surprised you are taking on the topic of FPN - since so far you have completely avoided the term.

 

Unfortunately, the literature is fairly clear on using FPN to refer to spatial noise in the bias and in dark current - but the literature is divided on the use of the term for PRNU - since it is not a noise term intrinsic to the sensor - but involves the photon signal also.

 

In addition, the odd mosaic stuff in the image you refer to - is probably due to the bayer pattern itself - and has nothing to do with FPN or PRNU in the sensor.  It appears to be an artifact of showing a bayer pattern gray scale image at lower resolution.

 

I did a study of PRNU in ASI600 and was surprised it was so small.  

 

https://www.cloudyni...-asi1600-study/

 

In that study I found the PRNU in an asi1600 to be on the order of 0.56% - which is very small and would never show in the images above.

 

So - the screendoor stuff in the image above should be ignored and has nothing to do with FPN.

 

FPN is important and shows in the bias and darks.  PRNU shows in the calibrated flats - but is usually a very small noise term - and only applies when there is a strong light signal in the image.

 

Frank

 

[addendum]

 

If this is a monochrome camera then a Bayer pattern shouldn't apply as a factor here - but at the same time I think the grid pattern is due to something odd about the way the image has been resampled.  But I don't think it is due to PRNU or FPN.


Edited by freestar8n, 15 December 2017 - 05:26 AM.

  • Jon Rista likes this

#27 cfosterstars

cfosterstars

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,893
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 15 December 2017 - 10:48 AM

You've received some good advice but I have a few things to add.

 

1)  First of all, you shouldn't have to stretch an image of your flat to display it.  You should be exposing the frame so that roughly 50-80% of the full well depth is filled.  That will be quite a bright image and will not require any stretching.  You are way underexposed if you have to stretch the image to see it.

 

2)  Here is an annotated image of what you are seeing in your flat.  The blotchy/rectilinear background that you see is something called "Fixed Pattern Noise," which is a property of the sensor itself.  It is caused by variations in gain between pixels (it can actually be caused by variations in amplifier gain, small variations in the size of the pixels, or in responsivity due to variations in the material.)  Most systems have enough optical vignetting to make FPN hard to see but you've done a nice job of bringing it out in this image (well done!)  FPN is not really a noise source since it is fixed in time but it's called "noise" because it can be treated like a noise source when evaluating temporal noise by analyzing ensembles of pixels across the sensor.  FPN is proportional to the signal strength and it will be the dominant factor causing variations in the signal across the sensor with a bright source.  CMOS sensors typically have larger FPN values than CCD sensors but neither will have much less than 1% variation due to FPN.  The reason that you shoot flats is to correct for optical vignetting, dust motes (a special case of vignetting,) and to correct for FPN.  Since FPN is proportional to the signal, it divides out of the image just like vignetting.  The bottom line is you want to record all this stuff in your flats so you are on the right track!

 

John

John

 

Please look at these galleries. THey show a lot more detail on the flats including unstretched images and histograms. I know that the exposure time is good and that it is at 50% of the full well depth. YOu can see from the histograms. If you look at unstretched flats, they show basically nothing at all and that is the problem. They show nothing irregular, but the wont calibrate correctly:

 

 
 
Finshed Rig EON side
Album: ZWO filter reflection issue
31 images
2 comments



#28 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 15 December 2017 - 11:33 AM

 

You've received some good advice but I have a few things to add.

 

1)  First of all, you shouldn't have to stretch an image of your flat to display it.  You should be exposing the frame so that roughly 50-80% of the full well depth is filled.  That will be quite a bright image and will not require any stretching.  You are way underexposed if you have to stretch the image to see it.

 

2)  Here is an annotated image of what you are seeing in your flat.  The blotchy/rectilinear background that you see is something called "Fixed Pattern Noise," which is a property of the sensor itself.  It is caused by variations in gain between pixels (it can actually be caused by variations in amplifier gain, small variations in the size of the pixels, or in responsivity due to variations in the material.)  Most systems have enough optical vignetting to make FPN hard to see but you've done a nice job of bringing it out in this image (well done!)  FPN is not really a noise source since it is fixed in time but it's called "noise" because it can be treated like a noise source when evaluating temporal noise by analyzing ensembles of pixels across the sensor.  FPN is proportional to the signal strength and it will be the dominant factor causing variations in the signal across the sensor with a bright source.  CMOS sensors typically have larger FPN values than CCD sensors but neither will have much less than 1% variation due to FPN.  The reason that you shoot flats is to correct for optical vignetting, dust motes (a special case of vignetting,) and to correct for FPN.  Since FPN is proportional to the signal, it divides out of the image just like vignetting.  The bottom line is you want to record all this stuff in your flats so you are on the right track!

 

John

John

 

Please look at these galleries. THey show a lot more detail on the flats including unstretched images and histograms. I know that the exposure time is good and that it is at 50% of the full well depth. YOu can see from the histograms. If you look at unstretched flats, they show basically nothing at all and that is the problem. They show nothing irregular, but the wont calibrate correctly:

 

 

The reason that your flats don't calibrate correctly is that you have a lot of stray light.  You need to remove the camera, point your scope at a bright source (like a window or your flat panel) and look backwards through the system.  If you look carefully, you'll see reflections from various components such as the inside of extender tubes, adapters, and such. You need to either flock these surfaces or paint them with flat black paint.  Be careful to catch all of the sources of stray reflections and that will fix your problem.

 

John



#29 cfosterstars

cfosterstars

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,893
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 15 December 2017 - 11:49 AM

 

 

You've received some good advice but I have a few things to add.

 

1)  First of all, you shouldn't have to stretch an image of your flat to display it.  You should be exposing the frame so that roughly 50-80% of the full well depth is filled.  That will be quite a bright image and will not require any stretching.  You are way underexposed if you have to stretch the image to see it.

 

2)  Here is an annotated image of what you are seeing in your flat.  The blotchy/rectilinear background that you see is something called "Fixed Pattern Noise," which is a property of the sensor itself.  It is caused by variations in gain between pixels (it can actually be caused by variations in amplifier gain, small variations in the size of the pixels, or in responsivity due to variations in the material.)  Most systems have enough optical vignetting to make FPN hard to see but you've done a nice job of bringing it out in this image (well done!)  FPN is not really a noise source since it is fixed in time but it's called "noise" because it can be treated like a noise source when evaluating temporal noise by analyzing ensembles of pixels across the sensor.  FPN is proportional to the signal strength and it will be the dominant factor causing variations in the signal across the sensor with a bright source.  CMOS sensors typically have larger FPN values than CCD sensors but neither will have much less than 1% variation due to FPN.  The reason that you shoot flats is to correct for optical vignetting, dust motes (a special case of vignetting,) and to correct for FPN.  Since FPN is proportional to the signal, it divides out of the image just like vignetting.  The bottom line is you want to record all this stuff in your flats so you are on the right track!

 

John

John

 

Please look at these galleries. THey show a lot more detail on the flats including unstretched images and histograms. I know that the exposure time is good and that it is at 50% of the full well depth. YOu can see from the histograms. If you look at unstretched flats, they show basically nothing at all and that is the problem. They show nothing irregular, but the wont calibrate correctly:

 

 

The reason that your flats don't calibrate correctly is that you have a lot of stray light.  You need to remove the camera, point your scope at a bright source (like a window or your flat panel) and look backwards through the system.  If you look carefully, you'll see reflections from various components such as the inside of extender tubes, adapters, and such. You need to either flock these surfaces or paint them with flat black paint.  Be careful to catch all of the sources of stray reflections and that will fix your problem.

 

John

 

John,

 

Thanks for the feedback. I am PAINFULLY well aware of the stray light issue. It is the subject of a huge other thread. I don't know if you are following it, but there is a ton of data on that thread - everything you suggest and more is there:

 

https://www.cloudyni...ring-artifacts/

 

This thread documents all the work I have already done and many others have done with this issue. The main issue is that the ZWO filters are not coated to the edge and leak like a sieve. This has not been that straight forward since it is multiple issues, but the filters are the big issue. Each incremental improvement has made a substantive improvement in the imaging, but I have not gotten everything fixed yet. Work in progress---

 

Thanks,

 

Chris


Edited by cfosterstars, 15 December 2017 - 11:52 AM.


#30 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 15 December 2017 - 11:52 AM

John - I am quite surprised you are taking on the topic of FPN - since so far you have completely avoided the term.

 

Unfortunately, the literature is fairly clear on using FPN to refer to spatial noise in the bias and in dark current - but the literature is divided on the use of the term for PRNU - since it is not a noise term intrinsic to the sensor - but involves the photon signal also.

 

In addition, the odd mosaic stuff in the image you refer to - is probably due to the bayer pattern itself - and has nothing to do with FPN or PRNU in the sensor.  It appears to be an artifact of showing a bayer pattern gray scale image at lower resolution.

 

I did a study of PRNU in ASI600 and was surprised it was so small.  

 

https://www.cloudyni...-asi1600-study/

 

In that study I found the PRNU in an asi1600 to be on the order of 0.56% - which is very small and would never show in the images above.

 

So - the screendoor stuff in the image above should be ignored and has nothing to do with FPN.

 

FPN is important and shows in the bias and darks.  PRNU shows in the calibrated flats - but is usually a very small noise term - and only applies when there is a strong light signal in the image.

 

Frank

 

[addendum]

 

If this is a monochrome camera then a Bayer pattern shouldn't apply as a factor here - but at the same time I think the grid pattern is due to something odd about the way the image has been resampled.  But I don't think it is due to PRNU or FPN.

 

I didn't notice the "-C" on the camera so I stand corrected and agree that in this case, it is more likely due to the Bayer filter; however, this is almost exactly what FPN looks like for a CMOS sensor.  The slides are numbered so I can't reference the page but Richard Crisp shows it very clearly in this presentation--about half way through:

 

http://www.narrowban...sp2013_talk.pdf

 

I still disagree with calling FPN a noise term because it is not a noise term.  However, It can be treated as a noise term when using an ensemble of pixels to work out the statistical properties of an image.  Previous authors have sub-divided the parameters into just "signal" and "noise" terms and in this case we have an ambiguous parameter that acts like a signal term in that it can be directly subtracted from the image and a noise term when analyzing the temporal statistics of many sensors across the image.  

 

 

 

 

You've received some good advice but I have a few things to add.

 

1)  First of all, you shouldn't have to stretch an image of your flat to display it.  You should be exposing the frame so that roughly 50-80% of the full well depth is filled.  That will be quite a bright image and will not require any stretching.  You are way underexposed if you have to stretch the image to see it.

 

2)  Here is an annotated image of what you are seeing in your flat.  The blotchy/rectilinear background that you see is something called "Fixed Pattern Noise," which is a property of the sensor itself.  It is caused by variations in gain between pixels (it can actually be caused by variations in amplifier gain, small variations in the size of the pixels, or in responsivity due to variations in the material.)  Most systems have enough optical vignetting to make FPN hard to see but you've done a nice job of bringing it out in this image (well done!)  FPN is not really a noise source since it is fixed in time but it's called "noise" because it can be treated like a noise source when evaluating temporal noise by analyzing ensembles of pixels across the sensor.  FPN is proportional to the signal strength and it will be the dominant factor causing variations in the signal across the sensor with a bright source.  CMOS sensors typically have larger FPN values than CCD sensors but neither will have much less than 1% variation due to FPN.  The reason that you shoot flats is to correct for optical vignetting, dust motes (a special case of vignetting,) and to correct for FPN.  Since FPN is proportional to the signal, it divides out of the image just like vignetting.  The bottom line is you want to record all this stuff in your flats so you are on the right track!

 

John

John

 

Please look at these galleries. THey show a lot more detail on the flats including unstretched images and histograms. I know that the exposure time is good and that it is at 50% of the full well depth. YOu can see from the histograms. If you look at unstretched flats, they show basically nothing at all and that is the problem. They show nothing irregular, but the wont calibrate correctly:

 

 

The reason that your flats don't calibrate correctly is that you have a lot of stray light.  You need to remove the camera, point your scope at a bright source (like a window or your flat panel) and look backwards through the system.  If you look carefully, you'll see reflections from various components such as the inside of extender tubes, adapters, and such. You need to either flock these surfaces or paint them with flat black paint.  Be careful to catch all of the sources of stray reflections and that will fix your problem.

 

John

 

John,

 

I am well aware of the stray light issue. It is the subject of a huge other thread. I dont know if you are following it, but there is a ton of data on that thread:

 

https://www.cloudyni...ring-artifacts/

 

This thread documents all the work I have already done and many others have done with this issue. The main issue is that the ZWO filters are not coated to the edge and leak like a sieve. This has not been that straight foward since it is multiple issues, but the filters are the big issue. 

 

 

I agree that edge reflections can be a source of stray light as well and I'm glad that you are aware of it.  As I said, if you fix the stray light issues, your flats should calibrate just fine.

 

John



#31 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 15 December 2017 - 12:19 PM

John - I am quite surprised you are taking on the topic of FPN - since so far you have completely avoided the term.

 

Unfortunately, the literature is fairly clear on using FPN to refer to spatial noise in the bias and in dark current - but the literature is divided on the use of the term for PRNU - since it is not a noise term intrinsic to the sensor - but involves the photon signal also.

 

In addition, the odd mosaic stuff in the image you refer to - is probably due to the bayer pattern itself - and has nothing to do with FPN or PRNU in the sensor.  It appears to be an artifact of showing a bayer pattern gray scale image at lower resolution.

 

I did a study of PRNU in ASI600 and was surprised it was so small.  

 

https://www.cloudyni...-asi1600-study/

 

In that study I found the PRNU in an asi1600 to be on the order of 0.56% - which is very small and would never show in the images above.

 

So - the screendoor stuff in the image above should be ignored and has nothing to do with FPN.

 

FPN is important and shows in the bias and darks.  PRNU shows in the calibrated flats - but is usually a very small noise term - and only applies when there is a strong light signal in the image.

 

Frank

 

[addendum]

 

If this is a monochrome camera then a Bayer pattern shouldn't apply as a factor here - but at the same time I think the grid pattern is due to something odd about the way the image has been resampled.  But I don't think it is due to PRNU or FPN.

 

I didn't notice the "-C" so I stand corrected.  I agree that the pattern is probably mostly due to the Bayer filter plus FPN.   Flats are a good way to look at FPN and you will see a similar (though finer) rectilinear screen-door pattern for most CMOS sensors. There aren't any page numbers but Richard Crisp clearly shows FPN for a CMOS camera in this presentation--about half way down:

 

http://www.narrowban...sp2013_talk.pdf

 

I have never agreed with calling FPN a noise term because it is not a noise term.  Previous authors have subdivided most everything into either being a "signal" or a "noise" term but FPN is a special case that can be treated either way.  It can act like a signal and be directly removed from an image or it can be treated as a noise term (adding in quadrature) when analyzing temporal statistics using an ensemble of pixels across the array.  I personally would have called it something like "fixed pattern gain" to avoid confusion.  Now we are saddled with referring to things like "DFPN" and "DFPN shot noise."

 

To be precise FPN is proportional to signal and does not show up in the dark or bias data.  Dark FPN (DFPN) is different than light FPN and will include both dark current signal and trap leakage terms.  I haven't said anything about PRNU so I'm not going to get into that here.

 

John



#32 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 15 December 2017 - 07:11 PM

I have never agreed with calling FPN a noise term because it is not a noise term.  Previous authors have subdivided most everything into either being a "signal" or a "noise" term but FPN is a special case that can be treated either way.  It can act like a signal and be directly removed from an image or it can be treated as a noise term (adding in quadrature) when analyzing temporal statistics using an ensemble of pixels across the array.  I personally would have called it something like "fixed pattern gain" to avoid confusion.  Now we are saddled with referring to things like "DFPN" and "DFPN shot noise."

 

To be fair, John, there is a LOT of literature out there that uses the terms FPN and DFPS to refer to noise. Every time I come across the terms, that's how they are used.

 

I also want to note that it's just DFPN. I don't think we should try introducing DFPN Shot Noise. That is simply dark current noise, which is the same thing you've been familiar with all along: temporally random noise that is the square root of the dark current. DFPN is the fixed component of the dark signal offset, due to DSNU, the non-uniform response of each pixel to that dark signal. The offset grows linearly with time (and therefor exposure), varying in space (randomly, non-randomly, a combination of the two).

 

It is considered a noise by many, in the amateur community as well as the scientific community, because it does not represent the target signal you are after. Anything OTHER than the deep space object signal can be considered a noise.



#33 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 15 December 2017 - 09:10 PM

I think I confused things by thinking it was a color/bayer camera - but it is also confusing because ASI1600MM-C is the mono version and the C means "Cool."  The color version is ASI1600MC-C.

 

I don't know where the mottling/screendoor effect comes from in the flat - but I do see similar things when I look at reduced versions of my flats with the asi1600.  And my noise study of that camera relied on highpass filter to study the prnu - so I may have filtered out this longer range effect.  I may take another look at it.

 

But getting to the main questions of this thread - I thought the OP, Chris, had found the main problem with his system - and that is 1)  filters leaking light around the edge and 2)  reflections off an insufficiently blackened tube in the optics.  So I'm not sure what problems remain.

 

Separately, I do see arcs and other anomalies in my flats - especially narrow band - but the flats still work.

 

Here is an example:

 

This is an Ha flat with EdgeHD11 at f/10

 

hourglasshaSmall.jpg

 

This is an Oiii flat with EdgeHD11 at f/10

 

hourglassoiiiSmall.jpg

 

Here is a stacked, linear, stretched image in Oiii and there is no sign of the arcs

 

ngc6188oiiiSmall.jpg

 

The flats were taken with blue sky and no t-shirt - and despite the circular arcs they seem to work - so the flats are faithfully capturing the way light arrives on the sensor.  I don't know exactly what causes the arcs - but they aren't a problem as long as they are identically behaved in the light as in the flats.

 

These are 1.25" Astrodon 3nm filters in an Atik EFW2 filter wheel.

 

If the problems in the system have been identified it may be fine that arcs and other oddities remain.  If the flats still don't calibrate well, I recommend blue sky flats because they are very bright and allow short exposures - and should be less prone to light leaks as a result.

 

As for FPN and so forth - my main point is that the variation in dark current across the sensor acts as a noise term in every sense of the word - the only thing slightly unusual about it as a noise term is that it is constant in every exposure.  As a noise term it adds in quadrature with other noise terms in the image.  It just happens to be repeatable so you can measure it and subtract it - and there should be nothing confusing about it at all.  The terminology has been honed in the sensor literature just like in any other field - for maximum clarity.

 

The only slight subtlety is that PRNU is different - because it multiplies the photon signal and isn't an independent noise term like FPN.  So I refer to it as another pattern noise in the image - but it isn't really an FPN term.

 

In all this terminology my usage is consistent with the literature and I find it has maximum clarity to describe what is going on and how the various noise terms contribute to the final image.

 

Frank


Edited by freestar8n, 15 December 2017 - 11:06 PM.


#34 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 15 December 2017 - 09:36 PM

One other point - the sharply defined spots in the flat are due to stuff directly on the sensor coverslip - which is in the dry chamber of the camera and below the window.  Stuff on the window will be much more blurred - and corresponds to the larger donuts visible.  Then you see some really large donuts - and I think they are on the filter itself.

 

Note that all the specks and donuts are the same in the Ha and Oiii flats - except for the very large donuts.  So that is consistent with only those big donuts being due to the filters themselves.

 

The only ones that are really a problem are the ones on the sensor itself - but dithering largely gets rid of them.  If you look in the lower left of my image you see two separate black smudges caused by the big black blob down there - and there are two of them because the stack consisted of two separate exposure periods with slightly different framing - and each set was dithered.

 

Unfortunately I have the earlier version of the ASI1600 that unscrews easily to expose the dessicant tablets.  So I think it is more susceptible to particles reaching the sensor surface.  And it's hard to clean because the tablets get exposed.  But the specks are annoying and I will probably clean it.

 

All the other donuts and arcs don't appear to be a problem - with blue sky flats.

 

Frank


Edited by freestar8n, 15 December 2017 - 09:38 PM.


#35 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 15 December 2017 - 11:36 PM

Frank, I suspect because you do sky flats, even if you do have some other reflections from the imaging train, the light from the sky is far enough away that it is more collimated like the light from space.

 

I have found that reflections from the imaging train are more problematic and may not properly correct when the flat surface is close to the scope when taking flats. The light is then not collimated the same, it may be totally scattered, who knows for sure. But it doesn't reflect at the same angles, so the flat structure ends up different...slightly, by a lot, depends on the system. 



#36 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 16 December 2017 - 12:04 AM

Hi Jon-

 

I agree it's important to have the light source exactly duplicate light from the sky, and I don't think flat panels or t-shirts do that well.  But at the same time I'm not convinced the rings are due to a reflection.  They may be - but I'm not sure.

 

The components of the imaging train are shown here:

 

http://www.astrogeek...GFreestar8n.pdf

 

and nothing is very constricted - particularly at f/10 where the light cone is fairly narrow.

 

Plus, the rings are completely different for Ha vs. Oiii - and I think they are absent in other filters.

 

So - I don't know what the cause is - but I think that if it were reflections then it would be very hard for the flats to work well in calibration.  I think in that case they would act similarly to a pupil ghost - and those are notoriously hard to deal with because they just don't calibrate away - and sky or panel flats won't matter because the pupil ghost just can't be duplicated.

 

So my current thinking is they are weird and unknown, but somewhat common - and not indicative of a problem.  I could be wrong and maybe there is a reflection involved - but right now I don't think so.

 

Frank



#37 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 December 2017 - 12:09 AM

I think I confused things by thinking it was a color/bayer camera - but it is also confusing because ASI1600MM-C is the mono version and the C means "Cool."  The color version is ASI1600MC-C.

 

I don't know where the mottling/screendoor effect comes from in the flat - but I do see similar things when I look at reduced versions of my flats with the asi1600.  And my noise study of that camera relied on highpass filter to study the prnu - so I may have filtered out this longer range effect.  I may take another look at it.

 

But getting to the main questions of this thread - I thought the OP, Chris, had found the main problem with his system - and that is 1)  filters leaking light around the edge and 2)  reflections off an insufficiently blackened tube in the optics.  So I'm not sure what problems remain.

 

Separately, I do see arcs and other anomalies in my flats - especially narrow band - but the flats still work.

 

Here is an example:

 

This is an Ha flat with EdgeHD11 at f/10

 

attachicon.gifhourglasshaSmall.jpg

 

This is an Oiii flat with EdgeHD11 at f/10

 

attachicon.gifhourglassoiiiSmall.jpg

 

Here is a stacked, linear, stretched image in Oiii and there is no sign of the arcs

 

attachicon.gifngc6188oiiiSmall.jpg

 

The flats were taken with blue sky and no t-shirt - and despite the circular arcs they seem to work - so the flats are faithfully capturing the way light arrives on the sensor.  I don't know exactly what causes the arcs - but they aren't a problem as long as they are identically behaved in the light as in the flats.

 

These are 1.25" Astrodon 3nm filters in an Atik EFW2 filter wheel.

 

If the problems in the system have been identified it may be fine that arcs and other oddities remain.  If the flats still don't calibrate well, I recommend blue sky flats because they are very bright and allow short exposures - and should be less prone to light leaks as a result.

 

As for FPN and so forth - my main point is that the variation in dark current across the sensor acts as a noise term in every sense of the word - the only thing slightly unusual about it as a noise term is that it is constant in every exposure.  As a noise term it adds in quadrature with other noise terms in the image.  It just happens to be repeatable so you can measure it and subtract it - and there should be nothing confusing about it at all.  The terminology has been honed in the sensor literature just like in any other field - for maximum clarity.

 

The only slight subtlety is that PRNU is different - because it multiplies the photon signal and isn't an independent noise term like FPN.  So I refer to it as another pattern noise in the image - but it isn't really an FPN term.

 

In all this terminology my usage is consistent with the literature and I find it has maximum clarity to describe what is going on and how the various noise terms contribute to the final image.

 

Frank

 

1)  Well, I was confused as well about which camera the OP has.  I originally read it as mono-cooled but then when you pointed out the -C, I though that I had the designator wrong.  Thanks for the clarification.

 

2)  I believe that you were the one that suggested Janesick as an authoritative reference so perhaps you should go back to review section 3.4 (pp 30-33) of "Photon Transfer."  FPN, as he defines it, has nothing to do with dark current.  It is caused by variations in charge collection efficiency and responsivity from pixel to pixel that results in sensitivity variations across the sensor.  Did you look at Richard Crisp's data?  He uses the same definition in all of his (many) articles.  Both of these guys refer to the fixed spatial pattern formed by dark current signal as either "Dark FPN" or just "DFPN."  This is how I have used the terms.

 

The problem is that different authors use the same terminology to mean different things.  Here is one such example:  Gerald Holst and Terrence Lomheim, in "CMOS/CCD Sensors and Camera Systems," (SPIE Press, 2nd ed, 2011, p181-183) use the term FPN to refer to the spatial variation in dark signal as just "FPN" and the effects of pixel to pixel responsivity variation across the sensor that vary directly as signal strength as "PRNU."  Conversely, these authors refer to DFPN as simply "FPN."   It sounds like this is the terminology that you are using.  So...not only are we calling signals noise; but, we are naming things completely differently.  It seems that you can toss any concept of honing terminology for "maximum clarity" right out the window.  

 

Putting all of that aside, in all likelihood, the pattern that you see has nothing to with the light source or the filters.  The cross hatch pattern that you see in your flat data is caused by FPN as defined by Janesick (or PRNU as defined by Holst and Lomheim) in your sensor.  I think that if you make a good flat using the sky and display the data as you did here, you will see the exact same pattern--as Crisp has already demonstrated using sky flats.  That pattern is the result of the characteristics of the sensor--not the way that it is illuminated.

 

John



#38 cfosterstars

cfosterstars

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,893
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 16 December 2017 - 12:20 AM

 

I think I confused things by thinking it was a color/bayer camera - but it is also confusing because ASI1600MM-C is the mono version and the C means "Cool."  The color version is ASI1600MC-C.

 

I don't know where the mottling/screendoor effect comes from in the flat - but I do see similar things when I look at reduced versions of my flats with the asi1600.  And my noise study of that camera relied on highpass filter to study the prnu - so I may have filtered out this longer range effect.  I may take another look at it.

 

But getting to the main questions of this thread - I thought the OP, Chris, had found the main problem with his system - and that is 1)  filters leaking light around the edge and 2)  reflections off an insufficiently blackened tube in the optics.  So I'm not sure what problems remain.

 

Separately, I do see arcs and other anomalies in my flats - especially narrow band - but the flats still work.

 

Here is an example:

 

This is an Ha flat with EdgeHD11 at f/10

 

attachicon.gifhourglasshaSmall.jpg

 

This is an Oiii flat with EdgeHD11 at f/10

 

attachicon.gifhourglassoiiiSmall.jpg

 

Here is a stacked, linear, stretched image in Oiii and there is no sign of the arcs

 

attachicon.gifngc6188oiiiSmall.jpg

 

The flats were taken with blue sky and no t-shirt - and despite the circular arcs they seem to work - so the flats are faithfully capturing the way light arrives on the sensor.  I don't know exactly what causes the arcs - but they aren't a problem as long as they are identically behaved in the light as in the flats.

 

These are 1.25" Astrodon 3nm filters in an Atik EFW2 filter wheel.

 

If the problems in the system have been identified it may be fine that arcs and other oddities remain.  If the flats still don't calibrate well, I recommend blue sky flats because they are very bright and allow short exposures - and should be less prone to light leaks as a result.

 

As for FPN and so forth - my main point is that the variation in dark current across the sensor acts as a noise term in every sense of the word - the only thing slightly unusual about it as a noise term is that it is constant in every exposure.  As a noise term it adds in quadrature with other noise terms in the image.  It just happens to be repeatable so you can measure it and subtract it - and there should be nothing confusing about it at all.  The terminology has been honed in the sensor literature just like in any other field - for maximum clarity.

 

The only slight subtlety is that PRNU is different - because it multiplies the photon signal and isn't an independent noise term like FPN.  So I refer to it as another pattern noise in the image - but it isn't really an FPN term.

 

In all this terminology my usage is consistent with the literature and I find it has maximum clarity to describe what is going on and how the various noise terms contribute to the final image.

 

Frank

 

1)  Well, I was confused as well about which camera the OP has.  I originally read it as mono-cooled but then when you pointed out the -C, I though that I had the designator wrong.  Thanks for the clarification.

 

2)  I believe that you were the one that suggested Janesick as an authoritative reference so perhaps you should go back to review section 3.4 (pp 30-33) of "Photon Transfer."  FPN, as he defines it, has nothing to do with dark current.  It is caused by variations in charge collection efficiency and responsivity from pixel to pixel that results in sensitivity variations across the sensor.  Did you look at Richard Crisp's data?  He uses the same definition in all of his (many) articles.  Both of these guys refer to the fixed spatial pattern formed by dark current signal as either "Dark FPN" or just "DFPN."  This is how I have used the terms.

 

The problem is that different authors use the same terminology to mean different things.  Here is one such example:  Gerald Holst and Terrence Lomheim, in "CMOS/CCD Sensors and Camera Systems," (SPIE Press, 2nd ed, 2011, p181-183) use the term FPN to refer to the spatial variation in dark signal as just "FPN" and the effects of pixel to pixel responsivity variation across the sensor that vary directly as signal strength as "PRNU."  Conversely, these authors refer to DFPN as simply "FPN."   It sounds like this is the terminology that you are using.  So...not only are we calling signals noise; but, we are naming things completely differently.  It seems that you can toss any concept of honing terminology for "maximum clarity" right out the window.  

 

Putting all of that aside, in all likelihood, the pattern that you see has nothing to with the light source or the filters.  The cross hatch pattern that you see in your flat data is caused by FPN as defined by Janesick (or PRNU as defined by Holst and Lomheim) in your sensor.  I think that if you make a good flat using the sky and display the data as you did here, you will see the exact same pattern--as Crisp has already demonstrated using sky flats.  That pattern is the result of the characteristics of the sensor--not the way that it is illuminated.

 

John

 

John,

 

The ASI1600MM-C is a mono camera. That is what I have and that is what has been generating these flats issues. The rings and edge effects - as you point out - are definitely due to reflections - and from multiple sources at different degrees of severity. At the current stage of debug, I am only left with color artifacts in the very corner and coming from mostly my OIII filter. My previous fix had the Ha working well but the SII was terrible. The second iteration now has both the Ha and SII mostly clean, but the OIII is now the limiter. I still have some very subtle concentric rings on all filters that I believe are coming from diffuse reflection off the focuser draw tube that is already blackened but not enough. I am going to recoat it blacker with BLACK 2.0 paint when I get a few more items for my rig. I have to do a tear down anyway to improve a few things so I am going to get all done in one shot. 

 

Anyway, I will be posting my results from the latest filter mask/tape job shortly on a new thread. They are clearly better than before and easily my best images to date - not that they are wart free, but much better.

 

On the subject of flats, I have taken my flats four different ways: Flatman EM panel, Home made light box, tee shirt sky flats and bare bright sky flats without a diffuser - there was no measureable difference in the quality of the issues. They all produced the same degree of artifacts in the final images. Once I clean up all the reflections, I will redo the test by processing the same lights with four different sets of flats. I really want to determine the effect for myself with a controlled experiment.


Edited by cfosterstars, 16 December 2017 - 12:24 AM.


#39 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 12:21 AM

I generally try to go by Janesick's definitions. FPN for signal and gain (PRNU) related fixed pattern, DFPN for dark signal (DSNU) related fixed pattern. 



#40 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 12:23 AM

 

Plus, the rings are completely different for Ha vs. Oiii - and I think they are absent in other filters.

 

I guess that isn't all that surprising. The former two are narrow band...so they might only transmit certain reflections. The rest are broadband, so they would transmit a lot more reflection.

 

I am not otherwise disagreeing, though. There can be many reasons for artifacts in flats. No question about that. ;) 



#41 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 December 2017 - 12:24 AM

I also want to note that it's just DFPN. I don't think we should try introducing DFPN Shot Noise. That is simply dark current noise, which is the same thing you've been familiar with all along: temporally random noise that is the square root of the dark current. DFPN is the fixed component of the dark signal offset, due to DSNU, the non-uniform response of each pixel to that dark signal. The offset grows linearly with time (and therefor exposure), varying in space (randomly, non-randomly, a combination of the two).

 

It is considered a noise by many, in the amateur community as well as the scientific community, because it does not represent the target signal you are after. Anything OTHER than the deep space object signal can be considered a noise.

 

1)  I didn't introduce DFPN Shot Noise.  That comes from Janesick, "Photon Transfer", p167.  This is an effort to separate temporal noise from spatial noise, which results from using the ensemble of pixels across the sensor to assess measurement uncertainty.

 

2)  I come from a metrology background where uncertainty in a measurement is well defined as the statical to measurement variations.  There are wanted signals and unwanted signals both of which can be measured to within some uncertainty.  As such, I strongly disagree with defining noise as anything that is unwanted.  But I also recognize that opinions in the amateur world are hard to change and I don't have the energy or time to try to straighten this out so work with it anyway you like.

 

John



#42 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 December 2017 - 12:31 AM

Frank, I suspect because you do sky flats, even if you do have some other reflections from the imaging train, the light from the sky is far enough away that it is more collimated like the light from space.

 

I have found that reflections from the imaging train are more problematic and may not properly correct when the flat surface is close to the scope when taking flats. The light is then not collimated the same, it may be totally scattered, who knows for sure. But it doesn't reflect at the same angles, so the flat structure ends up different...slightly, by a lot, depends on the system. 

 

Jon,

The sky is an extended, diffuse source.  It does not provide collimated light.  The main difference between the sky and a panel is that the panel will be Lambertian; whereas the sky is uniform.

 

John



#43 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 12:37 AM

I generally go by the SNR equation:

 

SNR = S/N

 

Anything in the denominator there is noise. The noise in a single uncalibrated sub would be:

 

N = SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

That leaves your final object SNR formula as:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

The only thing in the numerator here is the object signal. Everything else, including the FPN and DFPN terms, are noise. The signal from the sky, which sure is a signal, has an IMPACT on the final SNR that leaves it only behaving as a noise. Same goes for dark current. It, too, is a signal, but the IMPACT on the final SNR leaves it only behaving as a noise.

 

When we calibrate, we remove the FPN components, so:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2)

 

We end up back at the more familiar SNR formula. Technically speaking, calibration doesn't eliminate the FPN. Even if it perfectly removes the fixed sensor patterns, the random noise pattern of the master dark itself becomes a new FPN term. Stack enough undithered subs, and spatially random remainder of the noise in the dark (subtract one master dark from another, and you'll get this) will become the limiting factor on your SNR, as it will be in every calibrated sub. Thing is, this remnant FPN is totaly spatially random, it is otherwise imperceptible from temporally random noise...because it was derived from temporally random noise. Is it a signal? Or a noise? It's definitely FPN! :p



#44 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 December 2017 - 12:38 AM

I generally try to go by Janesick's definitions. FPN for signal and gain (PRNU) related fixed pattern, DFPN for dark signal (DSNU) related fixed pattern. 

Janesick defines FPN as the result of variation in sensitivity.  Holst and Lomheim define PRNU as due to gain variation.  If gain refers to the amplifier then I can see how it would affect a CCD differently but in a CMOS device, I believe that both will combine to affect the signal in exactly the same way.

 

Please explain the difference between DFPN and DSNU.

 

John



#45 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 12:39 AM

 

Frank, I suspect because you do sky flats, even if you do have some other reflections from the imaging train, the light from the sky is far enough away that it is more collimated like the light from space.

 

I have found that reflections from the imaging train are more problematic and may not properly correct when the flat surface is close to the scope when taking flats. The light is then not collimated the same, it may be totally scattered, who knows for sure. But it doesn't reflect at the same angles, so the flat structure ends up different...slightly, by a lot, depends on the system. 

 

Jon,

The sky is an extended, diffuse source.  It does not provide collimated light.  The main difference between the sky and a panel is that the panel will be Lambertian; whereas the sky is uniform.

 

John

 

Ok, fair enough. Point remains though. Reflections in flats from panels vs. flats from the sky can differ. I've had more problems with well diffused panel flats than sky flats. 



#46 Peter in Reno

Peter in Reno

    Voyager 1

  • *****
  • Posts: 12,310
  • Joined: 15 Jul 2008
  • Loc: Reno, NV

Posted 16 December 2017 - 12:47 AM

I have used triple layer T-shirt flats using sunlight for at least five years without issues. It even works under cloudy skies.

 

Peter



#47 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,997
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 December 2017 - 12:50 AM

 

I generally try to go by Janesick's definitions. FPN for signal and gain (PRNU) related fixed pattern, DFPN for dark signal (DSNU) related fixed pattern. 

Janesick defines FPN as the result of variation in sensitivity.  Holst and Lomheim define PRNU as due to gain variation.  If gain refers to the amplifier then I can see how it would affect a CCD differently but in a CMOS device, I believe that both will combine to affect the signal in exactly the same way.

 

Please explain the difference between DFPN and DSNU.

 

John

 

FPN will be affected by gain in terms of the magnitude of the "noise" that FPN represents. Lets just say the PRNU results in spatially random noise that follows a gaussian distribution for an otherwise perfectly flat evenly lit neutral surface, after stacking enough subs to average into oblivion the shot noise. As gain increases, the standard deviation of that distribution will also increase. Not all FPN is purely gaussian...in fact, I've found most has some recognizable pattern to it, underneath optical issues like dust motes and vignetting. So you are correct, FPN the result of both gain and signal.

 

DFPN is dark FPN. DSNU is, like PRNU, the non-uniform response of each pixel to dark signal (Dark Signal Non-Uniformity). DFPN arises because of DSNU. If there was no DSNU, then the dark offset introduced by dark current would be entirely uniform, and thus a static offset across the entire sensor...no spatially random noise apperance, no hot or cold pixels, no banding, no glows. You would have shot noise, though (which again could be averaged into oblivion!) ;) DFPN exists because of DSNU like FPN exists because of PRNU.



#48 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 December 2017 - 01:05 AM

I generally go by the SNR equation:

 

SNR = S/N

 

Anything in the denominator there is noise. The noise in a single uncalibrated sub would be:

 

N = SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

That leaves your final object SNR formula as:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2 + (Sobject*PRNU)^2 + (Sdark*DSNU)^2)

 

The only thing in the numerator here is the object signal. Everything else, including the FPN and DFPN terms, are noise. The signal from the sky, which sure is a signal, has an IMPACT on the final SNR that leaves it only behaving as a noise. Same goes for dark current. It, too, is a signal, but the IMPACT on the final SNR leaves it only behaving as a noise.

 

When we calibrate, we remove the FPN components, so:

 

SNR = Sobject/SQRT(Nobject^2 + Nsky^2 + Ndark^2 + Nread^2)

 

We end up back at the more familiar SNR formula. Technically speaking, calibration doesn't eliminate the FPN. Even if it perfectly removes the fixed sensor patterns, the random noise pattern of the master dark itself becomes a new FPN term. Stack enough undithered subs, and spatially random remainder of the noise in the dark (subtract one master dark from another, and you'll get this) will become the limiting factor on your SNR, as it will be in every calibrated sub. Thing is, this remnant FPN is totaly spatially random, it is otherwise imperceptible from temporally random noise...because it was derived from temporally random noise. Is it a signal? Or a noise? It's definitely FPN! tongue2.gif

 

The total signal that you get is the object signal plus the unwanted signal from the sky.  The uncertainty in the sum of those two signals decreases the resulting SNR.  In this case, you cannot remove the unwanted signal--all you can do is to increase SNR to reduce the effect of the unwanted signal.  Regardless, sky fog is not noise.  It is unwanted signal (with it's own contribution to the total noise.)

 

As for your calculation, this is where conflating spatial noise with temporal noise creates problems.  Right up front, your definition of noise isn't right.  You are adding the spatial rms variation of FPN into the noise term and in this case, that's not right.  Those are fixed values (hence the 'F' in the name.)  Flat calibration does indeed remove FPN and that's part of the reason that you calibrate the image.  So...I agree with your final SNR equation.  BTW, both FPN and DFPN are removed during calibration.  Subtracting both introduces a shot noise contribution, but that's orders of magnitude smaller than what FPN variations introduce across the sensor.

 

John



#49 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 16 December 2017 - 01:28 AM

I'm happy to have a discussion about this noise terminology elsewhere - but this is definitely a side topic from the OP's - and I don't have a sense he is interested.  So I recommend another thread.  I have spawned threads on pattern noise numerous times so I'm happy for someone else to give it a shot.

 

One reason to get back to the flats issue is that I'm still not sure what the OP's problem is with his flats.  I'm also not sure what role reflection has played in the rings - or if they are now gone now after darkening things.

 

[Edit]  I just saw his note above among the other notes about noise - and he says that darkening things has reduced the rings, but they still remain and he hopes to get rid of them.  So if he can in fact get rid of the rings, and if they look a lot like mine, then that will be interesting to me.  But it doesn't explain why flats work for me despite the rings.

 

The issue of the checkerboardish pattern applies to his flats and mine and my main point is - I had never seen it before and I don't see it at all in my flats when I view them at 100%.  I also don't see it in uncalibrated lights - whereas I do see vignetting and dust - etc.  It's a very subtle thing if it is present - but certainly flats are good for more than just vignetting and dust donuts - because they correct for prnu also - whatever form it takes.

 

Frank


Edited by freestar8n, 16 December 2017 - 01:36 AM.


#50 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,994
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 16 December 2017 - 01:35 AM

I have used triple layer T-shirt flats using sunlight for at least five years without issues. It even works under cloudy skies.

 

Peter

There are definitely people who have no problems with flats and they are easy to take and work fine - but for others it can be more delicate.  

 

Peter - I forget, but did you ever work with sct's?  Refractors may make things easier.

 

And Jon Rista is mostly DSLR lenses?

 

The OP appears to have and SCT like John and me.

 

Frank




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics