70 replies to this topic

[AXAF]

Getting back to my original point..problem solved.

 

First, John, you were correct. I had a serious light leak. I simply placed my flat panel on top of

my refractor, with the instrument in the vertical position. When I took a 60-s dark, obviously with

the panel turned off, and ambient lighting in the room, the dark frame was completely

saturated. Turned off the room lights, took another dark frame, and it looked as it should.

 

Second, I was not calibrating my flats against darks and biases properly.

 

Here is a stack of 100 frames. No more artifacts.

 

-Gary

 

 

Attached Thumbnails

• 

[jhayes_tucson]

Getting back to my original point..problem solved.

 

First, John, you were correct. I had a serious light leak. I simply placed my flat panel on top of

my refractor, with the instrument in the vertical position. When I took a 60-s dark, obviously with

the panel turned off, and ambient lighting in the room, the dark frame was completely

saturated. Turned off the room lights, took another dark frame, and it looked as it should.

 

Second, I was not calibrating my flats against darks and biases properly.

 

Here is a stack of 100 frames. No more artifacts.

 

-Gary

 

Gary,

Good work!  That looks perfect.  

 

I look forward to seeing the final image...

 

John

[Allaboutastro]

 

People often talk about noise in their images and are concerned about PRNU causing noise in the dark regions - where PRNU is likely not involved at all.  It could be if the sky background is really bright - but it's unlikely.  PRNU would only reveal itself in smooth, bright parts of the image. 

 

This is where I agree with Frank.  No matter the source of FPN, there needs to be a practical distinction between the two.  PRNU, as it's been known historically in our application, is only a source of concern in the already higher SNR areas of the image.  In other words, noise limits SNR across three chief regimes, moving through our exposures as first read-noise limited, then sky-limited, and then PRNU limited, whereas it can even be argued if it's actually a limitation in most of the types of images we do (we typically stop our exposures once sky-limited).   

 

I remember Richard Crisp arguing this point many years ago, and unless he's rethought many of his statements, he was once very firm in his belief that PRNU would NOT be properly removed via flat-field unless the flats were taken at upwards of 60 to 70% illumination of the sensor.   Is Crisp wrong about that?  Has he changed his view point?   Seriously, I'm not as up on the current papers as you guys...nor do I have the time to become so. 

 

But the nomenclature has always referred to PRNU differently than FPN.   That said, unless I am guilty of using antiquated information, I think it's important to leave the terms separate because how it FPN and PRNU shows itself in an image are wildly different. 

[Jon Rista]

Jon,

I agree with what most of what you are saying; however, FPN is a signal modulator and although Janesick treats it as a noise term, that's only in the context of trying to characterize the spatial uniformity of a sensor.  FPN represents a systematic uncertainty in the measurement that can be arithmetically calibrated out of an image so it does not represent temporally uncorrelated noise (uncertainty)--in spite of the unfortunate fact that it contains the word "noise" in its name.  Of course the calibration process needed to remove the effects of FPN does add some measurement uncertainty to the result (as Janesick points out in chapter 8 , p111 of ref (1)).

 

John

I agree. 

 

I only say that dithering in effect IMPARTS a temporal component to FPN in a stack. Because dithering shifts the position of the image in the frame, once the image contents are aligned (registered), the FPN then is no longer aligned in the stack. Each time you add a new frame to the stack, the pattern is in a different position, and the value of a given pixel no longer has a known fixed component. Thus, the FPN contributions of that pixel are now uncertain. Therefor, dithering effectively makes fixed pattern noise another temporally random noise. It may not be totally uncorrelated, depends on the nature of the pattern (FPN due to PRNU is usually more spatially random, DFPN may not be...consider:)

 

ASI1600 master dark:

 

Manually "dithered" ASI1600 master dark (I just used the PI cropping tool to shift the contents of the dark frames around before stacking):

 

 

This is where I agree with Frank.  No matter the source of FPN, there needs to be a practical distinction between the two.  PRNU, as it's been known historically in our application, is only a source of concern in the already higher SNR areas of the image.  In other words, noise limits SNR across three chief regimes, moving through our exposures as first read-noise limited, then sky-limited, and then PRNU limited, whereas it can even be argued if it's actually a limitation in most of the types of images we do (we typically stop our exposures once sky-limited).   

 

I remember Richard Crisp arguing this point many years ago, and unless he's rethought many of his statements, he was once very firm in his belief that PRNU would NOT be properly removed via flat-field unless the flats were taken at upwards of 60 to 70% illumination of the sensor.   Is Crisp wrong about that?  Has he changed his view point?   Seriously, I'm not as up on the current papers as you guys...nor do I have the time to become so. 

 

But the nomenclature has always referred to PRNU differently than FPN.   That said, unless I am guilty of using antiquated information, I think it's important to leave the terms separate because how it FPN and PRNU shows itself in an image are wildly different. 

I've read some of Crisp's papers in the past, and I thought he too seemed to use FPN to refer to "gain FPN" which is FPN due to photon response non-uniformity and gain, and DFPN to refer to "dark and offset FPN" which is FPN due to sensor bias pattern and dark signal non-uniformity. 

 

The way you are using FPN here, it makes me think you are referring to DARK SIGNAL FPN, not FPN due to photo response and gain. If I am talking only about FPN due to DSNU and sensor bias, I use the term DFPN. A lot of literature seems to use the term FPN only to refer to what is alternatively called "gain FPN", which is FPN resulting from gain and PRNU. Some papers and articles never even cover offset or dark FPN at all, and the only FPN of concern is related to PRNU. 

 

I do believe that what Richard Crisp has said about sensor illumination for adequate flats has not changed. I also believe that there was more to it than that, that not only did the sensor need to be strongly illuminated, but you needed SIGNIFICANT total signal to adequately model FPN due to PRNU/gain...millions of electrons total per pixel in the integrated flat. I don't know if I have that article handy, but I've never tried to create flats with quite that strong of a signal. I just dither, which is a heck of a lot easier. 

Edited by Jon Rista, 05 January 2020 - 01:07 AM.

[freestar8n]

I'm glad the OP has figured out the issue.  My main point here is that if you see "the flats don't work" then something is wrong either in the way the flats were captured or in the calibration process.  With cmos cameras it is convenient to take flats in bright situations with short exposures - but you do need to be extra careful about light leaks.  This is especially true when using narrowband filters that have little light entering after the filter - but a lot could leak in between the filter and the sensor - particularly with OAG.

 

I masked off my imaging train by looking at the imaging sensor in daytime with a cover over the front of the telescope and a 3nm Ha filter in the filter wheel.  In addition, when I take flats I cover the back end of the scope with a large black trash bag that lets the imaging train rotate within it.  This is all very simple and crude - but effective.

 

I don't think there is a need to cite anyone in particular regarding PRNU vs. FPN since both usages discussed here can be found in professional writings - but in different contexts.  I think it's great when people talk about FPN and dithering - but there has been an increased concern about PRNU in particular - when I think it almost never plays a visible role in CN - except for the particular case of some cmos cameras that have a form of grid pattern on them.  And as with other manifestations of PRNU - it won't show in the dark parts of the image.

 

PRNU is the reason you shouldn't smooth flats "to make them better" as I have sometimes heard people doing.  But in practice there should only be a very small variation in pixel response in a sensor and it would only impact precise photometry.  I don't think I have ever "seen" PRNU in an exposure I have made.  It's certainly going to be there - but it probably won't be noticeable.  Whereas dark current FPN is a huge issue and shows up all the time.

Frank 

Edited by freestar8n, 05 January 2020 - 01:29 AM.

[jhayes_tucson]

This is where I agree with Frank.  No matter the source of FPN, there needs to be a practical distinction between the two.  PRNU, as it's been known historically in our application, is only a source of concern in the already higher SNR areas of the image.  In other words, noise limits SNR across three chief regimes, moving through our exposures as first read-noise limited, then sky-limited, and then PRNU limited, whereas it can even be argued if it's actually a limitation in most of the types of images we do (we typically stop our exposures once sky-limited).   

 

I remember Richard Crisp arguing this point many years ago, and unless he's rethought many of his statements, he was once very firm in his belief that PRNU would NOT be properly removed via flat-field unless the flats were taken at upwards of 60 to 70% illumination of the sensor.   Is Crisp wrong about that?  Has he changed his view point?   Seriously, I'm not as up on the current papers as you guys...nor do I have the time to become so. 

 

But the nomenclature has always referred to PRNU differently than FPN.   That said, unless I am guilty of using antiquated information, I think it's important to leave the terms separate because how it FPN and PRNU shows itself in an image are wildly different. 

 

This is where some participants on CN sow a lot of misinformation and confusion.  First off, the key issue should be with photon shot noise--not read noise.  The optical signal that generates photon shot noise comes from the object that you are imaging plus sky glow along with other unwanted signals like satellites, airplanes, street lights, and cosmic rays.  All of those unwanted signals decrease the contrast of the desired signal (irradiance from your target object) but if you are doing things right, photon noise should be the biggest contributor to noise in your images.  Photon noise is the result of the quantum nature of light and has nothing to do with the detector.  Remember that noise is simply the random uncertainty in any measurement.  Other unwanted signals that contribute to noise in an image come from the sensor (and electronics) itself.  Dark and bias current are two such unwanted signals, which also contribute additional shot noise by the mere fact that they are quantized sources of photoelectrons.  Read noise is comprised of all noise that is independent of signal and ideally, it should be a very small contributor.  In my view, everyone interested in this stuff should read up on measurement theory first, and this is an excellent place to start:  http://hep.ucsb.edu/..._18f/Taylor.pdf.

 

FPN is signal dependent and is the result of PRNU.  PRNU modulates signal and its effect can be calibrated out with flat fielding--just like vignetting.  PRNU is most definitely not a "different kind of FPN!"  Go read ref #1.  I've taken Richard Crisp's SPIE short course (in August, 2019) and I didn't see anything in his course or in his notes that deviated from Janesick.  In fact, he gave out both ref  #1 and ref #2 as a part of his course.  He freely references Janesick who wrote about FPN and flat fielding in ref #2, which was published in 2000 so this stuff has been around for at least 20 years.  It's not new.

 

John

Edited by jhayes_tucson, 05 January 2020 - 11:12 AM.

[Jon Rista]

Getting back to my original point..problem solved.

 

First, John, you were correct. I had a serious light leak. I simply placed my flat panel on top of

my refractor, with the instrument in the vertical position. When I took a 60-s dark, obviously with

the panel turned off, and ambient lighting in the room, the dark frame was completely

saturated. Turned off the room lights, took another dark frame, and it looked as it should.

 

Second, I was not calibrating my flats against darks and biases properly.

 

Here is a stack of 100 frames. No more artifacts.

 

-Gary

Glad you got it figured out!

[freestar8n]

This is where some participants on CN sew a lot of misinformation and confusion. 

Hmm - what stitch do they use?

 

There is no point in citing references that support your perspective since I have already provided references that support both perspectives.

 

I pointed you and Jon to Janesick to get the basic ideas of FPN across.  Thank goodness we are at least not arguing over signal vs. noise as two distinct things.  Now I am trying to discuss something more nuanced - so I'm not too surprised there is resistance again.  All I can suggest is to read more extensively on these topics.

 

There are two main types of pattern noise: Fixed, which is the same in every exposure regardless of illumination - and noise due to PRNU.  The noise resulting from PRNU is not the same in every exposure - so many references make a point to distinguish it from FPN - as I recommend - supported by such references.

 

I often see people in CN expressing concern about PRNU and dithering as in this thread - but can anyone point to an example of a subexposure where PRNU is a visible problem?  The only one I know about is in some CMOS sensors that show a grid pattern.  People are making a fuss about prnu, but I view it as something simply to be aware of - and as an important reason for good flats.  But I have never seen it in a subexposure because it only would show in bright parts of the image - and other noise terms, particularly shot noise, dominate.

 

That is a key reason to support distinguishing PRNU *and noise resulting directly or indirectly from its presence* from FPN - specifically limited to noise present without illumination.

 

So - any examples of PRNU, anyone?

 

Frank

[Allaboutastro]

Glad the OP fixed his issues.  Thanks also for the refresher on the topic of PRNU.   You guys certainly know your stuff. 

 

I too would like to see examples of PRNU.   To me, I've always regarded PRNU as a variance in pixel QE.  For example, if a sensor is 65% QE at 656.3nm, then we should expect some pixels to be 63% and others to be 67% (or some such).  As such, you would need wide areas of illuminated content in our images to perceive it, something that isn't usually achieved in a standard astroimage.  If this isn't a healthy way to think of it, then I'd like to change that.  

 

Just for some perspective, Crisp used to post a lot on various forums, one of which was the Astronomy Magazine forum where I was a moderator.  Crisp would express concerns about PRNU and always raised the issue that flats, taken right, had to be well illuminated for that reason.  I contended that if you showed me examples of PRNU and how it was ruining my image, then I would address it...otherwise, I would be happy taking subs that were shot-noise limited, keeping my flat's somewhere around 50% illumination...which was most certainly the goal with a 15 e- read noise camera like my STL-11000M.   However, Crisp's influence was always part of my advice to others when they asked about flat illumination.  "Err to the right side of the histogram," I would say. 

 

But just for clarification (at least in my own mind), in discussing the noise "regimes" mentioned earlier.   PRNU, if left uncalibrated, should become the dominant noise source at some point, where it overcomes shot-noise, quadratically.   However, when you consider that most of our images don't contain a majority of high (enough) SNR regions and the fact that whatever flat-fields we take likely does a reasonable job of calibrating out those QE variances of individual pixels, then I have never seen PRNU as a concern.  Albeit, I'd love to nail down this topic better...so I'm loving the discussion. 

[Allaboutastro]

 

The way you are using FPN here, it makes me think you are referring to DARK SIGNAL FPN, not FPN due to photo response and gain. If I am talking only about FPN due to DSNU and sensor bias, I use the term DFPN. A lot of literature seems to use the term FPN only to refer to what is alternatively called "gain FPN", which is FPN resulting from gain and PRNU. Some papers and articles never even cover offset or dark FPN at all, and the only FPN of concern is related to PRNU. 

 

Jon...I definitely understand the distinction.  But the way we always talk about such things leads more to confusion that anything else.  FPN is the catch-all term and I wish there was a universally accepted nomenclature for what you call DFPN and FPN (as it arises from PRNU).  Truly, I don't care much about the latter but rather the former, since that's really what we are seeing as pattern noise within our uncalibrated images.   At that point of acceptance, I'd be just fine getting rid of the FPN term altogether and just calling it PRNU.   I realize that's pretty much what we have...PRNU and DSNU...but it's still way confusing, especially when we try to teach it to others. 

Edited by Allaboutastro, 05 January 2020 - 12:12 PM.

[jhayes_tucson]

I agree. 

 

I only say that dithering in effect IMPARTS a temporal component to FPN in a stack. Because dithering shifts the position of the image in the frame, once the image contents are aligned (registered), the FPN then is no longer aligned in the stack. Each time you add a new frame to the stack, the pattern is in a different position, and the value of a given pixel no longer has a known fixed component. Thus, the FPN contributions of that pixel are now uncertain. Therefor, dithering effectively makes fixed pattern noise another temporally random noise. It may not be totally uncorrelated, depends on the nature of the pattern (FPN due to PRNU is usually more spatially random, DFPN may not be...consider:)

 

...

 

Jon,

First, I agree that dithering at a distance greater than the autocorrelation distance of the spatial patter, will combine statistically uncorrelated spatial components, which with enough frames, will convert the structure visible in the pattern into a more random spatial distribution. This is true for FPN and for the dark signal--as you've shown.  

 

I want to add another key take away. It's important to remember that FPN is directly proportional to the signal while photon shot noise is proportional to the square root of the signal, which means that spatial variations across the sensor due to FPN will dominate the spatial variations introduced by photon noise over the dynamic range of sensor (ref 1, section 3.4 p31.)  Therefore, if you want the cleanest possible result, it's a bad idea to simply use dithering as a way to avoid flat fielding.  The best approach is to gather dithered data and then to first calibrate the data using darks and flats before stacking the subs.   Simply stacking uncalibrated dithered data will indeed reduce the effects of FPN but not by as much as when the data is first calibrated.   In the end, I completely agree that dithering will further reduce and spatially randomize the additional noise introduced by the calibration process.

 

John

Edited by jhayes_tucson, 06 January 2020 - 01:23 AM.

[jhayes_tucson]

Jon...I definitely understand the distinction.  But the way we always talk about such things leads more to confusion that anything else.  FPN is the catch-all term and I wish there was a universally accepted nomenclature for what you call DFPN and FPN (as it arises from PRNU).  Truly, I don't care much about the latter but rather the former, since that's really what we are seeing as pattern noise within our uncalibrated images.   At that point of acceptance, I'd be just fine getting rid of the FPN term altogether and just calling it PRNU.   I realize that's pretty much what we have...PRNU and DSNU...but it's still way confusing, especially when we try to teach it to others. 

 

Jay,

Unfortunately the term "FPN" is often tossed about very imprecisely; however, we came to a tacit agreement a while back here on CN to use the term as defined by Janesick, which is very precise and unambiguous.  That is why Jon and I are talking the same language.  Furthermore Janesick's definition is the same term that I've personally heard Crisp use in his course; although, I can imagine that he may have gotten sloppy about it at one time or another.  Using the term "FPN" as a catch all is confusing and won't work to make this stuff tractable.

 

PRNU stands for "Pixel Response Non-Uniform".  It represents a variation in pixel responsivity across the array and is normally expressed as a percentage of mean responsivity.  (Responsivity is similar to QE; however, it has different units.  It converts irradiance into Amps [or micro-Amps] as the output of a detector [i.e. a pixel]  The expression for signal is:  S(m,n) = R(m,n) A I(m,n) I(m,n) where R(m,n) is the responsivity of a pixel at location (m, n) in units of Amps/Watt, A = the pixel area in m², and I(m,n) is the average irradiance over pixel(men) in Watts/m².)  A typical value for PRNU might be 1%.  You can't see PRNU directly.  Instead you see the effect of PRNU on a signal and that effect is call Fixed Pattern Noise (FPN.)

 

Janesick calls the dark current spatial distribution DFPN.  I personally think that this is a terrible name because dark current is a signal; it is not noise. Dark current is a quantum phenomenon governed by Poisson statistics so there will be uncertainty (shot noise) when you measure it.  Attaching a name that includes the word "noise" to a signal makes it very difficult to then talk about the noise in a measurement of that signal.  We can resolve the problem (by at least a little) by referring to the noise inherent in dark current as "dark shot noise."  Remember, in the case of Janesick, he is describing how to characterize the uniformity of performance of the ensemble of sensors that make up an imaging array.  In that case, he views spatial variations as a noise term--even when the variation actually come from a signal.  In my view, things become a lot more direct (and understandable) if we simply talk about "dark current" and "dark shot noise", which are terms that Janesick also uses.

 

John

Edited by jhayes_tucson, 06 January 2020 - 11:29 PM.

[Jon Rista]

Jon,

First, I agree that dithering at a distance greater than the autocorrelation distance of the spatial patter, will combine statistically uncorrelated spatial components, which with enough frames, will convert the structure visible in the pattern into a more random spatial distribution. This is true for FPN and for the dark signal--as you've shown.  

 

I want to add another key take away. It's important to remember that FPN is directly proportional to the signal while photon shot noise is proportional to the square root of the signal, which means that spatial variations across the sensor due to FPN will dominate the spatial variations introduced by photon noise over the dynamic range of sensor (ref 1, section 3.4 p31.)  Therefore, if you want the cleanest possible result, it's a bad idea to simply use dithering as a way to avoid flat fielding.  The best approach is to gather dithered data and then to first calibrate the data using darks and flats before stacking the subs.   Simply stacking uncalibrated dithered data will indeed reduce the effects of FPN but not by as much as when the data is first calibrated.   In the end, I completely agree that dithering will further reduce and spatially randomize the additional noise introduced by the calibration process.

 

John

Oh, I was NEVER recommending NOT doing flat fielding. I believe one should always do dark and flat calibration, regardless of how "clean" the sensor may seem to be. That said, I've encountered enough remnant FPN in my time, from numerous cameras (including CCDs, I have found that beyond a few years, lots of CCDs get pretty "dirty" and can use all the help they can get!), so I rely heavily on both calibration and dithering. Depending on the camera, I also use cosmetic correction (which is actually a tool in PI that can be batch-run on frames during the pre-processing phase to clean any remnant pattern junk, if you have any obvious stuff after calibration), so I definitely don't skimp on my pre-processing, nor do I ever recommend anyone else skimp (this should be evident in thousands of my posts on these forums throughout the years. ;P)

 

I just wanted to make it clear that aside from drizzling, which not everyone (probably not even a majority) of people do...the real reason we dither is to deal with FPN, or whatever remains of it after calibration.

Edited by Jon Rista, 06 January 2020 - 09:51 PM.

[jhayes_tucson]

Oh, I was NEVER recommending NOT doing flat fielding. I believe one should always do dark and flat calibration, regardless of how "clean" the sensor may seem to be. That said, I've encountered enough remnant FPN in my time, from numerous cameras (including CCDs, I have found that beyond a few years, lots of CCDs get pretty "dirty" and can use all the help they can get!), so I rely heavily on both calibration and dithering. Depending on the camera, I also use cosmetic correction (which is actually a tool in PI that can be batch-run on frames during the pre-processing phase to clean any remnant pattern junk, if you have any obvious stuff after calibration), so I definitely don't skimp on my pre-processing, nor do I ever recommend anyone else skimp (this should be evident in thousands of my posts on these forums throughout the years. ;P)

 

I just wanted to make it clear that aside from drizzling, which not everyone (probably not even a majority) of people do...the real reason we dither is to deal with FPN, or whatever remains of it after calibration.

 

Yes, I recognize that you understand the point, but I added the second paragraph to amplify the issue for those who have not studied Janesick--mainly because is it such an important point.  It's not uncommon to hear from folks on CN who believe that using dithering to smear noise and applying DBE to remove gradients is an equivalent replacement for flat fielding.  That's absolutely not true for any imaging sensor and it's especially not true for most CMOS sensors where there the number of amplifiers is the same as the number of pixels.

 

John

[freestar8n]

I will ask again. For those who feel it is critical to view prnu as a form of fpn somehow - have you seen it visibly as a noise term in your images?

I see speckle and streaks in the background and by all references that is due to dark current and bias fpn. But no one would blame it on prnu.

Yet there is confusion in cn that prnu is an important and visible noise term.

I think it is important but not in the tangible and visible way that fpn is.

And that’s why it’s important to distinguish the two. And go along with a significant body of literature usage.

So - any examples where prnu was a problem and a good flat fixed it? Other than dust motes and vignetting of course. And also ignoring pathological checkerboard patterns.

Fpn is pretty ubiquitous. Prnu hardly ever a visible factor.

Frank

Edited by freestar8n, 06 January 2020 - 10:43 PM.

[Jon Rista]

Yes, I recognize that you understand the point, but I added the second paragraph to amplify the issue for those who have not studied Janesick--mainly because is it such an important point.  It's not uncommon to hear from folks on CN who believe that using dithering to smear noise and applying DBE to remove gradients is an equivalent replacement for flat fielding.  That's absolutely not true for any imaging sensor and it's especially not true for most CMOS sensors where there the number of amplifiers is the same as the number of pixels.

 

John

Yep! Whole-heartedly agree! 

 

In fact, I've been struggling for a week now to try and get some corrected flats for a bunch of the data I acquired last year. I did not have a lot of clear sky, but what clear skies I did have I imaged through every night from twilight to twilight, and have well over 100 hours (maybe even 200 hours) of data on several objects. However, somewhere along the line, my image train orientation changed, and I only have good flats for about a third of the data. No amount of DBE or anything is capable of fixing the FPN issues left behind by subs that were either improperly calibrated, or left only dark calibrated. So, I've largely been left with hundreds of hours data that I cannot use...because of bad or missing flats. Flats are critical to correcting FPN properly...not to mention, they are the only way to correct for other field-related issues...vignetting, dust motes, etc.

[freestar8n]

Flats are critical to correcting FPN properly

Do you have an example where flats corrected FPN in a noticeable way?  Can you show the FPN present and then going away after the flat?

 

Flats correct the image by multiplication rather than subtraction.  The FPN leading to visible noise in the background is removed by subtraction.  Only noise in the bright parts of the image would be helped by flat correction.

 

I hope that people following along this stuff see my point.  PRNU is an important issue to be aware of in sensors and in the calibration process - but if you see noise in your images - it is not likely due to PRNU, and flats likely aren't the problem - unless they are not well matched with the lights, as in the case of this thread.

 

If anyone thinks they have visible noise caused by PRNU I would be interested to see it.

 

If you don't have good flats you won't be able to correct dust motes and vignetting well - but if it's because it is moving around and changing - then it most certainly isn't "fixed."

 

Frank

[Jon Rista]

Do you have an example where flats corrected FPN in a noticeable way?  Can you show the FPN present and then going away after the flat?

 

Flats correct the image by multiplication rather than subtraction.  The FPN leading to visible noise in the background is removed by subtraction.  Only noise in the bright parts of the image would be helped by flat correction.

 

I hope that people following along this stuff see my point.  PRNU is an important issue to be aware of in sensors and in the calibration process - but if you see noise in your images - it is not likely due to PRNU, and flats likely aren't the problem - unless they are not well matched with the lights, as in the case of this thread.

 

If anyone thinks they have visible noise caused by PRNU I would be interested to see it.

 

If you don't have good flats you won't be able to correct dust motes and vignetting well - but if it's because it is moving around and changing - then it most certainly isn't "fixed."

 

Frank

CMOS sensors often exhibit non-random spatial FPN patterns due to PRNU. For example, my IMX183:

 

 

It certainly has spatially random patterns resulting from PRNU (and gain and light), but there are other fixed structures, some small around 4-5 pixels in size, some tens of pixels in size, some hundreds to thousands of pixels in size. This is a working flat, all the structure in this field exists in the lights and with strong enough signal, can be a problem. It is not all dust motes either. This pattern has been entirely fixed since I got the camera, while various motes have come and gone. This is the result of PRNU. And if I don't calibrate with flats, once I stack enough subs, these structures leave me with rather dirty subs that are not clean or smooth, hard to process, etc. 

 

I don't have any example integrations at the moment as I deleted them all. I've been working on this data for some time, and have yet to actually get a quality integration, and I had no interest in keeping the bad ones around. I finally managed to acquire some new flats today, so once I get a chance to process the data again, I'll see if I can get an integration with the bad pre-processing first. 

Edited by Jon Rista, 07 January 2020 - 05:34 PM.

[freestar8n]

Yes all that fine structure is pattern noise in the flat due to prnu. It is perhaps a few percent modulation of the pixel response across the sensor and it has very real but small impact on the noise in an image.

But my point is - it is almost never actually visible in Astro images at all. It has nothing to do with the background noise that is so effectively handled by good darks and dithering. That noise is fpn. The noise you show here is the combination of the illumination with the prnu and it is completely separate and fairly innocuous.

The noise introduced by prnu is scene dependent and not fixed even though the prnu pattern is fixed and constant.

As I have been saying - I don’t think images are visibly impacted by prnu.

Frank

[BenKolt]

PRNU stands for "Pixel Response Non-Uniform".  It represents a variation in pixel responsivity across the array and is normally expressed as a percentage of mean responsivity.  (Responsivity is similar to QE; however, it has different units.  It converts irradiance into Amps [or micro-Amps] as the output of a detector [i.e. a pixel]  The expression for signal is:  S(m,n) = R(m,n) A I(m,n) I(m,n) where R(m,n) is the responsivity of a pixel at location (m, n) in units of Amps/Watt, A = the pixel area in m², and I(m,n) is the average irradiance over pixel(men) in Watts/m².)  A typical value for PRNU might be 1%.  You can't see PRNU directly.  Instead you see the effect of PRNU on a signal and that effect is call Fixed Pattern Noise (FPN.)

John:

 

Quick correction.  For the signal equation you put irradiance in there twice.  Should be S(m,n) = R(m,n) A I(m,n), and using your  MKS units the signal S(m,n) comes out as [A] or [C/s].  One can convert that to [e-/s], which is commonly used in this line of work as related to camera sensors.

 

All:

 

John made the point above that demonstrates one of the distinctions between FPN and Shot Noise as the former increases linearly with signal, the latter as the square root with signal.  This can be seen when constructing a Photon Transfer Curve (PTC) of a camera sensor.  Make a log-log plot of FPN and Shot Noise vs. Signal, and through the linear response region of the sensor the former will have a slope of +1, the latter +1/2 when the data is properly calibrated.  John gave references to Janesick and Crisp for details on how to do this.  That +1 shows the direct relation of FPN with signal illumination as has been discussed, the +1/2 denotes the poissonian distribution of shot noise.

 

I bring up the PTC because learning to make these with those references as guide was the most constructive way for me to learn about all these terms and better understand their sources, how they interrelate and, most importantly, how and why I really need to properly calibrate with dark subtraction and flat-fielding as well as the benefit of dithering.  For me the exercise helped to set my mind straight and use better practices.

 

Some of these debates have been going on for quite some time, but I still recommend that we all pay attention to them and listen and learn and do our best to anchor the concepts properly.  But at the same time, putting the concept debates aside, note there is near universal agreement on the necessity of proper flat-fielding calibration as well as the strong recommendation of dithering.  I wouldn't re-emphasize this last point if not for the fact that even today I came across yet another post (no need to link to it since this was a beginner who I'm sure will learn in time) where the imager stacked uncalibrated frames and tried to post-process his way to a better result.  I for one will continue to gently advise not to do that!  And I'm happy to explain why using to the best of my ability proper terminology.

 

Lastly, good job to the OP for working his way through this.  Your last calibrated image was quite good with maybe a slight bit of roll-off over- (or was it under-?) correction.  I suffer from that often, although yours is minimal.  It's also possible you just have some sky gradient to deal with.  (Now, sky glow I know for sure is signal, albeit unwanted signal!)  I look forward to seeing your finished images.

 

Best Regards,

Ben

[jhayes_tucson]

John:

 

Quick correction.  For the signal equation you put irradiance in there twice.  Should be S(m,n) = R(m,n) A I(m,n), and using your  MKS units the signal S(m,n) comes out as [A] or [C/s].  One can convert that to [e-/s], which is commonly used in this line of work as related to camera sensors.

 

All:

 

John made the point above that demonstrates one of the distinctions between FPN and Shot Noise as the former increases linearly with signal, the latter as the square root with signal.  This can be seen when constructing a Photon Transfer Curve (PTC) of a camera sensor.  Make a log-log plot of FPN and Shot Noise vs. Signal, and through the linear response region of the sensor the former will have a slope of +1, the latter +1/2 when the data is properly calibrated.  John gave references to Janesick and Crisp for details on how to do this.  That +1 shows the direct relation of FPN with signal illumination as has been discussed, the +1/2 denotes the poissonian distribution of shot noise.

 

I bring up the PTC because learning to make these with those references as guide was the most constructive way for me to learn about all these terms and better understand their sources, how they interrelate and, most importantly, how and why I really need to properly calibrate with dark subtraction and flat-fielding as well as the benefit of dithering.  For me the exercise helped to set my mind straight and use better practices.

 

Some of these debates have been going on for quite some time, but I still recommend that we all pay attention to them and listen and learn and do our best to anchor the concepts properly.  But at the same time, putting the concept debates aside, note there is near universal agreement on the necessity of proper flat-fielding calibration as well as the strong recommendation of dithering.  I wouldn't re-emphasize this last point if not for the fact that even today I came across yet another post (no need to link to it since this was a beginner who I'm sure will learn in time) where the imager stacked uncalibrated frames and tried to post-process his way to a better result.  I for one will continue to gently advise not to do that!  And I'm happy to explain why using to the best of my ability proper terminology.

 

Lastly, good job to the OP for working his way through this.  Your last calibrated image was quite good with maybe a slight bit of roll-off over- (or was it under-?) correction.  I suffer from that often, although yours is minimal.  It's also possible you just have some sky gradient to deal with.  (Now, sky glow I know for sure is signal, albeit unwanted signal!)  I look forward to seeing your finished images.

 

Best Regards,

Ben

Thanks Ben!  That's a typo.  For some reason the first time I entered I(m, n), the text disappeared as I typed so I thought that I might have accidentally erased it...so I re-entered it.  I don't know how any of that happened but it doesn't matter.  There should definitely be only a single I(m, n) term in the expression.  It looks like I can't fix the original post.  Thanks for that feature CN.

 

John

Edited by jhayes_tucson, 07 January 2020 - 10:09 PM.

[jhayes_tucson]

Do you have an example where flats corrected FPN in a noticeable way?  Can you show the FPN present and then going away after the flat?

 

Flats correct the image by multiplication rather than subtraction.  The FPN leading to visible noise in the background is removed by subtraction.  Only noise in the bright parts of the image would be helped by flat correction.

 

I hope that people following along this stuff see my point.  PRNU is an important issue to be aware of in sensors and in the calibration process - but if you see noise in your images - it is not likely due to PRNU, and flats likely aren't the problem - unless they are not well matched with the lights, as in the case of this thread.

 

If anyone thinks they have visible noise caused by PRNU I would be interested to see it.

 

If you don't have good flats you won't be able to correct dust motes and vignetting well - but if it's because it is moving around and changing - then it most certainly isn't "fixed."

 

Frank

 

No Frank.  PRNU modulates the signal by multiplication, which causes FPN--just as inadequately sized apertures block light, which multiplies irradiance to create vignetting.  FPN is mostly definitely not additive.  FPN is calibrated out of the signal just like vignetting is corrected--by division.  Please get a copy of Janesick and read it.  For your convenience, I gave you all of the pages to look at in ref #1.

 

John

[freestar8n]

John. I have emphasized prnu multiplies the signal. That is why it it is so different from dark current fpn. And it is why I don’t consider it, along with many texts. Fpn is additive as a noise term. Prnu is not.

Again - see those speckles in Jon’s flat image above? That is noise in the flat due to prnu.

Can you show me an image that reveals that noise from the flat?

I can show tons of images with noise from dark current fpn. The impact from prnu is tiny and only visible in bright and smooth parts of the image.

If someone points to noise in the background and says it is prnu they would be wrong. I’m trying to avoid that confusion.

But it’s great you are now using the term fpn. That was a long time coming. But now I think you are using it too much.

Frank

[jhayes_tucson]

Frank,

First, dark current is not FPN.  Dark current is an unwanted signal that generates its own "dark shot noise", which is a noise term.   Because dark current is a signal, it can be subtracted directly from the image.  Second, of course Jon's flat image contains FPN due to PRNU.  That's how the effects of PRNU get calibrated out of an image by division.  It is identical to calibrating out vignetting.  Go look at the math in ref #1 on the pages that I referenced.  Once the image is calibrated, you will not see FPN in the output (within the uncertainty of the calibration process.)

 

Finally, my position has always been that FPN is not a real noise term--any more than vignetting is a noise term.  FPN is merely called a noise term in the context of evaluating the uniformity of the ensemble of pixels contained in an imaging sensor.  That's a big part of what the Photon Transfer process evaluates.  I am merely using the term FPN precisely as it is used in Janesick--a "professional" reference that you yourself have endorsed and one that I thought you understood.

 

 

John

[freestar8n]

John please remember I know exactly how this stuff works - in detail - and am well read on it and have been for years. It has been a struggle in cn due to resistance of accepted terminology. But there has been progress.

Your resistance to fpn as a noise term directly conflicts with it being a noise term. But it is. That’s what the ‘N’ stands for. And you are using that term - finally.

But regardless of the terminology I think it should be clear to everyone prnu is not pertinent to concerns of good flats or dithering. That is caused by dark current fpn. No one has provided an example per my request of prnu being visible in an image.

People absolutely should use good flats and good darks. But if you see speckles and streaks in the background it is not related to flats or prnu.

It’s a very simple but important point for understanding how different noise sources behave in the calibration process.

Frank