80 replies to this topic

[sharkmelley]

I've recently come across a very odd problem which was very difficult to diagnose.  I'm still not sure I fully understand precisely what happens nor how to avoid it.  I noticed the problem with my Sony A7S and if I'm right about the cause then it will potentially affect Nikon cameras as well.  Concentric coloured banding:

 

 

I took 30 flats at ISO 100 and averaged them to create a (bias subtracted) master flat. Now, you would think that one of the original (bias subtracted) flats would be properly calibrated by this master flat.  But dividing followed by stretching gives the following (displaying the green channel only): 

 

 

Ignore the horizontal stripes on the left hand side - they are caused by the Sony A7S split sensor.  It is the concentric rings that are of interest.

 

Looking at part of the histogram of that flat I see the following:

 

 

Certain values are under-represented (the histogram notches) and this fairly regular pattern continues throughout the histogram.  Here's another plot where I highlight only those under-represented values - the histogram notches:

 

 

So each notch in the histogram of the file being calibrated causes a concentric ring in the calibrated file. 

 

The obvious question you may be asking is why the master flat doesn't have the notches in the same place.  The thing is, I took that set of flats against the dimming twilight sky.  So each flat had a slightly different brightness.  Looking at the histogram of each flat, each one has the same notches at ADU values at 522, 558, 597 etc.  But in the image these values appear in different locations so once the flats have been averaged together, the notches are completely absent from the histogram of the master flat:

 

 

To clarify a further point, if I choose one of the other flats to calibrate then the concentric rings will have moved - they move to the location where the pixel values correspond to the fixed notches in the histogram.

 

The same thing happens when calibrating lights - a concentric ring will appear in the background where the pixel values correspond to the notches in the histogram. 

 

So is this a problem for stacked images?  It is more than likely that the background sky brightness changes slightly from exposure to exposure and therefore so does the position of the concentric rings.  This should reduce their amplitude in the final stacked image because they then average out.  But if the lighting for all your flats is identical then potentially the master flat itself could end up with concentric rings.

 

One final point.  The Sony A7S has regular histogram notches in all three channels.  Nikon cameras have regular histogram notches (or spikes) in the red and blue channels because of the way they are scaled.  I don't know about the green channel.  So it is likely that Nikons can show the same behaviour. To the best of my knowledge, Canon cameras don't suffer from the same histogram notches and histogram missing values.

 

When I started to investigate this issue I had assumed that it was the raw compression (using histogram combing) that was the root of the problem.  This turned out not to be the case - instead it was the semi-regular notches in the histogram that was causing the problem.  What, exactly, causes the histogram notches in the Sony A7S, I haven't a clue.

 

Mark

Edited by sharkmelley, 26 November 2016 - 09:34 PM.

[ccs_hello]

Mark,

 

In Nikon DSLR, its EXPEED ISP is playing the image science adjustment (optimize the R and B channels dynamic range to be equal to the "brighter" G channel.)

G-ch has no scaling thus no "comb-like" glitches, while B and R do have a fixed multiplier (e.g., 1.14x.)  Since the image output has to be in integer values, there will be some numbers missing thus the "comb" histogram shows on B and R histogram.

By applying Nikonhacker firmware "sensor raw" patch, such comb notches disappear (and R and B channel histograms have lower spread.)

 

On a7s, I suspect there is a multiplier but a very small number applied thus the comb notches are further spread out.

Since they are applied on all RGBG channels and the flat is concentric ring like, I suspect it's camera's BIONZ-X ISP doing its magic:

compensating on microlense's off to direct optical path (especially off the image center.)  <-- microlenses are always directly above individual sensel's imaging surface.

 

Hope this is a good guess.

 

Clear Skies!

 

ccs_hello

[bobzeq25]

Have you explored the magnitude of the rings?  Or seen just how much you have to stretch to get them to appear?

 

Reason I ask.  I have a Nikon D5500, which has a spiky histogram at ADUs above 1500 or so.  If I load a flat exposed above that into Pixinsight, and use the script ImageAnalysis - FlatContourPlot, I get the rings.  But a look at the image (which shows a bunch of very small dust spots I've never seen before) and the scale next to it shows this is a truly monster stretch, more than you'd ever do.   If instead I do a STF Autostretch (boosted), which is pretty stout, no rings.  The same is true if I load it into StarTools, and do an Autostretch, another stretch which is very strong.  My assumption is this is a minor effect, which would explain why I've never seen it on an image.

 

It looks to me like maybe the rings are close to (or beyond) being buried in the noise?  But that's just an uneducated guess.  I'm guessing FlatContourPlot somehow deals with the noise.

Edited by bobzeq25, 26 November 2016 - 11:16 PM.

[17.5Dob]

I saw rings like that "occasionally" , using my Nikon D7100 on my APO and the huge attendant chain of imaging peripherals.  Using the same Nikon camera and anything other than the APO, I've never seen those. Just slightly altering the imaging chain from one night to the next eliminated it in the APO. I saw it pop up "really" bad only ~ 3 times. The other 200 + times I saw nothing, so I don't think it's anything camera related

Edited by 17.5Dob, 27 November 2016 - 12:22 AM.

[DesertRat]

Mark,
If you were to collect enough flats of slightly differing intensities would the missing values get filled in by simple averaging?  Or is your plot showing that is happening already just due to photon noise?  Because I see dips and not missing values.  Its possible it is just a smoothing or sliding average showing up in that plot - don't know.

 

The green channel in a lossy compression NEF also has the same missing values as the red & blue channels.  The scaling you mention for r&b sounds like simple digital gain, and the encoding table kicks in later.  At least I think so - could be wrong.

 

Bob,

 

The PI FlatCountourPlot script is simply a tool to investigate regions and their shape.  Could be used on any image really.  It posterizes the data and then draws contours like a topo map.  If you take a perfect noiseless flat and run the script it looks like this:

 

 

This I am afraid does not help diagnose the problem here, but may have some utility.

 

Glenn

[ccs_hello]

Mark,

 

I thought about one thing that could be related to the comb notches...

 

Your test condition is ISO100.

 

In modern SONY image sensors, the EXMOR image sensor SOC has the built-in 

analog gain before its column-parallel A/D converter, and a digital gain (just a multiplier on the A/D output's ADU.)

 

Both SONY EXMOR's analog gain module and its digital gain module are positive gain units (can make signal larger, final ADU number larger)

but never make the final ADU output smaller.

To make the number smaller (multiply by a fractional number), that job has to be done by the BIONZ ISP.

In a camera design, when gain is needed, analog gain is applied first, then digital gain is used on as needed basis.

Analog is linear, so thru A/D converter, the resulted number will always be in continuous number range.

 

Typically, such image sensor, when gain is 0dB (i.e., no gain applied), the native sensor ISO is somewhere around ISO 400 but not a precisely a good fit to the ISO scale the camera used, so the fine-tuning on gain is performed.

 

When you use ISO100, BIONZ is performing the math to divide down the sensor ADU number (while sensor gain is 0 dB.)  This will generate notches.

 

Clear Skies!

 

ccs_hello

[nikao]

How are you shooting the flats? 

It looks very much like Newton's rings to me.. could there be a reflection going on somewhere?

[sharkmelley]

Interesting and useful feedback from folk, as always!  Some replies below:

 

 

In Nikon DSLR, its EXPEED ISP is playing the image science adjustment (optimize the R and B channels dynamic range to be equal to the "brighter" G channel.)

G-ch has no scaling thus no "comb-like" glitches, while B and R do have a fixed multiplier (e.g., 1.14x.)  Since the image output has to be in integer values, there will be some numbers missing thus the "comb" histogram shows on B and R histogram.

By applying Nikonhacker firmware "sensor raw" patch, such comb notches disappear (and R and B channel histograms have lower spread.)

 

 

Interestingly, our investigation in Jerry's thread (http://www.cloudynig...ipping-related/) shows that the Nikon D5300 has a multiplier of around 0.9 in the red and blue channels and this leads to regular spikes in the histogram instead of the usual gaps in the R&B histograms.

 

 

 

Have you explored the magnitude of the rings?  Or seen just how much you have to stretch to get them to appear?

 

Reason I ask.  I have a Nikon D5500, which has a spiky histogram at ADUs above 1500 or so.  If I load a flat exposed above that into Pixinsight, and use the script ImageAnalysis - FlatContourPlot, I get the rings.  But a look at the image (which shows a bunch of very small dust spots I've never seen before) and the scale next to it shows this is a truly monster stretch, more than you'd ever do.   If instead I do a STF Autostretch (boosted), which is pretty stout, no rings.  The same is true if I load it into StarTools, and do an Autostretch, another stretch which is very strong.  My assumption is this is a minor effect, which would explain why I've never seen it on an image.

 

It looks to me like maybe the rings are close to (or beyond) being buried in the noise?  But that's just an uneducated guess.  I'm guessing FlatContourPlot somehow deals with the noise.

In this example it didn't take much stretch to make them visible - they're easily visible after a stretch of 10x.  However I did "cheat" by using a low ISO so the data were very clean and allowed the rings to show through the noise.  What I've presented here are my preliminary conclusions about a genuine mechanism at work. 

 

There are many questions that remain unanswered. 

For instance:

1) Under what circumstances does this effect become apparent in a stacked image?

2) Are there other causes of concentric rings? (The answer is almost certainly yes!)

For me, this effect appears only very occasionally in a real image.  I always assumed it was something to do with Sony's compression of raw data.  It now seems that the cause was something totally different that I wasn't expecting.

I still don't understand fully what is going on, that's why I'm hoping that others will provide some insight.  However, the correlation of the ring positions to the notches in the histogram is quite striking.

The PixInsight FlatContourPlot certainly looks quite useful to show the overall shape of the vignetting in the flats.

 

 

 

Mark,
If you were to collect enough flats of slightly differing intensities would the missing values get filled in by simple averaging?  Or is your plot showing that is happening already just due to photon noise?  Because I see dips and not missing values.  Its possible it is just a smoothing or sliding average showing up in that plot - don't know.

 

 

That's exactly what was happening in this example.  The flats had slightly different intensities, so in the master flat all the missing values were filled in by simple averaging.  But the missing values still exist in the single frame that is being calibrated and the positions of the concentric rings seem to line up with those notches.

 

 

 

Mark,

 

I thought about one thing that could be related to the comb notches...

 

Your test condition is ISO100.

 

<SNIP>

 

Typically, such image sensor, when gain is 0dB (i.e., no gain applied), the native sensor ISO is somewhere around ISO 400 but not a precisely a good fit to the ISO scale the camera used, so the fine-tuning on gain is performed.

 

When you use ISO100, BIONZ is performing the math to divide down the sensor ADU number (while sensor gain is 0 dB.)  This will generate notches.

 

 

 

The Sony A7S has histogram notches at all the ISOs I've tried.

 

Mark

Edited by sharkmelley, 27 November 2016 - 04:45 AM.

[nikao]

We are talking flats here right? (not bias?) 

I'm confused by the fact that you guys are only talking about the camera itself, while with flats it is highly relevant to know how you shot them. (what light source and which optics?)

[sharkmelley]

How are you shooting the flats? 

It looks very much like Newton's rings to me.. could there be a reflection going on somewhere?

Thanks - I should have said more about how I shot the flats. This particular data comes from using a modified Sony A7S on the Tak Epsilon 180ED.  These were sky flats taken in evening twilight with an opaque white diffuser on the front of the scope.

 

I think any concentric banding is going to give the same appearance as Newton's Rings!  However, the evidence doesn't point to this as a cause.

 

Mark

[nikao]

 

How are you shooting the flats? 

It looks very much like Newton's rings to me.. could there be a reflection going on somewhere?

Thanks - I should have said more about how I shot the flats. This particular data comes from using a modified Sony A7S on the Tak Epsilon 180ED.  These were sky flats taken in evening twilight with an opaque white diffuser on the front of the scope.

 

I think any concentric banding is going to give the same appearance as Newton's Rings!  However, the evidence doesn't point to this as a cause.

 

Mark

 

What evidence?

Why would any in camera/sensor mechanics give you concentric banding? considering everything is made up out of squared patterns (RGGB) I don't see an easy explanation there, without considering the whole imaging train. 

 

I would be interested to see shots of the sky without a white diffuser and see if you still have those rings.....

[sharkmelley]

 

What evidence?

 

Why would any in camera/sensor mechanics give you concentric banding?

 

 

I tried to explain this when discussing the histogram notches.

 

An explanation along the lines of Newton's rings would need to address how the position of the rings alters with the fading twilight.

 

Mark

Edited by sharkmelley, 27 November 2016 - 05:11 AM.

[nikao]

 

 

What evidence?

 

Why would any in camera/sensor mechanics give you concentric banding?

 

 

I tried to explain this when discussing the histogram notches.

 

An explanation along the lines of Newton's rings would need to address how the position of the rings alters with the fading twilight.

 

Mark

 

Ok, I might not understand your post, but how did you determine the rings are moving on other pixels?

Could you post a few of the flats? I'd be interested to see what's going on here

 

Also, notice the pattern in the left upper quadrant of the flats. That's quite interesting as well. Do you know what is going on there?

 

And just to be sure; did you cover the view finder of the camera?

[sharkmelley]

 

 

 

Also, notice the pattern in the left upper quadrant of the flats. That's quite interesting as well. Do you know what is going on there?

 

 

 

That is banding triggered by the fact that the A7S full frame sensor (and probably many others) has a left half and right half.  The horizontal banding was quite a big issue for me because it doesn't calibrate out.  It tooks ages until I worked out how to get around it.  Full details of the investigation performed is here:

http://www.cloudynig...or-and-banding/

 

Mark

Edited by sharkmelley, 27 November 2016 - 05:40 AM.

[nikao]

Just checked a masterflat of my own, and I seem to see some sort of concentric banding. 

However; If I move the histogram to stretch it I see the circles move! quite weird effect, could it be due to the computer monitor or is it some sort of optical illusion?

Anyways, just thought it might be worth noting that the circles seem to move depending on how I stretch the histogram. So can't say that there is a 'real' displacement and might be consistent with the fading light for your conditions?

 

Did a quick screencapture of this effect. Will be available shortly here https://youtu.be/5vsmXOasZ_w

Edited by nikao, 27 November 2016 - 05:53 AM.

[DesertRat]

Nikao,
The fringes look like Newton's but we do not think they are at this time.  The reason for their mostly circular shape is that we have symmetry of the aperture and with it a radial fall off.

 

Mark,
If you have determined that the location of the steps is correlated with missing values then we are done. Correct me if I am wrong.  I would think that the reason they show up at f/2.8 and not so much at f/4 and slower is simply that for these slower optics the fringes migrate outwards to being mostly invisible.  Because the slope is significantly less for the slower optics.  I suppose if you look at anything close enough you will find some effect even at f/4 or slower.

 

It has seemed from this and Jerry's problem that this issue mostly arises when you have a significant enough radial falloff.

 

Its tempting to do a flat simulation, if you have not already.  Assume a set of idealized radial profiles from f/2, f/4 and f/8 optics.  Digitize the data at 12 or 14 bits and encode them using a lookup table with ~ 1k or 4k entries like the NEF tables, or any scheme that has enough missing values to matter.

 

Since even terrestrial photographers have complained about these lossy compression schemes its no wonder they cause problems for AP.  I can see how its useful for HD video however and having an option for lossy or not makes sense.

 

Glenn

 

PS: Should have noted this:  The microlens array has a color dependency giving a different falloff profile for r,g &b. Hence the colorization of these fringes.

Edited by DesertRat, 27 November 2016 - 12:43 PM.

[sharkmelley]

Mark,
If you have determined that the location of the steps is correlated with missing values then we are done. Correct me if I am wrong.  I would think that the reason they show up at f/2.8 and not so much at f/4 and slower is simply that for these slower optics the fringes migrate outwards to being mostly invisible.  Because the slope is significantly less for the slower optics.  I suppose if you look at anything close enough you will find some effect even at f/4 or slower.

 

It has seemed from this and Jerry's problem that this issue mostly arises when you have a significant enough radial falloff.

 

 

Yes, you're exactly correct.  

 

However I would add that I don't think this has anything at all to do with lossy compression.  The missing histogram values causing the problem are spaced well apart in the histogram unlike the combed histogram effect caused by the lossy compression.

 

The thing I find interesting is why the effect is not often seen.  It seems that most people never hit the issue and I personally have very rarely hit the issue.  I think the problem happens only when there is good spatial separation in the image between the rings that correspond to the missing histogram values.

 

Possibly, when imaging with higher ISOs, the noise (read noise and shot noise) is sufficiently high that the pixels with the "missing" values tend to mix in with one another spatially in the image and so the concentric ring becomes indistinct.  Also, because of slight changes in brightness from exposure to exposure the rings move slightly and this blurs their effect when the image is integrated.  This is only conjecture on my part.

 

Mark

[sharkmelley]

 

Ok, I might not understand your post, but how did you determine the rings are moving on other pixels?

 

Could you post a few of the flats? I'd be interested to see what's going on here

 

 

Here's an animation of the rings moving position as the lighting fades slightly:

https://drive.google...YWVZa0VMclh3cFk

 

Mark

[nikao]

 

 

Ok, I might not understand your post, but how did you determine the rings are moving on other pixels?

 

Could you post a few of the flats? I'd be interested to see what's going on here

 

 

Here's an animation of the rings moving position as the lighting fades slightly:

https://drive.google...YWVZa0VMclh3cFk

 

Mark

 

Looks very similar to my screencapture ...

[DesertRat]

Mark,

Its clear this behavior is made more visible with faster optics and quicker radial falloff.

 

What is not clear is this distinction you addressed between missing values and histogram combing.  I know these compression schemes are generally performed in 2 steps, and the details are different between cameras. Its the details I am not familiar with.  So I guess any help I can offer is quite limited by that ignorance.

 

What is really mysterious is why you see it at times and others not so much.  Could this be you hit on a sequence of flat frames which were distributed in rg&b levels and with time just at some magic level so that combined with any non-linearity and noise effects, that all this stuff just cancelled out?  Yikes!  Or hopefully when you do get these fringes - that is the exception and not the rule.

 

Now what I don't find mysterious is that some report never seeing this with cameras having lossy compression.  Possibly because they never looked hard enough.  I am not sure about that conjecture either.

 

If you can ever write some of this stuff up it might be of great value to the community.

 

Glenn

Edited by DesertRat, 27 November 2016 - 02:40 PM.

[bobzeq25]

 

Bob,

 

The PI FlatCountourPlot script is simply a tool to investigate regions and their shape.  Could be used on any image really.  It posterizes the data and then draws contours like a topo map.  If you take a perfect noiseless flat and run the script it looks like this:

 

syn_flat1k_contourPlot.png

 

This I am afraid does not help diagnose the problem here, but may have some utility.

 

Glenn

Agreed.  What I was referring to was that the contours on my flat were _tiny_.  I suppose the degree of vignetting has relevance also.

 

I also agree that I've not looked "hard enough".  Because I've never seen a practical problem.

Edited by bobzeq25, 27 November 2016 - 04:17 PM.

[sharkmelley]

 

What is not clear is this distinction you addressed between missing values and histogram combing.  I know these compression schemes are generally performed in 2 steps, and the details are different between cameras. Its the details I am not familiar with.  So I guess any help I can offer is quite limited by that ignorance.

 

 

Good point.  Let me explain with some histograms of Sony A7S data.

 

 

In the above chart we see the transition from the zone where every integer is present in the histogram to the zone where all the odd integers are absent.  The missing values are caused by the lossy compression.  However in the zone where all integers are present, there are regular "notches" where some integers are under-represented.  It is this effect that is causing the concentric rings in the calibrated light frames.  The Nikon D7000 (for example) has actual missing histogram values in the red and blue channels (because of scaling) and these occur widely spaced like the notches in the Sony A7S.

 

In the next A7S chart we see a transition from the zone where every other value is present to the zone where every 4th values is present:

 

 

Again this is a symptom of the variable gap lossy compression scheme.  The next chart shows the transition from where every 4th value is present to the zone where every 8th value is present (the gaps have increased further):

 

 

I hope this makes the distinction clear.  In the lossy compression zones, at least 50% of the values are missing.  In the "regular" zone occasional values are missing or under-represented.  The cause of these widely spaced missing values is not clear but they do seem to create the concentric ring artefacts in calibrated light frames.

 

As an aside, Nikon cameras usually let the user choose whether or not to use lossy compression.  Bill Claff has done a lot of work to characterise which histogram values are present when lossy compression is selected:

http://www.photonsto...Compression.htm

 

Mark

Edited by sharkmelley, 27 November 2016 - 06:39 PM.

[DesertRat]

Hi Mark.
That is much clearer and your finding of this correlation well done.

 

I have always viewed these DSLR's as a little inscrutable.  Even ones without lossy compression have been found to preprocess data in one way or another, which few of us here appreciate and fewer who understand it..

 

Bill Claff has presented data for the Nikons and I have reviewed some of it.  In addition to that I would need a better understanding of this scaling operation.  At least at one time in a CFA the red filter was the most efficient and any adjustments were made to the green and blue.  But white balance today is normally used via digital gain adjustments to r & b, using green as a reference.  I would need to no more about the details of the compression if I were to pursue any further analysis.

 

Thanks for bringing this data forward and sharing your findings!

 

Glenn

[bobzeq25]

Mark - the seemingly most popular Nikons here, the D5XXX series, do not let you turn off lossy compression.  But, since it's biased toward compressing highlights, I believe the effect is minimal in astro.  It's conceivable it could bite in some specific situation, but that's even more conjecture than the previous statement, which is based on results.

Edited by bobzeq25, 27 November 2016 - 11:58 PM.

[sharkmelley]

Mark - the seemingly most popular Nikons here, the D5XXX series, do not let you turn off lossy compression.  But, since it's biased toward compressing highlights, I believe the effect is minimal in astro.  It's conceivable it could bite in some specific situation, but that's even more conjecture than the previous statement, which is based on results.

I agree.  I think it is unlikely that the lossy compression will ever produce visible artefacts because the variable gap histogram compression is a well engineered solution.  If you think about what it is doing, it is introducing quantisation into the data - small amounts of quantisation at the low end and higher amounts of quantisation at the high end.  Quantisation error is something we talk about all the time in astro-imaging in our endless discussions about which ISO to use and whether we should we use unity gain ISO etc.  So we well understand the effects of quantisation error.  We know that if the size of the quantisation error is swamped by other sources of noise then it will not have any effect.  Looking at the NEF compressed histograms (at least in 14bit mode) the combined read noise and shot noise is easily sufficient to mask the quantisation error introduced by the histogram gaps.

 

Mark