Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Concentric coloured rings caused by flats

  • Please log in to reply
79 replies to this topic

#51 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 06:21 PM

The following is what I think is going on in the Sony A7S.  I have to assume that the notches in the histogram are caused by the camera applying a scaling factor to digital values that come out of the ADU (Analogue Digital Unit).  Why on earth it does this I don't know but the evidence strongly suggests that it does.

 

Below are 4 columns of figures. Column 1 is a list of integer values with no gaps - it represents a nice linear gradient coming from some theoretical image.  Column 2 has a scalar multiplier of 1.11 applied (this is just an arbitrary value as an illustration).  If the camera could write floating point numbers into the raw file then those are the numbers it would write. But it can't, so it converts them to the nearest integer, shown in column 3.  Notice that the gaps now appear - the values 15 and 25 are missing.  Column 4 is the ratio of columns 2 and 3.  A ratio greater than 1 is where the integer is higher than the "true" floating point number and a ratio less than 1 is where the integer is less than the "true" floating point number.

 

10   11.10   11   0.991
11   12.21   12   0.983
12   13.32   13   0.976
13   14.43   14   0.970
14   15.54   16   1.030
15   16.65   17   1.021
16   17.76   18   1.014
17   18.87   19   1.007
18   19.98   20   1.001
19   21.09   21   0.996
20   22.20   22   0.991
21   23.31   23   0.987
22   24.42   24   0.983
23   25.53   26   1.018
24   26.64   27   1.014
25   27.75   28   1.009
26   28.86   29   1.005
27   29.97   30   1.001

 

When we take lots of flat frames, there will (usually) be subtle intensity variations in each one which means that when we average them out in the creation of a master flat we get something very close to the true floating point data represented by column 2.  So when we divide the actual raw integer data (column 3) by this master we are left with the ripple effect shown in column 4. 

 

Here is a graph of that effect: 

ConcentricRingGraph.jpg

 

Note how the amplitude of the ripple decreases as the values become higher.  It is this ripple that we observe as concentric rings in the exposure calibrated with a flat.

 

One other point to note is that in a real image this "ripple" will be masked by shot noise which will hide the ripple. To show the ripple (i.e. the concentric rings), we need underexposure which keeps the pixel values low and the amplitude of the ripple high.  To show the ripple we also want a very low ISO to avoid too much shot noise masking the data ripple.  This is why in my real world examples in the earlier post I deliberately used a low ISO and underexposure.

 

One final point.  It is possible to take a whole load of flat frames with a low ISO and accidentally underexposed.  A master flat created from these can have the ripple actually encoded into the master flat.  This will introduce concentric rings into the final stacked image.  When it happened to me the only solution was to shoot a new set of flats.

 

The way to mitigate the concentric ring effect is to use higher ISOs and/or make sure data are well exposed.  I have proved that concentric rings can easily be found in the Sony A7S because of this assumed scaling of integer data in all channels (i.e. R, G and B). 

 

Mark


Edited by sharkmelley, 29 November 2016 - 06:24 PM.

  • DesertRat and bsavoie like this

#52 ccs_hello

ccs_hello

    Voyager 1

  • *****
  • Posts: 10,623
  • Joined: 03 Jul 2004

Posted 29 November 2016 - 07:58 PM

Mark,

 

Your post#1 chart seems to indicate the gaps are either 36 or 37 ADUs.  I'd suggest the scaler is 1.03 (i.e., 3% gain added.)

In my D5100 analysis (R abd B channels), it's either 7 or 8 ADUs.  That should be 14% gain scaling up to match up G-channel's dynamic range.

 

Sony's 3% adjustment probably is due to ISO calibration.  In my view, SONY engineers should know pretty well about it and should be able to release a patch if DPR community makes enough noise about it.

 

(Or ask nikonhacker team to jump ship to become the sonyhacker team :) )

Clear Skies!

 

ccs_hello



#53 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 08:22 PM

 

Sony's 3% adjustment probably is due to ISO calibration.  In my view, SONY engineers should know pretty well about it and should be able to release a patch if DPR community makes enough noise about it.

 

That's an interesting thought - yes it might well be for ISO calibration purposes.

 

Mark



#54 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 30 November 2016 - 09:54 AM

Hmm I see something similar in my Nikon D810A / Star 71 imaging set ... not as severe and not the same by a mile but I see it ...

Need to investigate ...

 

 

 

attached exaggerated processing

iso 400 3 minute sub, so you can call it under exposed?

 

/Yves

Attached Thumbnails

  • test.jpg

Edited by vdb, 30 November 2016 - 09:56 AM.

  • sharkmelley and bobzeq25 like this

#55 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 30 November 2016 - 10:51 AM

Hmm I see something similar in my Nikon D810A / Star 71 imaging set ... not as severe and not the same by a mile but I see it ...

Need to investigate ...

 

 

 

attached exaggerated processing

iso 400 3 minute sub, so you can call it under exposed?

 

/Yves

That is a really clear example - I really wasn't expecting to see it from a Nikon.  Yes you have stretched the data but no more than you need to stretch the data to show the faint cosmic structures. 

 

Is it possible for you to upload one of the raw lights and one of the raw flats.  I'd love to take a look to see if it is possible to determine the cause.

 

Mark



#56 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 30 November 2016 - 11:13 AM

Mark & Yves,
I'm thinking, as I posted just now in Jerry's thread on the N5300, that there may be something in the decompression that's causing this.

 

If the decompression alters the data, specifically the gamma response on the low end, these fringes could result.  Such an encoding error would bin gradients into discrete bins resulting in fringes.

 

Just a thought.  Possibly employ a different engine than DCRAW, like the original vendors codec, to perform the first stage.

 

Glenn


Edited by DesertRat, 30 November 2016 - 11:14 AM.


#57 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,349
  • Joined: 27 Oct 2014

Posted 30 November 2016 - 11:29 AM

Hmm I see something similar in my Nikon D810A / Star 71 imaging set ... not as severe and not the same by a mile but I see it ...

Need to investigate ...

 

 

 

attached exaggerated processing

iso 400 3 minute sub, so you can call it under exposed?

 

/Yves

The easiest and best way to determine this is to look at where the skyfog peak is on the (usual) stretched histogram of one sub.  With a low read noise camera like the 810a, 25% to the right would be my benchmark for proper exposure.

 

I think to truly see if this could be an issue for me, I have to try actual imaging at a variety of exposures, and hence a variety of stretches.  With the fastest optics I have.  That may also tell me an easy way to avoid it.  I have done some images at less than 25 % and not seen it, though.  Example below.  I stretched this as far as I could until the image deteriorated.  Don't recall any rings.  3 minutes at ISO 200.  A bit slower than F6.  It depends on sky brightness (why using the skyfog peak is good), but that's clearly less exposed than the image above.  So maybe I got "saved" by the F number and/or the small amount of vignetting?

 

http://www.astrobin....237518/?nc=user

 

So maybe it's a combination of all of the following.  Low F number.   Large amount of vignetting (the full frame 810a would show that more than the D5XX).  Underexposure. 

 

That might explain why it's so rare, and suggest an effective way to avoid it is simply to be careful to expose properly when using low F numbers with optics that exhibit substantial vignetting.


Edited by bobzeq25, 30 November 2016 - 11:40 AM.


#58 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 30 November 2016 - 02:12 PM

Just talked to someone who also uses a D810A, had the same problem, solution was in his case to take flats that have histogram far to the right, like 1/4 of the right side.

 

/Yves


  • bobzeq25 likes this

#59 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 30 November 2016 - 02:21 PM

Just talked to someone who also uses a D810A, had the same problem, solution was in his case to take flats that have histogram far to the right, like 1/4 of the right side.

 

/Yves

That makes complete sense to me.  If you look at your image, in each ripple you can see the red ramping up towards the centre then a sharp drop, ramping up then a sharp drop.  That's the wrong direction if the ripple is in the light frames.  So it means the ripple has been encoded into the master flat itself.

 

I had the same problem on my Sony A7S which was solved by exposing the flats well over to the right.

 

I'd still be interested to see one of your raw flats just to understand exactly what kind of "histogram gapping" is causing it.

 

Mark


Edited by sharkmelley, 30 November 2016 - 02:23 PM.

  • bobzeq25 likes this

#60 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,349
  • Joined: 27 Oct 2014

Posted 30 November 2016 - 02:32 PM

Bottom lines for me.  The problem is rare, especially on D5XXX.  And there are easy solutions when it occurs, that don't involve limiting your targets or your optics.

 

Thanks for your work and your expertise on this, Mark.


Edited by bobzeq25, 30 November 2016 - 02:33 PM.

  • bsavoie likes this

#61 nikao

nikao

    Viking 1

  • -----
  • Posts: 513
  • Joined: 14 Jun 2012

Posted 30 November 2016 - 03:08 PM

great topic and analysis! very insightful. 

 

About underexposing; When I was shooting in Namibia with my D600 at ISO200 for 8 or 10 minutes it was really underexposed. Never saw any problem though. 

Flats were shot at 50% histogram. 


  • bsavoie and bobzeq25 like this

#62 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 01 December 2016 - 07:07 PM

Hmm I see something similar in my Nikon D810A / Star 71 imaging set ... not as severe and not the same by a mile but I see it ...

Need to investigate ...

 

post-97748-0-77843700-1480517644.jpg

Hi Yves,

 

As you said to me in a message, the rings are visible even in a single exposure.  I've taken the single exposure you sent me and enhanced them here (with no calibrating frames):

 

Red

yves_red.jpg

 

 

Green

yves_green.jpg

 

 

Blue

yves_blue.jpg

 

The problem is that the rings do not correlate with contours in the image and definitely not with gaps in the histogram.  The histograms do have the usual gaps but they just don't correlated with the image.

 

Here is the red channel again, just with a simple stretch:

 

yves_red_gradient.jpg

 

You can see a top to bottom gradient but the rings totally ignore that gradient.

 

So I think there's probably an alternative explanation for your rings.

 

Mark


Edited by sharkmelley, 01 December 2016 - 07:13 PM.

  • Miguelo likes this

#63 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 02 December 2016 - 01:24 AM

Thanks Mark, I processed M45 with and without flats, and the results are the same so yes the flat calibration makes them more visible but that could be related to the dividing process, they are in the lights, I never saw them when taking iso 800 5 minute subs ... or probably I never searched for them so I need to revisit some previous captures

Attached Thumbnails

  • Screen Shot 2016-12-02 at 07.17.49.jpg

Edited by vdb, 02 December 2016 - 01:24 AM.


#64 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 02 December 2016 - 04:06 AM

Just got confirmation from another D810A user, 1 minute subs, concentric rings as well,

so we need to establish a minimum value where it's gone, my guess is 7000 or so, as I don't see it in my M31 one exposures, iso 800, 5 minute subs. f4.9

 

/Yves



#65 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 04 December 2016 - 08:44 AM

Doubled my light exposure time from 180 sec to 360 and gone are the concentric circles, even with same master flat ...

 

/Yves



#66 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 04 December 2016 - 11:30 AM

Doubled my light exposure time from 180 sec to 360 and gone are the concentric circles, even with same master flat ...

 

/Yves

That's good news!

 

There is definitely weirdness happening in the histograms of the D810A - at least the histograms generated from the FIT files produced by SGP.  What I am seeing is the following:

 

NikonD810A_R.JPG

NikonD810A_G.JPG

NikonD810A_B.JPG

 

It looks like scaling is taking place in all three channels.  Is this behaviour also seen in the raw NEF files from the D810A?

 

Mark


Edited by sharkmelley, 04 December 2016 - 11:31 AM.


#67 vdb

vdb

    Surveyor 1

  • -----
  • Posts: 1,567
  • Joined: 08 Dec 2009

Posted 04 December 2016 - 11:57 AM

Good news and not, someone else is reporting circles with one object and not with other ... really bizar ...



#68 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 11 April 2017 - 06:48 PM

Some progress on this issue.  After some detailed analysis of the histogram gaps I think it is possible to repair the Sony A7S data to undo most of the damage done by the in-camera raw processing i.e. the in-camera digital multiplication of the data by 1.027.  If so, it might prevent the banding from appearing or at least greatly attenuate it.  I've written a prototype in PixInsight and the first test results of a real (non-astro) image can be found here:

https://www.dpreview...s/post/59404098

 

Once the algorithm has been refined I'll test it on some actual astro-data.

 

Mark



#69 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 30 April 2017 - 02:37 AM

Finally I've had some time to do further testing on this.  In doing so I've made a new discovery: using bulb mode on the A7S has a much more severe potential banding problem than non-bulb mode.  But I'm running ahead of myself here.

 

I'm using the same set of flats data that I used for the original post. In the testing below, I used 10 flats to create a master flat and used this master flat (and a bias of course) to calibrate the 11th flat.  After stretching the result (no colour saturation was done) here is the result:

 

a7s_concentric_banding_before.jpg

 

 

I also picked up the same 11th flat and processed it in Photoshop by setting the white balance and then pushing the saturation up to nearly 100%.  Here is the result:

 

a7s_concentric_banding_photoshop.jpg

 

 

The ring structure is clearly visible.  This demonstrates that it is a real problem and not some artifact of my processing.

 

I decided to look at the histogram of the data and this is when I made the discovery that all those flats had been made in silent shutter mode.  Silent shutter on the A7S reduces the camera to 12bit mode, just like continuous shooting and bulb modes do.  Here is the histogram:

 

A7S_12bit_Histogram_small.jpg

 

 

Looking at that histogram I made an important discovery.  There is an extra set of histogram gaps with period 150ADU - I've highlighted their positions in yellow.  These are in addition to the usual set of histogram gaps with period 37.5ADU.  So it appears that in 12bit mode (silent shutter or bulb mode or continuous shooting) there are two sets of digital scaling going on.  In fact it is the long period (150ADU) digital scaling that is causing the concentric bands in the original image. 

 

So I had to adjust my correction algorithm to remove this long period scaling (as best as possible). I applied the algorithm to the all 11 flats then created the master flat.  The 11th (corrected flat) was then divided by the (corrected) master flat.  Here's the result:

 

a7s_concentric_banding_after.jpg

 

 

On the right hand side of the sensor, the banding has disappeared almost entirely which makes me think I have correctly identified the underlying cause of the banding.  However, on the left hand side of the sensor there is a big mess.  This is probably because the camera does some additional processing to cope with the fact that the sensor is built in two halves.  I guess the in-camera processing has to blend the left and right in some "seamless" manner so the sensor design does not affect "normal" photos.  It leaves some nasty artifacts that I have encountered previously on A7S astro-images (https://www.cloudyni...sor-and-banding).  These left hand side artifacts are unique to each camera.

 

If I subtract this corrected result from the original this is what I see:

 

a7s_concentric_banding_difference.jpg

 

 

You can clearly see the rings that have been removed by the algorithm.

 

So why hasn't this A7S concentric banding in 12bit mode been discovered by other astrophotographers?  Does everyone avoid bulb mode?  Or is it the case that those who do use bulb mode don't bother to use calibration frames?  It's a bit of a mystery.

 

I took some "normal" terrestrial photos in 12bit mode and I was able to see ring-like tonal variations in the sky and in other featureless expanses.  I'm pretty much convinced that it is this effect that is causing the rings observed by various commentators in A7 series cameras e.g.
https://diglloyd.com...arthShadow.html

 

With the bulb mode spatial filtering causing star eating behaviour and with the digital scaling causing coloured concentric bands the Sony A7S is a very "interesting" camera to work with!

 

Mark


Edited by sharkmelley, 30 April 2017 - 02:50 AM.

  • Maurice Toet, SandyHouTex and entilza like this

#70 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 01 May 2017 - 02:36 AM

With a bit of further analysis I can give some very specific advice for A7S users based on the fact that the greater the shot noise the more the peaks and troughs of the concentric rings are reduced in amplitude.  Some simulations I performed showed that when the standard deviation of the shot noise in ADU increases to around half the period of the histogram gaps then the concentric rings more or less disappear.

 

Let's assume we are aiming to have the back of camera histogram peak at around the 1/3 from the left hand side.  Then what ISO do we need to give the necessary shot noise?

 

Very roughly, a 1/3 histogram peak corresponds to an ADU of around 1600 out of 16384. 

In non bulb mode, ISO 800 or greater is sufficient to suppress the concentric rings caused by the histogram gaps of period 37.5

In bulb mode, ISO 10000 or greater is required to suppress the concentric rings caused by the bulb mode histogram gaps of period 150.  This includes silent shutter and continuous shooting modes.

 

Since concentric banding can appear in both the lights and the flats, the above advice applies to both lights and flats. 

 

Mark


  • SandyHouTex likes this

#71 jammeymc

jammeymc

    Ranger 4

  • -----
  • Posts: 328
  • Joined: 19 Mar 2016

Posted 01 May 2017 - 11:10 PM

Thank you Mark. Just so i understand, safe bet in bulb mode is iso 10K or greater, for both flats and lights? More importantly, your also saying that, at this level, you can in fact image safely with a histogram peak of 1/3 or greater?  With my skies, at 10K, i probably will never need bulb mode to be honest. Lastly, in non-bulb mode, it's safe to image at the 1/3 histogram peek @ 800 or higher?

 

I love the A7S but have been very discouraged to astro image with it. This is good and bad news -  It seems the bad is that long single frame integration's will be almost impossible with this camera, wouldn't you agree? Appreciate all your had work in nailing this issue down for all of us A7S owners.....



#72 jammeymc

jammeymc

    Ranger 4

  • -----
  • Posts: 328
  • Joined: 19 Mar 2016

Posted 01 May 2017 - 11:16 PM

One more question.......@ non bulb iso 800, does this apply for silent shutter as well?



#73 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 02 May 2017 - 12:30 AM

The above recommendations should prevent the concentric banding.  Bulb mode, silent shutter and continuous shooting are grouped together because they all cause the camera to drop down to 12 bits and introduce the more pernicious widely spaced histogram gaps.

 

However in practice I would recommend to always use ISO 2000 and above, even in non-bulb mode, to take advantage of the low noise sensor read out mode that is used for ISO 2000 and above.

 

There is one more caveat, depending on how badly the camera suffers from the split sensor horizontal bands on the left hand side.  My A7S suffers from it but I have seen examples of the A7S with no left hand side banding at all.  I have also seen examples far worse than mine.  To overcome it I currently use ISO 10000 and push the back of camera histogram just to the right of the hallway point.  Unlike the concentric banding issue I have no maths to back up this suggestion because, to be honest, I have no clue what kind of in-camera processing is causing it.  At some point I might go back and re-analyse this issue, armed with the knowledge I now have about the cause of the concentric banding. 

 

Mark


Edited by sharkmelley, 02 May 2017 - 12:31 AM.

  • charotarguy likes this

#74 jag32

jag32

    Apollo

  • *****
  • Posts: 1,073
  • Joined: 30 May 2017

Posted 05 October 2017 - 07:23 AM

Man, with these issues combined with star eater, it seems like more than firmware updates are needed and maybe it's best to just wait for the A7S Mk III and hope Sony has worked out their issues with this. 

 

Shooting in bulb mode at over 10000 iso to mitigate this issue is just not an acceptable solution. 


Edited by jag32, 05 October 2017 - 07:31 AM.


#75 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 05 October 2017 - 09:07 AM

Man, with these issues combined with star eater, it seems like more than firmware updates are needed and maybe it's best to just wait for the A7S Mk III and hope Sony has worked out their issues with this. 

 

Shooting in bulb mode at over 10000 iso to mitigate this issue is just not an acceptable solution. 

It depends on your optics and your sky conditions.  For my (fast) scope and (dark) sky conditions I need ISO 10000 in order to get the histogram halfway across the graph with a 30second exposure.  The required ISO to achieve this will vary quite a lot from user to user.

 

I don't think the split sensor issue will go away in the near future.  Maybe Sony might change their approach to dealing with the artifacts caused by it.

 

Mark


Edited by sharkmelley, 05 October 2017 - 09:10 AM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics