Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Concentric coloured rings caused by flats

  • Please log in to reply
79 replies to this topic

#26 srosenfraz

srosenfraz

    Fly Me to the Moon

  • *****
  • Posts: 5,649
  • Joined: 06 Mar 2011
  • Loc: United States

Posted 28 November 2016 - 11:13 AM

Food for thought - if the quantisation error is greater at the high (bright) end of the scale, then I would expect that there would be a fair amount of quantisation error in a flat.  After all, our flats are generally well exposed.  Perhaps this aspect of the lossy compression is a factor?



#27 nikao

nikao

    Viking 1

  • -----
  • Posts: 513
  • Joined: 14 Jun 2012

Posted 28 November 2016 - 11:32 AM

Food for thought - if the quantisation error is greater at the high (bright) end of the scale, then I would expect that there would be a fair amount of quantisation error in a flat.  After all, our flats are generally well exposed.  Perhaps this aspect of the lossy compression is a factor?

Quantization error is 0.29LSB (least significant bit), depending on your gain this is really really low and in most cases will be swambed by read noise and shot noise



#28 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,348
  • Joined: 27 Oct 2014

Posted 28 November 2016 - 01:27 PM

Food for thought - if the quantisation error is greater at the high (bright) end of the scale, then I would expect that there would be a fair amount of quantisation error in a flat.  After all, our flats are generally well exposed.  Perhaps this aspect of the lossy compression is a factor?

Quantisation error is relative to the signal intensity.  Which is why it's clever that Nikon introduces compression by skipping more ADUs as the ADU increases, and not skipping any at the low end.  See this.

 

http://www.photonsto...Compression.htm

 

So flats are not a problem with quantisation.


Edited by bobzeq25, 28 November 2016 - 01:34 PM.


#29 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 28 November 2016 - 01:39 PM

 

Food for thought - if the quantisation error is greater at the high (bright) end of the scale, then I would expect that there would be a fair amount of quantisation error in a flat.  After all, our flats are generally well exposed.  Perhaps this aspect of the lossy compression is a factor?

Quantization error is 0.29LSB (least significant bit), depending on your gain this is really really low and in most cases will be swambed by read noise and shot noise

 

That is only true for no lossy compression, no missing values and histogram combing.  The quantization error here is variable and dependent on intensity.  Your figure is for native analog to digital conversion, before any encoding, and is normally much less than intrinsic read noise and swamped by other noise sources.

 

Although I have used the term quantization error myself it is not strictly correct.  Its an encoding error with variable step size.

 

Glenn


Edited by DesertRat, 28 November 2016 - 02:08 PM.

  • nikao likes this

#30 nikao

nikao

    Viking 1

  • -----
  • Posts: 513
  • Joined: 14 Jun 2012

Posted 28 November 2016 - 01:41 PM

 

 

Food for thought - if the quantisation error is greater at the high (bright) end of the scale, then I would expect that there would be a fair amount of quantisation error in a flat.  After all, our flats are generally well exposed.  Perhaps this aspect of the lossy compression is a factor?

Quantization error is 0.29LSB (least significant bit), depending on your gain this is really really low and in most cases will be swambed by read noise and shot noise

 

That is is only true for no lossy compression, no missing values and histogram combing.  The quantization error here is variable and dependent on intensity.  Your figure is for native analog to digital conversion, before any encoding, and is normally much less than intrinsic read noise and swamped by other noise sources.

 

Glenn

 

Thanks for the clarification! :)



#31 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 28 November 2016 - 02:47 PM

 

 

So flats are not a problem with quantisation.

What Mark has demonstrated is that lossy compression in any camera will produce these kinds of artifacts (circular and/or banding) when certain conditions exist:

 

1) Faster optics with higher radial falloff
2) Collecting lots of data with subsequent stretching to bring forth details about a small range of values.  This is typically the case under moderate LP and RGB imaging where the feature of interest is buried at or even below background sky noise.  This requires significant stretching, and many frames normally.

 

Now the characteristics of these artifacts may differ as a function of the lossy scheme, and intensity levels, but I don't see how they can be avoided in general if the 2 conditions above are in play.  Because they follow as a property of numbers with missing or attenuated discrete levels and their manipulation, subtracting and division.

 

My take on all of this is to avoid lossy compression if possible.  And if that is what you are dealt with then imaging goals need changing.  For example concentrating on brighter stuff, like stars, some nebula and comets. Working at f/4 or slower helps too.

 

None of this is completely hopeless, but the work to deal with it is often a hard limit.  This behavior acts like a variable pattern noise, and if it could be modeled with any precision it could in theory be accounted for. And it is in some of HST's cameras (variable pattern noise there is electrical in nature not lossy compression), but we don't have their resources.  What one can do with PI is pretty far reaching, but not entirely satisfactory.

 

Glenn


Edited by DesertRat, 28 November 2016 - 03:25 PM.


#32 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 28 November 2016 - 06:12 PM

I'm very close to having a complete explanation of how these rings are caused. If I'm right it's nothing to do with lossy compression but everything to do with digital values being multiplied by a scalar.

Mark
  • ccs_hello likes this

#33 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,348
  • Joined: 27 Oct 2014

Posted 28 November 2016 - 06:47 PM

Have you seen this?  Scroll down to the images with round banding.

 

http://www.strollswi...iso-vs-pushing/


  • DesertRat likes this

#34 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 06:08 AM

In essence, my argument is that the Sony A7S applies a scaling to the digital values that creates widely spaced histogram gaps and associated ripples in the data. These ripples become obvious after calibration with a good quality master flat.

Mark

#35 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • Posts: 7,521
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 29 November 2016 - 07:49 AM

In essence, my argument is that the Sony A7S applies a scaling to the digital values that creates widely spaced histogram gaps and associated ripples in the data. These ripples become obvious after calibration with a good quality master flat.

Mark

Mark, is that the same thing that's happening with the Nikon D5300?

 

Jerry



#36 ccs_hello

ccs_hello

    Voyager 1

  • *****
  • Posts: 10,623
  • Joined: 03 Jul 2004

Posted 29 November 2016 - 08:46 AM

Jerry,

 

Nikon engineers/image scientists decided to perform (just) the R and B channel scaling up, to compensate for weaker signal (less opaque Bayer CFA), see the diagrams here:

http://www.cloudynig...-2#entry5266359

 

It is different than SONY's, which is uniformly applying a weak scaling (less gaps but longer distance in between rounding-up/rounding-down gaps)

on all R, G. and B channels.

 

Nikonhacker firmware has a feature to defeat many of the in-camera processing, on some selected camera models.

 

Clear Skies!

 

ccs_hello


Edited by ccs_hello, 29 November 2016 - 09:00 AM.


#37 ccs_hello

ccs_hello

    Voyager 1

  • *****
  • Posts: 10,623
  • Joined: 03 Jul 2004

Posted 29 November 2016 - 08:52 AM

SONY ARW (or Nikon NEF) lossy version of the image compression is using companding techniques (mapped thru a Huffman codebook.)

The resulted missing output codewords (seen as the gaps in histograms) will bot be uniformly distributed. 

Scaling (digital multiply), on the other hand, will.

 

Clear Skies!

 

ccs_hello



#38 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 09:39 AM


In essence, my argument is that the Sony A7S applies a scaling to the digital values that creates widely spaced histogram gaps and associated ripples in the data. These ripples become obvious after calibration with a good quality master flat.

Mark

Mark, is that the same thing that's happening with the Nikon D5300?

Jerry

No, I don't believe that Nikon cameras are affected by this. These particular histogram gaps are a Sony A7S thing. Maybe other Sony cameras are affected - I don't yet know.

Mark

Edited by sharkmelley, 29 November 2016 - 09:43 AM.


#39 jammeymc

jammeymc

    Ranger 4

  • -----
  • Posts: 328
  • Joined: 19 Mar 2016

Posted 29 November 2016 - 10:44 AM

I didn't read through all the replies but i had very similar rings show up in my A7S image stack during my stretching process . Turns out, i had some internal reflections that was going on due to my image train.  In the end, the problem was my particular eos to nex adapter. I changed adapters and problem was solved.   



#40 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,348
  • Joined: 27 Oct 2014

Posted 29 November 2016 - 11:08 AM

 

 

 

So flats are not a problem with quantisation.

What Mark has demonstrated is that lossy compression in any camera will produce these kinds of artifacts (circular and/or banding) when certain conditions exist:

 

1) Faster optics with higher radial falloff
2) Collecting lots of data with subsequent stretching to bring forth details about a small range of values.  This is typically the case under moderate LP and RGB imaging where the feature of interest is buried at or even below background sky noise.  This requires significant stretching, and many frames normally.

 

Now the characteristics of these artifacts may differ as a function of the lossy scheme, and intensity levels, but I don't see how they can be avoided in general if the 2 conditions above are in play.  Because they follow as a property of numbers with missing or attenuated discrete levels and their manipulation, subtracting and division.

 

My take on all of this is to avoid lossy compression if possible.  And if that is what you are dealt with then imaging goals need changing.  For example concentrating on brighter stuff, like stars, some nebula and comets. Working at f/4 or slower helps too.

 

None of this is completely hopeless, but the work to deal with it is often a hard limit.  This behavior acts like a variable pattern noise, and if it could be modeled with any precision it could in theory be accounted for. And it is in some of HST's cameras (variable pattern noise there is electrical in nature not lossy compression), but we don't have their resources.  What one can do with PI is pretty far reaching, but not entirely satisfactory.

 

Glenn

 

I'm just not seeing the problem.  This is a bright target, but I stretched it hard to capture the outer regions.  The limit on the stretch was, as usual, diffuse noise, not the appearance of rings.  Skies pretty light polluted, mag per arc sec mid 18s.  I suppose someone might say F4.8 is enough to avoid the issue?  Or that the vignetting is not high enough to cause it?  I don't think that's it.

 

I have imaged the night sky wide field with camera lenses, also, at lower F numbers.  Still, have never seen rings while processing.  Those aren't very good, so I choose not to post them, you can look them up on my astrobin if you like.  But here's a decent one from astrobin with a fast camera lens, I'm sure there are many more there.

 

http://www.astrobin....3148/0/?nc=user

 

The fact remains that many, many people are imaging with D5XXX cameras with Nikon's particular lossy compression scheme, and just not seeing the issue.  I think that:  Jerry's image is not a compression issue, but either a processing isssue (people have processed his data in other ways, and gotten a good image, without the rings) or a lens issue.  I'm not convinced the rings Mark has pulled up (I can get them too, but only by vastly overstretching the data beyond anything one would want to do, they're tiny) are an issue.  Rather than this being a major issue that should limit one's targets, I see it as an outside possibility that is yet to proven as a practical problem in doing astrophotography with the D5XXXs.

 

One which has not been shown here to exist in an actual AP image.  I appreciate the theoretical discussion by people who do not have a suitable camera to experiment with, much less doing those experiments with actual AP.  Just think that, as a practical issue, some people are taking this _way_ too far.  

 

Limit targets?  Clusters, comets, and _some_ nebula?  Please.  I'll do that when I actually see an issue.  Don't expect that to happen, ever.

 

Bottom line.  In the below $1000 category, I think the Nikon D5300/D5500 is the obvious best choice.  Good sensitivity very low noise (I don't usually do darks).  I think anyone pushed away from the camera by this theoretical discussion of an unlikely problem, one that has not been seen by the substantial number of people who use this camera, and have made a great many images with it, is making a mistake.

 

Personal opinion.  I think Nikon's particular method of doing lossy compression, at the high end of the ADU range, is not an issue for astrophotography. 

Attached Thumbnails

  • M31 SV70T small.jpg

Edited by bobzeq25, 29 November 2016 - 11:41 AM.

  • Traveler and nikao like this

#41 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 29 November 2016 - 12:26 PM

Mark,
Assume you were to assemble flats from a large number of varying levels.  Further you stack lights similarly from exposures 15s, 30s .. 90s.  Or maybe 30, 31, 32, .. 45s.  Would this help by smoothing the distribution?  Not sure but since you have a low read noise camera you could presumably control this to some extent, since there is little penalty from large N.  Sounds lke a lot of work, and darks would be a problem but scaling them might help.

 

I recognize with a split sensor you almost have to calibrate the equivalent of two data streams in series, though that is a bit of an exaggeration.

 

Need to find out more about this encoding effect before I can be of much more help, as limited as that has been so far.
 
Bob,
Mark is employing at full frame camera at f/2.8.  You have a smaller sensor at f/4.8.  That is a big difference, some would say huge.  The slope of fall-off is significantly different, and the slope magnifies these artifacts geometrically into visibility.

 

Agree great images are possible with the N5300 & N5500.  I would even say an APOD is possible with these cameras, or with the Sony A7S.  These issues do not happen all the time, but when they do they apparently come just at the wrong time, another strike from Murphy.

 

NEF lossy compression should not create what Mark is reporting here, in the presence of noise and sufficient stacking, NEF "quantization" should be totally filled in.  Marks bands are different in a way I do not fully understand.

 

This is not theory, and I am an experimentalist.  Have lots of cameras of all types, had successes along the way with 10X the number of fails.  Have seen banding issues from any number of cameras, including uncompressed raws.  They can come from any number of sources, too many to detail here.

 

Glenn



#42 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,348
  • Joined: 27 Oct 2014

Posted 29 November 2016 - 01:14 PM

Mark,
Assume you were to assemble flats from a large number of varying levels.  Further you stack lights similarly from exposures 15s, 30s .. 90s.  Or maybe 30, 31, 32, .. 45s.  Would this help by smoothing the distribution?  Not sure but since you have a low read noise camera you could presumably control this to some extent, since there is little penalty from large N.  Sounds lke a lot of work, and darks would be a problem but scaling them might help.

 

I recognize with a split sensor you almost have to calibrate the equivalent of two data streams in series, though that is a bit of an exaggeration.

 

Need to find out more about this encoding effect before I can be of much more help, as limited as that has been so far.
 
Bob,
Mark is employing at full frame camera at f/2.8.  You have a smaller sensor at f/4.8.  That is a big difference, some would say huge.  The slope of fall-off is significantly different, and the slope magnifies these artifacts geometrically into visibility.

 

Agree great images are possible with the N5300 & N5500.  I would even say an APOD is possible with these cameras, or with the Sony A7S.  These issues do not happen all the time, but when they do they apparently come just at the wrong time, another strike from Murphy.

 

NEF lossy compression should not create what Mark is reporting here, in the presence of noise and sufficient stacking, NEF "quantization" should be totally filled in.  Marks bands are different in a way I do not fully understand.

 

This is not theory, and I am an experimentalist.  Have lots of cameras of all types, had successes along the way with 10X the number of fails.  Have seen banding issues from any number of cameras, including uncompressed raws.  They can come from any number of sources, too many to detail here.

 

Glenn

So get a Nikon and experiment.  <grin>

 

What is this banding thing you speak of?  With the D5500 I'm unfamiliar with it.  <_big_ grin>

 

More seriously Marks rings (and the ones I can get with unrealistic methods) are not limited to the center of the field.  And to say these issues "do not happen all the time"  with D5300/D5500s is a true masterpiece of understatement.  "Rare" would still be an understatement.  Vast.

 

The D5XXXs are all APS-C, maybe that's why this is not a problem.  Maybe.

 

"Filling in" is what I see when my flats are stacked and bias subtracted.  Another possible reason why I don't see the problem.  My theoretical explanation above was suggesting it might be buried in the noise, but that's just speculation. 


Edited by bobzeq25, 29 November 2016 - 01:33 PM.


#43 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 02:22 PM

Nikons won't suffer this concentric ring issue because (as far as I am aware) they do not have the widely spaced histogram notches/aps that afflict the Sony A7S.  Yes, Nikons do have histogram gaps (or spikes in the case of the D5300) in the Red/Blue channels but they are too narrowly spaced to cause the rings.

 

It's a clear night here tonight so I hope to shoot some genuine astro-data with rings at ISO 100.  If I succeed then it should support my histogram gap theory.

 

Mark


Edited by sharkmelley, 29 November 2016 - 02:23 PM.

  • bobzeq25 likes this

#44 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 29 November 2016 - 03:13 PM

Bob wrote:

And to say these issues "do not happen all the time"  with D5300/D5500s is a true masterpiece of understatement.  "Rare" would still be an understatement.  Vast.

At least I am a master of something! :flowerred:

 

Agree the NEF encoding is not as prone to these issues because its table lookup appears continuous.

 

The Sony A7S table lookup is not as continuous, rather it steps from one ADU "skipping zone" to another.  It is this stepwise response which will make these fringes more likely to happen, I think.  In this case, if a certain number of ifs are satisfied, the artifact may be the expected result.  Now how frequently it does occur I have no way of knowing, and lack the knowledge to analyze it further.

 

Sure I would love to have Marks scope and camera to experiment.  I don't have the time and too many other things to work on already.  I have my own problems!  Got a Nikon already with no compression...

 

Mark,
I was wondering how much if any help would dithering the lights be?  Normally they are good for filtering out the last vestiges of pattern noise at the < 1% level not corrected by calibration.  What you report is different, and at the center of the field the falloff is very slight, increasing as you go out. Its not clear to me how much dithering would be required, how many pixels, if it were to help at all.

 

Glenn



#45 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 03:28 PM

Here we are - a teaser for you ;)

 

This is a 120sec exposure at ISO 100.  Sony A7S on Tak Epsilon at F/2.8  Just the usual calibration and stretching applied.

 

A7S_StarsAndRings.jpg

 

Mark


Edited by sharkmelley, 29 November 2016 - 03:29 PM.

  • bobzeq25 likes this

#46 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,348
  • Joined: 27 Oct 2014

Posted 29 November 2016 - 03:37 PM

Here we are - a teaser for you ;)

 

This is a 120sec exposure at ISO 100.  Sony A7S on Tak Epsilon at F/2.8  Just the usual calibration and stretching applied.

 

attachicon.gifA7S_StarsAndRings.jpg

 

Mark

I note in passing that here, the effect seems to decrease farther out in the field.

 

When I get a chance I'll tr this with my F2 16mm camera lens, which should have some vignetting wide open.



#47 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 04:36 PM

To show what happens as the exposure changes (at the same ISO 100):

 

30sec:

A7S_StarsAndRings_30sec.jpg

 

 

60sec:

A7S_StarsAndRings_60sec.jpg

 

 

120sec:

A7S_StarsAndRings.jpg

 

 

240sec - sorry about the aircraft!

A7S_StarsAndRings_240sec.jpg

 

Mark


Edited by sharkmelley, 29 November 2016 - 04:40 PM.


#48 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 29 November 2016 - 04:47 PM

Mark,
Please tell us you cranked up the saturation!

 

Can you determine if most of this is coming from the flat?  Or is it a compounding of everything?  Also probably not important, its odd that the pattern is not perfectly circular but given the anomaly maybe not surprising.  Perhaps the ellipticity is due to a gradient in the sky.

 

This is one of the weirder things I've seen, and I know you don't like to hear that!

 

Most real optics seldom follow simple radial falloff (like cos^4) but in general the slopes are greater as you go out.  I think as you do go out and this slope increases the "fringes" merge making the whole mess less visible.  Not sure.  When you add in the background level dependence its becomes almost nightmarish.

 

Now I have a headache considering these results and how to mitigate them!

 

Glenn

 



#49 sharkmelley

sharkmelley

    Soyuz

  • *****
  • topic starter
  • Posts: 3,725
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 29 November 2016 - 05:12 PM

Mark,
Please tell us you cranked up the saturation!

 

Can you determine if most of this is coming from the flat?  Or is it a compounding of everything?  Also probably not important, its odd that the pattern is not perfectly circular but given the anomaly maybe not surprising.  Perhaps the ellipticity is due to a gradient in the sky.

 

This is one of the weirder things I've seen, and I know you don't like to hear that!

 

Most real optics seldom follow simple radial falloff (like cos^4) but in general the slopes are greater as you go out.  I think as you do go out and this slope increases the "fringes" merge making the whole mess less visible.  Not sure.  When you add in the background level dependence its becomes almost nightmarish.

 

Now I have a headache considering these results and how to mitigate them!

 

Glenn

No, I didn't even touch the saturation, I didn't need to!

 

In this particular case, none of the effect is coming from the flat - I took great care creating that flat.  A carefully created synthetic flat would show the rings just as well.  Those concentric rings are actually "ripples" in the data in the light frames themselves.  I'm now convinced I understand what's going on and I'm creating an example to explain it.

 

Mark


  • bsavoie likes this

#50 DesertRat

DesertRat

    Fly Me to the Moon

  • *****
  • Posts: 6,266
  • Joined: 18 Jun 2006
  • Loc: Valley of the Sun

Posted 29 November 2016 - 05:16 PM

Egads!

 

I admire your tenacity. :waytogo: :waytogo:

 

If it were anyone else I would recommend the obvious easy solution.  But it requires additional funds. :(

 

Glenn




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics