107 replies to this topic

[sharkmelley]

Up until now I've always said that the red-channel concentric rings that are obvious at ISO 100 on the Canon EOS R do not appear at ISO 1600.  This was mentioned in my review of the Canon EOS R. Unfortunately, I was wrong.

 

This example comes from a recent 6 hour imaging session of IC348 with the EOS R on my Tak Epsilon 180ED with a SQM reading of 20.8:

 

 

IC348 is very faint, so it required a lot of stretching.  I've also increased colour saturation to a high level to emphasize the radial colour gradient and the sharply defined circular discontinuity in the outer parts of the image.

 

The position of the circular discontinuity coincides precisely with a very distinct notch that appears in the red channel of the histogram:

 

 

I've noticed these notches on previous occasions but they never proved to be a big issue - until now.     Clearly, some further investigation is required 

 

Mark

[Etendue645]

Thanks for the report. Seems like a lot of newer mirrorless cameras are showing these gradients like the FujiFilm GFX cameras. 

[ChristopherBeere]

Hey Sharky, i assume this is post calibration ?

 

Is it possible that the flats have affected it ?

 

How does it look stacked but uncalibrated ?

[sharkmelley]

Hey Sharky, i assume this is post calibration ?

 

Is it possible that the flats have affected it ?

 

How does it look stacked but uncalibrated ?

Yes, the image in the original post is post-calibration. Certainly the master flat has affected the result but it's difficult to separate the contribution of the lights to the ring from the contribution of the master flat.

 

Stacked but uncalibrated doesn't tell us much because of radial colour gradients (colour shading) caused by a combination of non-linearity in the sensor response and pixel crosstalk (i.e. the effect that some photons passing through a green filter end up in a red or blue pixel and vice versa).  For this reason my flats are very carefully created so they match the lights both in colour and brightness.  This simultaneously corrects for both effects.

 

Sharply defined rings also appear if I divide one master flat (average of 50 flats at ISO 1600) by another, where the two master flats were created at different brightnesses.  Again the position of rings coincides with the notch(es) in the red channel histogram.

 

With a bit more analysis and experimentation I might find a workaround.  Otherwise this is the end of the road for me struggling against DSLR/Mirrorless camera manufacturers cooking our raw data.  Next purchase will then be a dedicated scientific camera.

 

Mark

[ChristopherBeere]

For this reason my flats are very carefully created so they match the lights both in colour and brightness.  This simultaneously corrects for both effects.

Question, i can understand controlling brightness through exposure (i shoot my flats to the same exposure level as my lights using the same ISO - targeting the same 20-25% histogram origin x-axis) but how do i control colour in flats to match the lights ?

Edited by ChristopherBeere, 19 October 2023 - 02:16 PM.

[sharkmelley]

Question, i can understand controlling brightness through exposure (i shoot my flats to the same exposure level as my lights using the same ISO - targeting the same 20-25% histogram origin x-axis) but how do i control colour in flats to match the lights ?

To match colour, use a computer controlled display and adjust the R,G,B levels until they match your lights.  It's trial and error but the end result is incredibly effective (unless there are rings in the raw data!)

[DantheSpaceMan]

Yes, the image in the original post is post-calibration. Certainly the master flat has affected the result but it's difficult to separate the contribution of the lights to the ring from the contribution of the master flat.

 

Stacked but uncalibrated doesn't tell us much because of radial colour gradients (colour shading) caused by a combination of non-linearity in the sensor response and pixel crosstalk (i.e. the effect that some photons passing through a green filter end up in a red or blue pixel and vice versa).  For this reason my flats are very carefully created so they match the lights both in colour and brightness.  This simultaneously corrects for both effects.

 

Sharply defined rings also appear if I divide one master flat (average of 50 flats at ISO 1600) by another, where the two master flats were created at different brightnesses.  Again the position of rings coincides with the notch(es) in the red channel histogram.

 

With a bit more analysis and experimentation I might find a workaround.  Otherwise this is the end of the road for me struggling against DSLR/Mirrorless camera manufacturers cooking our raw data.  Next purchase will then be a dedicated scientific camera.

 

Mark

Perhaps this might be a long shot, but any chance using Rawtherapee with a master flat file and Limmse demosaic might come out with a different result? In other words, is there a chance a different demosiacing algorithem is able to avoid or address this issue, or would this not work? 

[sharkmelley]

Perhaps this might be a long shot, but any chance using Rawtherapee with a master flat file and Limmse demosaic might come out with a different result? In other words, is there a chance a different demosiacing algorithem is able to avoid or address this issue, or would this not work? 

Unfortunately, all raw convertors and demosaicing algorithms will have the same problem because the concentric rings are "baked into" the raw data by the camera's raw-data processing.  You can read more in my article on diagnosing baked-in concentric rings.  The only real solution is to reverse engineer the camera's raw-data processing and then attempt to "undo" the effects in the raw data itself but it's rarely possible to do either.  On the other hand, it did work successfully with the problem of Nikon's concentric rings caused by badly implemented lossy compression.

 

I have a hunch on one aspect of what may be going on with the Canon EOS R raw-data processing but I need to run a few more experiments to confirm.  In any case, I don't know whether this would lead to a successful workaround.

[Alen K]

I have been watching this “Lord of the Concentric Rings” saga with interest and some dismay. How is it that this evidently serious problem or more accurately group of similar-appearing problems stemming from multiple causes, which seems to be affecting numerous camera models from at least four manufacturers (others remain largely untested), is only recently coming to light? Is it a case of images being stretched more now in our quest to reveal ever fainter structures or something else?

Edited by Alen K, 20 October 2023 - 09:06 AM.

[ErwinL]

Sharply defined rings also appear if I divide one master flat (average of 50 flats at ISO 1600) by another, where the two master flats were created at different brightnesses.  Again the position of rings coincides with the notch(es) in the red channel histogram.

Would you say that the problem in processing the raw data is due to the red intensity distribution alone? That the appearance as a ring is only caused by a radial drop in intensity?

If this is the case, it could explain the different dependence on the ISO setting: In a dark sky, even at high ISO, the red component shifts down into the intensity range where a notch appears in the histogram.
Conversely, additional red light would shift the signal into an uncritical area of the histogram.

 

From your plots it appears to me that the notch in the red only includes one ADU value. Could it be that it is caused by a simple rounding problem?

[sharkmelley]

Would you say that the problem in processing the raw data is due to the red intensity distribution alone? That the appearance as a ring is only caused by a radial drop in intensity?

If this is the case, it could explain the different dependence on the ISO setting: In a dark sky, even at high ISO, the red component shifts down into the intensity range where a notch appears in the histogram.
Conversely, additional red light would shift the signal into an uncritical area of the histogram.

 

From your plots it appears to me that the notch in the red only includes one ADU value. Could it be that it is caused by a simple rounding problem?

Yes, this particular problem exists only in the red channel and it manifests as a radial step-change in red intensity.

 

A longer exposure simply pushes the histogram further to the right, where it encounters the next notch.  Experimentation has shown me that these notches are spaced fairly regularly (approx 310 apart) and the hypothesis I'm currently working on is that this might be caused by some kind of red channel data-scaling.  This kind of raw-data scaling was the cause of Sony A7S concentric rings.  

 

If the hypothesis is correct then it might be possible to rescale the raw data in a way that removes the rings.

[ErwinL]

If the hypothesis is correct then it might be possible to rescale the raw data in a way that removes the rings.

Hm, some kind of histogram equalisation, limited to small parts of the histogram perhaps?

 

Another approach I could think of is "dithering" the exposure time by a few percent to blur the notches.

I use an Arduino microcontroller on a cable remote with a homemade program to trigger my cameras, for which no Magic Lantern is available. It would be easy to adapt the code to vary the exposure. Let me know if you want to try something like this.
 

[sharkmelley]

Success!!  

 

Here's the before and after (both with the same crazy colour saturation applied):

 

 

For anyone interested, here's the PixInsight PixelMath expression I applied to the raw lights and raw flats before calibration:

• iif(x()%2==0 && y()%2==0,$T-floor(($T*65535-2048-0.5)/310)/65565,$T)

The "x()%2==0 && y()%2==0" sub-expression limits the effect to the red pixels and the effect of "$T-floor(($T*65535-2048-0.5)/310)/65565" is to shift the red-pixel histogram left by various amounts to undo the assumed scaling the camera has applied to the data.

 

The major caveat is that this works for my own Canon EOS R at ISO 1600.  It won't work at other ISOs and it may not work for other people's cameras because we don't know if the amount of red channel scaling is calibrated at the factory.

 

Mark

[ChristopherBeere]

Success!!  

 

Here's the before and after (both with the same crazy colour saturation applied):

 

CanonEOSR_ISO1600_IC348_Rings.jpg CanonEOSR_ISO1600_IC348_Rings_Solved.jpg

 

For anyone interested, here's the PixInsight PixelMath expression I applied to the raw lights and raw flats before calibration:

• iif(x()%2==0 && y()%2==0,$T-floor(($T*65535-2048-0.5)/310)/65565,$T)

The "x()%2==0 && y()%2==0" sub-expression limits the effect to the red pixels and the effect of "$T-floor(($T*65535-2048-0.5)/310)/65565" is to shift the red-pixel histogram left by various amounts to undo the assumed scaling the camera has applied to the data.

 

The major caveat is that this works for my own Canon EOS R at ISO 1600.  It won't work at other ISOs and it may not work for other people's cameras because we don't know if the amount of red channel scaling is calibrated at the factory.

 

Mark

Strong game Sherlock, it looks like you have cracked it 

[Rlakjdlsj]

Doing a lot of normal daylight landscape photography,  and making mistakes, I've seen those gaps in histograms when trying to stretch 'normal daylight' photographs with 'regular DSLRs'.  Stretching data too much to bring up detail in underexposed shadows will create those gaps, because there is not enough 'data.'   Increasing the ISO only reduced the dynamic range  (bit depths) and makes it worse.  I'm guessing your pixel math is attempting to fill in the gaps.  i.e.,  you can only push a camera designed for sunlit daylight photography so far.  

 

Its the old slinky analogy in this  Photoshop example fig. 4.15

 

https://www.adobepre...157747&seqNum=3

 

To paraphrase your English bard, the fault lies not in your camera but in your trying to use it to collect near zero photons of nebula   If thats what you want to do, your conclusion that your next camera will be a dedicated astro-cam is probably correct,

Edited by Ron359, 21 October 2023 - 12:01 AM.

[sharkmelley]

Hi Ron359, you bring up some interesting points. 

 

Firstly, stretching the data too much is not the issue here otherwise it would also affect images from dedicated astro-cams.

 

Secondly, I disagree with the statement that you can only push a camera designed for sunlit photography so far.  The sensors in consumer cameras are essentially the same (and in many cases identical) as the sensors in dedicated astro-cams.  Both are equally capable of producing artefact-free images of very faint nebulae (collecting near zero photons in each sub-exposure).   As you are aware, Canon cameras have been used for years to do this without any problem.

 

The problem with the EOS R (and the EOS Ra) is that the raw data are being adversely affected by in-camera processing.  In this particular case, integer data (in the red channel) are being scaled by a multiplier very close to unity i.e. in the range 0.997-1.013  This leads to regular gaps (i.e. missing values) or spikes in the histogram and the problem with this is that it also introduces discrete intensity steps into the image.  These steps are disguised by the read noise until many exposures are stacked together, which reduces the noise and reveals the steps.  In the case of the EOS R/Ra, there is also some other (unidentified) processing going on which "spreads out" the histogram gap or spike into a notch or blip but my core argument remains true.

 

To the best of my knowledge, this scaling of data is a fairly recent development in Canon cameras.  Nikon cameras have been doing raw channel scaling it for years but they never make the mistake of choosing multipliers so close to unity, so they are unaffected by this particular problem.  Of course, Nikon cameras have different raw-processing issues.  

 

The only reason (apart from sensor cooling) that it's becoming increasingly necessary to recommend dedicated astro-cams is to escape the effects of badly thought-out in-camera raw-data processing on the imaging of faint deep-sky objects. 

 

Mark 

[Rlakjdlsj]

Hi Ron359, you bring up some interesting points. 

 

Firstly, stretching the data too much is not the issue here otherwise it would also affect images from dedicated astro-cams.

 

Secondly, I disagree with the statement that you can only push a camera designed for sunlit photography so far.  The sensors in consumer cameras are essentially the same (and in many cases identical) as the sensors in dedicated astro-cams.  Both are equally capable of producing artefact-free images of very faint nebulae (collecting near zero photons in each sub-exposure).   As you are aware, Canon cameras have been used for years to do this without any problem.

 

The problem with the EOS R (and the EOS Ra) is that the raw data are being adversely affected by in-camera processing.  In this particular case, integer data (in the red channel) are being scaled by a multiplier very close to unity i.e. in the range 0.997-1.013  This leads to regular gaps (i.e. missing values) or spikes in the histogram and the problem with this is that it also introduces discrete intensity steps into the image.  These steps are disguised by the read noise until many exposures are stacked together, which reduces the noise and reveals the steps.  In the case of the EOS R/Ra, there is also some other (unidentified) processing going on which "spreads out" the histogram gap or spike into a notch or blip but my core argument remains true.

 

To the best of my knowledge, this scaling of data is a fairly recent development in Canon cameras.  Nikon cameras have been doing raw channel scaling it for years but they never make the mistake of choosing multipliers so close to unity, so they are unaffected by this particular problem.  Of course, Nikon cameras have different raw-processing issues.  

 

The only reason (apart from sensor cooling) that it's becoming increasingly necessary to recommend dedicated astro-cams is to escape the effects of badly thought-out in-camera raw-data processing on the imaging of faint deep-sky objects. 

 

Mark 

You are thinking  of the problem backwards.  Going back to fundamentals of ccd's clarifies the problems you experience with these cameras.  The reason there is "in-camera processing"in a 'prosumer camera,  is because its primary purpose is not pro-level astro-photography of extremely faint nebula, with very weak or few photons signals.  Your weak signal processing only enhances the 'lack of signal' the system has to work with.  And you're correct you may have the same problem with a dedicated astro-camera because you are expecting to buy a unicorn off the shelf and have it work as well as a $10-15K or more astro imager.  

 

Besides getting a dedicated astro-cam you have other variables to try.  You either have to collect more signal, with bigger aperture, and/or a lot more time to acquire more signal.   Your small Epsilon Tak aperture also has a large central obstruction which is likely creating the big donut when your weak signal is stretched to the extreme levels with flats.   Have you tried using the R8 or others you denigrate on a refractor, collecting many hours and see if you get the same 'donut ring'?   Besides giving up on cmos DSLRs,  you could  put it on a much larger aperture scope and collect many times more photons and see if the same problems occur.  Or just doing the math on how much more signal more aperture and/or time would give you could be done.  IMO, you've only shown us (again) that extreme stretching of extremely faint signal from a small aperture scope has found the limits of a consumer telescope and camera, proving the obvious, that its not even a close rival to a  pro-observatory or far more expensive pro-cmos or ccd camera.  

 

Wishing you clear and darker skies for your further endeavors in small scope imaging.  ; )  

 

--p.s...  If what you say about the gaps in histograms caused by newly added 'modern' in-camera processing, then I wouldn't have seen them in my photos with my old (but still using) Canon 6D or 7D 'daytime' images of dark shadows or sure wouldn't have been described in that decades old Adobe PS description using the "slinky toy" analogy.  

Edited by Ron359, 21 October 2023 - 01:47 PM.

[sharkmelley]

IMO, you've only shown us (again) that extreme stretching of extremely faint signal from a small aperture scope has found the limits of a consumer telescope and camera, proving the obvious, that its not even a close rival to a  pro-observatory or far more expensive pro-cmos or ccd camera.  

What you say is partially true.  Yes it's true that a combination of stacking, background subtraction and stretching does hit the limits of (some) consumer cameras. 

 

However, we need to understand the reason(s) for those limits i.e. why do some consumer cameras show artefacts such as coloured concentric rings whereas dedicated CMOS astro-cameras (often using the same sensor) do not?  It's not a question of "proving the obvious" because it's far from obvious why two cameras using the same sensor should behave so differently.  Instead, the answer is found in various types of undocumented in-camera processing of the raw data which doesn't exist in dedicated astro-cameras.

 

Examples of such in-camera processing are:

• Digital scaling, for instance on the first generation Sony A7/A7S/A7R cameras and unfortunately now the Canon EOS R/Ra and Canon EOS R8
• Badly implemented lossy compression such as Nikon's
• Image corrections (possibly intended as colour-shading corrections) found on Sony and some Nikon cameras
• Yet-to-be-understood raw-processing in many FujiFilm cameras

In my opinion, it's a worthy cause to identify and make-known those weaknesses to the wider community because it will help prevent fellow astro-photographers wasting their hard-earned cash on cameras that ultimately will not behave in the manner they deserve to expect.

 

Mark

[Rlakjdlsj]

Mark,  

 

I was curious about the nebula IC348 you imaged in the OP.  Looking it up on sky Safari, its described as a " bright nebula mag. 7.3 only "10' x 10' "across.  Certainly not an extended faint nebula of questionable magnitude. An not your description from #1:

 

 "IC348 is very faint, so it required a lot of stretching.  I've also increased colour saturation to a high level to emphasize the radial colour gradient and the sharply defined circular discontinuity in the outer parts of the image."

 

So did a search for other images, and the very first image google came up with, was this one at:

 

https://www.hansonas...m/ic-348ic-1985

 

  His description is VERY interesting in comparison:

"IC 348 (also known as IC 1985) is a rarely imaged open star cluster surrounded by a reflection nebula. The reason this object is not imaged is due to the very bright star glare washing most of the detail away. I spent countless hours slowly expanding this image to fight of the glare and reveal the wonderful image you see here."

 

As you can see there is a huge circular glare extending into the nebula from the bright star (Atik mag. 3.48) just out of the top of your image. Otherwise the orientation and star positions are similar to yours.  The scope used is much larger in aperture but is a RC  Planewave, so I presume, also has a relatively large secondary in proportion to your Tak which are known to created bit circular features from glare.  I've seen it often visually or imaging when I used a one of the fast f/6.3 SCT's.   

 

 Can it be such a odd unexpected coincidence that that the circular feature you get in your extremely stretched image is that very same glare from the star just out of your image field?  

Edited by Ron359, 21 October 2023 - 10:51 PM.

[sharkmelley]

Sure, IC348 itself is bright.  What I meant (and what I should have said) is that the surrounding areas of nebulosity and molecular clouds are very faint and require a lot of data stretching.  But this should have been clear enough from looking at the area covered by my image!  

 

Regarding the possibility of "circular glare", yes it would be a very odd and unexpected coincidence if it were produced by a star outside the image field.  The large circular feature that appears in my OP (original post) is an artefact that appears only in the red channel.  It would be very difficult to explain why the green and blue channels are unaffected by an optical phenomenon such as this.  In fact, the difference in responses between the colour channels forms part of the diagnostic technique I describe in my on-line article on diagnosing baked-in concentric rings

 

The position of the circular artefact in the OP image and the fact that it affects only the red channel is fully explained by what has been deduced about the EOS R in-camera raw-data processing i.e. the red channel scaling.  Furthermore, using code to "undo" this scaling (as far as possible) in the raw data of each light and flat removes the circular artefact, providing further supporting evidence that the cause is correctly identified.

 

For anyone interested in further technical discussions of digital channel scaling and how it leads to histogram gaps and consequent concentric rings then you can read my explanation of the issue in the context of the Sony A7S here:

• DPReview - Explanation of the sawtooth response
• DPReview - Contour banding discussion

In brief, the channel scaling creates the undesirable consequence whereby the mathematical expectation of (scaled) pixel values in various parts of the image no longer has a linear relationship with the amount of light falling on the sensor but the relationship actually has discontinuities.  This scaling is also the cause of ripples in what is known technically as the photon transfer curve:

• DPReview - Ripples in the photon transer curve

These ripples actually form another useful diagnostic technique.

 

Mark

Edited by sharkmelley, 22 October 2023 - 06:21 AM.

[Rlakjdlsj]

Perhaps you could re-post your previous review from a couple years ago (?),  of the Canon R. In which I seem to remember you gave it a 'rave' (for you) review and said it would be your camera of choice or something like that.  Its too difficult for me to use the CNs search.  Then folks can read it and get both sides and decide which camera review is better.    

Edited by Ron359, 22 October 2023 - 10:24 AM.

[sharkmelley]

Perhaps you could re-post your previous review from a couple years ago (?),  of the Canon R. In which I seem to remember you gave it a 'rave' (for you) review and said it would be your camera of choice or something like that.  Its too difficult for me to use the CNs search.  Then folks can read it and get both sides and decide which camera review is better.    

I provided the link to my EOS R review in the 2nd sentence of the original post! 

 

The relevant part of the review relating to histogram gaps, channel scaling and concentric rings is here:

https://www.cloudyni...ing/?p=11355453

 

At the time, I thought ISO 1600 was safe but unfortunately it turns out I was wrong.  However, it's still true that you're far less likely to encounter the problem at ISO 1600 than at ISO 100.  

[Rlakjdlsj]

I provided the link to my EOS R review in the 2nd sentence of the original post! 

 

The relevant part of the review relating to histogram gaps, channel scaling and concentric rings is here:

https://www.cloudyni...ing/?p=11355453

 

At the time, I thought ISO 1600 was safe but unfortunately it turns out I was wrong.  However, it's still true that you're far less likely to encounter the problem at ISO 1600 than at ISO 100.  

Thanks, I had overlooked the link as I was 'focused' on the problem & image of the OP.

 

 One thing I found is that your analysis of R and Ra and other model claims of firmware induced rings, about these cameras, have been beat to death in previous threads. Regarding the Ra. I've said this before, that likely lead to, or contributed to it being discontinued by Canon.  And then as now, it appears others cannot find the same problems in their images, particularly when not taken near bright stars, with other optical trains, and more uniform 'dark nebula' in exposures of hours when the problem should show up with extreme stretching.  Those include your own images in your review of the R link you provided.  I give you credit for the followup post #17 to this one (link below), admitting, you don't know whats going on either.

 

 If there is good news,  since astro is a tiny portion of the digital camera market, your  reviews of the R and other non-astro intended pro or consumer cameras,  as an 'internet influencer' should not dissuade the vast majority of digital photographers, even astro photographers, from considering them or buying them.   So far less likely to be discontinued for the inconsistent analysis and reviews you give them.  

 

https://www.cloudyni...ing/?p=10730912   post  #17 & #21

Edited by Ron359, 22 October 2023 - 03:18 PM.

[sharkmelley]

I give you credit for the followup post #17 to this one (link below), admitting, you don't know whats going on either.

True.  Back then I didn't understand the cause.  Now I do.  I call that progress!

[Rlakjdlsj]

True.  Back then I didn't understand the cause.  Now I do.  I call that progress!

 

 

Apparently not.

You don't explain why others fail to find the problem in their images or even in yours that I pointed out in my previous post.

 

Basic H.S.  science fair testing: Have a control and control the variables.    In more than 30 yrs of doing engineering and science data analysis, I learned early on, that bad data are worse than no data.  It always leads to the wrong conclusions.

Edited by Ron359, 22 October 2023 - 05:11 PM.