107 replies to this topic

[sharkmelley]

You don't explain why others fail to find the problem in their images or even in yours I pointed out in my previous post.

The reason for the "non-appearance" of rings in stacked images are very interesting but quite technical. I'll attempt to explain.  The rings caused by channel scaling (like we see with the EOS R) have a couple of important and interesting properties:

• Firstly, the step size of the (circular) discontinuity in pixel values is exactly 1 ADU (analogue to digital unit).

• Secondly, the position of the (circular) discontinuity in a single exposure coincides with where the pixel values match the spike or notch in the histogram.  But this only happens where the image has some kind of background gradient because otherwise there are no well defined areas in the image where these pixel values occur. Optical light fall-off (vignetting) is the most common cause of such background gradients and this results in a circle where the pixel values fulfill that criterion, leading to circular discontinuity.  Different background gradients will produce non-circular discontinuities.  The best way to visualize this is to consider that the discontinuity will follow contour lines in the background of the image.

 

Once the above 2 points are understood, there are many interesting corollaries:

• If the image noise is much larger than 1 ADU then the discontinuity will be disguised by the noise.  Moreover, instead of being a sharply defined 1 ADU step-like discontinuity, it will become more spread out.  This is part of the reason it's difficult to detect at high ISOs since the noise (in ADUs) is greater.
• Averaging many exposures together (i.e. stacking) will reduce the background noise and make the 1 ADU discontinuity easier to detect.
• If the background gradient is gentle then again the step is not sharply defined and becomes more spread out. In general, optical systems with sharp light fall-off will be more prone to this type of ring.  Optics with little or no vignetting will not show these rings.  However, note that in other cameras, concentric rings can be caused by raw data-processing issues where vignetting plays no role.
• Since the position of the ring is dependent on the background pixel values, it means the ring position will change with the recorded brightness.  Averaging together many such exposures will "dither" the ring position, spreading it out and making it difficult (or impossible) to detect.

So, as a vast over-simplification, rings generally will not be visible in a stacked image when shooting at high ISO, with non-vignetting optics and sky conditions that are changing.  Rings will be visible in a stacked image when shooting at low ISO with heavily vignetted optics where sky conditions are constant.  But a sufficient number of exposures will still need to be stacked, depending on the ISO being used for shooting. Between these two extremes, the rings may or not be apparent.

 

In my own case, I'm shooting with the Tak Epsilon which suffers from heavy vignetting.  But the brightness of my sky typically varies quite a lot during an imaging session.  Moreover, my flats are typically sky flats taken at dusk when the light levels are changing quite rapidly.  So the varying sky conditions usually give me some protection from sharply defined rings (though not from the background gradients which are caused by the red channel scaling).  The thing I did differently on this occasion was to take flats using a constant artificial light source.   So the ring you see in my image was caused by the flats.

 

All this is only a brief summary.

[Rlakjdlsj]

Thanks.  Given the limitations of your testing 'a limited range of black-boxes,' that lays out the many extreme special conditions that must be met for these consumer cameras to exhibit "the problem" you seem to find.

 

 Folks can take those conditions into account in decisions to purchase, or if it happens to show up in their image processing.  To this simple 'astro-nut 'it seems there are already relatively easy, 'standard work arounds' to prevent this showing up.  And others have already mentioned those.  

 

  It will be interesting to read your findings and comments when you start using and no doubt testing,  a dedicated, cooled, astro camera and narrow band imaging.  Until then.  Clear, dark skies.  

Edited by Ron359, 23 October 2023 - 11:26 AM.

[sharkmelley]

In the light of what's been discovered in this thread I've updated the information on the Canon EOS R/Ra in my online camera summary. It now mentions that this is a type of raw-data processing that can cause image rings when using vignetting optics.  To be fair though, since these cameras have full-frame sensors, almost all lenses and telescopes will have vignetting!

Edited by sharkmelley, 23 October 2023 - 06:37 AM.

[alan.dang]

I wonder if there is any sort of thermal scaling or non linearity with short or long subs that gets kicked in.

 

Weather has not allowed me to shoot the Ra in about a year, but I am pretty aggressive with my stretches and haven’t had the banding.  In your old notes, you mentioned that Pierre’s camera didn’t have it with ISO800.

 

Since you, sharkmelly, have your modified R, can you see if there is a difference with 30s vs 1 minute vs 2 minutes?  As I noted, I don’t recall seeing this with my own Ra, but thinking about it, I run much shorter subs that many due to my light pollution.

[Rlakjdlsj]

In the light of what's been discovered in this thread I've updated the information on the Canon EOS R/Ra in my online camera summary. It now mentions that this is a type of raw-data processing that can cause image rings when using vignetting optics.  To be fair though, since these cameras have full-frame sensors, almost all lenses and telescopes will have vignetting!

I point out that there is likely a large difference in the 'normal' R and Ra  models re any 'in-camera' firmware 'processing' of raw files.  The Ra must - obviously- have different in-camera processing because it was specifically designed and marketed to shift the white or color balance away from the deep red of Ha sensitivity, which made it useful for astro-images.  

 

The Ra was  able to shoot a 'normal' daylight white balance with no changes to the settings or major post-processing.   'Purists' would say it didn't work 100% but it was very close.  So 'something' must have been done in-camera firmware to be able to do both daylight white balance and also with increased Ha sensitivity for 'enhanced' astro imaging.  

 

So any comparison of the 2 models is like comparing 2 diff species of apples, one of which is different internally and produces  deep red color, extra sweet,' while  the other is just a 'normal' plain ol' apple.  

 

And yeah, even L  EF lenses for FF cameras are vignetting to a very minor extent, easily handled in post-processing or with flats if cropping a tiny amount still really bothers you.  Certainly shouldn't get a neg. check mark even for high end optics.  And since vignetting is created by  any lens or scope optics, vignetting is going to occur no matter the sensor, like any cooled, dedicated FF astro-camera any "serious" 'amateur' is going to afford. 

Edited by Ron359, 16 November 2023 - 12:17 PM.

[alan.dang]

Yes.  I have both the 60Da and Eos Ra and both do normal photos reasonably.

 

https://www.cloudyni...-ra/?p=10239435
 

Importantly, you cannot simply “click white balance” the raw files and get the same results with aftermarket modifications to a stock camera.

 

I missed the part where this might only be a problem with vignetting optics.  In that case, I haven’t seen the problem with my Ra since I use scopes that are capable of covering the KAF16800.  So it’s really about mismatched optics to the Ra (which is designed with the RF400 and RF85 in mind, among others.)

[piaras]

https://www.cloudyni...-m8-and-friend/

 

This was taken by accident at ISO 25XXX! Forgot to reset to lower ISO after frame and focus. 68 subs X 2 minutes Canon Ra. No LP filter. I was not looking for banding or rings in the individual subs. 
 

Do you want me to put up a sub?

Pierre

[sharkmelley]

I wonder if there is any sort of thermal scaling or non linearity with short or long subs that gets kicked in.

 

Weather has not allowed me to shoot the Ra in about a year, but I am pretty aggressive with my stretches and haven’t had the banding.  In your old notes, you mentioned that Pierre’s camera didn’t have it with ISO800.

 

Since you, sharkmelly, have your modified R, can you see if there is a difference with 30s vs 1 minute vs 2 minutes?  As I noted, I don’t recall seeing this with my own Ra, but thinking about it, I run much shorter subs that many due to my light pollution.

Yes, here's where I said I couldn't find any hint of rings on Pierre's EOS Ra at ISO 800:

https://www.cloudyni...ing/?p=10732073

 

I've just examined those ISO 800 raw frames again and they are free of the troublesome "blips" and "notches" in the histogram, though of course they existed at ISO 100.

 

When I bought my EOS R, I therefore already knew about the potential for rings but I hoped that it would be free from rings at ISO 800 and above. Unfortunately, I was mistaken. We now see the same ring behaviour with the EOS R8, even at high ISOs:

https://www.cloudyni...2#entry12942043

 

I have tested my EOS R at different length exposures and see no change in the channel scaling that is causing the problem.  The possibility you mentioned of different scaling at different temperatures is an interesting possibility but I haven't seen any evidence of different scaling in the range of ambient temperatures 0C-20C.

 

Another possibility exists, which is that individual cameras have different scaling parameters built-in, possibly as a factory calibration.  I've seen this with the hardcoded correction on the Nikon Z6, which differs from camera to camera.  However, we would need examples from a few cameras to begin to confirm/refute this idea.

 

I certainly wouldn't want to assert that the Pixelmath formula in post #13 that works for my own camera at ISO 1600 would work for other cameras because the ISO 1600 histogram notches in other cameras may be differently spaced to my own notches.

 

Mark

[sharkmelley]

Another possibility exists, which is that individual cameras have different scaling parameters built-in, possibly as a factory calibration.

Looking at the raw data I have, there is already some evidence that the scaling might differ between individual copies of the same camera.

 

At ISO 100, the spacing of the red histogram "blips" in Pierre's EOS Ra camera is approx 175:

 

 

The spacing in Nico's EOS Ra camera is approx 140:

 

 

Both are short exposure flat frames with an internal camera temperature of 30C. The scaling in the blue and green channels is also different between the two cameras. From extensive examination or raw files, Ilya Borg (the author of LibRaw and RawDigger) has often claimed that channel scaling is factory calibrated and the above data seems to support that hypothesis .

[ErwinL]

Looking at the raw data I have, there is already some evidence that the scaling might differ between individual copies of the same camera.

This scaling is probably achieved by tuning amplifier gains, as described in this Magic Lantern forum topic.

[tfluthy]

While I am not at the level that this conversation is progressing at, I would like to interject that I'm getting some very similar artifacts when taking exposures and processing in Siril.  However, I'm not using a R, or Ra camera, I have a relatively old Canon 6D with the newest firmware on it.

 

Is there evidence that older Canons have similar "baked in" raw photo processing, similar to the R and Ra cameras?  I specifically purchased a older 6D as it was a highly recommended as a entry level astrophotography camera.

 

The funny thing is that individual exposures look great, its only after stacking that they look terrible.

[Rlakjdlsj]

While I am not at the level that this conversation is progressing at, I would like to interject that I'm getting some very similar artifacts when taking exposures and processing in Siril.  However, I'm not using a R, or Ra camera, I have a relatively old Canon 6D with the newest firmware on it.

 

Is there evidence that older Canons have similar "baked in" raw photo processing, similar to the R and Ra cameras?  I specifically purchased a older 6D as it was a highly recommended as a entry level astrophotography camera.

 

The funny thing is that individual exposures look great, its only after stacking that they look terrible.

In my experience, there are two simple truths re digital imaging.   The 6D produces 'true raw 'un-doctored' raw files.  

 

And "overprocessing"  produces artifacts in all digital images.  This is especially true as it is seen more often in planetary "lucky imaging" which stacks thousands of frames and often heavily 'processed to bring out more and more sub-arc sec detail, creates all sort of arc like 'ringing' features that might look like 'cloud bands' or other features on the planets..

 

 But processing artifacts occur with digital deep sky images, the most common artifact often seen are dark halos around stars when sharpened too much.     Multi-Math-manipulating a huge matrix of  tiny e-signal strength  represented in pixels by shades of gray or tones of color,  is going to have 'consequences.'  

 

And then there are a whole list of subtle and not subtle arc or circular 'artifacts' that can be produced by the optics the camera is attached to,  most of which are 'round' like 'donuts' from any secondary mirror, vignetting from any optical design, refections off the tube,  cells or focuser or shadows and reflections from other anything near the optical path,   bright stars just out of the field, or in the field etc. Internal reflections occur when there are  many optical elements in refractors or camera lenses.   All form arcs or donuts even if very faint and 'invisible' visually, show up when their weak contrast with surrounding 'background' is stretched and 'enhanced' many times.  These artifacts need to be separated out first in testing a camera or software as causing a problem in images.   

Edited by Ron359, 20 November 2023 - 12:00 PM.

[sharkmelley]

In my experience, there are two simple truths re digital imaging.   The 6D produces 'true raw 'un-doctored' raw files. 

Untrue.  Digital scaling takes place in Canon 6D files, causing obvious histogram gaps.

 

While I am not at the level that this conversation is progressing at, I would like to interject that I'm getting some very similar artifacts when taking exposures and processing in Siril.  However, I'm not using a R, or Ra camera, I have a relatively old Canon 6D with the newest firmware on it.

Despite the known digital scaling, I have never seen any evidence of actual image rings from the 6D and I don't expect to find any.  On the other hand, if you upload a raw light and raw flat to a file-sharing site then I'll definitely take a look to see if there's anything obvious.

[Rlakjdlsj]

Untrue.  Digital scaling takes place in Canon 6D files, causing obvious histogram gaps.

 

 

Or as documented many years ago in Adobe PS,  the gaps are produced when 'overstretching' the bit data of the color bands of all digital images.  

 

 I haven't updated the firmware in my 6D for many years (ain't broke so, ain't fixed it), so no idea what Canon has done now.  worth looking into what they have 'added.'   I've some thought of having it modified for Ha since its 'aging out' as a good 'daylight' FF DSLR.  

Edited by Ron359, 20 November 2023 - 12:33 PM.

[tfluthy]

I can make it easier for everyone involved.  Here are two jpgs, one a unstacked photo, and the other after running my flats, biases, lights and darks through siril.

 

If you can let me know which rabbit hole I need to go down, I'm very happy to do so.  

 

Of the two, the larger one is what is produced by siril using the OSC script, or deep sky stacker.  The smaller one is what gets pulled off the camera as a single 120 second exposure.  (Note:  the siril images are pulled off of my Asi air as fits files, while the single exposure is a canon cr2 file)

Attached Thumbnails

• 
• 

[sharkmelley]

Or as documented many years ago in Adobe PS,  the gaps are produced when 'overstretching' the bit data of the color bands of all digital images.

For the avoidance of possible doubt, the histogram gaps in the Canon 6D raw files have nothing whatsoever to do with Adobe. 

[sharkmelley]

I can make it easier for everyone involved.  Here are two jpgs, one a unstacked photo, and the other after running my flats, biases, lights and darks through siril.

It's more than likely that you are seeing an optical issue e.g. extreme vignetting.  To analyse potential artefacts caused by in-camera raw-data processing, the raw CR2 files are necessary.

Edited by sharkmelley, 20 November 2023 - 12:57 PM.

[Rlakjdlsj]

It's more than likely that you are seeing an optical issue e.g. extreme vignetting.  For issues caused by in-camera raw-data processing, the raw CR2 files are necessary.

You could provide more details of your scope & camera setup.  sure looks like an optical train problem of vignetting.  that first one kinda looks like you're using afocal projection from a camera shooting into an eyepiece.   

[tfluthy]

Here is the link to the stacked raw image and example lights, darks, and flats, in fit format.

 

 

https://drive.google...4XO?usp=sharing

Edited by tfluthy, 20 November 2023 - 04:27 PM.

[sharkmelley]

Here is the link to the stacked raw image and example lights, darks, and flats, in fit format.

https://drive.google...4XO?usp=sharing

Thanks.  From those files it's clear that your main problem is that your optics are unable to illuminate the full frame of the Canon 6D.  Also you should check your flats because they are saturating the green channel in the central part of the image.

[tfluthy]

thank you so much... I'll work on that

[gregbradley]

Dedicated astro cameras are now much cheaper than mirrorless cameras. So using mirrorless for Deep space imaging does not have a strong case anymore. Also items like ASAIR wifi units that are mini computers and handle all the imaging steps including polar alignment, camera control and image storage and autoguiding means the portability of mirrorless/DSLR edge has also largely evaporated.

 

So trouble shooting around these cameras makers internal processing due to customer demand for lower noise in terrestial images seems to be an uphill battle.

 

Mirrorless are great for nightscape photography and dedicated astro are still awkward for that application.

 

Greg.

Edited by gregbradley, 24 November 2023 - 06:05 PM.

[Michael Covington]

 

With a bit more analysis and experimentation I might find a workaround.  Otherwise this is the end of the road for me struggling against DSLR/Mirrorless camera manufacturers cooking our raw data.  Next purchase will then be a dedicated scientific camera.

 

Mark

I hear you!  My 11-year-old Canon 60Da is very satisfying to use.  It's an older-technology sensor, but as far as I know, completely free of shenanigans.  Every newer DSLR/mirrorless seems to be weirder than the last.  And prices of dedicated astrocameras with really good sensors have fallen.  And there is now even one with the guide camera built in.  Probably will take the plunge soon...

I say that even though I wrote a book whose continued sales depend on interest in DSLR astrophotography.  And DSLRs are a great way to start if you already have one, or need one for daytime photography, or need the ability to operate without a computer attached (e.g., traveling).  I don't think they are still the most cost-effective way to do astrophotography for its own sake.

Edited by Michael Covington, 24 November 2023 - 06:14 PM.

[piaras]

Astro dedicated cameras have dropped, but what about the filters, filter wheels, and 4x7 times the exposures required to get a photo? I live in an area that is not conducive to doing this, so I will stick with my DSLR until circumstances change.

Pierre

[Michael Covington]

Plenty of astro dedicated cameras have Bayer-matrix color sensors just like DSLRs (in fact in many cases the same sensors).