Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Nikon Coloured Concentric Rings

  • Please log in to reply
873 replies to this topic

#851 lviatour

lviatour

    Mariner 2

  • -----
  • Posts: 269
  • Joined: 25 Aug 2022
  • Loc: Belgium

Posted 21 January 2025 - 10:26 AM

Early days, however, why is it that lviatour's data so far, is not showing any real concentric rings (bar that singular ring we are seeing, which I'm not convinced is it) whereas it's plagued by everyone else? It's important to understand this as it could be key to solving the problem, or ultimately shooting in a certain way that completely avoids them. 

If I understand correctly, I should be taking underexposed flats. Is that right? To do this, I need to go through ASIAir because I think working directly in FIT format without saving to the Z7 has an influence. I don't know how to do this at the moment as I'm on a business trip. This weekend, I can run some tests, but I really need to know exactly what to do.



#852 Infomastr

Infomastr

    Vostok 1

  • -----
  • Posts: 112
  • Joined: 02 Feb 2024

Posted 21 January 2025 - 11:57 AM

If I understand correctly, I should be taking underexposed flats. Is that right? To do this, I need to go through ASIAir because I think working directly in FIT format without saving to the Z7 has an influence. I don't know how to do this at the moment as I'm on a business trip. This weekend, I can run some tests, but I really need to know exactly what to do.

The ASIAir was the one thing that stood out to me as different between our processing.  I've been exclusively using raw NEFs from the camera (my ASIAir also saves the data in FIT format, but I've yet to test any of it).  Perhaps the data recorded by the ASIAir is not being affected by Nikon's hardcoded color correction? 



#853 primeshooter

primeshooter

    Viking 1

  • -----
  • Posts: 677
  • Joined: 19 Mar 2021

Posted 21 January 2025 - 05:06 PM

If I understand correctly, I should be taking underexposed flats. Is that right? To do this, I need to go through ASIAir because I think working directly in FIT format without saving to the Z7 has an influence. I don't know how to do this at the moment as I'm on a business trip. This weekend, I can run some tests, but I really need to know exactly what to do.

Mark will likely come back to explain more as he is the best to answer this one. It's very interesting right now, we are on tenterhooks here ;-)


Edited by primeshooter, 21 January 2025 - 05:07 PM.


#854 james7ca

james7ca

    Hubble

  • *****
  • Posts: 13,618
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 21 January 2025 - 06:37 PM

Actually, if memory serves I think there was another ASIAir user who claimed to not have the concentric rings issue with their Nikon DSLR (but in that case I don't think there was any final confirmation on the results). However, I'm sure that the ASIAir isn't using some kind of "hack" to avoid Nikon's standard RAW processing, I suspect they just save Nikon's RAW data to FIT after the former has been transferred to the ASIAir.

 

That said, I do wonder whether Nikon's color shading corrections that appear to be the source of this problem are set by calibrations done individually on each camera. So, each camera will be somewhat different and I suppose that a few cameras may even leave the factory without this calibration (by mistake) or with a "failed" calibration (all zeros or all ones, etc.). Since these colored rings don't seem to show up in most traditional types of photography it's possible that if this factory calibration were nulled no one would notice anything different in the results until that camera was used for astrophotography. Not a very likely scenario, but it might be a posssibility.

 

[UPDATE]

I searched back on CN and found the other report about the ASIAir and it seems to have been made by user lviatour back in May 2024. So, the same person that is currently claiming a lack of concentric rings in their result. However, back then Mark did an expanded stretch on one of lviatour's sample images and it showed rings (although the cause was uncertain). So, it kind of looks like this is a repeat of earlier reports that seemed somewhat doubtful. Here is a link to that earlier reply by Mark: https://www.cloudyni...2#entry13769779

[/UPDATE]


Edited by james7ca, 21 January 2025 - 07:40 PM.

  • whwang likes this

#855 whwang

whwang

    Fly Me to the Moon

  • *****
  • Posts: 5,138
  • Joined: 20 Mar 2013

Posted 21 January 2025 - 07:51 PM

It will be a BIG finding if indeed saving to FITS format can avoid the problem.  I kind of doubt, but I will be very excited if the solution is indeed that simple.  So please help to find out how to do it in ASIAir.  If possible, please also provide a low-exposure flat taken using standard method without ASIAir and saved in NEF format.  This comparison may tell us something useful.



#856 vidrazor

vidrazor

    Fly Me to the Moon

  • *****
  • Posts: 6,705
  • Joined: 31 Oct 2017
  • Loc: North Bergen, NJ, USA

Posted 22 January 2025 - 03:33 AM

The ASIAir was the one thing that stood out to me as different between our processing.  I've been exclusively using raw NEFs from the camera (my ASIAir also saves the data in FIT format, but I've yet to test any of it).  Perhaps the data recorded by the ASIAir is not being affected by Nikon's hardcoded color correction? 

I doubt it. The A to D conversion is still happening in the camera.
 



#857 FrankieT

FrankieT

    Vostok 1

  • -----
  • Posts: 148
  • Joined: 08 Jan 2019
  • Loc: Switzerland

Posted 22 January 2025 - 07:07 PM

I took a series of flats this evening using a Nikon D800 and a lens recognized by the camera with the back-of-camera histogram approx 1/4 from the left. The camera was set to 14-bit lossless compressed, ISO 100, f/5.6, 1 sec and the usual settings like LENR etc disabled.

 

First, a control image was taken normally with the camera and saved to the SD card (NEF format) to verify if the typical concentric ring pattern was present. Then, all memory cards were removed from the camera and, using KStars/Ekos, I took one image encoded in native NEF format and another encoded in 16-bit fits format. Both of these images were stored locally on the PC.

 

The concentric ring pattern for the ratio R/G was the same in all test images.

 

I don't have ASIAir but I believe the device communication framework, like Ekos, is based on INDI (I could be wrong on this however). Therefore, I'm skeptical that the ASIAir or fits format plays a role.

 

 

 

 

 

 

 

 



#858 Infomastr

Infomastr

    Vostok 1

  • -----
  • Posts: 112
  • Joined: 02 Feb 2024

Posted 22 January 2025 - 07:30 PM

I'm also dubious, but felt it was worth bringing up because it was an obvious difference in procedure.  Also, the ASIAir does modify images recorded to it; they appear to be bit shifted (or rescaled?) to 16 bit and do not have the same histogram as the (14 bit) raw NEF files.



#859 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 36,443
  • Joined: 27 Oct 2014

Posted 22 January 2025 - 08:32 PM

Some imagers have found the rings to be a substantial problem, some not. And it's not clear why. I had them in just one image of several. I don't know why.

#860 Michael Covington

Michael Covington

    Author

  • *****
  • Freeware Developers
  • Posts: 9,872
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 22 January 2025 - 10:53 PM

Some imagers have found the rings to be a substantial problem, some not. And it's not clear why. I had them in just one image of several. I don't know why.

It's a function of doing a particular amount of stretching with a particular background level. 

Also, the Nikon D5300 has them only when an electronic lens is attached -- not when attached to a telescope or non-electronic lens.  Other Nikons may have a similar quirk.


  • james7ca likes this

#861 bobharmony

bobharmony

    Viking 1

  • *****
  • Posts: 911
  • Joined: 28 Oct 2017
  • Loc: Connecticut, USA

Posted 23 January 2025 - 09:54 AM

It's a function of doing a particular amount of stretching with a particular background level. 

Also, the Nikon D5300 has them only when an electronic lens is attached -- not when attached to a telescope or non-electronic lens.  Other Nikons may have a similar quirk.

A note on the D5300 - it does have a ring issue when attached to a telescope.  The issue with it is that it doesn't have a lossless compression option and the lossy algorithm causes the rings.  I know this because I've been using one for several years and was one of Marks testers when he was developing the fix for the problem.

 

BTW, if someone has discovered a way to record D5300 data without the lossy compression, I am all ears!

 

Bob



#862 FrankieT

FrankieT

    Vostok 1

  • -----
  • Posts: 148
  • Joined: 08 Jan 2019
  • Loc: Switzerland

Posted 23 January 2025 - 06:00 PM

Also, the ASIAir does modify images recorded to it; they appear to be bit shifted (or rescaled?) to 16 bit and do not have the same histogram as the (14 bit) raw NEF files.

Ok that's interesting. In what way are the histograms different and how do you compare the two? I'm asking because scaling the raw 14-bit image to 16-bits is a non-destructive operation that doesn't alter the information content. I'd be surprised if the ASIAir modifies the raw data in a destructive way when it writes the .fits file because then it's not raw camera data anymore.



#863 Infomastr

Infomastr

    Vostok 1

  • -----
  • Posts: 112
  • Joined: 02 Feb 2024

Posted 23 January 2025 - 09:26 PM

Ok that's interesting. In what way are the histograms different and how do you compare the two? I'm asking because scaling the raw 14-bit image to 16-bits is a non-destructive operation that doesn't alter the information content. I'd be surprised if the ASIAir modifies the raw data in a destructive way when it writes the .fits file because then it's not raw camera data anymore.

Unfortunately this appears to be another dead end.  I took the same image from both the camera (NEF) and ASIAir (FIT) and examined them in darktable and Siril.  The histogram for the FIT was further to the right and wider than that of the NEF.  Despite bit shifting the FIT file back to 14 bit, image division with the NEF file (which had been converted to 16 bit FIT file but had no other processing) was producing some strange results.  Closer examination revealed that the ASIAir and camera had different image orientation (not immediately apparent when I was looking at unstretched flats), which required correction via horizontal/vertical mirroring.  Once this was done, image division via PixelMath showed that the images are in fact, identical, and that the only manipulation done by the ASIAir is a simple bit shift (framing may be entirely user dependent, but worth noting).


  • james7ca and FrankieT like this

#864 FrankieT

FrankieT

    Vostok 1

  • -----
  • Posts: 148
  • Joined: 08 Jan 2019
  • Loc: Switzerland

Posted 24 January 2025 - 12:43 PM

Unfortunately this appears to be another dead end.  I took the same image from both the camera (NEF) and ASIAir (FIT) and examined them in darktable and Siril.  The histogram for the FIT was further to the right and wider than that of the NEF.  Despite bit shifting the FIT file back to 14 bit, image division with the NEF file (which had been converted to 16 bit FIT file but had no other processing) was producing some strange results.  Closer examination revealed that the ASIAir and camera had different image orientation (not immediately apparent when I was looking at unstretched flats), which required correction via horizontal/vertical mirroring.  Once this was done, image division via PixelMath showed that the images are in fact, identical, and that the only manipulation done by the ASIAir is a simple bit shift (framing may be entirely user dependent, but worth noting).

Ok, that's as expected then. It's good to know so thanks for verifying.

 

As an aside, Siril always decodes .fits files using the bottom-up row order convention. However, some applications encode image data top-down. If the ASIAir encodes .fits top-down then that might explain the image orientation differences you encountered. It's easy to check by opening the fits header in Siril and looking for the 'ROWORDER=' keyword. If present, it will be set to either 'TOP-DOWN' or 'BOTTOM-UP'.


  • Infomastr likes this

#865 FrankieT

FrankieT

    Vostok 1

  • -----
  • Posts: 148
  • Joined: 08 Jan 2019
  • Loc: Switzerland

Posted 26 January 2025 - 10:50 AM

If I understand correctly, I should be taking underexposed flats. Is that right? To do this, I need to go through ASIAir because I think working directly in FIT format without saving to the Z7 has an influence. I don't know how to do this at the moment as I'm on a business trip. This weekend, I can run some tests, but I really need to know exactly what to do.

That's right, but the most effective approach is simply to follow Mark's Test Protocol. This will yield a set of flats consistent with those of other cameras that do produce concentric rings. For completeness, it would be helpful to repeat the test protocol without the ASIAir and produce a second set of flats in NEF format. Otherwise any conclusions drawn will just be conjecture and the discussion will go around in circles.


  • whwang likes this

#866 bkesler

bkesler

    Lift Off

  • -----
  • Posts: 11
  • Joined: 17 Feb 2023
  • Loc: Bakersfield, CA USA

Posted 06 February 2025 - 01:13 AM

What ISO was used for that flat?  It's not recorded in the FITS header but a quick calculation suggests ISO 100.  I'm not seeing any rings but that is entirely expected for such a well exposed flat, which is one of the reasons my test protocol suggests a series of flats ranging from normally exposed to very underexposed.

Hi Mark,

 

I have been using my Nikon Z8 for astrophotography since the summer of 2023 and have not noticed the ring problem until last week. My imaging has been Milky Way with various lenses and DSO with my Nikkor 500mm Reflex f/8 lens. Last week I tried DSO with my Rokinon 135mm @ f/2.8 saw the rings for the first time (stack of 625 images, 20s at ISO 500, f2.8). I followed your testing protocol for my 500mm Reflex lens (not recognized by Z8), Rokinon lens, and Rokinon lens with contacts taped. The EXIF data for the Rokinon with contacts taped shows it as Nikkor Reflex at f/8 because it defaulted to the 1st non-CPU lens stored on my Z8, but it is the Rokinon at f/2.8. I always shot in uncompressed RAW, with all in-camera data processing turned off, ISO 500. The flats linked below were shot with the same settings except for ISO 64 (native ISO for Z8). Here is the link to the three sets of files, your analysis would be appreciated.

 

https://drive.google...oMp?usp=sharing



#867 sharkmelley

sharkmelley

    Cosmos

  • *****
  • topic starter
  • Posts: 8,187
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 06 February 2025 - 12:16 PM

We already know for certain that the Nikon Z8 applies a hardcoded correction to the raw data:

 

https://www.cloudyni...ngs/?p=13420852

 

The correction is applied whether or not the optics are recognised by the firmware, so taping over the electronic contacts makes no difference.

 

The files you uploaded are a good set of test files and they show that your Nikon Z8 is definitely applying the correction in each case: Nikkor 500mm and Rokinon 135mm (taped and untaped).

 

Here's a example from each set of files, dividing the (bias-subtracted) red channel by (bias-subtracted) green and also the (bias-subtracted) blue channel by (bias-subtracted) green:

 

CN_bkesler_NikonZ8.jpg



#868 bkesler

bkesler

    Lift Off

  • -----
  • Posts: 11
  • Joined: 17 Feb 2023
  • Loc: Bakersfield, CA USA

Posted 06 February 2025 - 12:53 PM

We already know for certain that the Nikon Z8 applies a hardcoded correction to the raw data:

 

https://www.cloudyni...ngs/?p=13420852

 

The correction is applied whether or not the optics are recognised by the firmware, so taping over the electronic contacts makes no difference.

 

The files you uploaded are a good set of test files and they show that your Nikon Z8 is definitely applying the correction in each case: Nikkor 500mm and Rokinon 135mm (taped and untaped).

 

Here's a example from each set of files, dividing the (bias-subtracted) red channel by (bias-subtracted) green and also the (bias-subtracted) blue channel by (bias-subtracted) green:

 

attachicon.gif CN_bkesler_NikonZ8.jpg

Thank you for your analysis. It is curious that I have not seen this problem until now. I shot over 14 hours of 120s subs with my Nikkor 500mm Reflex last month and there were no rings visualized in the stacked image. Because the lens is a fixed f/8, the exposures were to the far, far left on the histogram. The exposures with the Rokinon at f/2.8 were more central on the histogram. It may be that the difference in post processing is bringing out the rings with some images and not with others. I believe they are present in all images, as you have shown, just the difference in processing is not bringing them to a noticeable level in some cases. I will go back and take a closer look at the images from my Nikkor taken last month and over stretch them to see if the rings manifest.

 

Also, this has me wondering, since I have never taken subs with my Nikkor that were less than 90s (and have not seen the rings), and the subs with the Rokinon were 20s (rings apparent), is there an exposure length that if you exceed, the camera does not apply the correction algorithm to the raw data? This may explain why Iviatour is not seeing the rings in his images. Thanks again for your work and insight on this topic. 


Edited by bkesler, 06 February 2025 - 01:25 PM.


#869 Infomastr

Infomastr

    Vostok 1

  • -----
  • Posts: 112
  • Joined: 02 Feb 2024

Posted 06 February 2025 - 03:46 PM

Thank you for your analysis. It is curious that I have not seen this problem until now. I shot over 14 hours of 120s subs with my Nikkor 500mm Reflex last month and there were no rings visualized in the stacked image. Because the lens is a fixed f/8, the exposures were to the far, far left on the histogram. The exposures with the Rokinon at f/2.8 were more central on the histogram. It may be that the difference in post processing is bringing out the rings with some images and not with others. I believe they are present in all images, as you have shown, just the difference in processing is not bringing them to a noticeable level in some cases. I will go back and take a closer look at the images from my Nikkor taken last month and over stretch them to see if the rings manifest.

 

Also, this has me wondering, since I have never taken subs with my Nikkor that were less than 90s (and have not seen the rings), and the subs with the Rokinon were 20s (rings apparent), is there an exposure length that if you exceed, the camera does not apply the correction algorithm to the raw data? This may explain why Iviatour is not seeing the rings in his images. Thanks again for your work and insight on this topic. 

From what I've seen with the Z6ii, the exposure length only substantially impacts rings if it results in severe under- or over-exposure.  ISO has a much greater impact (I need at least 1600 to mitigate the effect, but 3200 is still visibly better). 



#870 sharkmelley

sharkmelley

    Cosmos

  • *****
  • topic starter
  • Posts: 8,187
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 07 February 2025 - 12:48 AM

Also, this has me wondering, since I have never taken subs with my Nikkor that were less than 90s (and have not seen the rings), and the subs with the Rokinon were 20s (rings apparent), is there an exposure length that if you exceed, the camera does not apply the correction algorithm to the raw data? 

What we know for certain is that for the affected Nikon cameras, every exposure will have rings embedded in the data, caused by the functional form of the hardcoded correction.  However some stacked images will show rings and others will not, which is due to many different factors.

 

This page on my website is essential reading for a better understanding: 

 

It demonstrates that the spacing and the position of each ring is dependent on the level of the recorded exposure (i.e. the pixel values of the background).  The greater the exposure (i.e. higher pixel values), the more rings there are and the closer they are spaced. Each ring is caused by a discrete step of 1 ADU (digital unit) in the data but generally speaking this step is not visible in the exposure because it is hidden in the background noise.  It is only when multiple exposures are stacked together that the signal-to-noise ratio improves sufficiently to make the ring visible (after stretching the stack).

 

So, for instance, if you take multiple flats with a light source of constant brightness, the rings will be in the same place in each flat and the excellent signal-to-noise ratio of the master flat will mean your master flat will contain rings.  This is one reason why I take my flats using a fading light source such as the dusk sky.  This moves the positions of the rings from flat to flat and then they are averaged away in the master flat.

 

The same problem occurs with the lights.  If your sky brightness varies during the imaging session then the ring positions will move from exposure to exposure and will average out in the final stack.  Most of my problems have occurred from imaging sessions where I have been "unlucky" enough to have constant sky brightness.

 

The other important thing is that increasing the exposure level (i.e. the pixel values of the background) will generate more closely spaced rings and the rings are weaker relative to the background pixel values, which makes them less obvious.  Also for a given number of rings in an exposure, a high ISO makes them weaker (i.e. less then 1 ADU) - this is something I'm trying to understand a bit better by performing Monte-Carlo simulations on synthetic data.  Rings at low ISO are sharper and better defined than the rings at high ISO, which is why my test protocol uses low ISO.

 

The above points are why my recommendation is to shoot flats with the back-of-camera histogram as far to the right as possible without saturating the pixels, preferably at a high ISO e.g. ISO 1600.  Using a light source of variable brightness (e.g. dusk sky) is helpful but not always necessary.  Also, shoot your lights with a combination of sufficiently high ISO and exposure length to put the peak of the back-of-camera histogram at least halfway across.  Also, hope that the sky brightness is not too constant when imaging.  

 

Perversely, there is an alternative strategy of using base ISO and exposures that are short enough to stop any rings forming at all.  By looking at the Z8 data you supplied, this will be the case if the (bias-subtracted) pixel values in the background average around 50.  But, generally speaking, this leads to exposures that are impracticably short.


Edited by sharkmelley, 07 February 2025 - 12:58 AM.


#871 Infomastr

Infomastr

    Vostok 1

  • -----
  • Posts: 112
  • Joined: 02 Feb 2024

Posted 07 February 2025 - 12:19 PM

So, for instance, if you take multiple flats with a light source of constant brightness, the rings will be in the same place in each flat and the excellent signal-to-noise ratio of the master flat will mean your master flat will contain rings.  This is one reason why I take my flats using a fading light source such as the dusk sky.  This moves the positions of the rings from flat to flat and then they are averaged away in the master flat.

This is a pretty interesting concept, almost like dithering against rings.  I wonder if this can be done synthetically, i.e. adjusting the light output of a tracing pad or adding small amounts of additional diffusion material during a flat sequence?



#872 whwang

whwang

    Fly Me to the Moon

  • *****
  • Posts: 5,138
  • Joined: 20 Mar 2013

Posted 07 February 2025 - 07:12 PM

This is indeed exposure dithering.  If you take flats manually, this should be fairly easy.  I don't know if any of the programs like NINA, SGP, etc can automize this.


  • sharkmelley likes this

#873 martz

martz

    Sputnik

  • -----
  • Posts: 34
  • Joined: 26 Aug 2020

Posted 08 February 2025 - 08:28 AM

This is a pretty interesting concept, almost like dithering against rings.  I wonder if this can be done synthetically, i.e. adjusting the light output of a tracing pad or adding small amounts of additional diffusion material during a flat sequence?

 

 

This is indeed exposure dithering.  If you take flats manually, this should be fairly easy.  I don't know if any of the programs like NINA, SGP, etc can automize this.

Can't this be done via pixel math--i.e., increasing/decreasing the intensity of an individual flat and using the resulting frames to generate a master?

 

If so, what would be a good expression; what would be a proper amount to increase/decrease between flats to accomplish the desired result of getting rid of the rings?

 

Thanks.



#874 sharkmelley

sharkmelley

    Cosmos

  • *****
  • topic starter
  • Posts: 8,187
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 08 February 2025 - 09:13 AM

Scaling the data is a nice thought but it won't work. The rings are already embedded in the data. If you scale the data, leaving the result in floating point representation then the rings remain in the same location. If you scale the data and truncate (or round) back to integer representation then the existing rings remain and you risk creating new ones, since that's what caused the rings in the first place.
  • martz likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics