873 replies to this topic

[fmeschia]

That's great news and it confirms what we thought - the D5500 has a wider range of exposures without rings being a problem.

 

Mark

Yes indeed. Anecdotally, I only see dimly colored rings in my astro images when I shoot from home (Bortle 7), and my background DNs are in the vicinity of the 1500-DN transition. I have never seen those rings when I shoot from my dark site. I always thought it was some stray light in the backyard getting somehow in the image train, now thanks to you analysis I know the true origin.

Thanks!

[limeyx]

That's great news and it confirms what we thought - the D5500 has a wider range of exposures without rings being a problem.

 

Mark

So the (D5300) conclusion is to expose lights less and flats more ? I am starting to lose track

[sharkmelley]

So the (D5300) conclusion is to expose lights less and flats more ?

Exactly.  My advice for using the D5300 hasn't changed:

• https://www.cloudyni...ngs/?p=10744747

Mark

[Readerp]

Mark

 

I have a D5300 and a D810.

 

Would you look at my typical flat from each to see if the issue is present?

 

I use the 14-bit compressed lossless format NEF with the D810. There is also a 14 bit uncompressed which has huge files.

Perhaps that data might be useful?

 

-Pete

Edited by Readerp, 26 January 2021 - 09:23 PM.

[Kenbuddy]

Well Cr*p I JUST got the D5300 last year specifically because everyone seemed to be using and recommending it and I really did not want to go buy a new camera thats for all intents and purposes "the same" (minus this issue and some live view potential fixes)

 

This is disappointing.

Yup. I've had the D5300 for several years and have had few problems. As has been mentioned many times above, the rings don't always appear. In fact, they usually don't. But I've had a couple of sessions now that were completely ruined by them, which started my investigation and led me to this thread. Biggest shock to me was that the D5300 uses lossy compression! I always assumed the NEF files were uncompressed. I had no idea it was using lossy compression. Huge disappointment! I mean, what was Nikon thinking? So now I'm thinking about upgrading to a D7500. That way I can keep my lenses but get an all around better camera. I also have gotten used to the APS-C sized sensor. Not ready to go full-frame yet, and when I do, I don't know if it will be Nikon.

[celeron787]

Anyone has any stacked files with the D5300 to share? I'm in the midst of getting a modded D5300 and would like to see how much effort is needed to remove/reduce those rings in post-processing.

[Alrakis]

Is it possible that the change in types of street lighting (to LEDs) provided more background light in the green part of the spectrum and this is causing our issues to appear now? 

 

Chris 

[Alrakis]

Also I am curious if there are other settings that might have an effect on this issue, such as:

 

White balance

Set picture control

Color space

Active D-lighting

 

Just curious

 

Chris 

[fmeschia]

Also I am curious if there are other settings that might have an effect on this issue, such as:

 

White balance

Set picture control

Color space

Active D-lighting

 

Just curious

 

Chris 

None of them should have any effect.

[fmeschia]

Anyone has any stacked files with the D5300 to share? I'm in the midst of getting a modded D5300 and would like to see how much effort is needed to remove/reduce those rings in post-processing.

In my experience, they take some effort to remove. DBE is not very effective on them, the only sure-fire method I know is to use Vicent Peris’s multiscale gradient correction (https://pixinsight.c...ent-correction/), with the support of wider-field imaging. But we’re going off-topic...

Edited by fmeschia, 27 January 2021 - 12:03 AM.

[sharkmelley]

I have a D5300 and a D810.

 

Would you look at my typical flat from each to see if the issue is present?

 

I use the 14-bit compressed lossless format NEF with the D810. There is also a 14 bit uncompressed which has huge files.

Perhaps that data might be useful?

I'm happy to take a look. 

 

Mark

[sharkmelley]

Anyone has any stacked files with the D5300 to share? I'm in the midst of getting a modded D5300 and would like to see how much effort is needed to remove/reduce those rings in post-processing.

Depending on the number of rings in the image, typical gradient removal e.g. Pixinsight's ABE and DBE will have huge difficulty removing them. 

 

However I think there is a very good possibility that an analytic lookup table approach will work.  In essence this would be a table of corrections - one for every possible pixel value.  It would be applied to every raw file before calibration and would "undo" the systematic biases to the raw data values caused by the lossy compression.

 

For those with a technical mindset, the lossy compression causes ripples in the photon transfer curve - a good example of the effect can be seen here:

https://www.photonst...htm#Nikon D5200

These ripples are fixed in position and magnitude and affect every exposure.  Once they have been characterized and written to a table, compensation can be applied.

 

You can see the effect of a similar algorithm I designed for the Sony A7S here:

https://www.dpreview...s/post/59505728

Unfortunately that algorithm only worked on the right hand half of the sensor because the camera firmware was doing something even more strange on the left-hand half of the sensor.

 

Be warned though that this is unlikely to happen any time soon.  I don't have an urgent motivation for such a time consuming project because I never use my Nikon D5300.  I stopped using it when I realised the raw data filtering for hot pixel suppression (HPS) caused green stars:

https://www.cloudyni...d-star-colours/

 

Mark

Edited by sharkmelley, 27 January 2021 - 04:37 AM.

[Kevin_A]

I would imagine that this "lossy issue" would all apply to the Nikon Z7 as well?

I will definitely switch my Z6 from lossy compression to no compression at all, but the Z7 change may eat up the hard drive space a little too much for me to use it for astro too! hahahaha

[sharkmelley]

Yes this lossy compression issue applies to the Z7 as well.  Use "lossless compression" because there's no need to use "no compression".

 

Mark

[Kevin_A]

Yes this lossy compression issue applies to the Z7 as well.  Use "lossless compression" because there's no need to use "no compression".

 

Mark

Thanks Mark,

I guess I haven't had enough coffee this morning and read it as a compression issue including lossless! Thanks for clearing it up! Back to my coffee! hahahaha

[Kevin_A]

I think I got caught up with the last comment from Bill Claff in his conclusion about Nikon NEFs where he stated....

"However, if you will be post processing images with a great deal of highlight content then you may want to choose uncompressed NEF rather than compressed NEF to reduce any chance of posterization."

I think I read into that as also including lossless... my bad!

Edited by Kevin_A, 27 January 2021 - 12:31 PM.

[limeyx]

I must admit, I am having trouble correlating my data with the advice on the D5300

 

I have two stacks that have rings and two that I dont believe do (one has some dark areas by the Cone, and the M81/M82 has red mottling but the pattern is very different)

 

The image statistics in PI are all right around 1000 with no real correlation to under-exposed == better

 

Perhaps I am missing something but I believe the data is in the lights

 

Since I shot this data, I have started taking 3+sec flats instead of sub-second (1/10 or 1/2 etc) but I honestly have no idea if that is a factor here

 

Super irritating. I am 100% "OK" with refractor compared to lens so I am hoping that holds (if the rain will ever stop)

 

Rings have proved impossible to remove. Only tool that works is ... crop but thats not always possibe!

[sharkmelley]

I must admit, I am having trouble correlating my data with the advice on the D5300

Upload one of your raw lights and one of your raw flats and I'll take a look.

 

Mark

[Mike in Rancho]

About 2/3 down this page is an "aside" within a larger paper that theorizes on Nikon's NEF compression strategy and why they came up with this.

http://theory.uchica...3.html#bitdepth

 

The only way I've found to deal with it so far, when I get them, is to fiddle around in Startools wipe module.  Pretty much requires maximum or near-maximum settings - this can move the rings around a bit or push them out wider, leaving color casts that are easier to deal with.  Of course that erases more than you'd like to, often.  And then less of a stretch in the ensuing step, perhaps leaving a remaining single ring that can be selected and rejected later.

 

The clouds cleared temporarily last night, letting me take some quick sky shots through my old achro refractor.  I didn't hook up any cables, so the mount was unpowered and static, and the camera wasn't tethered so I was limited to the 30 second intervalometer.  As such, I varied ISO to get differing exposure levels, from 100 to 1000.

 

Basically I was looking for the sweet spot.  However, after looking things over I believe the BOC histogram is some kind of luminance reading.  So, you can be under the recommended 1/4 and possibly still have your data into Mark's danger zone.  BYN's histogram is similar to BOC except you can change from luminance to RGB, and I would suggest using that to stay under 1/4.

 

Here is where my ISO400 test shot ended up on BOC, and in IRIS.  (note - I think the +EC can be ignored in M mode, I was just using that to test viewscreen brightness levels)

 

 

So if using BOC I would think maybe 1/6 would be safer.  On all the subs I took, the bad spiking (which I think is what Mark is talking about) starts at just past 1030.  A lesser spike usually starts in the upper 1020's.

 

On my test shots (a moving M42 lol), the BOC peak corresponded with the following ADU ranges in IRIS:

1/16:  640-760 with peaks at 655 and 725

1/8: 670-910 with peaks at 710 and 845

1/4: 760-1220 with peaks at 820 and 1090

1/3: 875-1625 with peaks at 980, 1280, and 1415

3/8: 950-1850 with peaks at 1075, 1450, and 1620

1/2: 1000-2200 with peaks at 1200, 1700, and 1900

 

I don't know if there's a rule of thumb here or if they are very subject-dependent.

 

I subtracted a dark off each one, and could not induce the rings in any of them with stretching.  Maybe because it's not a stack?  Or perhaps it needs flats divided to really bring them out?  I might try different flat tests on them later.

 

Interesting idea on some kind of post-acquisition repair procedure to reverse the compression strategy.  The true data is lost, as you say, but I guess it would be some sort of interpolation to smooth out the problematic spikes and craters?  And could that be done while remaining lossy-sized, just with the anomalies leveled off, or filled out to be a full-sized 14bit RAW?

[limeyx]

Upload one of your raw lights and one of your raw flats and I'll take a look.

 

Mark

NO RINGS (M81/M82)

 

Light: https://drive.google...iew?usp=sharing

Flat: https://drive.google...iew?usp=sharing

RINGS (M45)

 

Light: https://drive.google...iew?usp=sharing

Flat: https://drive.google...iew?usp=sharing

[sharkmelley]

NO RINGS (M81/M82)

 

Light: https://drive.google...iew?usp=sharing

Flat: https://drive.google...iew?usp=sharing

RINGS (M45)

 

Light: https://drive.google...iew?usp=sharing

Flat: https://drive.google...iew?usp=sharing

Your M81/M82 image has almost no vignetting. It's the main reason why rings are not forming.

 

The M45 image has sufficient vignetting to form rings but it's mainly happening in the red channel of the flat.

 

In the images below, I bias subtracted, debayered, divided the red channel by the green and then stretched:

 

 

There's a slight hint of a ring in the light but it's very obvious in the flat - the red channel is very weak in the flat and it hits a "messy" part of the histogram.

 

Mark

Edited by sharkmelley, 27 January 2021 - 05:04 PM.

[limeyx]

Your M81/M82 image has almost no vignetting. It's the main reason why rings are not forming.

 

The M45 image has sufficient vignetting to form rings but it's mainly happening in the red channel of the flat.

 

In the images below, I bias subtracted, debayered, divided the red channel by the green and then stretched:

 

M45Rings.jpg

 

There's a slight hint of a ring in the light but it's very obvious in the flat - the red channel is very weak in the flat and it hits a "messy" part of the histogram.

 

Mark

Well that's interesting and I have some more info

 

I was convinced these rings are in the lights. Partly because someone here -- I think it may have been you, analyzed some of my other shots (M31) I think and concluded the rings were in the lights

 

Differences between the two

 

M45 - Nikkon 300MM prime lens. The flat is also 1/8 sec (I think this is also ISO200)

 

M81 - AT60ED refractor (camera is the same). The flat is 3 sec (but still around 50% ADU) and ISO400

 

So it's really tough to narrow down what it is

 

if it is exacerbated by vignetting then maybe the refractor will help, but I think I see other people with non-DSLR lenses also seeing the rings

 

I do have some data with the DSLR lens on the Cone nebula which I believe does not have the rings

[sharkmelley]

I was convinced these rings are in the lights. Partly because someone here -- I think it may have been you, analyzed some of my other shots (M31) I think and concluded the rings were in the lights.

I found your M31 example here:

https://www.cloudyni...ong/?p=10503187

 

I just right now downloaded the stacked TIF and it definitely contains rings.  But without examining the original raw files it's not possible to tell if the problem is in the lights or flats or both.

 

Mark

Edited by sharkmelley, 27 January 2021 - 05:45 PM.

[limeyx]

I found your M31 example here:

https://www.cloudyni...ong/?p=10503187

 

I just right now downloaded the stacked TIF and it definitely contains rings.  But without examining the original raw files it's not possible to tell if the problem is in the lights or flats or both.

 

Mark

Let me see if I can replicate your test.

 

I *think* I stacked just the lights and still saw it but I could be wrong

Edited by limeyx, 27 January 2021 - 06:02 PM.

[Alrakis]

The interesting thing to me is that I just acquired this camera (D5300) and started seeing this ringing issue pretty much right away and came to CN and found out this was an issue that others had. However, what is odd to me is that everyone is discussing this now about a camera that has been out for 7 years. Looking it up it seems to have popped up every so often in the intervening time period, but only now is someone (thank you Mark) really digging into it in a scientific way and finding out that it is endemic to other Nikon cameras and based on the compression algorithm.

 

Overall, I am hoping we can come up with a rational explanation and a way to prevent this issue with the cameras that have lossy compression built in for raw files.

 

Chris