873 replies to this topic

[sharkmelley]

Hmmm... I've been assuming the rings on my D5500 are due to the low-level correction... and they were seen with a completely non-electronic lens (an Askar on a T-ring).  Do I need to keep pursuing the possibility they were a reflection or something?  Do they occur with any camera with no lens on it?

Take a series of flats using the test protocol I suggested and you'll soon know for sure.

[Michael Covington]

I've taken test exposures and sent them to you.  You are most welcome to share the results with this group.  Doing a hasty version of the analysis here, I think I do have a ring, in blue/green but not in red/green, and in exactly the position where I had a mysterious blue ring in an extremely stretched picture.  But that was taking some shortcuts in the analysis.  I look forward to seeing what you get!

[Mike in Rancho]

You're absolutely right. There are rings in the blue channel of your exposures taken with the Nikkor 55-200 lens:

 

Top row is your stock D5300 and bottom row is your full spectrum D5300.  Camera firmware is v1.03.

 

That's a very useful result.  Again the D5300 hardcoded correction (on both cameras!) is applied when the firmware recognises the lens but not with a telescope.

 

Mark

Thank you Mark!  Very interesting.  And though I read your website links for more details, I'm clearly still not doing the analysis part correctly.  But at least I can get some faint rings to show in the worst of the files.

 

I haven't used lenses much of late.  The last one I did, duoband of NA+Pelican, came out okay but that's a fairly bright field-filling nebula.  Around the same time with the same lens, I did have difficulty with a wide-field galaxy, such that either flat field correction, gradient extraction, or both, was rendered near impossible.  I gave up on that data on the assumption my lens must have been picking up stray light.  I wonder now...

 

But, I will now follow the tips for ISO or exposure on lights and flats if using a lens.

 

More troubling in my case, since most of my data has been through a scope, were the colored waves.  Noted by Piotr a few pages back, with him using a telescope also.  If those effects are also due to hard-coded internal NEF meddling, recognizing an attached lens seems unnecessary as a trigger.  They are not easy to suppress in processing either, if they even can be.  One obvious example I had was of M42 with the D5300 and Newtonian, so not requiring any sort of severe stretching.  One of the red stripes was baked across the top of the nebulosity.  Pretty in its own way, but not exactly true color. 

 

If I shoot some telescope OSC with the D5300 again, hopefully the same guidance will help with the waves.

[sharkmelley]

I've taken test exposures and sent them to you.  You are most welcome to share the results with this group.  Doing a hasty version of the analysis here, I think I do have a ring, in blue/green but not in red/green, and in exactly the position where I had a mysterious blue ring in an extremely stretched picture.  But that was taking some shortcuts in the analysis.  I look forward to seeing what you get!

Thanks.

 

Yes, you definitely have D5500 rings in the blue channel with no lens:

 

 

Firmware version is v1.02

If you shoot exposures with back-of-camera histograms further right, you will see more and more rings.

 

Mark

[Michael Covington]

Thanks, Mark!

[Michael Covington]

OK, here's where I've ended up.  I very much thank Mark for discovering the low-level color correction.  I had experienced it on a couple of occasions and thought I must have a circular reflection somewhere in my optical train.  It is a relief to find out that that is not so.

 

It is disconcerting to learn that my Nikon D5500 has this limitation, but after doing careful comparison tests, my conclusion is that the rings only show up in extremely stretched images, and the same amount of stretch brings out other defects in other sensors (e.g., Canon's "tartan") or simply strong noise.  It is not hard to avoid the problem by exposing more generously (at a higher ISO if necessary) and not stretching so much.

I still get better pictures with the Nikon than the Canon, so I'm going to keep using it.  

Note that I haven't tried a newer, low-noise Canon (80D or later).  This is a 2012 Canon versus a 2019 Nikon.  Also, I gather that many Nikon cameras are much worse than mine.

[Piotr K.]

Hey guys, sorry for my absence - I had other issues to fix
 

Piotr, one thing I remember from way back was that BYN said it could not support connecting to a firmware hacked D5300, though of course I never had a hacked Nikon to test that out.  I wonder if any such fix would impair acquisition automation through BYN and/or NINA.

Thanks for this info! However, after applying the Nikon Hacker firmware patch, I don't see any problems in controlling my D5100 with ASIair Pro - all is working as before. And I don't use BYN or NINA (at least for now).

Mark - here's a link to 7 flats I've just took wit my D5100, ED80 + SW FF/R 0,85x and Lacerta FlatBox:

https://we.tl/t-7hGMcnD97p

As recommended, I used ISO 100, and exposures from 1/60 to 1/4000 - the first one is a bit too bright than the "normal photography exposure level", but I had to start from 1/60, to be able to shoot 7 flats in total (1/4000 s is the shortest time, and the flat will probably be damaged by the shadow of the mirror). Lacerta FlatBox is very bright, and I don't have any dimmer able to make it darker without flickering. Hopefully the flats will be OK for Mark's procedure of determining the presence of rings / hardcoded correction.

My firmware at the moment:

A 1.02
B 1.01
L 1.006

Patched with Nikon Hacker patch for D5100, with options "NEF compression OFF" and "Disable Nikon star eater" enabled.

[sharkmelley]

Mark - here's a link to 7 flats I've just took wit my D5100, ED80 + SW FF/R 0,85x and Lacerta FlatBox:

https://we.tl/t-7hGMcnD97p

My firmware at the moment:

A 1.02
B 1.01
L 1.006

Patched with Nikon Hacker patch for D5100, with options "NEF compression OFF" and "Disable Nikon star eater" enabled.

Here are the results and I'm sorry to report that the rings are still there:

 

 

This is surprising to me.  I really hoped that the hack would also prevent the hardcoded correction and its rings.

 

The previous set of results was back in post #467

 

Mark

Edited by sharkmelley, 25 March 2023 - 06:18 PM.

[Michael Covington]

Are we absolutely certain that these are hardcoded corrections, and not simply gradients in the sensor's response itself?

[sharkmelley]

Are we absolutely certain that these are hardcoded corrections, and not simply gradients in the sensor's response itself?

Any conjecture/hypothesis needs to fit the known facts. Personally I can't see how the suggestion of a "gradient in the sensor response" fits the facts:

• How does the "sensor gradient" result in rings/posterization which change position according to the level of recorded values?
• Why is the "sensor gradient" so different in the red and blue channels?
• In the case of some D5300 cameras tested in this thread, why does the "sensor gradient" appear only in the case of a lens and not a telescope?
• Why does the "sensor gradient" affect only Nikon cameras and not Sony cameras or dedicated astro-cameras even though the sensors used are in many cases (more or less) identical e.g. the IMX410 sensor used in the Nikon Z6, Sony A7III, ZWO ASI2400MC, QHY410C

Mark

[Michael Covington]

I'm about to have to rush off and don't have time to think all of this through at the moment, but some responses to consider:

(1) Wouldn't any gradient result in posterization of that type, if it is simply a very shallow gradient in response?
(2) Crosstalk is different at different wavelengths (see the article you cited).
(3) The light hits the sensor at a greater range of angles when coming out of the exit pupil of a lens.

(4) is the important one.  If we have found nothing like this in very similar sensors in non-Nikon cameras, that's a reason for attributing it to Nikon's firmware.  But it is surprising that it varies so much from camera to camera, but not from firmware version to firmware version.

[sharkmelley]

Certainly crosstalk and optical angle of incidence will affect the amount of light received and recorded by the sensor pixels.  However, a well-behaved imaging system will produce pixel values whose expectations are directly proportional to photon flux of the incoming light.  Of course the pixel values measured by the ADC will be subject to photon shot noise, read noise and are quantized but the quantization does not create posterization.  Instead, the more and more exposures of the same scene that are averaged together, the closer and closer we get to the true analogue photon flux.  Astrophotographers rely on this good behaviour of the imaging system to extract faint signals, including signals whose strength is just a fraction of the quantization.

 

The fact that we are seeing rings/steps/posterization in Nikon images is because something is breaking this fundamental property of the well-behaved imaging system so that the expected values of a pixel are no longer directly proportional to the incoming light.  

 

Mark

[bobzeq25]

I'm about to have to rush off and don't have time to think all of this through at the moment, but some responses to consider:

(1) Wouldn't any gradient result in posterization of that type, if it is simply a very shallow gradient in response?
(2) Crosstalk is different at different wavelengths (see the article you cited).
(3) The light hits the sensor at a greater range of angles when coming out of the exit pupil of a lens.

(4) is the important one.  If we have found nothing like this in very similar sensors in non-Nikon cameras, that's a reason for attributing it to Nikon's firmware.  But it is surprising that it varies so much from camera to camera, but not from firmware version to firmware version.

I think a bigger factor than camera to camera variation is imager to imager technique variation.  Many people have imaged happily with Nikon D5300+ cameras and no issues.  With my D5500 I ran into it exactly once.

[Michael Covington]

Certainly crosstalk and optical angle of incidence will affect the amount of light received and recorded by the sensor pixels.  However, a well-behaved imaging system will produce pixel values whose expectations are directly proportional to photon flux of the incoming light.  Of course the pixel values measured by the ADC will be subject to photon shot noise, read noise and are quantized but the quantization does not create posterization.  Instead, the more and more exposures of the same scene that are averaged together, the closer and closer we get to the true analogue photon flux.  Astrophotographers rely on this good behaviour of the imaging system to extract faint signals, including signals whose strength is just a fraction of the quantization.

 

The fact that we are seeing rings/steps/posterization in Nikon images is because something is breaking this fundamental property of the well-behaved imaging system so that the expected values of a pixel are no longer directly proportional to the incoming light.  

 

Mark

Gotcha.  Key point: Smooth unevenness in the sensor gets smoother as many frames are stacked. (It would be posterized in a single frame, but smoother in a stack of frames, averaged.)  A 1-ADU subtraction or addition in postprocessing remains the same in every frame and does not get smoother with stacking, so it is visible (posterized) when sufficiently stretched.  And we are observing the rings in stacks of dozens of frames.

[Michael Covington]

I think a bigger factor than camera to camera variation is imager to imager technique variation.  Many people have imaged happily with Nikon D5300+ cameras and no issues.  With my D5500 I ran into it exactly once.

I plan to keep using my D5500 and avoiding situations where very extreme stretching is needed.  But I can understand this as a reason not to recommend Nikon so highly, and to ask them to please, please implement a "truly raw" mode.

[FrankieT]

Apologies in advance if the following test was already performed and the results posted.

 

As described in #566, I was able to reproduce the rings with my D850 following Mark's test protocol. This weekend, I had an old Nikkor 55mm Ai manual lens on the camera - it's a non-cpu lens so the lens information is entered into the camera manually. I don't use it often and certainly not for astrophotography; nevertheless, I took a couple of underexposed flats anyway but the rings were not visible in any of the test images.

 

So, I then mounted the same lens I used in #566 and took one underexposed flat image as usual (1st column in the image below) and another with the electrical contacts covered (2nd column). All other settings were identical with the peak of the camera histogram approximately 1/4 from the left. The images in the top and bottom rows are the red and blue channels divided by the green respectively.

 

In this example, it seems that the D850 did not apply a correction when the firmware didn't recognise the lens.

 

Nikon D850, C v1.02, LD v2.017   AF Nikkor 180mm f/2.8 IF-ED Lens data v0204

ISO 64, f/5.6, lossless compressed

 

[Michael Covington]

Very interesting.  By the way, I have a 1971-vintage 50-mm f/2 Nikkor lens (pre-AI) that is one of my sharpest astrophotography lenses.

[bobzeq25]

I plan to keep using my D5500 and avoiding situations where very extreme stretching is needed.  But I can understand this as a reason not to recommend Nikon so highly, and to ask them to please, please implement a "truly raw" mode.

Why on earth would they?  We're a completely insignificant market for them.  Understatement.  Terrestrial imagers don't have the problem, and appreciate lower memory size requirements.

 

If you want true RAW, get an astrophotography specific camera.  DSLRs often modify RAW in different ways, that cater to their real market, terrestrial.  Astro camera manufacturers cater to _their_ market.

 

Nikon figured this out, and still took a flier at the AP market with the D810A.  I'd be surprised if that effort ever made them money, much less significant money.

[sharkmelley]

In this example, it seems that the D850 did not apply a correction when the firmware didn't recognise the lens.

 

That's very useful info!  So it appears that the D850 behaves the same as the D5300 i.e. the hardcoded-correction rings only appear with a lens recognised by the firmware.

 

By the way, this is how the Sony hardcoded correction works - it is only applied for the lenses that the firmware recognises.

[whwang]

So can I just tape the connectors on a Nikon lens and make it unrecognizable to D850?  Or the lens will be unusable this way?  How about Sony?

 

Do we know if Z6 and Z7 behave the same as D850?

[FrankieT]

Very interesting.  By the way, I have a 1971-vintage 50-mm f/2 Nikkor lens (pre-AI) that is one of my sharpest astrophotography lenses.

Incredible how well 50 yr old lenses still perform on modern technology. I inherited my 70's-vintage 55mm f1.2 Ai. It's a beautiful lens with that enduring nikon build quality of yesteryear - it will definitely outlast me! As expected, it has too much coma and chromatic aberration wide-open (it's not a noct) but it's sharp and it's capabilities far exceed my photographic skills. I never really tried it stopped down for astro, perhaps I should give it a go...but I digress.
 

[Michael Covington]

That's very useful info!  So it appears that the D850 behaves the same as the D5300 i.e. the hardcoded-correction rings only appear with a lens recognised by the firmware.

 

By the way, this is how the Sony hardcoded correction works - it is only applied for the lenses that the firmware recognises.

That is how Nikon ought to do it.  After all, if the camera doesn't recognize the lens, it has no idea where the lens's exit pupil is, and so doesn't know whether light hits the sensor from appreciably different angles off center.

[FrankieT]

So can I just tape the connectors on a Nikon lens and make it unrecognizable to D850? Or the lens will be unusable this way?

I have tried taping the contacts on a couple of Nikon and 3rd-party F-mount lenses and I'm quite confident that the D850 does not recognize the lens once the contacts are covered (I'm not sure how it could). Unless I enter the lens information manually, the D850 does not embed any relevant lens information like the model, focal length or aperture setting in the exif data or display such information via the camera UI. Also as expected, the camera is unable to control focus or lens aperture automatically, which may render some lenses partly unusable.

 

The 180mm Nikon lens I tested has a manual focus ring and an aperture ring. With the contacts covered, this lens can still be fully operated manually, which is probably sufficient for most astrophotography. However, I also have an F-mount lens without a manual aperture ring. Instead, the aperture is controlled electronically by the camera and is only stopped down when a photo is taken. I don't know how, or even if it's possible, to set the aperture manually on this lens when the contacts are covered so it might only be possible to use it wide-open.

Edited by FrankieT, 27 March 2023 - 08:56 AM.

[sharkmelley]

So can I just tape the connectors on a Nikon lens and make it unrecognizable to D850?  Or the lens will be unusable this way?  How about Sony?

 

Do we know if Z6 and Z7 behave the same as D850?

Depending on the lens, it will probably be unusable with the contacts taped over.  It was the same problem with Sony,  various people performed the "tape over the contacts" experiment to prove that Sony was applying an undocumented hardcoded image correction (causing coloured concentric polygons) when the firmware recognised the lens.

 

The Z6 does not behave in the same way as the D850.  Unfortunately, the Z6 applies the hardcoded image correction even when no lens is present e.g. when used on a telescope.  It was my Z6 on the Takahashi Epsilon that first alerted me to Nikon's undocumented hardcoded image correction.  I don't have any evidence of how the Z7 behaves without a lens but my guess is that it will behave just like the Z6.

 

Mark

[bobzeq25]

That is how Nikon ought to do it. 

They don't bother because it wouldn't make them more money.  They're not in business to do the esthetically correct thing.

 

BTDTGTTS.  I used to own a pony car.  Kids on forums would routinely explain how the manufacturer "should" have made their ponycar.  The manufacturer knew a lot better about how to make money than they did.

Edited by bobzeq25, 27 March 2023 - 02:46 PM.