873 replies to this topic

[Mike in Rancho]

The problem with the 5100 is entirely different.  The black point is set at zero, which chops half your bias off.  Plays havoc with calibration.  The hack moves it to 600ADU, and the D5300 is already there.

 

ISO 100 seems to have quirks.  I use 200, and do so consistently, it's general purpose.

 

There's no need for dark flats for the D5300/5500/5600.  Bias works fine.  This is also true for many cameras, the ZWO1600 (which does need dark flats) seems to have caused an overreaction spilling onto cameras which do not need dark flats.
 

Should we just use one though?  As in, feed my master ISOxxx bias into DSS, or use DF's, but not both?  I had been using every calibration frame DSS allowed, but recently tried out just DF's.  It seemed better, but I could be fooling myself of course.  Looking at the formula DSS says it uses, it would seem in that when you use everything the bias just ends up washing out anyway, but I'm not real sure.

[Mike in Rancho]

It seems like they (he?) are not. I thought maybe if resources were available, I might take a stab at a hack, but it looks to be way over my head. I think it's just the one guy writing stuff. In any case, I did submit a request to Nikon Support to do a firmware update to make lossless NEF compression available on the D5300, but that's probably a longshot. But we'll see what they say...

 

That's right, the D5500 has a lot more room at the lower end where the rings will not cause a problem.  The weird histogram artefacts that cause the rings start when the histogram contains every third value. On the D5300 this happens when values hit 970 whereas on the D550 problems won't begin until values hit 2125.  The bias level is 600 for both cameras. 

 

It's an excellent reason to buy the D5500 instead of the D5300.

 

Mark

 

Thank you Mark for such a detailed analysis, and congratulations for getting to the bottom of it.

Checking my D5500 – it seems the lossy compression algorithm starts “dropping” values at 1552 DN. Still that’s better headroom than the D5300.

TBH I never understood why, even at the bias level, the D5300 drops every other DN. It doesn’t make sense to keep every value only at a DN level that the camera will never use...

Looking at the specs, the 5300 and 5500 appear to be practically twins.  5500 has GPS.  5500 has a higher "native" max ISO that is only an "extended" max ISO on the 5300.  I don't know if there's any hardware behind that improvement.  Same Exspeed 4 processor.

 

As such, at minimum wouldn't it seem possible for the 5300 to use the 5500's better lossy NEF algorithm, thus expanding the headroom before the troubles arise?  Either as an offering by Nikon (though why would they?), or an enterprising programmer...

[limeyx]

Should we just use one though?  As in, feed my master ISOxxx bias into DSS, or use DF's, but not both?  I had been using every calibration frame DSS allowed, but recently tried out just DF's.  It seemed better, but I could be fooling myself of course.  Looking at the formula DSS says it uses, it would seem in that when you use everything the bias just ends up washing out anyway, but I'm not real sure.

Definitely interested here. My feelings

 

1. If you use darks and flat-darks then I don't think Bias adds anything (it would only need to be used to scale darks, so as long as the exposure time matches, you are good)

2. I did a 180second, 60 second and 30 second Dark exposure. Probably around 60F ambient (I have no idea what temp the sensor was)

Then I subtracted the bias from the dark with PixelMath and used statistics. All results were <2ADUs difference -> Therefore you could conclude darks give a minimal benefit and since you cannot reasonably do darks offline with this camera, more imaging time is a better use of the time ? Especially if you dither

3. When I dont use darks, I do use bias

 

So usually I do

Lights, Bias and Flats

 

On my M81 I did the gamut (adding darks and flat-darks)

I have stacked the data multiple ways and analyzing whether there is a significant difference in the stacks

 

I can see flats working for 100% sure (no surprise). If I get time, I will post up a better analysis

[limeyx]

Looking at the specs, the 5300 and 5500 appear to be practically twins.  5500 has GPS.  5500 has a higher "native" max ISO that is only an "extended" max ISO on the 5300.  I don't know if there's any hardware behind that improvement.  Same Exspeed 4 processor.

 

As such, at minimum wouldn't it seem possible for the 5300 to use the 5500's better lossy NEF algorithm, thus expanding the headroom before the troubles arise?  Either as an offering by Nikon (though why would they?), or an enterprising programmer...

You would think. I don't thing either firmware has been modded yet, so it takes someone to kickstart that and it does NOT look easy to do so unless there is a hidden option in the Nikon SDK - I would imagine someone would have already found that

[sharkmelley]

Thank you Mark for such a detailed analysis, and congratulations for getting to the bottom of it.

Checking my D5500 – it seems the lossy compression algorithm starts “dropping” values at 1552 DN. Still that’s better headroom than the D5300.

TBH I never understood why, even at the bias level, the D5300 drops every other DN. It doesn’t make sense to keep every value only at a DN level that the camera will never use...

Yes, it's correct that the D5500 starts dropping values at 1552 DN and that from 1700 DN the histogram contains only even values.  But the larger gaps only begin to appear at 2125 DN.  This is where the rings are likely to begin.

 

Of course this could easily be confirmed by carefully shooting a series of flats at one stop intervals at ISO 100 with a wide open dumb lens (i.e. with plenty of vignetting but no electronic connection) like I did with the D5300 in the original post.

 

Mark

Edited by sharkmelley, 26 January 2021 - 05:48 PM.

[limeyx]

Yes, it's correct that the D5500 starts dropping values at 1552 DN and that from 1700 DN the histogram contains only even values.  But the larger gaps only begin to appear at 2125 DN.  This is where the rings are likely to begin.

 

Of course this could easily be confirmed by carefully shooting a series of flats at one stop intervals at ISO 100 with a wide open lens (i.e. with plenty of vignetting) like I did with the D5300 in the original post.

 

Mark

So it looks like D7500 is the cheapest Nikon I can find that does lossless RAW

From new, they start at ~$1200 (modded) from LifePixel

 

Amazon has them at $650 used + $275 mod fee = $925

eBay is no better

 

Doesn't seem at all worth it unless you pick up a real bargain somewhere. May as well get a "real" astro-cam

[fmeschia]

Yes, it's correct that the D5500 starts dropping values at 1552 DN and that from 1700 DN the histogram contains only even values.  But the larger gaps only begin to appear at 2125 DN.  This is where the rings are likely to begin.

 

Of course this could easily be confirmed by carefully shooting a series of flats at one stop intervals at ISO 100 with a wide open dumb lens (i.e. with plenty of vignetting but no electronic connection) like I did with the D5300 in the original post.

 

Mark

I’d be happy to collect the data for you, but aren’t the flats testing the opposite end of the DN spectrum, not the one closer to the bias point?

[sharkmelley]

I’d be happy to collect the data for you, but aren’t the flats testing the opposite end of the DN spectrum, not the one closer to the bias point?

No. For this specific testing purpose the flats should cover the whole dynamic range from saturation down to extreme under-exposure, in one stop intervals.  Just like the original post in this thread.

 

Mark

[fmeschia]

No. For this specific testing purpose the flats should cover the whole dynamic range from saturation down to extreme under-exposure, in one stop intervals.  Just like the original post in this thread.

 

Mark

Ah, got it. I will run the tests, I also have a “dumb” Nikkor 50/1.8.

[Alrakis]

No. For this specific testing purpose the flats should cover the whole dynamic range from saturation down to extreme under-exposure, in one stop intervals.  Just like the original post in this thread.

 

Mark

So, if I understand, you would want a set of flats from the left side of the histogram to the right side utilizing one ISO setting (say ISO200) with the something like a 50mm lens set at the widest f-stop and controlling exposure with just shutter speed. Is this correct?

 

Chris 

[Alrakis]

Ah, got it. I will run the tests, I also have a “dumb” Nikkor 50/1.8.

Thank you Francesco. I am looking forward to this experiment

 

Chris 

[sharkmelley]

Ah, got it. I will run the tests, I also have a “dumb” Nikkor 50/1.8.

I'm happy to analyse the resulting raw data exposures.

 

Mark

[bobzeq25]

Well Cr*p I JUST got the D5300 last year specifically because everyone seemed to be using and recommending it and I really did not want to go buy a new camera thats for all intents and purposes "the same" (minus this issue and some live view potential fixes)

 

This is disappointing.

Not a big deal unless your name is sharkmelley.  <smile>

 

Many people have made many lovely images with a D5300, and have not had the problem.  Look at astrobin.  All 36 (three-six) pages of them.  <smile>  How many pieces of equipment have 36 pages of nice images?

 

https://www.astrobin...search/?q=D5300

 

Don't worry about it.  If you run into the problem you can take steps to reduce it.  And you may not run into it at all, I doubt many on the 36 pages of nice images did.

Edited by bobzeq25, 26 January 2021 - 06:17 PM.

[bobzeq25]

So it looks like D7500 is the cheapest Nikon I can find that does lossless RAW

From new, they start at ~$1200 (modded) from LifePixel

 

Amazon has them at $650 used + $275 mod fee = $925

eBay is no better

 

Doesn't seem at all worth it unless you pick up a real bargain somewhere. May as well get a "real" astro-cam

That's generally true these days.  A cooled 533 is $900, and is a serious improvement to most any DSLR. 

 

For some people that's "too much money".  But, in general, a DSLR is now an entry level tool for people who already have one.

Edited by bobzeq25, 26 January 2021 - 06:20 PM.

[limeyx]

Not a big deal unless your name is sharkmelley.  <smile>

 

Many people have made many lovely images with a D5300, and have not had the problem.  Look at astrobin.  All 36 (three-six) pages of them.  <smile>  How many pieces of equipment have 36 pages of nice images?

 

https://www.astrobin...search/?q=D5300

 

Don't worry about it.  If you run into the problem you can take steps to reduce it.  And you may not run into it at all, I doubt many on the 36 pages of nice images did.

Sadly I have had it a LOT with the DSLR lens. Unfortunately it's hard to tell what caused it. I *think* moonglow really exacerbates it (maybe indirectly by lightening the lights ?) but it's just a guess

 

So far 1/1 times I got to use the refractor have not had this (even with the moon out -- just a lot of moonglow) but I did have the "red mottling" which I *think* is different

 

On the astro-cam side - yeah, on my list but I don't really want to pay that price for a small sensor cam ideally. I love the APS-C format but ... that starts to get pricey

[fmeschia]

Here’s the result. Like you, Mark, I have set the exposure of the center frame so that the BOC histogram was near the 1/4 point. ISO set at 100.

 

[fmeschia]

And ISO 200. Again, the image in the middle was exposed so that the BOC histogram peaks around 1/4 scale.

 

[limeyx]

Here’s the result. Like you, Mark, I have set the exposure of the center frame so that the BOC histogram was near the 1/4 point. ISO set at 100.

So is the difference in these frames the ISO or the exposure ? Or both ? 

[fmeschia]

So is the difference in these frames the ISO or the exposure ? Or both ? 

Only exposure. Exposure time, to be precise.

[sharkmelley]

Here’s the result. Like you, Mark, I have set the exposure of the center frame so that the BOC histogram was near the 1/4 point. ISO set at 100.

In your result DSC_180, DSC_0179, DSC_0178 and even DSC_0177 all show the same set of rings.  This suggests that those rings are embedded in the brightest flat - i.e. the one you are using as reference.

 

Mark

[fmeschia]

In your result DSC_180, DSC_0179, DSC_0178 and even DSC_0177 all show the same set of rings.  This suggests that those rings are embedded in the brightest flat - i.e. the one you are using as reference.

 

Mark

Good point. Should I maybe use an even more saturated flat?

[sharkmelley]

Good point. Should I maybe use an even more saturated flat?

Yes, either that or maybe use DSC_0179, or DSC_0180 as your reference.

[fmeschia]

Yes, either that or maybe use DSC_0179, or DSC_0180 as your reference.

Thanks. I ended up using another flat, more exposed. You were absolutely right, as you can see here:

[sharkmelley]

Here's another data point, this time the Nikon Z6.

 

 

I took 2 identical raw flat frames in quick succession with a wide open lens, one using the usual lossy compression and one using lossless compression.  They were bias subtracted and divided one by the other.  Further proof that the lossy compression is the problem.

 

Mark

[sharkmelley]

Thanks. I ended up using another flat, more exposed. You were absolutely right, as you can see here:

That's great news and it confirms what we thought - the D5500 has a wider range of exposures without rings being a problem.

 

Mark