Nikon D7000 read noise, gain and thermal noise
Posted 23 February 2013 - 03:25 PM
Posted 23 February 2013 - 08:16 PM
The patcher (Dec 2012: patcher v 1.9) currently handles D3100, D5100, and D7000 with various capabilities.
For example, for 5100, NEF is now changed from
- NEF Compressed (approx 16-20MB sized file) to
- NEF UnCompressed (about 33MB per picture file)
- (not yet available) NEF Lossless
Posted 24 February 2013 - 12:07 PM
I'll work on this today, you were testing at ISO200, right?
I actually used ISO 800 but ISO 200, 400 or 800 would be fine.
Posted 25 February 2013 - 09:01 PM
Many Review sites are using DCRAW based raw file decoding, while the camera mfg. alter the actual OBP (Optically Black Pixel) values and change these to a mfg supplied fix value to make the picture "clean". There is a new RAW decoding library (looking my note now) which can defeat that...
Found my note: it's LibRAW
I am using RawDigger
and uncheck the "masked pixels" (OBP) area and inspect data as much as I can figure it out.
Posted 26 February 2013 - 01:10 AM
It might be worth trying LibRaw but I strongly suspect the black level is subtracted, with the resultant clipping, before the raw NEF file is written.
Posted 01 March 2013 - 01:39 PM
Posted 01 March 2013 - 03:21 PM
Posted 02 March 2013 - 05:17 AM
I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether.
The darks might still have some value - they will help cancel out any pixels that have a higher dark current than average (i.e. they tend to stand out in long exposures) but are not obvious enough to be removed by the Hot Pixel Suppression algorithm.
On the other hand, if you are stacking images, sigma stacking will probably remove them just as effectively - as long as you use dithering during image acquisition.
Posted 03 March 2013 - 11:01 AM
Posted 03 March 2013 - 12:54 PM
So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?
Do you mean let the camera perform its own in-camera dark subtraction? This might work but it really does depend on whether the camera subtracts a clipped dark frame or a non-clipped dark frame from the light frame. But even if this approach works, the problem is that it doubles the shooting time, so a 5 minute exposure then takes 10 minutes. From my own point of view, doubling my acquisition time is not a sacrifice I wish to make.
Posted 03 March 2013 - 01:48 PM
Posted 03 March 2013 - 04:26 PM
Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?
That's an insightful question. Unfortunately the answer is: "it depends".
It depends on what the main limiting factor is in your images i.e. what noise source is dominant. In my own case, my main usage is full colour imaging on a fast (F/2.8) scope and I find the noise from the light pollution swamps the thermal noise and read noise (my sky is around mag 5.5 or so). In such situations, the overall quantum efficiency of the sensor is more important than the dark current or the read noise.
But if you are imaging at slower F-ratios, or on a warm night, or have very dark skies then the thermal noise might well be the dominant source of noise and so I would agree it could well be the case that you would need fewer sub-exposures from a low noise camera than you would from a higher thermal noise camera to get the same overall image quality i.e. to achieve the same signal-to-noise ratio in the final stacked image.
As an aside, read noise is not usually a factor because exposure length can be increased until the noise from sky background or from thermal noise dominates the read noise. However if you want to do shorter exposures e.g. for faster moving objects or to create a night sky video then read noise could easily be the dominant factor.
However the question still remains, does Nikon in-camera dark subtraction use a clipped or a non-clipped dark? I can't think of an easy way to test this. If it uses a clipped dark then in-camera dark subtraction doesn't solve the problem of clipping.
Posted 03 March 2013 - 05:06 PM
I'd suggest wait for few days. Some talent folks in Nikonhacker (Simeon) and others (LibRAW, RawDigger, etc.) are exploring other hidden capabilities in D7000 and D5100. The result may be a completely surprise than the current guessing.
In a 100,000 ft level, I think the approach is trying to undo what the camera body is doing but trying to get the raw image sensor data as-is, or at least understanding the camera processing such that an (sub-)optimal setting can be chosen for astro situations.
Posted 03 March 2013 - 05:08 PM
Posted 03 March 2013 - 06:22 PM
Posted 03 March 2013 - 11:40 PM
1. Out of camera it's NEF RAW Compressed. The 14-bit sensor data is first lossy Huffman encoded (0 - 16383 mapped to 3073 levels; on low side, 0 - 449 are 1:1 mapped) then resulted data compressed. For D5100, the size is about 16-22MB. There are two other Nikon methods, both will not use Huffman encoding table (thus gapless and not lossy). Lossless Compressed (i.e., data is compressed) and Uncompressed (not compressed. file size always 33MB). With patcher v1.12, the latter two method can be chosen as well (can pick one and only one.) ** Note in camera setting, do not use RAW+JPEG or it will hang until battery removed.
Note: the latter two are not optimized for speed or space.
Also note: RawDigger currently having difficulty showing NEF Uncompressed due to overflow-beyond spec. But will be fixed at the next release 0.9.15.
2. Can show OBP area if chose such option in patcher. The values are high (about 596-610), I think subtract these by 595 may start to make sense.
BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different. But we know P does some magic work on raw as well.
3. N does R and B channels magnitude scaling (1.4x). There seems to be a method to defeat that. But will have to wait. BTW, P does not do that.
4. The now better HPS (Hot Pixel Suppression) method is doing a great job (looks cleaner) than teh previous HPS (nick named star eater). The patch seems to be completely defeat the HPS. (Still fairly clean.)
5. There seems to be a way to completely defeat the camera processing, but we'll have to wait.
This is my interim report and getting excited.
All credits go to these talented developers.
Posted 04 March 2013 - 02:52 AM
It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created.
If the camera is set to perform in-camera dark subtraction but is switched off before the dark exposure has completed (the so-called "Mode 3") then the raw NEF that has been written to the card has both the HPS and the biasing (that causes the clipping) already applied. This might indicate that a processed (and clipped) dark frame is subtracted from a processed (and potentially clipped) light frame. For our purposes, the order in which it performs these operations is crucial.
Posted 05 March 2013 - 02:45 AM
BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different.
And on the K-01 do you see clipping of long exposure darks in the same way as the Nikon? Or do you see a full "bell curve" when you display the histogram of values?
Posted 06 March 2013 - 07:50 AM
This post: Nikon (14 bit NEF-Lossless) D5100 ISO800 5 min
Next post: K-01 (12 bit DNG) ISO800 5 min
Note: different Y-axis (pixel count) scale
Posted 06 March 2013 - 07:52 AM
Histogram: Pentax K-01 (12 bit) ISO800 5 min
Posted 06 March 2013 - 05:40 PM
You can definitely see in the Nikon graphs that only the right hand tail of the bell curve exists. The left tail has been clipped to zero.
In the Pentax distribution there is definitely some kind of secondary peak at around 100. I don't know the explanation for that but when you examine the y-axis scale you realise this secondary peak only involves a relatively small percentage of the total pixels - it is a peak that exists in the higher value pixels. The pixel range being displayed (the x-axis) is from 37 to 222. If instead, the x-axis range is set to say 3 to 222 (like the Nikon) you'll again see the main result is a right hand tail of the bell curve i.e. that the Pentax is clipping in a very similar way to the Nikon.
I have seen Nikon D7000 histograms (ISO 800) with a slight secondary peak at around 400 (for a 14bit raw) which corresponds to 100 for a 12 bit raw. I think if you take a 14bit Pentax K-01 raw frame at ISO 800 (instead of 12bit) you will see the secondary peak at around 400, matching the Nikon. All assuming the exposure is kept at 5 minutes.
Posted 06 March 2013 - 10:17 PM
650D, 5 min, ISO800
Posted 07 March 2013 - 01:11 AM
I've attached my own results for a 5min ISO800 dark frame from my Canon 600D which internally sets its bias level at a value of 2048.
Posted 12 March 2013 - 05:17 PM
One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?