I have been working through some of the implications of the digital spatial filtering and have come to a rather extraordinary conclusion. For some cameras, master dark subtraction is totally ineffective because of the digital spatial filtering. For these cameras, don't bother taking darks!
For a long time there has been anecdotal evidence that for some cameras calibration with darks gives no improvement to images. I have always found this quite puzzling but I now think I know the reason why.
I'll take the Nikon D5500 as an example of this. You may be familiar with the charts where I plot of pixel values against the maximum of neighbouring pixels. Here is the chart for the Nikon D5500:
To explain what this chart means, some understanding of the Bayer matrix is required. A red pixel has 8 neighbours of the same colour in a 5x5 block of pixels centred on that pixel. The same for a blue pixel - it has 8 neighbours of the same colour. A green pixel has 12 neighbours of the same colour in a 5x5 block. So in the plot above I have plotted every pixel value against the maximum of its neighbours of the same colour.
The pink line on the chart has slope=2 and it is very obvious that there is no pixel above that line. In other words, allowing for the bias level of 500, no pixel has a value more than twice the maximum of it neighbours. This is because of the "hot pixel suppression" spatial filtering algorithm that is applied to all exposures 0.25sec and longer. Any pixel brighter than this has its value capped.
Now think about the darks we use for calibration. We average darks together to create a master dark. Some pixels are consistently brighter than others because they have a higher dark current. This is known as the thermal fixed pattern noise (FPN). The idea of the master dark is to allow this FPN to be subtracted from the light frames to remove the effect of those brighter pixels.
Now consider what happens when the camera performs spatial filtering on raw image data. Those brighter pixels in each dark frame tend to be isolated, so the effect of the spatial filtering is to cap their values down to match their same colour neighbours. When we average these dark frames together, the master dark will also have FPN that has been severely capped in value.
Intuitively you might think it's not a problem because those same bright pixels in the light frames will also have their values capped by the spatial filtering, so the subtraction of the master dark will work quite happily. Unfortunately that's not the case. I'll explain why by means of an example.
Typically you might expose your light frames so the peak of the back-of-camera histogram is a quarter of the way from the left hand side. Ignoring the bias level, that will give a pixel value of around 1000 (for 14 bit camera like the D5500). So the sky fog in this example has a level of 1000. There will be some photon shot noise associated with this and in addition some pixels will be brighter because of the FPN. But will the spatial filtering cap these brighter pixels? The answer is no. We have already seen from the chart above that the D5500 spatial filtering does not cap a pixel value unless its value is more than twice the level of its neighbours. Since the neighbours have values of around 1000 then a pixel would need to reach a value of 2000 before it is capped. Only the very brightest of the hot pixels will reach such a level. In other words, the vast majority of the hot and warm pixels visible in the fixed pattern noise will not be capped in the light frames. But they are capped in the dark frames.
A one line summary is the following:
Subtracting a master dark of capped FPN values from a light frame of uncapped FPN values will achieve nothing useful.
Note that the D5500 is just one example camera. Unfortunately the spatial filtering algorithms vary from Nikon camera to Nikon camera and they differ from the Sony spatial filtering. Each algorithm needs to be analysed individually to determine whether or not dark subtraction will be a waste of effort. Having said that, even if it is a waste of time for removing FPN, dark subtraction is still useful for removing any amp-glow that a camera might have.
Edited by sharkmelley, 15 January 2019 - 06:55 PM.