Jump to content


Photo

Nikon D7000 read noise, gain and thermal noise

  • Please log in to reply
76 replies to this topic

#26 Hemlock

Hemlock

    Vostok 1

  • -----
  • Posts: 123
  • Joined: 09 Feb 2012
  • Loc: Finger Lakes, New York

Posted 23 February 2013 - 03:25 PM

Or what is 800, or both? There's a lot of number flying around here :) and all this math is a little over my head - let me know, and we can figure out a file transfer method later on.

#27 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 23 February 2013 - 08:16 PM

Nikon hacker Simeon now has a firmware patcher for D7000 (alpha code) to fetch the sensor data (almost as if its Mode 3), except that the only way to do it is through USB cable (issue a special command). I.e., need a PC.
http://simeonpilgrim...ikon-patch.html

<edit>
The patcher (Dec 2012: patcher v 1.9) currently handles D3100, D5100, and D7000 with various capabilities.
For example, for 5100, NEF is now changed from
- NEF Compressed (approx 16-20MB sized file) to
- NEF UnCompressed (about 33MB per picture file)
- (not yet available) NEF Lossless

#28 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 24 February 2013 - 12:07 PM

I'll work on this today, you were testing at ISO200, right?


I actually used ISO 800 but ISO 200, 400 or 800 would be fine.

Mark

#29 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 25 February 2013 - 09:01 PM

...
Many Review sites are using DCRAW based raw file decoding, while the camera mfg. alter the actual OBP (Optically Black Pixel) values and change these to a mfg supplied fix value to make the picture "clean". There is a new RAW decoding library (looking my note now) which can defeat that...


Found my note: it's LibRAW
See http://www.libraw.org/about#better

I am using RawDigger
http://www.rawdigger.com/
and uncheck the "masked pixels" (OBP) area and inspect data as much as I can figure it out.

Clear Skies!

ccs_hello

#30 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 26 February 2013 - 01:10 AM

RawDigger shows the same "clipped to zero" pixels across the main sensor area. But in the optically black area it would appear that we are seeing the original, direct off sensor, un-subtracted values.

It might be worth trying LibRaw but I strongly suspect the black level is subtracted, with the resultant clipping, before the raw NEF file is written.

Mark

#31 TimN

TimN

    Apollo

  • -----
  • Posts: 1408
  • Joined: 20 Apr 2008
  • Loc: Ontario, Canada

Posted 01 March 2013 - 01:39 PM

Ok, I found the previous discussion interesting - at least the parts I could actually understand :o I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether. This would be great as I'm happy with the camera and if the darks aren't adding any value it would be one less step. BTW, I rarely - if ever - image as warm as 20 degrees Celcius. Usually between -25 and +15 Celcius.

#32 RedLionNJ

RedLionNJ

    Viking 1

  • -----
  • Posts: 893
  • Joined: 29 Dec 2009
  • Loc: Red Lion, NJ, USA

Posted 01 March 2013 - 03:21 PM

I must say, I'm finding this thread fascinating. I currently use a Canon (550D), but find the noise characteristics to be less-than-superlative (esp on a warm night) and am in the process of investigating other models for astro purposes, including the D7000 body. While the post-capture histrionics carried out by the Canons before delivering up a 'raw' image are legendary. I've yet to see any model which produces a true 'raw' output. Reading here about the different-but-equally-annoying Nikon processes is not reassuring me any consumer DSLR can be made to spit out a truly raw image :(

Grant

#33 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 02 March 2013 - 05:17 AM

I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether.


The darks might still have some value - they will help cancel out any pixels that have a higher dark current than average (i.e. they tend to stand out in long exposures) but are not obvious enough to be removed by the Hot Pixel Suppression algorithm.

On the other hand, if you are stacking images, sigma stacking will probably remove them just as effectively - as long as you use dithering during image acquisition.

Mark

#34 TimN

TimN

    Apollo

  • -----
  • Posts: 1408
  • Joined: 20 Apr 2008
  • Loc: Ontario, Canada

Posted 03 March 2013 - 11:01 AM

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?

#35 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 03 March 2013 - 12:54 PM

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?


Do you mean let the camera perform its own in-camera dark subtraction? This might work but it really does depend on whether the camera subtracts a clipped dark frame or a non-clipped dark frame from the light frame. But even if this approach works, the problem is that it doubles the shooting time, so a 5 minute exposure then takes 10 minutes. From my own point of view, doubling my acquisition time is not a sacrifice I wish to make.

Mark

#36 TimN

TimN

    Apollo

  • -----
  • Posts: 1408
  • Joined: 20 Apr 2008
  • Loc: Ontario, Canada

Posted 03 March 2013 - 01:48 PM

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?

#37 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 03 March 2013 - 04:26 PM

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?


That's an insightful question. Unfortunately the answer is: "it depends".

It depends on what the main limiting factor is in your images i.e. what noise source is dominant. In my own case, my main usage is full colour imaging on a fast (F/2.8) scope and I find the noise from the light pollution swamps the thermal noise and read noise (my sky is around mag 5.5 or so). In such situations, the overall quantum efficiency of the sensor is more important than the dark current or the read noise.

But if you are imaging at slower F-ratios, or on a warm night, or have very dark skies then the thermal noise might well be the dominant source of noise and so I would agree it could well be the case that you would need fewer sub-exposures from a low noise camera than you would from a higher thermal noise camera to get the same overall image quality i.e. to achieve the same signal-to-noise ratio in the final stacked image.

As an aside, read noise is not usually a factor because exposure length can be increased until the noise from sky background or from thermal noise dominates the read noise. However if you want to do shorter exposures e.g. for faster moving objects or to create a night sky video then read noise could easily be the dominant factor.

However the question still remains, does Nikon in-camera dark subtraction use a clipped or a non-clipped dark? I can't think of an easy way to test this. If it uses a clipped dark then in-camera dark subtraction doesn't solve the problem of clipping.

Mark

#38 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 03 March 2013 - 05:06 PM

Mark and folks,

I'd suggest wait for few days. Some talent folks in Nikonhacker (Simeon) and others (LibRAW, RawDigger, etc.) are exploring other hidden capabilities in D7000 and D5100. The result may be a completely surprise than the current guessing.

In a 100,000 ft level, I think the approach is trying to undo what the camera body is doing but trying to get the raw image sensor data as-is, or at least understanding the camera processing such that an (sub-)optimal setting can be chosen for astro situations.

Clear Skies!

ccs_hello

#39 TimN

TimN

    Apollo

  • -----
  • Posts: 1408
  • Joined: 20 Apr 2008
  • Loc: Ontario, Canada

Posted 03 March 2013 - 05:08 PM

interesting Mark. I think I read when - in the overall process - the in- camera dark subtraction is applied. Unfortunately, I'm away from home and I'm getting only a moment - now and then - to get access to the net. So, no time for research. However, a couple of quick thoughts. It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created. This would be after the image has the in-camera raw subtracted. Anyway, I'll try and do some more research when I return later this week.

#40 TimN

TimN

    Apollo

  • -----
  • Posts: 1408
  • Joined: 20 Apr 2008
  • Loc: Ontario, Canada

Posted 03 March 2013 - 06:22 PM

Sorry ccs_hello, I must have sent my last post before I read yours. Will wait for the more knowledgable folks to get back to us.

#41 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 03 March 2013 - 11:40 PM

There are some nice work on going. I'd like to report what I know on D5100 related:

1. Out of camera it's NEF RAW Compressed. The 14-bit sensor data is first lossy Huffman encoded (0 - 16383 mapped to 3073 levels; on low side, 0 - 449 are 1:1 mapped) then resulted data compressed. For D5100, the size is about 16-22MB. There are two other Nikon methods, both will not use Huffman encoding table (thus gapless and not lossy). Lossless Compressed (i.e., data is compressed) and Uncompressed (not compressed. file size always 33MB). With patcher v1.12, the latter two method can be chosen as well (can pick one and only one.) ** Note in camera setting, do not use RAW+JPEG or it will hang until battery removed.
Note: the latter two are not optimized for speed or space.
Also note: RawDigger currently having difficulty showing NEF Uncompressed due to overflow-beyond spec. But will be fixed at the next release 0.9.15.


2. Can show OBP area if chose such option in patcher. The values are high (about 596-610), I think subtract these by 595 may start to make sense.
BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different. But we know P does some magic work on raw as well.


3. N does R and B channels magnitude scaling (1.4x). There seems to be a method to defeat that. But will have to wait. BTW, P does not do that.


4. The now better HPS (Hot Pixel Suppression) method is doing a great job (looks cleaner) than teh previous HPS (nick named star eater). The patch seems to be completely defeat the HPS. (Still fairly clean.)


5. There seems to be a way to completely defeat the camera processing, but we'll have to wait.

This is my interim report and getting excited.

All credits go to these talented developers.

Links:
http://nikonhacker.c...ewforum.php?f=2
http://simeonpilgrim...ikon-patch.html
http://www.rawdigger.com/news

Clear Skies!

ccs_hello

#42 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 04 March 2013 - 02:52 AM

It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created.


If the camera is set to perform in-camera dark subtraction but is switched off before the dark exposure has completed (the so-called "Mode 3") then the raw NEF that has been written to the card has both the HPS and the biasing (that causes the clipping) already applied. This might indicate that a processed (and clipped) dark frame is subtracted from a processed (and potentially clipped) light frame. For our purposes, the order in which it performs these operations is crucial.

Mark

#43 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 05 March 2013 - 02:45 AM

BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different.


Interesting.
And on the K-01 do you see clipping of long exposure darks in the same way as the Nikon? Or do you see a full "bell curve" when you display the histogram of values?

Mark

#44 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 06 March 2013 - 07:50 AM

Showing the clipping effect

IMX071

This post: Nikon (14 bit NEF-Lossless) D5100 ISO800 5 min
Next post: K-01 (12 bit DNG) ISO800 5 min

Note: different Y-axis (pixel count) scale

Attached Files



#45 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 06 March 2013 - 07:52 AM

IMX071

Histogram: Pentax K-01 (12 bit) ISO800 5 min

Attached Files



#46 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 06 March 2013 - 05:40 PM

Those are interesting graphs.

You can definitely see in the Nikon graphs that only the right hand tail of the bell curve exists. The left tail has been clipped to zero.

In the Pentax distribution there is definitely some kind of secondary peak at around 100. I don't know the explanation for that but when you examine the y-axis scale you realise this secondary peak only involves a relatively small percentage of the total pixels - it is a peak that exists in the higher value pixels. The pixel range being displayed (the x-axis) is from 37 to 222. If instead, the x-axis range is set to say 3 to 222 (like the Nikon) you'll again see the main result is a right hand tail of the bell curve i.e. that the Pentax is clipping in a very similar way to the Nikon.

I have seen Nikon D7000 histograms (ISO 800) with a slight secondary peak at around 400 (for a 14bit raw) which corresponds to 100 for a 12 bit raw. I think if you take a 14bit Pentax K-01 raw frame at ISO 800 (instead of 12bit) you will see the secondary peak at around 400, matching the Nikon. All assuming the exposure is kept at 5 minutes.

Mark

#47 ccs_hello

ccs_hello

    Fly Me to the Moon

  • *****
  • Posts: 6331
  • Joined: 03 Jul 2004

Posted 06 March 2013 - 10:17 PM

It would only be fair to add C into the picture.

650D, 5 min, ISO800

Attached Files



#48 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 07 March 2013 - 01:11 AM

That histogram looks wrong for the Canon - the Canon does not clip its raws (unless the 650D behaves differently). Is it possible the "Subtract Black" option has been selected in RawDigger? This option should be switched off to see the true raw data values.

I've attached my own results for a 5min ISO800 dark frame from my Canon 600D which internally sets its bias level at a value of 2048.

Mark

Attached Files



#49 Plane

Plane

    Explorer 1

  • -----
  • Posts: 51
  • Joined: 09 Dec 2009

Posted 11 March 2013 - 04:59 PM

FYI, I've tested the dark current from optical black region on sensor with the raw hack mode turned on.

http://landingfield....haracteristics/
Giving me 0.15eps @ 21°C
Doubling temp 6~6.5°C

#50 sharkmelley

sharkmelley

    Lift Off

  • -----
  • Posts: 24
  • Joined: 19 Feb 2013

Posted 12 March 2013 - 05:17 PM

That's a great idea to use the optical black region for calculating the read noise and dark current - I'll give that a try myself.

One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?

Mark






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics