Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
Get a Cloudy Nights T-Shirt Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | (show all)
sharkmelley
member


Reged: 02/19/13

Nikon D7000 read noise, gain and thermal noise
      #5689403 - 02/19/13 06:04 PM

Apologies for the length of my first post but I hope you find it worthwhile.

I have been doing DSLR astrophotography for a few years mainly using an old modified Canon 350D. However I recently heard that on the D7000 model, Nikon have updated the infamous "Star Eater" Hot Pixel Suppression algorithm for long exposures which used to remove not only single hot pixels but clumps of up to 4 adjacent pixels i.e. stars. It now only removes isolated hot pixels in long exposures.
If you want to know more, see this post on DPReview:
http://forums.dpreview.com/forums/post/37071846

However, it's been almost impossible to find any firm data on its suitability for astrophotography, so with the help of a friend who owns one, I decided to put it through its technical paces.

Taking pairs of frames at different shutter speeds of a diffuse white object allows the gain and read noise to be calculated. Then taking 24 dark frames of 5 minutes each allows the dark current and hence thermal noise to be investigated.

My results are summarised here:



The figures in bold are figures I have calculated (most of the analysis was done at ISO 800) and the rest were inferred by assuming the gain doubles for each halving of ISO.
I find it odd that the gain for the Red and Blue channels is different from the Green channel. But my figures are confirmed by examining the histograms of pixel values in the raw NEF files. The Green channel has every integer pixel value represented in the histogram but there are gaps in histograms for Red and Blue. The frequency of the gaps (14 gaps / 100 values) agrees with the ratio of gains of the Green and R/B channels which is approx 1.14



Unfortunately this is not the only pre-processing of data that Nikon performs before writing raw (NEF format) images. The 5 minute dark frames had well over 50% of pixels with zero value - it seems to vary between 50% and 80% zero valued pixels from raw darks I have seen from different examples of the D7000. Does this mean that the dark current is ultra-low? Unfortunately not. It means that the values have had a constant value subtracted from them, clipped at zero, to make the thermal noise appear less intrusive. If you plot the pixel values from a dark frame as a histogram you should normally see the familiar "bell-shaped" normal distribution curve. But all that remains in the D7000 dark frames is the high end tail of this distribution - more on this later. However by taking the difference between pairs of successive frames and then trying to fit the tail to a complete distribution (lots of maths!) I was able to approximate how the dark current (and hence thermal noise) increases over the 2 hours taken to perform 24 exposures of 5 minutes each - a typical astro-imaging scenario.



For interest, compare this with a number of Canon cameras I have tested:



So there are 2 pieces of really good news:

1) The read noise of this camera is more or less constant at an extraordinarily low level for all ISO values. At ISO 200 you have a gain of almost unity (very useful because electrons only appear in whole numbers!) with incredibly low read noise. If you are shooting raw frames (which all astro-imagers ought to be doing) there is no reason at all to use higher ISOs - you would get no improvement in overall image quality and your images would saturate more quickly (i.e. the dynamic range reduces). I would actually argue that there should be no obvious difference in quality if you do all your imaging at ISO 100 on the D7000.

2) The dark current is also very low indeed. Notice that these measurements were taken at an ambient tmeperature of 20C. For every drop of 6-7C, the dark current will reduce by a factor of 2 and the thermal noise will reduce by a factor of 1.4 (square root of 2) On a cold night (typical in the UK!) this camera is capable of producing long exposure images with extraordinary low noise levels with huge dynamic range. Shooting in normal daylight daylight conditions, exposures will be short and thermal noise is not measurable.

But there are 3 pieces of bad news - all related to the way that Nikon pre-processes the data before writing the raw NEF file.

1) The HPS algorithm, although improved, can still mean that using short focal length lenses, a star occupying a single pixel will be erased. However, with many sub-exposures stacked together this is unlikely to be a big issue.

2) The Red and Blue channels are scaled by 1.14 (why?!) leading to the gaps in the histogram. This is only a slight annoyance to the purist like me and probably has no visible effect.

3) The bigger problem is the clipping of pixels to zero. If you use this camera at slow F-ratios, on a warmish night in a very dark place you will see pixels with zero value. Data in dark shadowy areas (e.g. dust lanes of nebulae) is being thrown away and you'll never be able to recover it by stacking how ever many images you stack. Why does Nikon decide to throw your your precious data away? In practice, this may or may not be a problem - it really depends on the scope/lens you are using and the conditions you are shooting under. But it is certainly likely to affect results from narrowband filters (e.g. H-alpha). I would be really interested to hear experiences from anyone who may have already encountered this.
But the bigger problem associated with pixel clipping is that it is actually impossible to shoot proper calibration frames i.e. dark frames and bias frames. Both darks and biases will contain at least 50% zero value pixels. You might go through the motions of taking many dark frames and summing into a single "master" dark but that master dark is not correct, the values have been distorted by the clipped pixels. They won't do a proper job of removing thermal effects from your image. The same with bias frames - they will not do a proper job of removing "pattern noise". However it is difficult to quantify what effect this has on the final stacked image. Also I am trying to think of a way of "biasing" dark frames and vias frames with very low levels of light to push all the pixel values up to the level where no clipping takes place.

So, in conclusion, the sensor (Sony Exmor IMX071) in this camera is an extraordinary sensor with extremely low levels of read noise and low levels of thermal noise - especially on cold nights. It is a real shame that the issue of clipped pixels exists but then again some imagers will never notice this nor the effects it has on their images. You never know, maybe some day a firmware "hack" will come along that enable this to be switched off.

Clear skies!

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5689899 - 02/19/13 10:43 PM

Hi Mark! Welcome to Cloudy Nights!

Thanks for your thorough analysis on D7000 and brought up Marianne Oelund's thread on DPR. Her study was not so well known here in CN (I only read her's last week on my IMX071 study <-- long story I'll describe more in later days.) But I highly recommend CNer's to go there reading it. Basically it's about N's modification on its infamous "Star Eater" (need mode 3 to help) feature, i.e., Hot-Pixel Suppression (HPS) algorithm. Then N "enhanced" it (the new 8-neighbor algorithm) P.S., it's been going for a while and a few feel its acceptable and mode 3 is no longer needed. Luckily it kind of align with N's change from CCD based to CMOS based DSLRs.

Let's exchange some notes:

Back to black level clipping, My gut feeling is that many mfgs are doing the same. Prof. Craig Stark's paper showed that on C's.
Many Review sites are using DCRAW based raw file decoding, while the camera mfg. alter the actual OBP (Optically Black Pixel) values and change these to a mfg supplied fix value to make the picture "clean". There is a new RAW decoding library (looking my note now) which can defeat that and let the user select the black level for proper sensor characterization.

Going back to D7000, my study on IMX071 is that Pentax K-5 and K-01, Sony NEX-5N, and others seem to track the identical sensor performance curve (using DXOmark) so may be it's not the body doing the black clip (and is not just N by itself) but the sensor itself. But at this point, it's just my guessing.

Now the IMX071 itself. Now we know its internal PGA (programmable gain amplifier) is no longer doing extra gain any more after ISO1600. Anything beyond is just digital scaling (left shift, multiply by two)! (Pentax at this stage run software enhancement, appears to look like one extra stop gain).

Anyway, I am getting a cam using IMX071. I'll know more.

Please keep us apprised with great info!

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
guyroch
Vendor (BackyardEOS)
*****

Reged: 01/22/08

Loc: Under the clouds!
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5689924 - 02/19/13 11:01 PM

That's a good informative *first* post. Welcome to CN.

I have a feeling your contribution will create some good conversation.

Guylain


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5689934 - 02/19/13 11:07 PM

Mark,


RE:
2) The Red and Blue channels are scaled by 1.14
N' being N is its rather rigid way of treating itself as a leading edge mfg. in imaging space. Sticking to precision is its job - to sell cameras (while scaling may ruin the perfect scientific use of image sensor , i.e., allowing us to use post-processing.) Oh, the reason... recalibrate to compensate the not so perfect Bayer filter spectrum response to fit the ideal color gamut.
BTW, N's HPS is famous, isn't it.

RE:
3) The bigger problem is the clipping of pixels to zero.
Let me dig out the reference of using the RAW decoding (replacing DCRAW) that can ignore "manipulated" OBP value and see how it goes.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
Astronewb
Pooh-Bah
*****

Reged: 09/19/11

Loc: Connecticut
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5689958 - 02/19/13 11:25 PM

Thanks so much for the interesting read. I image with a Nikon D5100 (same sensor), so I found it very interesting. My 'sweet spot' may be dropped from 400 to 250 or 320 now.

Great read...cheers

Paul


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Astronewb]
      #5689984 - 02/19/13 11:48 PM

About D7000 and D5100 ISO selection...

Indeed for max dynamic range, use low ISO (100 or 200), then the burden will be mount tracking, etc.

Since the sensor used has a very low noise, for shorter exposure time in imaging, I'd say up to ISO 1600 with reduced D.R. Beyond that there is absolute no gain since it's just digital scaling (while chop more bit off D.R.)

For low light LiveView/observ type of useage, go ahead use max. ISo possible. Per DXOmark's performance curve, at 25600 you still get 6 stop D.R. and reasonable S/N which IMHO is still very usable.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
cherokawa
sage
*****

Reged: 10/19/08

Loc: Torrance, CA
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5690043 - 02/20/13 12:30 AM

Quote:

About D7000 and D5100 ISO selection...

Indeed for max dynamic range, use low ISO (100 or 200), then the burden will be mount tracking, etc.

Since the sensor used has a very low noise, for shorter exposure time in imaging, I'd say up to ISO 1600 with reduced D.R. Beyond that there is absolute no gain since it's just digital scaling (while chop more bit off D.R.)





Please bear with me - I'm a newbie at this. But doesn't unity gain at ISO 200 mean that at any ISO beyond this there is only digital padding? In other words, 5 mins at ISO 200 will have the same SNR as 5 mins at ISO 1600 - just that the ISO 200 image will be dimmer and this can always be stretched in post-proc? Just trying to get my understanding straight.

And Mark, thanks for a fantastic and informative post.


Post Extras: Print Post   Remind Me!   Notify Moderator  
orlyandico
Postmaster
*****

Reged: 08/10/09

Loc: Singapore
Re: Nikon D7000 read noise, gain and thermal noise new [Re: cherokawa]
      #5690253 - 02/20/13 06:54 AM

http://www.sensorgen.info/NikonD7000.html

read noise is about 3 e- and the well depth is 50K e- at ISO 100.

at ISO 200 the well depth is 19000 e- (as expected).

QE is 48%

some other cameras -

http://www.sensorgen.info/

(model / QE / read noise / well depth)
EOS_5D 25% 4.2 55297
EOS_5D_MkII 33% 3.2 64600
EOS_5D_MkIII 49% 2.4 67531

none of these cameras come close to the 80% QE of the early-2000's Kodak ST-3200ME. Any reason for that? or is the above QE simply due to the Bayer sensor? or due to the small pixels? the 5D Mk I has large pixels (8um) but miserable QE (25%)

Seems that the D5200 has the best numbers on the above web site..


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: cherokawa]
      #5690310 - 02/20/13 07:51 AM

Re: (Does)(if) unity gain at ISO 200 mean that at any ISO beyond this there is only digital padding?

Not the case, the PGA does the amplification until there is (1) no significant benefit (yes, the noise part starts to degrade S/N, but only to the extent that "gain" helps in fishing out the signals) to amplify it further more
(2) noise starts to be a dominate term (too high a gain on PGA) such that digitally scale would become a better straegy

For IMX071, the cross-over point is around ISO1250 or so.

Again, my point: use low ISO for more dynamic range and better quality astrophotography. Can go a bit higher if you want to use that apparatus to help you to "see" (frame and focus) or get image quicker (e.g., mount limitation etc.) If you can tolerate more degradation, use even higher ISO (i.e., in-camera higher than ISO1600 or use post-processing techniques)

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: cherokawa]
      #5690318 - 02/20/13 07:58 AM

Quote:

Quote:

About D7000 and D5100 ISO selection...

Since the sensor used has a very low noise, for shorter exposure time in imaging, I'd say up to ISO 1600 with reduced D.R. Beyond that there is absolute no gain since it's just digital scaling (while chop more bit off D.R.)





Please bear with me - I'm a newbie at this. But doesn't unity gain at ISO 200 mean that at any ISO beyond this there is only digital padding? In other words, 5 mins at ISO 200 will have the same SNR as 5 mins at ISO 1600 - just that the ISO 200 image will be dimmer and this can always be stretched in post-proc? Just trying to get my understanding straight.






I may be wrong about this but I have a strong suspicion that the raw digital data from the chip is simply multiplied by different constants for each ISO. The Exmor sensor has a ADC unit for every column of detectors. If I'm right, nearly every ISO should have the same SNR but of course the dynamic range will change. So higher ISOs would gain you nothing at all.


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: orlyandico]
      #5690319 - 02/20/13 07:58 AM

DXOmark or sensorgen's figures reflect image system's figure of merit. It may have other components tinted by the camera mfgs.

Kodak's reflect the sensor itself. Note that the KAF-3200 is a monochrome and full-frame type CCD, not interline type of CCD nor CMOS APS image sensors.

I would caution that Q/E is simply a parameter in the sensor's figure of merit (pay attention to noise figure and dynamic range, pixel pitch, etc.)

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
Magellan
Pooh-Bah


Reged: 01/26/06

Loc: Enfield, NS Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5690532 - 02/20/13 10:04 AM

Wow what I post! great job!!

I have been looking for one of these for a 450D, do you have one by chance?


Post Extras: Print Post   Remind Me!   Notify Moderator  
srosenfraz
Post Laureate
*****

Reged: 03/06/11

Loc: United States
Re: Nikon D7000 read noise, gain and thermal noise new [Re: Magellan]
      #5690661 - 02/20/13 11:21 AM

Mark - welcome aboard and your first post sure is informative. Glad to have you join the crowd!

@Jeff - Best article I've seen is Craig Stark's article on CN where he analyzes much about the 450D. I think it'll have a lot of the information for which you're looking.


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5691261 - 02/20/13 04:26 PM

Quote:

Back to black level clipping, My gut feeling is that many mfgs are doing the same. Prof. Craig Stark's paper showed that on C's.





That's not my experience of Canons and I've analysed them in quite a lot of detail. They have a fixed bias e.g. 512, 1024, 2048 which does cause it's own problems but not nearly so bad as Nikon's clipping to zero. Do you have a reference for Craig Stark's paper?

Quote:


There is a new RAW decoding library (looking my note now) which can defeat that and let the user select the black level for proper sensor characterization.





Sounds interesting.

Quote:


Going back to D7000, my study on IMX071 is that Pentax K-5 and K-01, Sony NEX-5N, and others seem to track the identical sensor performance curve (using DXOmark) so may be it's not the body doing the black clip (and is not just N by itself) but the sensor itself. But at this point, it's just my guessing.





If we could obtain some long exposure dark frames from these cameras we would know for certain. The thing about this clipping to zero is that it is only noticeable on long exposures with no light (or very little light) reaching the sensor.

Quote:


Now the IMX071 itself. Now we know its internal PGA (programmable gain amplifier) is no longer doing extra gain any more after ISO1600. Anything beyond is just digital scaling (left shift, multiply by two)!





Marianne Oelund indicates this happens after ISO 950:
http://forums.dpreview.com/forums/thread/2947801
I think I was wrong with my guess about the on-chip ADC being 16bit - there is some analogue amplification taking place on the sensor. However I still think there is no advantage to using ISOs higer than ISO 200 which gives more or less a gain of unity. Once you are counting individual electrons there is nothing to be gained by trying to count half electrons or quarter electrons!

Quote:



Anyway, I am getting a cam using IMX071. I'll know more.





Me as well! My analysis was done on a friend's camera. One of the first tasks will be to mod it for H-alpha - I'm dying to know its H-alpha sensitivity.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5691823 - 02/20/13 09:58 PM

Mark,

Craig's paper in his website here which is a little bit revised. RE: the OBP swap out, you have to read the paper careful to derive your own conclusion.

RE: IMX071AQ family used in
D7000, D5100, K5, K-01, A580, NEX-5N, (perhaps NEX-C3), etc.
Indeed we need astro specific characteristics, especially on long exposure based low photon flux related testing.

(Or baby IMX071, the IMX109AQE used in
Panny GH3, E-PL3, E-PM2 and Oly OMD E-M5
They are fantastic too.

RE: the sensor on-chip column-parallel A/D Converters (few thousand of them) used in SONY Exmor sensor.
They are 12 bit only.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5692146 - 02/21/13 02:50 AM

Quote:

Craig's paper in his website here which is a little bit revised. RE: the OBP swap out, you have to read the paper careful to derive your own conclusion.

Quote:



Yes, I'm familiar with that paper. Although the Canons do shift the histogram (by a variable amount) the paper is not saying it results in the "clipping to zero" problem that the D7000 shows.




RE: the sensor on-chip column-parallel A/D Converters (few thousand of them) used in SONY EXMOR sensor
They are 12 bit only.





But the ADCs on the D7000 sensor are 14bit right? I'm now convinced that it is not a 16bit sensor.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
harbinjer
sage


Reged: 12/17/08

Loc: Southeastern Minnesota
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5692568 - 02/21/13 11:35 AM

I'm pretty sure that the K-01 and the K-30 don't use the same sensor are the K-5. It is similar in many ways, but it definitely has a lower dynamic range.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Astronewb
Pooh-Bah
*****

Reged: 09/19/11

Loc: Connecticut
Re: Nikon D7000 read noise, gain and thermal noise new [Re: harbinjer]
      #5692807 - 02/21/13 01:49 PM

One of the first tasks will be to mod it for H-alpha - I'm dying to know its H-alpha sensitivity

Here's an image of IC's 405 and 410 taken with a modded D5100, 5 minutes subs @ISO 400. They have pretty good dynamic range and low noise, now I want to try it at ISO 250!


IC405/IC410 Crop by Astronewb2011, on Flickr

Cheers,

Paul

Edited by Astronewb (02/21/13 01:50 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: harbinjer]
      #5693476 - 02/21/13 08:18 PM

Yep, discussing DSLR performance is tricky and sometimes heated in many forums. Study on DSLR body is not the sensor itself. (And RAW is not sensor raw!!!) One has to read in between the lines to make sense of it. Unfortunately, it is subject to interpretation.
Anyway, K-01 and K30 use the same sensor, K-5 is tiny, tiny bit improved, but I think it is using software techniques in the body.
K-5's ISO 80, 25600, and 51200 are in "extended" mode, which means (using your imagination,) software techniques.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5693492 - 02/21/13 08:23 PM

Mark,

RE: "Canons do shift the histogram (by variable amount)"
Isn't that the drop the low values first?


About the sensor dynamic range, Nikon follows its D300 tradition (in low-ISO, does a slow speed image acquisition with 14-bit resolution, while using a 12-bit sensor IMX021.) I think this time, it's going to be the same mystery, squeeze IMX071's 12-bit column-parallel ADC to become an effective 14-bit image acquisition.

My gut feeling is either N is using a software technique, post-sensor; or there is really using a (possible) undocumented IMX071 readout mode (non-destructive pixel readout) ( <-- I am guessing here, no solid proof its APS structure has the circuit do that) such that the same pixel got read twice (or more times) to estimate more sensor bits.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5693974 - 02/22/13 03:18 AM

Quote:

Mark,

RE: "Canons do shift the histogram (by variable amount)"
Isn't that the drop the low values first?




No.

Suppose in an example long exposure the thermal noise has a mean of 400 electrons. It will then have a standard deviation of 20 electrons - the square root of 400 (this relationship is a property of the Poisson process of photon arrivals). This means over 99% of the pixel values are within 3 standard deviations of 400 i.e. 99% are in the range 340-460. This electron count is then scaled by the gain and shifted so it's mean is the usual bias for that camera, say 512. It's extremly unlikely for any pixel to end up with a zero value. I don't think I've ever seen a Canon long exposure dark frame with pixels clipped to zero.

By contrast, the Nikon deliberately shifts the distribution to the point where at least 50% of the pixels are clipped to zero. But in typical cases, 80-85% pixels are zero-clipped.

My guess is that Nikon is using the electron counts in the "optical black" border of the sensor to determine the amount to subtract from all the pixels on the chip. The trouble is, the optical black border is the warmest area of the chip from my experiments, so it is has thermal electron counts higher than the rest of the sensor. So if the subtraction level is set so the average level of the pixels in the optical black border, it will be severely destructive to the pixel counts in the rest of the sensor and this is exactly what I am seeing, 80-85% zero-valued pixels in a dark frame.

It's not only darks that are affected. I can think of many real world astro-imaging situations where the D7000 will start pixel-clipping. Now, if you are taking single images or stacking a few frames together the effects may not be noticeable. Unfortunately, once you start stacking 100 sub-exposures of 5minutes together (which I do all the time for a typical image) then these effects will become very obvious.

I've now found this article online where the author(s) describe(s) exactly the same problem with the Nikon D300:
http://theory.uchicago.edu/~ejm/pix/20d/posts/tests/D300_40D_tests/

Their statement on the D300 agrees with my conclusion on the D7000:

"The nonlinear distortion of deep shadows in [Nikon] D300 raw data by clipping means that it will be a poor choice for astrophotography, or any application where pulling weak signals out of the noise at very low illumination levels is of importance."

In fact, the more I think about this the more I become doubtful about purchasing this camera. As far as I can see, this is probably the best sensor out there at the moment on a budget DSLR and I really want to like it. It has the capability of producing fantastic noise-free astro-images. But then the pre-processing of the data by the D7000 compromises it.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5694563 - 02/22/13 12:04 PM

Here's a image that may be of interest. I've taken average statistics from the final 20 dark frames of 5 minutes - from the 2 hour session for estimating dark current. In the image below, the brightness corresponds to the percentage of non-zero pixels.



I've annotated areas where 90%, 80% and 70% of the pixels were clipped to zero. In the corners only 40% were zero.

So this image also gives an indication of where the sensor is warmer i.e. the corners and sides.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5695075 - 02/22/13 04:14 PM

Great post - if anyone's interested in further testing darks, I'm pretty bored over here in Cloudchester!
At my disposal
- D7000
- D5100
- D300
- D600
(sorry, wish I had the 5200)


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5695575 - 02/22/13 08:32 PM

Thanks Hemlock.

The D5100 has the same sensor as the D7000 so I would be interested to see if it behaves the same as the D7000 and if the heat build up is similar. It would need 12-14 darks of 5 minutes at room temperature - each immediately following each other. Raws of course!

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5696805 - 02/23/13 03:16 PM

I'll work on this today, you were testing at ISO200, right?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5696821 - 02/23/13 03:25 PM

Or what is 800, or both? There's a lot of number flying around here and all this math is a little over my head - let me know, and we can figure out a file transfer method later on.

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5697227 - 02/23/13 08:16 PM

Nikon hacker Simeon now has a firmware patcher for D7000 (alpha code) to fetch the sensor data (almost as if its Mode 3), except that the only way to do it is through USB cable (issue a special command). I.e., need a PC.
http://simeonpilgrim.com/nikon-patch/nikon-patch.html

<edit>
The patcher (Dec 2012: patcher v 1.9) currently handles D3100, D5100, and D7000 with various capabilities.
For example, for 5100, NEF is now changed from
- NEF Compressed (approx 16-20MB sized file) to
- NEF UnCompressed (about 33MB per picture file)
- (not yet available) NEF Lossless


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5698158 - 02/24/13 12:07 PM

Quote:

I'll work on this today, you were testing at ISO200, right?




I actually used ISO 800 but ISO 200, 400 or 800 would be fine.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5700566 - 02/25/13 09:01 PM

Quote:

...
Many Review sites are using DCRAW based raw file decoding, while the camera mfg. alter the actual OBP (Optically Black Pixel) values and change these to a mfg supplied fix value to make the picture "clean". There is a new RAW decoding library (looking my note now) which can defeat that...




Found my note: it's LibRAW
See http://www.libraw.org/about#better

I am using RawDigger
http://www.rawdigger.com/
and uncheck the "masked pixels" (OBP) area and inspect data as much as I can figure it out.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5700885 - 02/26/13 01:10 AM

RawDigger shows the same "clipped to zero" pixels across the main sensor area. But in the optically black area it would appear that we are seeing the original, direct off sensor, un-subtracted values.

It might be worth trying LibRaw but I strongly suspect the black level is subtracted, with the resultant clipping, before the raw NEF file is written.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5707207 - 03/01/13 01:39 PM

Ok, I found the previous discussion interesting - at least the parts I could actually understand I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether. This would be great as I'm happy with the camera and if the darks aren't adding any value it would be one less step. BTW, I rarely - if ever - image as warm as 20 degrees Celcius. Usually between -25 and +15 Celcius.

Post Extras: Print Post   Remind Me!   Notify Moderator  
RedLionNJ
scholastic sledgehammer


Reged: 12/29/09

Loc: Red Lion, NJ, USA
Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5707362 - 03/01/13 03:21 PM

I must say, I'm finding this thread fascinating. I currently use a Canon (550D), but find the noise characteristics to be less-than-superlative (esp on a warm night) and am in the process of investigating other models for astro purposes, including the D7000 body. While the post-capture histrionics carried out by the Canons before delivering up a 'raw' image are legendary. I've yet to see any model which produces a true 'raw' output. Reading here about the different-but-equally-annoying Nikon processes is not reassuring me any consumer DSLR can be made to spit out a truly raw image

Grant


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5708193 - 03/02/13 05:17 AM

Quote:

I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether.




The darks might still have some value - they will help cancel out any pixels that have a higher dark current than average (i.e. they tend to stand out in long exposures) but are not obvious enough to be removed by the Hot Pixel Suppression algorithm.

On the other hand, if you are stacking images, sigma stacking will probably remove them just as effectively - as long as you use dithering during image acquisition.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5710386 - 03/03/13 11:01 AM

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5710601 - 03/03/13 12:54 PM

Quote:

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?




Do you mean let the camera perform its own in-camera dark subtraction? This might work but it really does depend on whether the camera subtracts a clipped dark frame or a non-clipped dark frame from the light frame. But even if this approach works, the problem is that it doubles the shooting time, so a 5 minute exposure then takes 10 minutes. From my own point of view, doubling my acquisition time is not a sacrifice I wish to make.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5710689 - 03/03/13 01:48 PM

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5710943 - 03/03/13 04:26 PM

Quote:

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?




That's an insightful question. Unfortunately the answer is: "it depends".

It depends on what the main limiting factor is in your images i.e. what noise source is dominant. In my own case, my main usage is full colour imaging on a fast (F/2.8) scope and I find the noise from the light pollution swamps the thermal noise and read noise (my sky is around mag 5.5 or so). In such situations, the overall quantum efficiency of the sensor is more important than the dark current or the read noise.

But if you are imaging at slower F-ratios, or on a warm night, or have very dark skies then the thermal noise might well be the dominant source of noise and so I would agree it could well be the case that you would need fewer sub-exposures from a low noise camera than you would from a higher thermal noise camera to get the same overall image quality i.e. to achieve the same signal-to-noise ratio in the final stacked image.

As an aside, read noise is not usually a factor because exposure length can be increased until the noise from sky background or from thermal noise dominates the read noise. However if you want to do shorter exposures e.g. for faster moving objects or to create a night sky video then read noise could easily be the dominant factor.

However the question still remains, does Nikon in-camera dark subtraction use a clipped or a non-clipped dark? I can't think of an easy way to test this. If it uses a clipped dark then in-camera dark subtraction doesn't solve the problem of clipping.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5711012 - 03/03/13 05:06 PM

Mark and folks,

I'd suggest wait for few days. Some talent folks in Nikonhacker (Simeon) and others (LibRAW, RawDigger, etc.) are exploring other hidden capabilities in D7000 and D5100. The result may be a completely surprise than the current guessing.

In a 100,000 ft level, I think the approach is trying to undo what the camera body is doing but trying to get the raw image sensor data as-is, or at least understanding the camera processing such that an (sub-)optimal setting can be chosen for astro situations.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5711020 - 03/03/13 05:08 PM

interesting Mark. I think I read when - in the overall process - the in- camera dark subtraction is applied. Unfortunately, I'm away from home and I'm getting only a moment - now and then - to get access to the net. So, no time for research. However, a couple of quick thoughts. It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created. This would be after the image has the in-camera raw subtracted. Anyway, I'll try and do some more research when I return later this week.

Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5711159 - 03/03/13 06:22 PM

Sorry ccs_hello, I must have sent my last post before I read yours. Will wait for the more knowledgable folks to get back to us.

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5711751 - 03/03/13 11:40 PM

There are some nice work on going. I'd like to report what I know on D5100 related:

1. Out of camera it's NEF RAW Compressed. The 14-bit sensor data is first lossy Huffman encoded (0 - 16383 mapped to 3073 levels; on low side, 0 - 449 are 1:1 mapped) then resulted data compressed. For D5100, the size is about 16-22MB. There are two other Nikon methods, both will not use Huffman encoding table (thus gapless and not lossy). Lossless Compressed (i.e., data is compressed) and Uncompressed (not compressed. file size always 33MB). With patcher v1.12, the latter two method can be chosen as well (can pick one and only one.) ** Note in camera setting, do not use RAW+JPEG or it will hang until battery removed.
Note: the latter two are not optimized for speed or space.
Also note: RawDigger currently having difficulty showing NEF Uncompressed due to overflow-beyond spec. But will be fixed at the next release 0.9.15.


2. Can show OBP area if chose such option in patcher. The values are high (about 596-610), I think subtract these by 595 may start to make sense.
BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different. But we know P does some magic work on raw as well.


3. N does R and B channels magnitude scaling (1.4x). There seems to be a method to defeat that. But will have to wait. BTW, P does not do that.


4. The now better HPS (Hot Pixel Suppression) method is doing a great job (looks cleaner) than teh previous HPS (nick named star eater). The patch seems to be completely defeat the HPS. (Still fairly clean.)


5. There seems to be a way to completely defeat the camera processing, but we'll have to wait.

This is my interim report and getting excited.

All credits go to these talented developers.

Links:
http://nikonhacker.com/viewforum.php?f=2
http://simeonpilgrim.com/nikon-patch/nikon-patch.html
http://www.rawdigger.com/news

Clear Skies!

ccs_hello

Edited by ccs_hello (03/04/13 12:23 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5711896 - 03/04/13 02:52 AM

Quote:

It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created.




If the camera is set to perform in-camera dark subtraction but is switched off before the dark exposure has completed (the so-called "Mode 3") then the raw NEF that has been written to the card has both the HPS and the biasing (that causes the clipping) already applied. This might indicate that a processed (and clipped) dark frame is subtracted from a processed (and potentially clipped) light frame. For our purposes, the order in which it performs these operations is crucial.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5713797 - 03/05/13 02:45 AM

Quote:


BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different.




Interesting.
And on the K-01 do you see clipping of long exposure darks in the same way as the Nikon? Or do you see a full "bell curve" when you display the histogram of values?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5715918 - 03/06/13 07:50 AM Attachment (37 downloads)

Showing the clipping effect

IMX071

This post: Nikon (14 bit NEF-Lossless) D5100 ISO800 5 min
Next post: K-01 (12 bit DNG) ISO800 5 min

Note: different Y-axis (pixel count) scale


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5715921 - 03/06/13 07:52 AM Attachment (29 downloads)

IMX071

Histogram: Pentax K-01 (12 bit) ISO800 5 min


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5716925 - 03/06/13 05:40 PM

Those are interesting graphs.

You can definitely see in the Nikon graphs that only the right hand tail of the bell curve exists. The left tail has been clipped to zero.

In the Pentax distribution there is definitely some kind of secondary peak at around 100. I don't know the explanation for that but when you examine the y-axis scale you realise this secondary peak only involves a relatively small percentage of the total pixels - it is a peak that exists in the higher value pixels. The pixel range being displayed (the x-axis) is from 37 to 222. If instead, the x-axis range is set to say 3 to 222 (like the Nikon) you'll again see the main result is a right hand tail of the bell curve i.e. that the Pentax is clipping in a very similar way to the Nikon.

I have seen Nikon D7000 histograms (ISO 800) with a slight secondary peak at around 400 (for a 14bit raw) which corresponds to 100 for a 12 bit raw. I think if you take a 14bit Pentax K-01 raw frame at ISO 800 (instead of 12bit) you will see the secondary peak at around 400, matching the Nikon. All assuming the exposure is kept at 5 minutes.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5717403 - 03/06/13 10:17 PM Attachment (26 downloads)

It would only be fair to add C into the picture.

650D, 5 min, ISO800


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5717603 - 03/07/13 01:11 AM Attachment (12 downloads)

That histogram looks wrong for the Canon - the Canon does not clip its raws (unless the 650D behaves differently). Is it possible the "Subtract Black" option has been selected in RawDigger? This option should be switched off to see the true raw data values.

I've attached my own results for a 5min ISO800 dark frame from my Canon 600D which internally sets its bias level at a value of 2048.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
member


Reged: 12/09/09

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5726479 - 03/11/13 05:59 PM

FYI, I've tested the dark current from optical black region on sensor with the raw hack mode turned on.

http://landingfield.wordpress.com/2013/03/11/imx071-characteristics/
Giving me 0.15eps @ 21C
Doubling temp 6~6.5C


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Plane]
      #5728734 - 03/12/13 06:17 PM

That's a great idea to use the optical black region for calculating the read noise and dark current - I'll give that a try myself.

One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
member


Reged: 12/09/09

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5729103 - 03/12/13 09:50 PM

Quote:

That's a great idea to use the optical black region for calculating the read noise and dark current - I'll give that a try myself.

One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?

Mark




Actually, its an estimation in worst case, the real average value indicates a 0.13eps from calculation and doubling temperature of 6.1C


Post Extras: Print Post   Remind Me!   Notify Moderator  
jfrech14
member
*****

Reged: 01/17/12

Loc: Virginia, USA
Re: Nikon D7000 read noise, gain and thermal noise new [Re: Plane]
      #5745809 - 03/20/13 05:57 PM

My d5100 is full spectrum modified and has a heat reduction system in it and I wanted to test the read noise. I used ImagesPlus to convert my NEF files to 32bit floating FITS and averaged the values of 10 flat frames and 10 bias frames to get average values of Red=1.356 e/pixel, Green=1.350 e/pixel and Blue=1.343 e/pixel with an average gain of about .06-.1 e/ADU. These were at ISO1600. Are these good values? Can you give me any insight to further analyze my camera because my images are so noisy and I am trying to find out if its my camera or my processing. Thanks

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: jfrech14]
      #5748008 - 03/21/13 04:47 PM

Quote:

Are these good values? Can you give me any insight to further analyze my camera because my images are so noisy and I am trying to find out if its my camera or my processing. Thanks




It's impossible to tell from the numbers you give. There are too many unknowns - for instance what scaling might ImagesPlus have performed when converting from 14bit integers to 32 bit float? A gain of 0.06-0.1 e/ADU is certainly not sounding right for ISO 1600.

I suggest you approach the problem in a different way. You don't say what length sub-exposures and what ISO you use - let's assume it is 5 minutes at ISO 1600.

There are 3 sources of noise: read noise, thermal noise and sky background glow. Let's call them r, t and b. I'm now going to assume a basic level of maths. These 3 sources of noise combine together in quadrature to give a total noise like this:
total_noise = square_root( r*r + t*t + b*b)

It is really straightforward to get relative estimates of each of these 3 sources of noise to see which one is the biggest.

Take 2 bias frames with the lens cap on i.e. both frames at 1/4000s at ISO 1600.
Using your favourite astro processing software load the two images and subtract one from the other (it might be necessary to add a fixed level e.g. 1000) to the first image to prevent values going negative (or being truncated) during subtraction. Now select an area of the resulting data somewhere near the middle of the frame (say 100x100 pixels) and let the software claculate the standard deviation. This is your read noise r. thius noise is not stated in any sensible units but it doesn't matter because everything else we do will be in the same units.

Now take 2 dark frames with the same ISO 1600 and 5 minutes long. It's best to let the camera cool to its normal outside imaging temperature before you do this. Again subtract one from the other (maybe with a fixed offset) exactly as you did before and calculate the standard deviation of an area near the middle. Call this d (for dark frame). Now this frame contains read noise and thermal noise. However, we already know the read noise.

The thermal noise is then given by:
t = square_root(d*d - r*r)

You say your camera is cooled in some way? If so, I would expect thermal noise to be less than read noise.

Now for the background sky glow. Attach the camera to the scope and point at a dark area of sky with as few stars as possible and no nebulosity. Make sure tracking is switched on so you don't get trailing of stars. Take 2 frames with the same ISO 1600 and 5 minutes long. Then do subtraction of the 2 frames just like before and calculate the standard deviation of an area somewhere near the middle with no hint of stars i.e. just sky background. Call this standard deviation m, it's the light frame standard deviation. This frame contains read noise, thermal noise and background sky noise. We already know the read and thermal noises.

The background sky noise is given by:
b = square_root(m*m - d*d)

There you have it - you have just calculated the relative sizes of the read noise, thermal noise and sky background noise. My guess is that unless you have pristine dark sky the background sky noise will be the largest noise source by a long way.

I'll be interested to see your results.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5753069 - 03/24/13 12:19 AM

Hi Mark,

I came across this interesting thread and would like to hear opinion from you. Additional to the various problem mentioned here, I also noticed that the raw files from D800 is nonlinear.

These images can demonstrate what I meant:
http://www3.asiaa.sinica.edu.tw/~whwang/misc/D800_flat_comparison.jpg
The result of flat fielding changes with the amount of exposure in the flat (all unsaturated). This should not happen, and indeed this was not the case on Canon cameras. I can use any unsaturated flat images to get perfect flat field results on my 5D2, regardless the ISO and exposure in the flats. The residual pattern in the top image is very similar to the flat field pattern. This strongly indicates that the residual pattern is a flat field effect, i.e., nonlinarity in the images. (FYI, the images are processed in DeepSkyStacker, which is based on DCRAW.)

The notorious clip at zero made me wonder if this nonlinear behavior has anything to do with the clip. So I decoded the raw files by myself (using DCRAW) and applied various offsets to the light, dark, and flat frames, and then re-process the images. This doesn't help to solve the problem no matter what offsets I adopted. I then apply simple gamma adjustments to the images to see if the nonlinear behavior is gamma-like. The result is that this still doesn't solve the problem.

I then shoot the wall in my house with various exposure times to test the linearity. Unfortunately the light source in my house is not perfect for this purpose. I also tried sunlight, but it is hard to get a perfect sunny day here with absolutely stable sunlight. With such difficult conditions, I still cannot get the data to properly measure the nonlinearity. However, the imperfect data strongly support that there is nonlinearity in the NEF files.

I had contacted Roger Clark about this, and he kindly offered help to analyze the files, once I get the proper test images. I will also analyze the images by myself. I will keep people updated about this here. If you have any insight into this, please let me know. I would also like to use this opportunity to warm people that there might be nonlinearity in other Nikon DSLRS, and this is worth looking into.

Finally, here are my measurements on the gain, dark, and read noise of D800:
http://alohaphotolog.blogspot.tw/2013/01/nikon-d800-performance-test.html
This is just a note for myself. So the explanation there may not be enough for you to fully understand what I was talking about. I apologize for this. Maybe in the future I can share more about this.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5753149 - 03/24/13 02:01 AM

Wei-How,

As plane reported, it is easier on testing D5100. There is a firmware hack which basically turn on camera's service mode which (1) disable N's HPS (Hot Pixel Suppression) algorithm, and (2) show both dummy pixels and OBP (Optical Black Pixel). Also NEF can be Lossless Compressed type (i.e., no more Huffman table used in RAW coding).

However, even with that, the main image area still
- uses grater than zero bias (i.e., not true sensor characteristics shown), and
- R-ch and B-ch with a multiplier always applied

I somehow doubt OBP values are involved in calculating bias. Zero clipping is.

I somehow trust libraw decoding than dcraw library for its faithful decoding and see OBP area (see "better" linkie. Rawdigger IMHO is very useful.
Later in future version 0.9.16 or 0.9.17 will be able to save the RAW output as TIFF.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5753711 - 03/24/13 11:18 AM

Hi Wei-How,

What you describe sounds very strange. I wonder if something on the camera or something in your processing sequence is apply a gamma curve or similar?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5753773 - 03/24/13 11:45 AM

Right. That's indeed very strange.

As I mentioned, I wondered if there is a mis-applied gamma curve. So I applied various different gamma back to the flat and light frames, but this does not help to solve the problem. My work flow in DeepSkyStacker is exactly the same as the one I used for Canon 5D2 and it worked perfectly there. So the problem is very likely in the camera.

Is there any uncommon camera menu item that may apply a gamma to the images? I tried to make things as raw as possible when I setup the camera. But I may have missed something.

Once I got the right light source, I will take the images for linearity test. We may learn something there. On the other hand, if anyone has a recent Nikon DSLR and has some proper light source, please also help to verify whether there is such a problem on other Nikon models.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5755382 - 03/25/13 01:36 AM

Hi Wei-Hao,

I've thought about this carefully and have come to the conclusion that the problem you are seeing cannot be caused by a response linearity in the camera. It is nothing to do with the Nikon black level clipping, either. You need to carefully look at your flats - ensure that no pixel is saturated and then look at your processing sequence - try performing the calibration operations (bias, flats, darks) manually instead of using DeepSkyStacker. I always use IRIS myself but it's not to everyone's taste. It might give a clue as to what is happening.

If that doesn't work, I'm happy to check it myself if you make available to me the flats, and a bias, dark and light.

Regards,

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5755396 - 03/25/13 01:53 AM

Hi Mark,

Because DSS works so perfectly on 5D2, it is hard to convince myself it is the culprit. I haven't had a chance to write my own program to dark-subtract and flat these files. Too busy these days. But if you like, it will be great if you can process the files with different tools. We will learn something from it.

Please ftp to ftp.asiaa.sinica.edu.tw and go to the directory upload/whwang/D800test .

Here are what the files are for.
#281-292: bias at ISO 200 (for flat)
#384-393: bias at ISO 800 (for light and dark)
#1043-1058: ISO 200 flat (the exposure time is determined by the camera's meter, so they should not be saturated).
#1147-1158: light frame for M13 at ISO 800
#1141-1146 and #1179-1182: dark frames taken before and after M13, at ISO800.

After I stacked the images and stretched them strongly, I can see residual patterns caused by imperfect flat fielding. Please let me know if you also see this or anything you notice.

ps, others are also welcome to play with the files. They will be removed in a week or two automatically.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5756751 - 03/25/13 06:24 PM

Hi Wei-Hao,

I've taken a look at a few of your raws - very nice M13 with very tight stars!

What I noticed is that the flats are very severely underexposed - maximum pixel value was around 600 (out of a possible maxiumum of around 16,000 for a 14bit image). If this is typical of your flat frames then there is a possible explanation. Just as yoususpected, the response curve of a DSLR sensor is not actually completely linear - there is usually some kind of non-linearity down at the "dark end" of the curve. The values I was seeing in your flat frame puts it down in that non-linear section of the response curve. Adding 1 or 2 stops to the exposure would push it back up into the linear part of the curve and the flats would perform better - this corresponds well with the behaviour you are seeing.

You need to investigate why you have such a severe degree of underexposure - I would recommend that the pixel values in the middle of the flat frame exposure should be around 10,000-12,000 i.e. very firmly in the linear part of the response curve.

Non-linearity down at the lower pixel levels is a well-known phenomenon. If you want to test linearity for yourself, then plot the square of the standard deviation against the mean level - choose an area near the middle of the frame and use a single colour channel e.g. R,B,G1 or G2. Do this over a large range of exposures. It is also best to use the difference between 2 frames at each exposure for calculating standard deviation because this removes discrepancies caused by PRNU (Pixel Response Non Uniformity).

Hope this helps! Thanks for drawing this interesting problem to my attention.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5757109 - 03/25/13 10:05 PM

Hi Mark,

The exposure is determined by the camera. The raw files may look very dark, but the JPG files all have the "correct" brightness (in the middle of the histogram shown on the camera's LCD picture info). So they are by no means under-exposed.

On the other hand, I did suspect about the amount of exposure. So, on the same day, I took flats of different exposures. I have flats that are 2 stops over exposed than these, and also flats that are 2 stops under exposed than these. None of them can perfectly flatten the light frames. The results look differently, but none of them are flat.

Also for your info, if you stack the M13 images and stretch them very strongly, you will see that the effect of imperfect flat does not only appear in the dark corners, it also appear near the center of the image where the exposure is high and where the images are already flatter to start with. This is the case on the results produced with various flat exposures. (Vary bad, right?)

At the very beginning, I also wondered if the nonlinearity only appears in the dark part. At that time, I used an F2.8 optics, which is fast, but also has much stronger vignetting at the corner. With this F2.8 scope, I shoot my light frames at ISO800, and the JPG histogram looked over exposed (by roughly 2 stops). I shoot my flat at ISO100, and over exposed them by 2.5 stops. It turns out, this combination produces very flat results. It's still not perfectly flat, but the residue is already at the level of sky gradient in stacked 1-hr pictures, so there is less need for further improvement. This made me think that the nonlinearity is at the darker part. Then, I took the set of M13 images (the ones you have) with an F7 scope. The M13 light frames at ISO 800 have under-exposed JPG histograms, and as I mentioned, there are various kinds of flats taken at ISO200. Unfortunately, none of them produces a flat results.

Combining all the above examples and some linearity tests that may not be properly executed, I tend to believe the problem is not simply linearity in the dark part. The nonlinear behavior may be ISO-dependent, and the part it affects (darker or brighter parts of the image) may also vary from ISO to ISO.

I am now looking for some good light source in our lab, and hope I can find time in April to do some proper tests.

Finally, talking about something slightly different. You mentioned using the difference of 2 frames. I know many people do this, but I think there are better methods. The 2-frame approach subtracts them from each other, and then measures the mean and stddev across different pixels in the subtracted frame. In my approach, I use N frames (N>10), and do not subtract frames from each other at all. For any given pixel, I measure its mean and stddev across the N frames. If there are M (M = 36Mega for D800) pixels in an image, I repeat this measurement M times on all pixels, and I end up with M measurements of means and stddevs. This can give me the behavior of every single pixel, and I believe this is a better way to account for any non-uniformity across the pixels (non-uniformity in QE, dark, readnoise, bias etc). For your reference, this is what I did to 5D2 using this method a while ago:
http://www3.asiaa.sinica.edu.tw/~whwang/misc/Canon5D2.pdf
The results are not too different from other people's measurements based on the 2-frame approach, but this is in principle a more accurate method.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5757382 - 03/26/13 01:41 AM

Quote:

The exposure is determined by the camera. The raw files may look very dark, but the JPG files all have the "correct" brightness (in the middle of the histogram shown on the camera's LCD picture info). So they are by no means under-exposed.




All I can do is to repeat what I said earlier. The pixel values peak at around 600 instead of approx 16,000. Judge for yourself whether or not that is underexposure.

By the way, the camera histogram is not a good judge, at least not until you have calibrated it against actual pixel values - you'll find the in-camera histogram does not use a linear scale.

I love your method of using multiple frames for determining the characteristics of the sensor down to the level of each individual pixel. I will certainly do some experiments like this. It's a laborious way of plotting the whole response curve though!

Regards,

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5757429 - 03/26/13 02:48 AM

Hi Mark,

Thank you. After you pointed this to me again, I realize something is going on. Using frame #1043 for example. This is how it looks like in Adobe's camera raw window, without changing any default parameters:
http://www3.asiaa.sinica.edu.tw/~whwang/misc/1043raw.png

It is not dark at all. The histogram shows that it is properly exposed, consistent with what I saw on the camera when I shoot it. The camera believe this is the right exposure for a proper image quality at this ISO.

On the other hand, I also converted the raw file into TIFF using dcraw. Without interpolating the Bayer array and any additional adjustment, the image has a mean value of 620, which sounds way too low. It is hard to believe this is a terribly under-exposed picture, otherwise everyone in the world will complain that D800 has low S/N even at ISO200. It is more likely that, for some reason, the raw file intrinsically has unusually low pixel values for a picture that is believed by the camera to be properly exposed.

This is new thing for me. It may have something to do with the nonlinearity. I will keep investigating this.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5757499 - 03/26/13 04:37 AM

The EXIF should indicate which camera settings were being used. This might give an explanation.

Mark

[Later edit: I've just taken a look at the EXIF - the exposure mode was Manual at 1/50sec ]

Edited by sharkmelley (03/26/13 03:54 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Jerry Lodriguss
Vendor


Reged: 07/19/08

Loc: Voorhees, NJ
Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5758610 - 03/26/13 04:35 PM

Hi Wei-Hao,

The histogram on the back of the camera and in Adobe Camera Raw and in Canon's DPP all show pixel values for the image after a typical "digital development" non-linear curve stretch has been applied.

The histogram on the back of the camera is actually from the stretched JPEG that has been processed with the camera settings inside the camera, so settings like contrast in the camera will affect the histogram.

Those pixel values are not raw linear values.

People who use these cameras for normal daytime photography want to know what the image will look like after the image has been processed so that the linear data is stretched to more closely approximate what human vision expects, which is closer to logarithmic (but not exactly log).

These histograms show them what the data looks like, and if the image is correctly exposed, after the non-linear stretch is applied.

Note that even with Images Plus Raw Development, the 14 bit data will be stretched to 16 bits unless you use Bayer Basic Raw as the development type.

It sounds like you need a lot more exposure for your flats.

But this still might not fix the problem if something else is going on.

Jerry


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5849039 - 05/08/13 11:08 PM

More on SONY Exmor CMOS image sensors:

From DxOMark
DxOMark-Reviews DxOMark-Camera-Sensor2.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6004723 - 08/03/13 12:11 PM

Today Auguest 3rd in fleaBay, there is a sale on factory refurb D5100 with 18-55 VR kit lens for $400 (180 day warranty).

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6020691 - 08/12/13 12:33 PM

Today (August 12) the price is ridiculously low (now $370).

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6393846 - 02/24/14 08:50 AM Attachment (3 downloads)

Saw it in the most recent Nikonhacker (beta patch)

D7100, D300s, and D300's "star eater" algorithm can be turned off.

See: http://simeonpilgrim.com/nikon-patch/nikon-patch-beta.html

All credits go to these talented Nikon Hackers!


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6394952 - 02/24/14 08:21 PM Attachment (6 downloads)

Add a histogram: black clipping is not part of the HPS fix (it should be unrelated anyway.) It seems to be the clipping point is a bit different than "this post" ( <-- which is pre- HPS fix).
There are more non-zero pixels, as expected.

Note: this histogram removes zero-value pixels.
HPS disabled.


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6395161 - 02/24/14 10:48 PM Attachment (5 downloads)

I think I've found a clue on D5100 (IMX071. It is doing in-camera "black level subtract" and the result is also zero shifted a tiny bit ( < 3 ADU) to remove residual value. While the new HPS is an improved version of spatial LPF.

Here is the analysis of an Oly E-PM2 which is using the "baby IMX071": the IMX109. This mirrorless stores (reasonably) raw as is in SD card.

Now with external software decoder (I use RawDigger) to perform raw file analysis:

First pic: without black level subtract


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6395163 - 02/24/14 10:49 PM Attachment (5 downloads)

Second pic: with black subtract and ADU = 0 pixels clipped:

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6582190 - 06/14/14 03:20 PM Attachment (6 downloads)

Add the new result when D5100 sensor's raw output is captured - without HPS,
- without black-clip (offset is about 425)
- without R and B channels scaling

Thanks for CN member Plain's great work on special firmware.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6582200 - 06/14/14 03:29 PM Attachment (4 downloads)

If I perform a black clipping at value = 425 and make sure X-axis is comparable, you'll notice thee sensor output is further spread out.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6582219 - 06/14/14 03:43 PM Attachment (1 downloads)

Pictures shown are not deBayered. Display format is RGBG sensel-for-pixel composite.

Left pic is the sensor raw with black-clip set at 425. <-- Sensor raw patch
Right pic is NEF output (without HPS applied.) <-- Std Nikonhacker patch w/ HPS disabled

I used higher gamma to make low-level values stand out for illustration purpose.

(On right picture) It appears that other than HPS algorithm, there is still some sort of image processing going on in NEF, even when HPS is disabled.


Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
member


Reged: 12/09/09

Read noise new [Re: ccs_hello]
      #6582228 - 06/14/14 03:47 PM

Tested read noise at ISO 320, where gain is 0.89e/ADU



Note that for CCD with only one readout amplifier, read noise should be identical for all pixels. Thus RMS is sufficient to characterize it. But for cmos sensor, read noise is a distribution since each pixel has its own read out circuit.

More information on my blog.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
member


Reged: 12/09/09

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #6582254 - 06/14/14 04:00 PM

Hi ccs_hello,

The black point is always at 128 in the patch. 425 is the result of your dark current accumulation.

As for preprocessing, the evidence we had now is the camera did preprocessing on the fly. There is a switch that controls the individual processing step. This includes R/B channel scaling, dark current subtraction, optical black pixel smoothing and something we do not well understand.

Dark current subtraction is the one that clip the average dark count to zero in the stock firmware. The average dark current is calculated from optical black pixels which is smoothed first to get rid of hot pixels.

Currently, all the preprocessing is off and what you get is sensor raw data.

Best,
astronomer (Nikonhacker)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | (show all)


Extra information
5 registered and 29 anonymous users are browsing this forum.

Moderator:  Dave M, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 4819

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics