Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
· Get a Cloudy Nights T-Shirt · Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu… uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | (show all)
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5693974 - 02/22/13 03:18 AM

Quote:

Mark,

RE: "Canons do shift the histogram (by variable amount)"
Isn't that the drop the low values first?




No.

Suppose in an example long exposure the thermal noise has a mean of 400 electrons. It will then have a standard deviation of 20 electrons - the square root of 400 (this relationship is a property of the Poisson process of photon arrivals). This means over 99% of the pixel values are within 3 standard deviations of 400 i.e. 99% are in the range 340-460. This electron count is then scaled by the gain and shifted so it's mean is the usual bias for that camera, say 512. It's extremly unlikely for any pixel to end up with a zero value. I don't think I've ever seen a Canon long exposure dark frame with pixels clipped to zero.

By contrast, the Nikon deliberately shifts the distribution to the point where at least 50% of the pixels are clipped to zero. But in typical cases, 80-85% pixels are zero-clipped.

My guess is that Nikon is using the electron counts in the "optical black" border of the sensor to determine the amount to subtract from all the pixels on the chip. The trouble is, the optical black border is the warmest area of the chip from my experiments, so it is has thermal electron counts higher than the rest of the sensor. So if the subtraction level is set so the average level of the pixels in the optical black border, it will be severely destructive to the pixel counts in the rest of the sensor and this is exactly what I am seeing, 80-85% zero-valued pixels in a dark frame.

It's not only darks that are affected. I can think of many real world astro-imaging situations where the D7000 will start pixel-clipping. Now, if you are taking single images or stacking a few frames together the effects may not be noticeable. Unfortunately, once you start stacking 100 sub-exposures of 5minutes together (which I do all the time for a typical image) then these effects will become very obvious.

I've now found this article online where the author(s) describe(s) exactly the same problem with the Nikon D300:
http://theory.uchicago.edu/~ejm/pix/20d/posts/tests/D300_40D_tests/

Their statement on the D300 agrees with my conclusion on the D7000:

"The nonlinear distortion of deep shadows in [Nikon] D300 raw data by clipping means that it will be a poor choice for astrophotography, or any application where pulling weak signals out of the noise at very low illumination levels is of importance."

In fact, the more I think about this the more I become doubtful about purchasing this camera. As far as I can see, this is probably the best sensor out there at the moment on a budget DSLR and I really want to like it. It has the capability of producing fantastic noise-free astro-images. But then the pre-processing of the data by the D7000 compromises it.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5694563 - 02/22/13 12:04 PM

Here's a image that may be of interest. I've taken average statistics from the final 20 dark frames of 5 minutes - from the 2 hour session for estimating dark current. In the image below, the brightness corresponds to the percentage of non-zero pixels.



I've annotated areas where 90%, 80% and 70% of the pixels were clipped to zero. In the corners only 40% were zero.

So this image also gives an indication of where the sensor is warmer i.e. the corners and sides.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5695075 - 02/22/13 04:14 PM

Great post - if anyone's interested in further testing darks, I'm pretty bored over here in Cloudchester!
At my disposal
- D7000
- D5100
- D300
- D600
(sorry, wish I had the 5200)


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5695575 - 02/22/13 08:32 PM

Thanks Hemlock.

The D5100 has the same sensor as the D7000 so I would be interested to see if it behaves the same as the D7000 and if the heat build up is similar. It would need 12-14 darks of 5 minutes at room temperature - each immediately following each other. Raws of course!

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5696805 - 02/23/13 03:16 PM

I'll work on this today, you were testing at ISO200, right?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Hemlock
super member


Reged: 02/09/12

Loc: Finger Lakes, New York
Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5696821 - 02/23/13 03:25 PM

Or what is 800, or both? There's a lot of number flying around here and all this math is a little over my head - let me know, and we can figure out a file transfer method later on.

Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5697227 - 02/23/13 08:16 PM

Nikon hacker Simeon now has a firmware patcher for D7000 (alpha code) to fetch the sensor data (almost as if its Mode 3), except that the only way to do it is through USB cable (issue a special command). I.e., need a PC.
http://simeonpilgrim.com/nikon-patch/nikon-patch.html

<edit>
The patcher (Dec 2012: patcher v 1.9) currently handles D3100, D5100, and D7000 with various capabilities.
For example, for 5100, NEF is now changed from
- NEF Compressed (approx 16-20MB sized file) to
- NEF UnCompressed (about 33MB per picture file)
- (not yet available) NEF Lossless


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Hemlock]
      #5698158 - 02/24/13 12:07 PM

Quote:

I'll work on this today, you were testing at ISO200, right?




I actually used ISO 800 but ISO 200, 400 or 800 would be fine.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5700566 - 02/25/13 09:01 PM

Quote:

...
Many Review sites are using DCRAW based raw file decoding, while the camera mfg. alter the actual OBP (Optically Black Pixel) values and change these to a mfg supplied fix value to make the picture "clean". There is a new RAW decoding library (looking my note now) which can defeat that...




Found my note: it's LibRAW
See http://www.libraw.org/about#better

I am using RawDigger
http://www.rawdigger.com/
and uncheck the "masked pixels" (OBP) area and inspect data as much as I can figure it out.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5700885 - 02/26/13 01:10 AM

RawDigger shows the same "clipped to zero" pixels across the main sensor area. But in the optically black area it would appear that we are seeing the original, direct off sensor, un-subtracted values.

It might be worth trying LibRaw but I strongly suspect the black level is subtracted, with the resultant clipping, before the raw NEF file is written.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5707207 - 03/01/13 01:39 PM

Ok, I found the previous discussion interesting - at least the parts I could actually understand I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether. This would be great as I'm happy with the camera and if the darks aren't adding any value it would be one less step. BTW, I rarely - if ever - image as warm as 20 degrees Celcius. Usually between -25 and +15 Celcius.

Post Extras: Print Post   Remind Me!   Notify Moderator  
RedLionNJ
scholastic sledgehammer


Reged: 12/29/09

Loc: Red Lion, NJ, USA
Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5707362 - 03/01/13 03:21 PM

I must say, I'm finding this thread fascinating. I currently use a Canon (550D), but find the noise characteristics to be less-than-superlative (esp on a warm night) and am in the process of investigating other models for astro purposes, including the D7000 body. While the post-capture histrionics carried out by the Canons before delivering up a 'raw' image are legendary. I've yet to see any model which produces a true 'raw' output. Reading here about the different-but-equally-annoying Nikon processes is not reassuring me any consumer DSLR can be made to spit out a truly raw image

Grant


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5708193 - 03/02/13 05:17 AM

Quote:

I have a D5100 and from what I am reading the darks would not be valid. Therefore, should I just skip the darks altogether.




The darks might still have some value - they will help cancel out any pixels that have a higher dark current than average (i.e. they tend to stand out in long exposures) but are not obvious enough to be removed by the Hot Pixel Suppression algorithm.

On the other hand, if you are stacking images, sigma stacking will probably remove them just as effectively - as long as you use dithering during image acquisition.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise [Re: sharkmelley]
      #5710386 - 03/03/13 11:01 AM

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise [Re: TimN]
      #5710601 - 03/03/13 12:54 PM

Quote:

So, If the problem is mainly clipping in darks and bias frames what about taking the darks in-camera after taking the subs. I think the dark is processed against the light before the algorithm is applied and a raw produced. Is this correct?




Do you mean let the camera perform its own in-camera dark subtraction? This might work but it really does depend on whether the camera subtracts a clipped dark frame or a non-clipped dark frame from the light frame. But even if this approach works, the problem is that it doubles the shooting time, so a 5 minute exposure then takes 10 minutes. From my own point of view, doubling my acquisition time is not a sacrifice I wish to make.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise [Re: sharkmelley]
      #5710689 - 03/03/13 01:48 PM

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise [Re: TimN]
      #5710943 - 03/03/13 04:26 PM

Quote:

Yes, but based on your observations of the reduced noise - read and thermal - wouldn't you need less light images to achieve the same results as a camera with significantly more noise? If so, I just wonder if there is that much more imaging time required - at least until we get a suitable hack?




That's an insightful question. Unfortunately the answer is: "it depends".

It depends on what the main limiting factor is in your images i.e. what noise source is dominant. In my own case, my main usage is full colour imaging on a fast (F/2.8) scope and I find the noise from the light pollution swamps the thermal noise and read noise (my sky is around mag 5.5 or so). In such situations, the overall quantum efficiency of the sensor is more important than the dark current or the read noise.

But if you are imaging at slower F-ratios, or on a warm night, or have very dark skies then the thermal noise might well be the dominant source of noise and so I would agree it could well be the case that you would need fewer sub-exposures from a low noise camera than you would from a higher thermal noise camera to get the same overall image quality i.e. to achieve the same signal-to-noise ratio in the final stacked image.

As an aside, read noise is not usually a factor because exposure length can be increased until the noise from sky background or from thermal noise dominates the read noise. However if you want to do shorter exposures e.g. for faster moving objects or to create a night sky video then read noise could easily be the dominant factor.

However the question still remains, does Nikon in-camera dark subtraction use a clipped or a non-clipped dark? I can't think of an easy way to test this. If it uses a clipped dark then in-camera dark subtraction doesn't solve the problem of clipping.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise [Re: sharkmelley]
      #5711012 - 03/03/13 05:06 PM

Mark and folks,

I'd suggest wait for few days. Some talent folks in Nikonhacker (Simeon) and others (LibRAW, RawDigger, etc.) are exploring other hidden capabilities in D7000 and D5100. The result may be a completely surprise than the current guessing.

In a 100,000 ft level, I think the approach is trying to undo what the camera body is doing but trying to get the raw image sensor data as-is, or at least understanding the camera processing such that an (sub-)optimal setting can be chosen for astro situations.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise [Re: sharkmelley]
      #5711020 - 03/03/13 05:08 PM

interesting Mark. I think I read when - in the overall process - the in- camera dark subtraction is applied. Unfortunately, I'm away from home and I'm getting only a moment - now and then - to get access to the net. So, no time for research. However, a couple of quick thoughts. It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created. This would be after the image has the in-camera raw subtracted. Anyway, I'll try and do some more research when I return later this week.

Post Extras: Print Post   Remind Me!   Notify Moderator  
TimN
Pooh-Bah


Reged: 04/20/08

Loc: Ontario, Canada
Re: Nikon D7000 read noise, gain and thermal noise [Re: TimN]
      #5711159 - 03/03/13 06:22 PM

Sorry ccs_hello, I must have sent my last post before I read yours. Will wait for the more knowledgable folks to get back to us.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | (show all)


Extra information
17 registered and 33 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 4764

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics