Return to the Cloudy Nights Telescope Reviews home pageAstronomics discounts for Cloudy Nights members
Get a Cloudy Nights T-Shirt Submit a Review / Article

Click here if you are having trouble logging into the forums

Privacy Policy | Please read our Terms of Service | Signup and Troubleshooting FAQ | Problems? PM a Red or a Green Gu uh, User

Astrophotography and Sketching >> DSLR & Digital Camera Astro Imaging & Processing

Pages: 1 | 2 | 3 | 4 | (show all)
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5711751 - 03/03/13 11:40 PM

There are some nice work on going. I'd like to report what I know on D5100 related:

1. Out of camera it's NEF RAW Compressed. The 14-bit sensor data is first lossy Huffman encoded (0 - 16383 mapped to 3073 levels; on low side, 0 - 449 are 1:1 mapped) then resulted data compressed. For D5100, the size is about 16-22MB. There are two other Nikon methods, both will not use Huffman encoding table (thus gapless and not lossy). Lossless Compressed (i.e., data is compressed) and Uncompressed (not compressed. file size always 33MB). With patcher v1.12, the latter two method can be chosen as well (can pick one and only one.) ** Note in camera setting, do not use RAW+JPEG or it will hang until battery removed.
Note: the latter two are not optimized for speed or space.
Also note: RawDigger currently having difficulty showing NEF Uncompressed due to overflow-beyond spec. But will be fixed at the next release 0.9.15.


2. Can show OBP area if chose such option in patcher. The values are high (about 596-610), I think subtract these by 595 may start to make sense.
BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different. But we know P does some magic work on raw as well.


3. N does R and B channels magnitude scaling (1.4x). There seems to be a method to defeat that. But will have to wait. BTW, P does not do that.


4. The now better HPS (Hot Pixel Suppression) method is doing a great job (looks cleaner) than teh previous HPS (nick named star eater). The patch seems to be completely defeat the HPS. (Still fairly clean.)


5. There seems to be a way to completely defeat the camera processing, but we'll have to wait.

This is my interim report and getting excited.

All credits go to these talented developers.

Links:
http://nikonhacker.com/viewforum.php?f=2
http://simeonpilgrim.com/nikon-patch/nikon-patch.html
http://www.rawdigger.com/news

Clear Skies!

ccs_hello

Edited by ccs_hello (03/04/13 12:23 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: TimN]
      #5711896 - 03/04/13 02:52 AM

Quote:

It would make no sense for Nikon to apply the dark subtraction already clipped - for the reasons you have already mentioned. Also, it makes sense to apply the Raw algorithm at the time the raw is created. Since Nikon, I believe, only creates one Raw for 1 image plus it's related in camera dark, it makes sense to apply the algorithm - clipping included - at the time the raw is created.




If the camera is set to perform in-camera dark subtraction but is switched off before the dark exposure has completed (the so-called "Mode 3") then the raw NEF that has been written to the card has both the HPS and the biasing (that causes the clipping) already applied. This might indicate that a processed (and clipped) dark frame is subtracted from a processed (and potentially clipped) light frame. For our purposes, the order in which it performs these operations is crucial.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5713797 - 03/05/13 02:45 AM

Quote:


BTW, I have a K-01 (same sensor IMX071), the OBP values are quite different.




Interesting.
And on the K-01 do you see clipping of long exposure darks in the same way as the Nikon? Or do you see a full "bell curve" when you display the histogram of values?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5715918 - 03/06/13 07:50 AM Attachment (34 downloads)

Showing the clipping effect

IMX071

This post: Nikon (14 bit NEF-Lossless) D5100 ISO800 5 min
Next post: K-01 (12 bit DNG) ISO800 5 min

Note: different Y-axis (pixel count) scale


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5715921 - 03/06/13 07:52 AM Attachment (27 downloads)

IMX071

Histogram: Pentax K-01 (12 bit) ISO800 5 min


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5716925 - 03/06/13 05:40 PM

Those are interesting graphs.

You can definitely see in the Nikon graphs that only the right hand tail of the bell curve exists. The left tail has been clipped to zero.

In the Pentax distribution there is definitely some kind of secondary peak at around 100. I don't know the explanation for that but when you examine the y-axis scale you realise this secondary peak only involves a relatively small percentage of the total pixels - it is a peak that exists in the higher value pixels. The pixel range being displayed (the x-axis) is from 37 to 222. If instead, the x-axis range is set to say 3 to 222 (like the Nikon) you'll again see the main result is a right hand tail of the bell curve i.e. that the Pentax is clipping in a very similar way to the Nikon.

I have seen Nikon D7000 histograms (ISO 800) with a slight secondary peak at around 400 (for a 14bit raw) which corresponds to 100 for a 12 bit raw. I think if you take a 14bit Pentax K-01 raw frame at ISO 800 (instead of 12bit) you will see the secondary peak at around 400, matching the Nikon. All assuming the exposure is kept at 5 minutes.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5717403 - 03/06/13 10:17 PM Attachment (23 downloads)

It would only be fair to add C into the picture.

650D, 5 min, ISO800


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5717603 - 03/07/13 01:11 AM Attachment (9 downloads)

That histogram looks wrong for the Canon - the Canon does not clip its raws (unless the 650D behaves differently). Is it possible the "Subtract Black" option has been selected in RawDigger? This option should be switched off to see the true raw data values.

I've attached my own results for a 5min ISO800 dark frame from my Canon 600D which internally sets its bias level at a value of 2048.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
journeyman


Reged: 12/09/09

Re: Nikon D7000 read noise, gain and thermal noise [Re: sharkmelley]
      #5726479 - 03/11/13 05:59 PM

FYI, I've tested the dark current from optical black region on sensor with the raw hack mode turned on.

http://landingfield.wordpress.com/2013/03/11/imx071-characteristics/
Giving me 0.15eps @ 21C
Doubling temp 6~6.5C


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: Plane]
      #5728734 - 03/12/13 06:17 PM

That's a great idea to use the optical black region for calculating the read noise and dark current - I'll give that a try myself.

One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Plane
journeyman


Reged: 12/09/09

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5729103 - 03/12/13 09:50 PM

Quote:

That's a great idea to use the optical black region for calculating the read noise and dark current - I'll give that a try myself.

One question: if you calculated a dark current of 0.007eps at -1C then doesn't 0.15eps at 21C seems a little high if the doubling temp is 6-6.5C?

Mark




Actually, its an estimation in worst case, the real average value indicates a 0.13eps from calculation and doubling temperature of 6.1C


Post Extras: Print Post   Remind Me!   Notify Moderator  
jfrech14
member
*****

Reged: 01/17/12

Loc: Virginia, USA
Re: Nikon D7000 read noise, gain and thermal noise new [Re: Plane]
      #5745809 - 03/20/13 05:57 PM

My d5100 is full spectrum modified and has a heat reduction system in it and I wanted to test the read noise. I used ImagesPlus to convert my NEF files to 32bit floating FITS and averaged the values of 10 flat frames and 10 bias frames to get average values of Red=1.356 e/pixel, Green=1.350 e/pixel and Blue=1.343 e/pixel with an average gain of about .06-.1 e/ADU. These were at ISO1600. Are these good values? Can you give me any insight to further analyze my camera because my images are so noisy and I am trying to find out if its my camera or my processing. Thanks

Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: jfrech14]
      #5748008 - 03/21/13 04:47 PM

Quote:

Are these good values? Can you give me any insight to further analyze my camera because my images are so noisy and I am trying to find out if its my camera or my processing. Thanks




It's impossible to tell from the numbers you give. There are too many unknowns - for instance what scaling might ImagesPlus have performed when converting from 14bit integers to 32 bit float? A gain of 0.06-0.1 e/ADU is certainly not sounding right for ISO 1600.

I suggest you approach the problem in a different way. You don't say what length sub-exposures and what ISO you use - let's assume it is 5 minutes at ISO 1600.

There are 3 sources of noise: read noise, thermal noise and sky background glow. Let's call them r, t and b. I'm now going to assume a basic level of maths. These 3 sources of noise combine together in quadrature to give a total noise like this:
total_noise = square_root( r*r + t*t + b*b)

It is really straightforward to get relative estimates of each of these 3 sources of noise to see which one is the biggest.

Take 2 bias frames with the lens cap on i.e. both frames at 1/4000s at ISO 1600.
Using your favourite astro processing software load the two images and subtract one from the other (it might be necessary to add a fixed level e.g. 1000) to the first image to prevent values going negative (or being truncated) during subtraction. Now select an area of the resulting data somewhere near the middle of the frame (say 100x100 pixels) and let the software claculate the standard deviation. This is your read noise r. thius noise is not stated in any sensible units but it doesn't matter because everything else we do will be in the same units.

Now take 2 dark frames with the same ISO 1600 and 5 minutes long. It's best to let the camera cool to its normal outside imaging temperature before you do this. Again subtract one from the other (maybe with a fixed offset) exactly as you did before and calculate the standard deviation of an area near the middle. Call this d (for dark frame). Now this frame contains read noise and thermal noise. However, we already know the read noise.

The thermal noise is then given by:
t = square_root(d*d - r*r)

You say your camera is cooled in some way? If so, I would expect thermal noise to be less than read noise.

Now for the background sky glow. Attach the camera to the scope and point at a dark area of sky with as few stars as possible and no nebulosity. Make sure tracking is switched on so you don't get trailing of stars. Take 2 frames with the same ISO 1600 and 5 minutes long. Then do subtraction of the 2 frames just like before and calculate the standard deviation of an area somewhere near the middle with no hint of stars i.e. just sky background. Call this standard deviation m, it's the light frame standard deviation. This frame contains read noise, thermal noise and background sky noise. We already know the read and thermal noises.

The background sky noise is given by:
b = square_root(m*m - d*d)

There you have it - you have just calculated the relative sizes of the read noise, thermal noise and sky background noise. My guess is that unless you have pristine dark sky the background sky noise will be the largest noise source by a long way.

I'll be interested to see your results.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5753069 - 03/24/13 12:19 AM

Hi Mark,

I came across this interesting thread and would like to hear opinion from you. Additional to the various problem mentioned here, I also noticed that the raw files from D800 is nonlinear.

These images can demonstrate what I meant:
http://www3.asiaa.sinica.edu.tw/~whwang/misc/D800_flat_comparison.jpg
The result of flat fielding changes with the amount of exposure in the flat (all unsaturated). This should not happen, and indeed this was not the case on Canon cameras. I can use any unsaturated flat images to get perfect flat field results on my 5D2, regardless the ISO and exposure in the flats. The residual pattern in the top image is very similar to the flat field pattern. This strongly indicates that the residual pattern is a flat field effect, i.e., nonlinarity in the images. (FYI, the images are processed in DeepSkyStacker, which is based on DCRAW.)

The notorious clip at zero made me wonder if this nonlinear behavior has anything to do with the clip. So I decoded the raw files by myself (using DCRAW) and applied various offsets to the light, dark, and flat frames, and then re-process the images. This doesn't help to solve the problem no matter what offsets I adopted. I then apply simple gamma adjustments to the images to see if the nonlinear behavior is gamma-like. The result is that this still doesn't solve the problem.

I then shoot the wall in my house with various exposure times to test the linearity. Unfortunately the light source in my house is not perfect for this purpose. I also tried sunlight, but it is hard to get a perfect sunny day here with absolutely stable sunlight. With such difficult conditions, I still cannot get the data to properly measure the nonlinearity. However, the imperfect data strongly support that there is nonlinearity in the NEF files.

I had contacted Roger Clark about this, and he kindly offered help to analyze the files, once I get the proper test images. I will also analyze the images by myself. I will keep people updated about this here. If you have any insight into this, please let me know. I would also like to use this opportunity to warm people that there might be nonlinearity in other Nikon DSLRS, and this is worth looking into.

Finally, here are my measurements on the gain, dark, and read noise of D800:
http://alohaphotolog.blogspot.tw/2013/01/nikon-d800-performance-test.html
This is just a note for myself. So the explanation there may not be enough for you to fully understand what I was talking about. I apologize for this. Maybe in the future I can share more about this.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
ccs_hello
Postmaster
*****

Reged: 07/03/04

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5753149 - 03/24/13 02:01 AM

Wei-How,

As plane reported, it is easier on testing D5100. There is a firmware hack which basically turn on camera's service mode which (1) disable N's HPS (Hot Pixel Suppression) algorithm, and (2) show both dummy pixels and OBP (Optical Black Pixel). Also NEF can be Lossless Compressed type (i.e., no more Huffman table used in RAW coding).

However, even with that, the main image area still
- uses grater than zero bias (i.e., not true sensor characteristics shown), and
- R-ch and B-ch with a multiplier always applied

I somehow doubt OBP values are involved in calculating bias. Zero clipping is.

I somehow trust libraw decoding than dcraw library for its faithful decoding and see OBP area (see "better" linkie. Rawdigger IMHO is very useful.
Later in future version 0.9.16 or 0.9.17 will be able to save the RAW output as TIFF.

Clear Skies!

ccs_hello


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: ccs_hello]
      #5753711 - 03/24/13 11:18 AM

Hi Wei-How,

What you describe sounds very strange. I wonder if something on the camera or something in your processing sequence is apply a gamma curve or similar?

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5753773 - 03/24/13 11:45 AM

Right. That's indeed very strange.

As I mentioned, I wondered if there is a mis-applied gamma curve. So I applied various different gamma back to the flat and light frames, but this does not help to solve the problem. My work flow in DeepSkyStacker is exactly the same as the one I used for Canon 5D2 and it worked perfectly there. So the problem is very likely in the camera.

Is there any uncommon camera menu item that may apply a gamma to the images? I tried to make things as raw as possible when I setup the camera. But I may have missed something.

Once I got the right light source, I will take the images for linearity test. We may learn something there. On the other hand, if anyone has a recent Nikon DSLR and has some proper light source, please also help to verify whether there is such a problem on other Nikon models.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5755382 - 03/25/13 01:36 AM

Hi Wei-Hao,

I've thought about this carefully and have come to the conclusion that the problem you are seeing cannot be caused by a response linearity in the camera. It is nothing to do with the Nikon black level clipping, either. You need to carefully look at your flats - ensure that no pixel is saturated and then look at your processing sequence - try performing the calibration operations (bias, flats, darks) manually instead of using DeepSkyStacker. I always use IRIS myself but it's not to everyone's taste. It might give a clue as to what is happening.

If that doesn't work, I'm happy to check it myself if you make available to me the flats, and a bias, dark and light.

Regards,

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
whwang
sage


Reged: 03/20/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: sharkmelley]
      #5755396 - 03/25/13 01:53 AM

Hi Mark,

Because DSS works so perfectly on 5D2, it is hard to convince myself it is the culprit. I haven't had a chance to write my own program to dark-subtract and flat these files. Too busy these days. But if you like, it will be great if you can process the files with different tools. We will learn something from it.

Please ftp to ftp.asiaa.sinica.edu.tw and go to the directory upload/whwang/D800test .

Here are what the files are for.
#281-292: bias at ISO 200 (for flat)
#384-393: bias at ISO 800 (for light and dark)
#1043-1058: ISO 200 flat (the exposure time is determined by the camera's meter, so they should not be saturated).
#1147-1158: light frame for M13 at ISO 800
#1141-1146 and #1179-1182: dark frames taken before and after M13, at ISO800.

After I stacked the images and stretched them strongly, I can see residual patterns caused by imperfect flat fielding. Please let me know if you also see this or anything you notice.

ps, others are also welcome to play with the files. They will be removed in a week or two automatically.

Cheers,
Wei-Hao


Post Extras: Print Post   Remind Me!   Notify Moderator  
sharkmelley
member


Reged: 02/19/13

Re: Nikon D7000 read noise, gain and thermal noise new [Re: whwang]
      #5756751 - 03/25/13 06:24 PM

Hi Wei-Hao,

I've taken a look at a few of your raws - very nice M13 with very tight stars!

What I noticed is that the flats are very severely underexposed - maximum pixel value was around 600 (out of a possible maxiumum of around 16,000 for a 14bit image). If this is typical of your flat frames then there is a possible explanation. Just as yoususpected, the response curve of a DSLR sensor is not actually completely linear - there is usually some kind of non-linearity down at the "dark end" of the curve. The values I was seeing in your flat frame puts it down in that non-linear section of the response curve. Adding 1 or 2 stops to the exposure would push it back up into the linear part of the curve and the flats would perform better - this corresponds well with the behaviour you are seeing.

You need to investigate why you have such a severe degree of underexposure - I would recommend that the pixel values in the middle of the flat frame exposure should be around 10,000-12,000 i.e. very firmly in the linear part of the response curve.

Non-linearity down at the lower pixel levels is a well-known phenomenon. If you want to test linearity for yourself, then plot the square of the standard deviation against the mean level - choose an area near the middle of the frame and use a single colour channel e.g. R,B,G1 or G2. Do this over a large range of exposures. It is also best to use the difference between 2 frames at each exposure for calculating standard deviation because this removes discrepancies caused by PRNU (Pixel Response Non Uniformity).

Hope this helps! Thanks for drawing this interesting problem to my attention.

Mark


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | (show all)


Extra information
3 registered and 14 anonymous users are browsing this forum.

Moderator:  Dave M, fishonkevin, WOBentley, tecmage 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 4255

Jump to

CN Forums Home


Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics