Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

EOS 60Da dark and bias frames

  • Please log in to reply
25 replies to this topic

#1 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 10 June 2019 - 04:26 PM

I'm trying to start processing output on the EOS 60Da (on a C14 + hyperstar).  I am getting images clearly through system but am having trouble with darks.  I'm acquiring with MaximDL and I tried getting a set of 30s dark and bias frames, the levels in the images look essentially identical with a level of about 2000 (the reported sensor temps are in the range of 30C).  I'm using the RAW Monochrome readout mode and ISO 800.

 

Can anyone spot the error in my interpretation?  I've turned off C.Fn II -1 Long exposure noise reduction, and disabled C.Fn II -2 High ISO speed noise reduction.  I am certain the camera is not taking a second dark on its own, the images suggest that there is some sort of automatic subtraction going on (I have no calibration groups set in MaximDL for this camera and the I am acquiring with "No Calibration").


Edited by mmnb, 11 June 2019 - 01:48 PM.


#2 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 10 June 2019 - 07:48 PM

Some useful info that suggests that there is some sort of internal dark scaling and subtraction (I don't see any hot pixel in my darks and I have sensor cleaning on). The results of the identical calibration are presented but no stats for the darks themselves.

 

"It seems that when the clean sensor now function is selected, the camera cleans the sensor then takes a short dark to map hot pixels. This is stored in the camera and used to correct  images for hot pixels. This, along with the dark current suppression built into the hardware, removes the need for dark frames."
 

http://www.nightandd...m27nodarks.html


Edited by mmnb, 10 June 2019 - 07:49 PM.


#3 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 14 June 2019 - 03:15 PM

Yes, the camera might well be taking hot pixel map but that is very different to a dark frame and doesn't remove the need for dark frames.  Once the camera has warmed up e.g. after 30 minutes of imaging, you will find a fixed pattern in your stretched dark frames (and the light frames) known as fixed pattern noise (FPN).  It is this FPN that the master dark removes.

 

However, if you are using dithered acquisition and using sigma rejection during stacking you might find that the darks are not making that much improvement and you might decide to not use them.  On the other hand, if you see "walking noise" in your stretched stacked image then you definitely need the darks.  You are using very fast optics so it is quite likely that the FPN is totally swamped by the noise from the background light pollution and it won't be affecting you much.

 

Mark


Edited by sharkmelley, 14 June 2019 - 03:17 PM.


#4 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 14 June 2019 - 07:26 PM

The inability to control temperature here and their opaque dark suppression methods make it difficult to find out what is going on.  The sensor temp was quite high and I had done a lot of imaging (so camera was warmed up) yet I could not see an obvious difference between the 30s dark and bias frames.  Are you saying that I should see a pattern in the stretched integrated darks that I won't see in the stretch integrated bias frames?

 

I am skeptical given the (casually inspected) mean levels (both frames really seem to be the same) but I will try and post results.


Edited by mmnb, 14 June 2019 - 07:39 PM.


#5 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 14 June 2019 - 07:56 PM

The inability to control temperature here and their opaque dark suppression methods make it difficult to find out what is going on.  The sensor temp was quite high and I had done a lot of imaging (so camera was warmed up) yet I could not see an obvious difference between the 30s dark and bias frames.  Are you saying that I should see a pattern in the stretched integrated darks that I won't see in the stretch integrated bias frames?

 

I am skeptical given the (casually inspected) mean levels (both frames really seem to be the same) but I will try and post results.

Yes I'm saying you will see a pattern in the stretched darks that you won't see in the stretched bias frames. All consumer cameras have thermal FPN despite any so-called "dark suppression methods".

 

It's true that the mean levels of the dark and bias frames will be the same - it's because the camera firmware is designed to make sure that exposures with no light hitting the sensor have an equal average darkness.  It's the behaviour that photographers expect from a consumer camera.  But even though the mean levels are the same, the standard deviation of the dark frames will be higher than the std dev of the bias frames.  This becomes more obvious the more you increase exposure length of the darks.

 

Mark


Edited by sharkmelley, 14 June 2019 - 07:58 PM.


#6 John_Gillies

John_Gillies

    Surveyor 1

  • *****
  • Posts: 1,544
  • Joined: 23 Apr 2003
  • Loc: Ontario, Canada

Posted 14 June 2019 - 08:00 PM

I'm sure I'll get some heat here from the experts but try just using Bias and Flats with your lights in your stacking process.  Since I started dithering I rarely take Darks any more and found my results not to differ when using Darks in the process.  (covering my head now!)



#7 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 02:28 AM

Both the integrated bias and dark frame sure do look like noise to me.  There are a few more hottish pixels in the integrated dark frame (right), a horizontal line or two, but I can't say I perceive any fixed pattern.

 

Image stats: both bias and dark are from 128 integrated frames. Darks were for 30s. Acquired in MaximDL using autosequence (ISO 800, "Raw Monochrome" (bayered) form).  No real difference in the stats:

 

bias_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2049.624
median      2049.977
avgDev      4.124
MAD         3.440
minimum     2013.407
maximum     15314.456

 

dark_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2049.039
median      2049.099
avgDev      4.214
MAD         3.397
minimum     2014.778
maximum     13886.218

Attached Thumbnails

  • darkbias.jpg


#8 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 17 June 2019 - 05:15 AM

The statistics for your integrated 30sec dark are more or less identical to the statistics for your integrated bias.  That is a totally unexpected result.

 

You could try taking MaximDL out of the equation.  If you take a raw CR2 bias file and a 30sec raw CR2 dark, does PI report any difference in the statistics between the two CR2 files?

 

Mark



#9 Kendahl

Kendahl

    Surveyor 1

  • *****
  • Posts: 1,605
  • Joined: 02 Feb 2013
  • Loc: Pinedale, Arizona

Posted 17 June 2019 - 10:05 AM

It's important to use darks taken at the same temperature as your lights. Since you can't control temperature, that means taking darks at a range of temperatures. For each light, use only the darks close to it in temperature. You need software that will show the temperature. Backyard EOS can embed it in the file name. My file name structure is FrameType_Date_Time_ISO_Duration_Temperature_Index. When using a camera lens, I include FStop between Duration and Temperature.



#10 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 11:06 AM

Well not totally unexpected given the subject of the post ;)

 

Until I hear anyone suggest otherwise, I don't think MaximDL is doing anything fishy that would alter the readout values. There is no question that the bias exposure is shorter than the dark (not clear to me exactly what happens with the bias frame on this camera). This is easier to explain with what Roger Clark and Blair McDonald are saying in the earlier link; there is some sort sort of opaque method that subtracts the dark frame.

 

The statistics for your integrated 30sec dark are more or less identical to the statistics for your integrated bias.  That is a totally unexpected result.

 

You could try taking MaximDL out of the equation.  If you take a raw CR2 bias file and a 30sec raw CR2 dark, does PI report any difference in the statistics between the two CR2 files?

 

Mark



#11 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 11:07 AM

Sure but that isn't the issue here. The dark and bias frames were acquired on the same night, so temperature variance is a small as it could be practically made yet the frames appear to be essentially identical.

 

It's important to use darks taken at the same temperature as your lights. Since you can't control temperature, that means taking darks at a range of temperatures. For each light, use only the darks close to it in temperature. You need software that will show the temperature. Backyard EOS can embed it in the file name. My file name structure is FrameType_Date_Time_ISO_Duration_Temperature_Index. When using a camera lens, I include FStop between Duration and Temperature.



#12 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 17 June 2019 - 12:01 PM

When I said that's a totally unexpected result, it was a polite way of saying it's more than likely a mistake has been made somewhere.  I have never seen such behaviour in a consumer camera i.e. where a dark has the same standard deviation as a bias, especially at a temperature of 30C.

 

That's why I suggested looking at some of the raw CR2 files.  The information in the EXIF will give valuable information about the shooting parameters.

 

Think about it for just a moment.  If this "dark suppression" technology exists and has the effect you are describing then why isn't this breakthrough built into all DSLR cameras since 2012 when the Canon 60Da was released?  Why don't all dedicated astro-cameras use it instead of using inconvenient and bulky power hungry cooling systems?   Also, here's a review of the 60Da: https://www.cloudyni...otography-r2727

Why did this reviewer see noisier dark frames than bias frames?

 

Mark


Edited by sharkmelley, 17 June 2019 - 12:29 PM.


#13 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 01:24 PM

I don't deny there could be a mistake somewhere (why I started this thread) but I haven't heard a compelling hypothesis to explain the observations yet. The "RAW Monochrome" readout setting in MaximDL seems to be simply be fits conversion:
 

https://diffractionl...l/Canon_EOS.htm

 

I don't see a good reason to believe that there is something happening to the numerical values of the image during fits conversion by MaximDL.  Which EXIF setting are you suggesting examining? 

 

I don't know why you would describe the dark suppression method that Roger Clark is suggesting about would be considered a breakthrough. The mechanism is fairly straightforward (taking a dark and storing it  during "sensor cleaning"), this isn't necessarily desirable but perhaps fine for many circumstances.

 

As for the review I don't think the comparison is straightforward: the darks are 10min (at which point the supposed internal dark might not scale properly), the graph for dark std. dev. is log scaled, the number of frames seem to be one or two etc.



#14 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 17 June 2019 - 03:18 PM

I don't deny there could be a mistake somewhere (why I started this thread) but I haven't heard a compelling hypothesis to explain the observations yet. The "RAW Monochrome" readout setting in MaximDL seems to be simply be fits conversion:
 

https://diffractionl...l/Canon_EOS.htm

 

I don't see a good reason to believe that there is something happening to the numerical values of the image during fits conversion by MaximDL.  Which EXIF setting are you suggesting examining? 

 

I don't know why you would describe the dark suppression method that Roger Clark is suggesting about would be considered a breakthrough. The mechanism is fairly straightforward (taking a dark and storing it  during "sensor cleaning"), this isn't necessarily desirable but perhaps fine for many circumstances.

 

As for the review I don't think the comparison is straightforward: the darks are 10min (at which point the supposed internal dark might not scale properly), the graph for dark std. dev. is log scaled, the number of frames seem to be one or two etc.

If you want a hypothesis, here's a hypothesis that might explain the observations:  MaximDL accidentally used very short exposures for the darks.

 

The EXIF settings I would start looking at would include (but not be limited to):

  • Exposure length
  • ISO
  • Noise reduction settings
  • Temperature 

 

Extraordinary claims require extraordinary evidence and the claim you are referring to, whether you realise it or not, is quite extraordinary.  Let's suppose for instance that the camera does indeed store some kind of dark frame.  Let's suppose it is a completely noise-free dark frame, representing only the thermal fixed pattern noise and let's suppose the camera can scale it (for exposure time and temperature) and subtract it perfectly.  Even then we would see a noise increase between bias and dark because of the physics of how electrons accumulate due to dark current.  Subtracting the mean level of dark current, even at an individual pixel level, does not subtract the associated noise.

 

I'm happy to accept that you may not see any visible difference between a stacked image using dark frames or not.  But the claim that long exposure darks have the same noise as bias frames is a very extraordinary claim.

 

Furthermore you can download some of Roger Clark's own data from his own Canon 6D that uses "dark current suppression" technology from a link in his article here: http://www.clarkvisi...ge.processing2/

Measure the noise in a couple of his dark frames and measure the noise in a difference frame between two darks (you may need to add an offset to prevent negative values).  Tell me what you find wink.gif . 

 

If you find the same as me, then you'll see it's an excellent demonstration of the effectiveness of dark subtraction.  I told Roger what I found and I never got an adequate response.

 

Mark


Edited by sharkmelley, 17 June 2019 - 04:01 PM.


#15 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 03:37 PM

You are misrepresenting my position, I never made the claim that noise should be same across dark and bias.

 

I'm just trying to understand what I see plainly in the images that have been acquired in a straight forward fashion.

 

The FITS header has all of those settings (I'm looking at the headers again right now to sanity check I hadn't done something silly like set the exposure for the bias frames to 30s or something), and they all check out to what I had remember setting them to except that there is nothing specific for noise reduction (though I did set noise reduction settings in the camera specifically as noted in the top of the thread).  I had watched several images come in and was even looking at single exposures, so I am confident about the exposure times.  The earlier link supports the idea that the camera may doing something that isn't transparent to the user here.  None of the links provided so far allow me to compare directly here.

 

If you want a hypothesis, here's a hypothesis that might explain the observations:  MaximDL accidentally used very short exposures for the darks.

 

The EXIF settings I would start looking at would include (but not be limited to):

  • Exposure length
  • ISO
  • Noise reduction settings
  • Temperature 

 

Extraordinary claims require extraordinary evidence and the claim you are making, whether you realise it or not, is quite extraordinary.  Let's suppose for instance that the camera does indeed store some kind of dark frame.  Let's suppose it is a completely noise-free dark frame, representing only the thermal fixed pattern noise and let's suppose the camera can scale it (for exposure time and temperature) and subtract it perfectly.  Even then we would see a noise increase between bias and dark because of the physics of how electrons accumulate due to dark current.  Subtracting the mean level of dark current, even at an individual pixel level, does not subtract the associated noise.

 

I'm happy to accept that you may not see any visible difference between a stacked image using dark frames or not.  But the claim that long exposure darks have the same noise as bias frames is a very extraordinary claim.

 

Mark



#16 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 17 June 2019 - 06:31 PM

John,

I see in your sig you use a 60Da? Do your master darks and biases seem essentially identical in terms of stats?

 

I'm sure I'll get some heat here from the experts but try just using Bias and Flats with your lights in your stacking process.  Since I started dithering I rarely take Darks any more and found my results not to differ when using Darks in the process.  (covering my head now!)



#17 John_Gillies

John_Gillies

    Surveyor 1

  • *****
  • Posts: 1,544
  • Joined: 23 Apr 2003
  • Loc: Ontario, Canada

Posted 17 June 2019 - 06:58 PM

I don't keep any of the extra files created from stacking.  That includes the masters as well.  I'm not quite as detailed as most so I honestly can't tell you the stats.  My suggestion would be to try your post- processing with and without darks and see if it makes a difference to you.  When I image, I do lights of each target (if there is more than one).  At the end of my session I do my bias. At daybreak I do my flats....every time.  I don't bank or create a library of darks, flats or bias.  I shoot new each session.  This may be a little unorthodox to some, but it works for me.



#18 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 18 June 2019 - 05:29 PM

So I think this is the definitive collection of links on the issue should anyone come by it:

http://adsabs.harvar...JRASC.106..212M
http://www.nightandd...m27nodarks.html

https://arxiv.org/pdf/1506.03097.pdf

 

The first two are from Blair MacDonald and the last is a draft form Zhang et al. assessing the EOS 60D for photometry. Zhang et al. claim their information is "...likely the most detailed publicly available description of the properties of the 60D" but MacDonald seems to have gotten a little farther than the Zhang et al.  It isn't totally clear to me how Zhang et al. are processing their frames from the EOS 60D, they seem to still be acquiring and subtracting darks despite  "not been able to find any documentation about how this [dark] suppression works". Summarizing across links:

 

-There are small regions on the sensor at the left and top margins that don't behave in expected ways (usually cropped out but worth mentioning here).

 

-The mean value for the dark frame value will not go up with increased exposure due to on-camera processing. It is not clear if this is additive bias or multiplicative scaling (MacDonald's article is ambiguous here, he says it is a bias but then continues to call it a 'scale factor' then again a 'bias shift').  He is somehow getting the offset from the white balance setting, not sure what value he is pulling out of the file to determine the offset. MacDonald says that he "needed to enable white balance to get the correction for bias shift", not clear to me what he means here since AFAICT the white balance is always on (p. 96 in EOS 60D manual); just a matter of which setting.

 

-I don't see units in McDonald's article, but he shows the dark frame std. dev. go up with increasing exposure.  I would believe that the difference between 0s and 30s (my bias and dark exposures) may be too small to really see in my frames or perhaps his numbers are post correction (it isn't clear to me that they are). FWIW I've pasted the FITS headers from PI for the first images in each set to confirm that the exposures are what I am claiming they are.

 

None of these have provided a credible way of subtracting the dark current from the lights and I do not observe any FPN in my integrated darks; I guess I'll just keep going with just bias and flat. I've reached out to both authors to see if my interpretation of what they are saying is correct.

 

 

 

PI only lets me copy when I view source:

 

Bias

 

["SIMPLE", "T", ""],
   ["BITPIX", "16", "8 unsigned int, 16 & 32 int, -32 & -64 real"],
   ["NAXIS", "2", "number of axes"],
   ["NAXIS1", "5184", "fastest changing axis"],
   ["NAXIS2", "3456", "next to fastest changing axis"],
   ["BSCALE", "1.0000000000000000", "physical = BZERO + BSCALE*array_value"],
   ["BZERO", "32768.000000000000", "physical = BZERO + BSCALE*array_value"],
   ["DATE-OBS", "\'2019-06-09T07:23:31\'", "YYYY-MM-DDThh:mm:ss observation start, UT"],
   ["EXPTIME", "0.00000000000000000", "Exposure time in seconds"],
   ["EXPOSURE", "0.00000000000000000", "Exposure time in seconds"], // row 10
   ["CCD-TEMP", "25.000000000000000", "CCD temperature at start of exposure in C"],
   ["XPIXSZ", "4.1699999999999999", "Pixel Width in microns (after binning)"],
   ["YPIXSZ", "4.2400000000000002", "Pixel Height in microns (after binning)"],
   ["XBINNING", "1", "Binning factor in width"],
   ["YBINNING", "1", "Binning factor in height"],
   ["XORGSUBF", "0", "Subframe X position in binned pixels"],
   ["YORGSUBF", "0", "Subframe Y position in binned pixels"],
   ["READOUTM", "\'RAW Monochrome\'", "Readout mode of image"],
   ["ISOSPEED", "\'ISO 800 \'", ""],
   ["IMAGETYP", "\'Bias Frame\'", "Type of image"], // row 20

 

 

Dark

["SIMPLE", "T", ""],
   ["BITPIX", "16", "8 unsigned int, 16 & 32 int, -32 & -64 real"],
   ["NAXIS", "2", "number of axes"],
   ["NAXIS1", "5184", "fastest changing axis"],
   ["NAXIS2", "3456", "next to fastest changing axis"],
   ["BSCALE", "1.0000000000000000", "physical = BZERO + BSCALE*array_value"],
   ["BZERO", "32768.000000000000", "physical = BZERO + BSCALE*array_value"],
   ["DATE-OBS", "\'2019-06-09T07:59:17\'", "YYYY-MM-DDThh:mm:ss observation start, UT"],
   ["EXPTIME", "30.000000000000000", "Exposure time in seconds"],
   ["EXPOSURE", "30.000000000000000", "Exposure time in seconds"], // row 10
   ["CCD-TEMP", "26.000000000000000", "CCD temperature at start of exposure in C"],
   ["XPIXSZ", "4.1699999999999999", "Pixel Width in microns (after binning)"],
   ["YPIXSZ", "4.2400000000000002", "Pixel Height in microns (after binning)"],
   ["XBINNING", "1", "Binning factor in width"],
   ["YBINNING", "1", "Binning factor in height"],
   ["XORGSUBF", "0", "Subframe X position in binned pixels"],
   ["YORGSUBF", "0", "Subframe Y position in binned pixels"],
   ["READOUTM", "\'RAW Monochrome\'", "Readout mode of image"],
   ["ISOSPEED", "\'ISO 800 \'", ""],
   ["IMAGETYP", "\'Dark Frame\'", "Type of image"], // row 20


Edited by mmnb, 18 June 2019 - 06:27 PM.


#19 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 18 June 2019 - 07:20 PM

I certainly wouldn't call that a definitive selection of links on the issue.

 

For instance Blair MacDonald comes up with a dark current estimate of 11e/sec.  This is bizarrely high.  I have rarely measured a dark current above 0.5e/sec at room temperature for a consumer camera, going right back to the original Canon Rebel (a.k.a. 300D).  Roger Clark hasn't measured anything that high - see his Figure 3 here:  http://www.clarkvisi...ion-technology/

 

Zhang says "we have not been able to find any documentation about how this [dark current] suppression works."  Firstly I wouldn't call this mechanism suppression of dark current at all.  The dark current still exists but the mean level of the accumulated dark current across the entire frame is subtracted during in-camera processing, after the event.  In any case, it is well known that the optical black pixels are used to measure this accumulated charge, even though Zhang found no documentation.  The reason for this subtraction is obvious:  photographers expect a black image when no light has reached the sensor, even for long exposures.  This is something that often confuses folk who are familiar with CCDs and then start to experiment with consumer cameras.

 

You say you do not observe any FPN in your integrated darks.  Maybe you haven't noticed those bright pixels?  Maybe you haven't noticed those dark horizontal and vertical striations?  If you subtract one dark from another (use a fixed offset to prevent negative values) you will notice those features disappear leaving you with what is essentially pattern-free random noise.  The features disappear because they are a fixed pattern that appears in every dark.

 

Mark


Edited by sharkmelley, 18 June 2019 - 07:34 PM.


#20 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 18 June 2019 - 08:05 PM

Perhaps 'definitive' should be replaced with 'most detailed accounts I can find on the issue'.

 

I don't take Zhang et al.'s use of "dark suppression" to mean actual suppression of current but any sort of processing done on-chip or post readout that accounts for the tiny increase mean ADU with longer exposures.  I don't see any documentation that could be used to support any mechanism of why the dark frames look like the way they do here (other than what has already been linked to).  Do you have a cite to support your proposed mechanism of subtraction in camera?  I can't find any evidence to support what you are saying is actually happening in the EOS 60D. 

 

Hard for me to say much about Macdonald's article given the questions I've already stated I have about it.

 

I do see two darker horizontal lines in my averaged dark with some extra hotter pixels. It seems that most sources define FPN in terms of columns though, so it isn't clear to me that what I am seeing is FPN.

https://www.dpreview...s/post/57527126

http://isl.stanford....392b/lect07.pdf



#21 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 19 June 2019 - 12:31 AM

Perhaps 'definitive' should be replaced with 'most detailed accounts I can find on the issue'.

 

I don't take Zhang et al.'s use of "dark suppression" to mean actual suppression of current but any sort of processing done on-chip or post readout that accounts for the tiny increase mean ADU with longer exposures.  I don't see any documentation that could be used to support any mechanism of why the dark frames look like the way they do here (other than what has already been linked to).  Do you have a cite to support your proposed mechanism of subtraction in camera?  I can't find any evidence to support what you are saying is actually happening in the EOS 60D. 

 

Hard for me to say much about Macdonald's article given the questions I've already stated I have about it.

 

I do see two darker horizontal lines in my averaged dark with some extra hotter pixels. It seems that most sources define FPN in terms of columns though, so it isn't clear to me that what I am seeing is FPN.

https://www.dpreview...s/post/57527126

http://isl.stanford....392b/lect07.pdf

Optical black subtraction is a standard part of basic image processing.  e.g. see here:

https://www.csie.ntu...alProcessor.ppt

 

Just Google for sensor "optical black" (keep "optical black" in quotes) and you'll find plenty of other references including patents.

 

FPN generally looks completely random to the human eye because it is caused by pixel to pixel variations in dark current and these are random manufacturing variations caused by many things including (lack of) substrate purity.  However in CMOS sensors it is often dominated by row and column effects whose cause may or may not be the OB subtraction.

 

There's another interesting article written by Craig Stark about in-camera subtraction in Canon cameras here:

http://www.stark-lab...onLinearity.pdf

The cameras were overcompensating their black level subtraction.

 

By the way, let's examine MacDonald's estimate of 11e/sec as the dark current at room temperature.  For your 30sec dark this would give 363 accumulated electrons giving a standard deviation of sqrt(363) = 19 electrons RMS using Poisson statistics.  Your read noise at ISO 800 is probably around 3 electrons so you can see that the noise from the dark current in the 60Da would be completely and utterly swamping the read noise.

 

On the opposite end of the spectrum, your assertion of no increase in standard deviation for 30sec darks is very difficult to accept until evidenced by actual raw CR2 files rather than files produced by some application where anything may be going on.

 

Mark


Edited by sharkmelley, 19 June 2019 - 12:33 AM.


#22 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 19 June 2019 - 01:04 AM

Great links.  I'll run the same darks/biases writing out RAW to the card in a few weeks to see if MaximDL is doing something in the conversion to FITS.



#23 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 19 June 2019 - 02:10 PM

I went back to look at the integration and I left a few scaling parameters on (you can see the max value is different in bias and dark in previous stats); turning all scaling normalization off makes a little more sense:

 

bias_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2049.538
median      2049.891
avgDev      4.122
MAD         3.437
minimum     2013.344
maximum     15305.945

 

dark_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2048.004
median      2048.070
avgDev      4.726
MAD         3.805
minimum     2009.625
maximum     15305.164

 

EDIT: MacDonald has also confirmed to me in private communication that the high gain from the RASC article was incorrect and due to a bug in Image Plus and was corrected in a later article. 


Edited by mmnb, 19 June 2019 - 02:17 PM.


#24 sharkmelley

sharkmelley

    Gemini

  • *****
  • Posts: 3,306
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 19 June 2019 - 04:06 PM

I went back to look at the integration and I left a few scaling parameters on (you can see the max value is different in bias and dark in previous stats); turning all scaling normalization off makes a little more sense:

 

bias_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2049.538
median      2049.891
avgDev      4.122
MAD         3.437
minimum     2013.344
maximum     15305.945

 

dark_integrated
            K
count (%)   100.00000
count (px)  17915904
mean        2048.004
median      2048.070
avgDev      4.726
MAD         3.805
minimum     2009.625
maximum     15305.164

 

EDIT: MacDonald has also confirmed to me in private communication that the high gain from the RASC article was incorrect and due to a bug in Image Plus and was corrected in a later article. 

 

Your new figures make a lot more sense though the difference in noise between the dark and the bias is still a bit on the low side, assuming you did this at room temperature.

 

MacDonald's explanation of his anomalously high result also makes sense.

 

Mark


Edited by sharkmelley, 19 June 2019 - 04:07 PM.


#25 mmnb

mmnb

    Sputnik

  • -----
  • topic starter
  • Posts: 41
  • Joined: 15 Jan 2018

Posted 19 June 2019 - 04:20 PM

Sensor temp is higher, but ambient was about 16C.


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics