Astronomer and Golfer - C11 and Vixen 140mm NA140SSF Refractor - G11 mount on permanent pier (in my moderately light polluted back yard) - Older Meade 8" SCT for travel
Mach1GTO / G11/G2 (stock) / AT6RC / AT10RC / TMB92SS / Astrodon 50D / STT-8300M / FW8G-STT / PixInsight / etc, etc. Astrobin - Flickr
Quote:Although the dark does not record signal, it does record noise
Quote:background in statistical analysis of data just helped me to understand why it works as well as it does
Televue 85/TRF-2008 field flattener, Meade LX200 10", Manfrotto 055SSB tripod/410 geared head/AstroTrac TT320X-AG/056 3D head, GM-8/Gemini, 10 Micron 1000 HPS, Canon 40D (unmodded), Canon 450D (modded w/Astronomiks clip-ins - UV/IR, OWB), Coronado SM60/Lunt B1200/WO diagonal, Baader Herschel Wedge Leeds Sky Clock Ripon Sky Clock
Quote:yeah, that's the guy.dark current is not noise. it's signal. unwanted signal, but not noise in the statistical sense. you should understand this since you seem to know something about statistics.after you register your lights, the dark signal is shifted around all over the place. a given pixel stack of the registered lights has dark signal from different sensor pixels! therefore it does not matter if it's difference in means or paired differences or whatever, because you're integrating different dark signal in each pixel stack.so the dark signal has been subtracted from each frame, but in a noisy fashion. this means that you've got some pixels where it was over-subtracted and some where it was under-subtracted. i don't see how you can hone in on the 'true' value of the dark signal at a given pixel location if you have shifted the dark signal all over the place before integration.please address this, because it is the flaw in your reasoning.
"My GOD, It's Full of Stars!" For the Obser: Atlas EQ-G w/moded SN8,JMI moto-focus, flocked tube,custom mirror cell.ED80 w/ACCUFocus,SSAG,DSI,EZ Guider. Modded Canon T1i,Pentax K-5II. For the field:14" Orion 14xxi,Flocked Tubes,Custom Paint,6 Fan Cooling,LED ground board lighting. Celestron CG5GT for film/digital photos w/EZ Guider, Atlas EQ-G w/an ES102ED/AT72 Piggy backed,EZ Guider. Various 1.25 & 2" Eyepieces.
Quote:Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?dave
Quote:Quote:dave Dark has both 'signal' (or at least something predictable and/or constant) and noise. So anything you do to reduce Dark has a noise component that can't be eliminated (but can be reduced). How would one go about removing the Dark component in a noise-free manner?as you say the uncertainty in the dark can not be totally eliminated, but it can be reduced dramatically......just like you do with your lights - you integrate a whole bunch of them so that you get down to the true value of the dark current at a given pixel location. you can also reject outlier pixels to clean up cosmic ray hits, etc.the dark signal in the light is noisy, but that's no reason to increase the noise in your calibrated sub by subtracting a noisy dark.
Quote:They are mathematically the same if there is no blocking factor. However, in an unregulated camera there is a blocking factor, temperature, and it can be very significant. I was puzzled for a long time why my dark libraries were hit and miss; sometimes doing okay, sometimes not. Then I took a close look at my source darks before I averaged them. They usually started out relatively cool, and then rapidly heated and the noise increased right along with it. If you want to see something interesting take a series of darks and look at the temperature and noise of each individual frame. No wonder my darks were hit'n miss, what ta' heck was I trying to average. Simple means and differences in means are built around uncertainty that is random and normally distributed. These distributions are far from normal, probably closer to an f-distribution. If you go crazy and average enough frames they'll migrate towards normal, but as long at the temperature is increasing there will be bias towards the high temperature side of the distribution. Now in the big scheme of things it is not that big of a deal as these methods are very simple and robust and they obviously work quite well. However, this is such a perfect real-world application for paired differences it is just beautifully simple and the way that Canon implements it is wonderful and smart. It also brings so many other statistical factors into ballance. For example, strictly speaking, when using a difference in means you really should use the same number of data points (source images) in each sample so that the variances match. The statistical work-around is to use pooled variances, but again the methods are sufficiently robust you can often away without it (and we do). All these little nuances go away when using paired differences since by definition you are always using the same number of darks and lights.Very simple, very clean, very effective. I like it. Enjoy.
Scopes: 190mm SW Mak-Newt @ f/5.3 & 90mm SV Raptor Triplet @ f/7Cameras: Atik 383L+ mono w/ EFW2 FW with Astrodon filters & Canon XSi 450D (Modded)Mount: Astro-Physics Mach1GTO mount guided with QHY5 KWIQ guiderSoftware: Sequence Generator Pro; PixInsight; Photoshop CS5Best investment: Stone Garden Observatory - Bortle 4 (Green)
Adam AP Mach 1, Rob Miller TRI36L, Celestron 8" Edge HD Hyperion 17mm, Celestron 40mm Plossl, TMB 7mm & 9mm WO 66, Lunt LS60/B1200 PT, CG-5 Clone STF-8300m Pro Package, Honis-Modified Canon XS, Un-modded T1i, SSAG, TSOAG9, SX LodestarLackawanna Astronomical SocietyMy Astrobin
Quote:It would seem to me that applying dark frames to a set of light frames AFTER the lights have been stacked and (more importantly) re-registered/whatever would be dramatically inferior to applying a dark to each light frame (without a registration change). And this would favor the Canon approach (with caveats about time loss and loss of flexibility).
Quote:SNIPThe master dark is applied to the individual lights before stacking.SNIP
Quote:Makes sense John. Thanks for sharing your perspective and experience. I personally don't do dark subtraction at all -- just bad pixel mapping and sigma-rejection based stacking. That probably puts me beyond the pale vis a vis the calibration orthodoxy. ...Keith
Quote:imagine that before you register your lights, you draw a big cross in the same place on each image, like a danish flag, but one pixel wide. let that cross represent the dark signal in the sub. as you register your lights, in each sub the cross is rotated by different angles and where the two lines cross is shifted around. now you average the registered lights. do you have a strong horizontal/vertical cross pattern on your stack, or a smeared out version of the cross?that's what i'm talking about. the dark signal, or the dark signal error no longer lies directly on top of itself after registration.
Astrophotograher's Guide to the Deep Sky
A Guide to DSLR Planetary Imaging
A Beginner's Guide to DSLR Astrophotography
An Advanced Guide to DSLR Astrophotography
Photoshop for Astrophotographers
Quote:It is my understanding that enabling the Canon (T3) 'long exposure noise reduction' is simply an automated dark frame (same exposure length) that follows the light frame (and then applied automatically to the light frame).
Somewhere (don't recall where) I read a recommendation that this function be disabled. Is there a problem here (would seem to me to be an automatic dark frame/bias frame) other than the fact that this will probably extend the time that the camera will spend in the imaging process? FWIW, I am a complete newbie here.
Quote:There is no way you can argue that using dark-sky time to collect signal and shooting separate darks on a cloudy night is not better than in-camera dark-frame subtraction.Jerry
Quote:i use one of those programs. absolutely, positively, no doubt about it, darks, bias and flats are applied to the subs BEFORE registration.i'm trying to show the flaw in John's reasoning here! my example about the cross is what happens to the dark signal in his method, not the method that we all use and is known to be correct, which is to calibrate images with master frames BEFORE registration.
Quote:yes, if you are going to make your own master calibration frames you really need to turn off in-camera dark subtraction. otherwise your darks are not darks, they are just dark current noise.jsines, your outline of the steps to calibrate and integrate images is correct.i'm glad jerry has weighed in here because frankly every time the in-camera dark subtraction comes up, i have this one-sided argument with john about how registration is the flaw in his theory. he never addresses that but just comes back with more statistics jargon.
Quote:Just so this isn't one-sided, I'm afraid I agree with John's logic and I respectfully disagree that there is some unacknowledged flaw in his theory. For every pixel there is a true value of dark signal that is dependent on factors such as temperature. If you take a bunch of darks at the same temperature however you don't get the same value every time - the variation around the true value is the noise.
Quote:The objective of calibration is to get the best estimate of the true value.
Quote:One way to do this is to stack a bunch of frames and take an average of each pixel's value which should converge on the true value the more frames you stack. That is the method that you promote, which should result in very little noise being injected into your light frame when it's calibrated. I think your logic is that this is the only way to get a good estimate of the true dark currant, and therefore when you register your lights and lose pixel-to-pixel mapping, you are losing the value of stacking darks. That's not what John is arguing. He is saying that a single matched dark could be as good of an estimate of your true dark signal as a stacked set of darks taken over a range of temperatures. If you accept that, then it's a red herring that your pixels get shuffled during registration because you aren't trying to get your best estimate of the true dark signal by stacking darks, your are trying to get the best estimate by subtracting the best-matched dark from your light. I don't see a flaw in this logic, it's just another way to skin the cat. Or you could use my method with dithering, bad pixel mapping with bias subtraction only, and sigma-based stacking and dispense with dark subtraction altogether. They are just different ways of trying to get a nice picture for those of us not doing photometry. ...Keith
Quote:in your experiments i guess an important thing is to figure out what the margin of error is in order to conclude that the results are truly the same or truly different. no idea how to compute that, though.