304 replies to this topic

[sharkmelley]

ummm, so I have some questions about this.  My understanding is that dark current (in each pixel) ramps up linearly with time, and then on top of that there is the sqrt(N) noise.  And your chart shows that things level off after about 1 hour of use.  Fixed pattern noise means that the signal will be the same even if you took a 1/1000th second exposure(?).  So I don't understand because if you subtract one frame from the next after things have "leveled off", that should remove both the fixed pattern noise and also the linearly ramping dark current, leaving only the sqrt(N) part.  Are you actually plotting the RMS value of this sqrt(N)?

Next question: Sooo, what are we supposed to do in the first hour worth of picture taking while the Z6 is still heating up?
 

Yes, dark current accumulates linearly during the exposure time.  The fixed pattern noise (FPN) in this case is technically known as the dark current non-uniformity (DCNU) i.e. the fact that some pixels have persistently higher dark current than others leading to brighter pixels in the dark exposure.  The dark current is then the standard deviation in the difference frame, squared, divided by 2, divided by the exposure time.

 

During an imaging session, the light frames taken while the camera is warming up can still be calibrated if you use dark scaling (aka dark optimisation).  Typically though, the camera is actually cooling down for for first few exposures because live view has heated it up during framing and focusing.

 

Mark

[Jerry Lodriguss]

During an imaging session, the light frames taken while the camera is warming up can still be calibrated if you use dark scaling (aka dark optimisation).  Typically though, the camera is actually cooling down for for first few exposures because live view has heated it up during framing and focusing.

Hi Mark,

 

I should start another topic on this, but how much do you know about how the heating is controlled in a traditional DSLR versus a MILC?

 

I guess the video pumping all those pixels into the processor is what really heats the new MILCs up (like the Canon R5 overheating problems when first released).

 

If they have to deal with that in the camera design, maybe the actual overall heat during a long astro exposure where the monitor isn't on, and the processor isn't really working will actually be less in a MILC than a DSLR.

 

Have you compared dark signal noise in long exposures between a MILC and DSLR?  Or does the scaling and filtering make that problematic?

 

Jerry

[sharkmelley]

I should start another topic on this, but how much do you know about how the heating is controlled in a traditional DSLR versus a MILC?

 

I guess the video pumping all those pixels into the processor is what really heats the new MILCs up (like the Canon R5 overheating problems when first released).

 

If they have to deal with that in the camera design, maybe the actual overall heat during a long astro exposure where the monitor isn't on, and the processor isn't really working will actually be less in a MILC than a DSLR.

 

Have you compared dark signal noise in long exposures between a MILC and DSLR?  Or does the scaling and filtering make that problematic?

It's difficult to generalise.  You will have seen my charts in post #2 where I measure the dark current during an image run of successive 5min exposures.  Those are all produced indoors at 20C with a lens cap in place and the camera already acclimatised - so I have standard conditions for doing the comparison.  However, they represent a worst case because in practice the temperature for thermal equilibrium when imaging outside would be less because of potential cooling by moving air currents and cooling by being connected to a heavy lens/telescope.  

 

It's certainly the case that live view heats up any camera (DSLR or MILC) very fast, so a typical imaging session is likely to start with a camera that is already quite warm because of live view used during focusing and framing.  Again this makes those charts a bit unrealistic.  I also think it varies quite a lot how much circuitry is active and generating heat when a long exposure is being taken.  The build quality of the camera also matters, for instance the Sony A7S has a lot of copper inside it for heat dissipation. 

 

Mark

[bokemon]

Yes, dark current accumulates linearly during the exposure time.  The fixed pattern noise (FPN) in this case is technically known as the dark current non-uniformity (DCNU) i.e. the fact that some pixels have persistently higher dark current than others leading to brighter pixels in the dark exposure.  The dark current is then the standard deviation in the difference frame, squared, divided by 2, divided by the exposure time.

 

OK, that makes sense.  BTW, have you compared this dark current noise to the noise just from the light pollution background?  In this case, I am wondering if a cooled camera has any benefits when doing broadband color imaging.

[sharkmelley]

 BTW, have you compared this dark current noise to the noise just from the light pollution background?  In this case, I am wondering if a cooled camera has any benefits when doing broadband color imaging.

It depends on the ambient temperature, the speed of the optics (i.e. the f-ratio) and the level of the light pollution.

 

In my own case, using fast f/2.8 optics in Bortle 3 skies with UK night-time temperatures, the dark current is never a problem and I would see no noticeable improvement by using a cooled camera.  The exception would be if I wanted to use narrowband filters.

 

Mark

[UKalwayscloudy]

All the results presented by me in this thread use raw files and all calibration (e.g. with flats) is done while the files are raw i.e. before demosaicing. You spoke about "desaturating flats" in LR before using them. However, the files produced by LR are non-linear and any attempt to use non-linear flats will never work correctly. If you are using flats in the wrong way like this then it's very likely they will be creating additional artifacts.

Mark

This has nothing to do with the particular methods used by Lightroom. I can eliminate the saturation in the flats in any number of ways and the final images from DSS benefit. If you don’t believe me maybe you should try it! I’ve noticed it particularly when trying to push the Starizona reducer in my SCT out to the limits of it circle of illumination where there is heavy vignetting. There is no reason why flats should not be corrupted by the in camera processing you have helpfully characterised, and it is worth exploring whether for that Simpler component there is a way of mitigating the effect so it does not add to the effect in the lights. I’m fiddling with the ISO choices as well.

[sharkmelley]

This has nothing to do with the particular methods used by Lightroom. I can eliminate the saturation in the flats in any number of ways and the final images from DSS benefit. If you don’t believe me maybe you should try it!

I have no reason to doubt you but I have no interest in trying it for one very simple reason - flats created by LightRoom will not correctly calibrate the data because they are non-linear.  The very fact that you are seeing de-saturated LR flats work better than more saturated ones is precisely because you are working in a non-linear domain.

 

To be clear, the TIFs created by LightRoom have the following issues that make them unsuitable for calibration purposes:

• They have been converted to a non-linear colour space.  This is especially bad when the colour space is sRGB because sRGB employs variable gamma - so RGB ratios alter with brightness.
• Additional tone curve distortions have been applied (to make the image more "arty")
• Additional colour saturation has been applied (again to make the image more "arty")

 

Mark

Edited by sharkmelley, 06 September 2020 - 04:44 AM.

[thibautflt]

I have no reason to doubt you but I have no interest in trying it for one very simple reason - flats created by LightRoom will not correctly calibrate the data because they are non-linear.  The very fact that you are seeing de-saturated LR flats work better than more saturated ones is precisely because you are working in a non-linear domain.

 

To be clear, the TIFs created by LightRoom have the following issues that make them unsuitable for calibration purposes:

• They have been converted to a non-linear colour space.  This is especially bad when the colour space is sRGB because sRGB employs variable gamma - so RGB ratios alter with brightness.
• Additional tone curve distortions have been applied (to make the image more "arty")
• Additional colour saturation has been applied (again to make the image more "arty")

 

Mark

Wow, very happy to find this topic ! Thank you mark, I learn a lot and... I think I'm facing the same problem ! I'm shooting with a D850, and I have this kind of ring. I can see them in the MasterFlat but not in a single flat, so rings may be very subtle. It is improved by shooting overexposed flat. The only difference is that rings are rather concentric, and I observe them only on the Red/Blue channel.

 

 

 

How do you proceed not to overexposed brightest stars if you overexposed your lights as well (I have to say I am a beginner in AP) ?

 

I wonder if it is the same with canon cameras.. if it is, maybe i'm gonna switch !

Attached Thumbnails

• 

[sharkmelley]

Wow, very happy to find this topic ! Thank you mark, I learn a lot and... I think I'm facing the same problem ! I'm shooting with a D850, and I have this kind of ring. I can see them in the MasterFlat but not in a single flat, so rings may be very subtle.

I would be interested to see one of the raw NEF files you used to generate that image.  

 

Mark

[thibautflt]

I would be interested to see one of the raw NEF files you used to generate that image.  

 

Mark

You can download it here : https://drive.google...iew?usp=sharing

 

An other strange thing is, when I apply the lens correction to a flat using LR, rings appear...

Attached Thumbnails

• 

Edited by thibautflt, 20 January 2021 - 12:42 PM.

[sharkmelley]

You can download it here : https://drive.google...iew?usp=sharing

It's a well exposed flat so underexposure is certainly not a factor. I couldn't find anything obvious that might cause the rings so it's a bit of a mystery.

 

Mark

[tonyt]

The concentric rings seem difficult to predict. I had an M45 image with prominent ring artifacts but I tested the Z6 on a faint target (sh2-308) with the same telescope (but higher ISO) and the ring banding didn't appear, even though the exposures were hard left on the histogram and the image is stretched hard and background enhanced.

 

One difference was that the FTZ adapter was used with M45 while a generic adapter used with this image - could the camera scaling be different depending on electrical contacts being made at the lens mount? (I imagine not, but though I'd ask)

 

 

[thibautflt]

It's a well exposed flat so underexposure is certainly not a factor. I couldn't find anything obvious that might cause the rings so it's a bit of a mystery.

 

Mark

I though it was an underexposed flat when I checked the histogram, which is almost clipping the left hand side. :o


Even if it is well exposed, can rings be present, but nearly invisible, and revealated by stacking ? Because I can see rings and discontinuity as you, by dividing 2 channels. And by increasing exposition, it (almost) vanishes.

Attached Thumbnails

• 

[sharkmelley]

I though it was an underexposed flat when I checked the histogram, which is almost clipping the left hand side. :o


Even if it is well exposed, can rings be present, but nearly invisible, and revealated by stacking ? Because I can see rings and discontinuity as you, by dividing 2 channels. And by increasing exposition, it (almost) vanishes.

Sorry, my mistake.  I took another look at the file you uploaded (DSC_7567.NEF) and I can see the rings when subtracting the bias and dividing the Blue channel by the Green channel:

 

 

It's the first evidence I've seen that the the Nikon D850 might be doing something weird.

 

You're absolutely right that rings can be almost invisible in a single exposure but can be revealed by stacking, which increases the signal to noise ratio.

 

Mark

[UKalwayscloudy]

Has anybody has the chance yet to explore these issues with the Z6ii? I’ve seen early reports that some of the linear banding has been mitigated with the Mkii but the issues described here are rather different. I’ve been working with an Altair 26C for the last period but am going to resume Z work soon.

[SandyHouTex]

Sorry, my mistake.  I took another look at the file you uploaded (DSC_7567.NEF) and I can see the rings when subtracting the bias and dividing the Blue channel by the Green channel:

 

NikonD850_rings.jpg

 

It's the first evidence I've seen that the the Nikon D850 might be doing something weird.

 

You're absolutely right that rings can be almost invisible in a single exposure but can be revealed by stacking, which increases the signal to noise ratio.

 

Mark

Not my beloved D850!  Say it isn’t so!

[tkottary]

Interesting point you make. All the rings I have seen so far has been with FTZ adapter but not when using redcat with baader adapter or while using cannon 135 art with dumb adapter.

 

The concentric rings seem difficult to predict. I had an M45 image with prominent ring artifacts but I tested the Z6 on a faint target (sh2-308) with the same telescope (but higher ISO) and the ring banding didn't appear, even though the exposures were hard left on the histogram and the image is stretched hard and background enhanced.

 

One difference was that the FTZ adapter was used with M45 while a generic adapter used with this image - could the camera scaling be different depending on electrical contacts being made at the lens mount? (I imagine not, but though I'd ask)

 

sh-2 308 200121 50% (Medium).jpg

[UKalwayscloudy]

Interesting point you make. All the rings I have seen so far has been with FTZ adapter but not when using redcat with baader adapter or while using cannon 135 art with dumb adapter.

I'm using the AstroHutech mirrorless filter adapter for 2in filters. Hmmm. I thought the problems here were to do with low level processing. 

[thibautflt]

Mark, and all people following. I have something new. When I tape over electronical contact, things change. I know most of you use telescope, but in my case as I mostly use wide angle lens (and small tele up to 200mm) so I don't need it. Here is what I just obtained. There is still the discontinuity but... rings seem to vanish.

Attached Thumbnails

• 

[sharkmelley]

Mark, and all people following. I have something new. When I tape over electronical contact, things change. I know most of you use telescope, but in my case as I mostly use wide angle lens (and small tele up to 200mm) so I don't need it. Here is what I just obtained. There is still the discontinuity but... rings seem to vanish.

That's an important piece of information, that the behaviour changes when electronic contact is disabled.  So it would appear that if the lens is recognised by the D850 firmware then some kind of correction is applied to the raw data even if lens corrections are switched off in the camera menu.  Sony mirrorless cameras also behave in this way.

 

The situation with the Nikon Z6 is worse because it applies a correction to the data even when the lens is unrecognised or if a telescope is attached instead of a lens.

 

Mark

[tonyt]

 

The situation with the Nikon Z6 is worse because it applies a correction to the data even when the lens is unrecognised or if a telescope is attached instead of a lens.

 

Mark

.... but is there a difference when the FTZ adapter or a generic dumb adapter is used? ie. if the FTZ adapter makes the contacts is that enough to cause corrections versus an adapter with no electronics, independent of what is on the lens side of the adapter? 

[sharkmelley]

.... but is there a difference when the FTZ adapter or a generic dumb adapter is used? ie. if the FTZ adapter makes the contacts is that enough to cause corrections versus an adapter with no electronics, independent of what is on the lens side of the adapter? 

That's correct - the FTZ adaptor will make the electronic connection to the lens but the dumb adaptor will not.  It's potentially possible that the Nikon Z6 will apply additional corrections when it recognises the lens via the electronic connection but I have no evidence of this.

 

Mark

[tonyt]

Interesting point you make. All the rings I have seen so far has been with FTZ adapter but not when using redcat with baader adapter or while using cannon 135 art with dumb adapter.

When I get a clear moonless night I'll repeat my m45 image with the dumb adapter and hopefully the ring will be gone - be nice if the fix is this simple, which it may be judging by the D850 results above.  

[sharkmelley]

When I get a clear moonless night I'll repeat my m45 image with the dumb adapter and hopefully the ring will be gone - be nice if the fix is this simple, which it may be judging by the D850 results above.  

We already know that the Nikon Z6 will produce rings with a dumb adaptor.  You don't need to wait for clear moonless night to prove it for yourself.  See my Nikon Z6 experiment here:

https://www.cloudyni...ting/?p=9641613

or my Nikon D5300 experiment here:

https://www.cloudyni...ncentric-rings/

 

Mark

Edited by sharkmelley, 23 January 2021 - 06:31 PM.

[whwang]

Hi,

 

The few recent posts confuse me a bit.  Does it mean on D850/D810, the problem will disappear if we tape the electronic contacts?  I remember when I used D810A on TAK E-180ED, I still see a ring in the flattened image.