Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Does the histogram at 50% produce noisy, poor-quality images?

  • Please log in to reply
17 replies to this topic

#1 Rudy Pohl

Rudy Pohl

    Ranger 4

  • -----
  • topic starter
  • Posts: 350
  • Joined: 27 Dec 2016
  • Loc: Ottawa, Ontario Canada

Posted 16 October 2019 - 11:16 AM

Hi there,

 

I am working on processing a 3-hour stack of images (not my own) of M45 to California nebula that should be producing a very nice image showing a good level of dust. The sky was a Bortle 4 to 4.5, the guiding was well under 1 arcsecond for the entire time, and yet the final stacked image was incredible noisy and impossible to process into a good-quality image,

 

Questions:

Could at least part of the reason be that the subs were 6 minutes in length with the resulting historgram was at the 50% mark producing very bright subs?

 

I have always under the impression that the optimal exposure for the subs has been reached when the histogram is at about 1/4 to 1/3 from the left edge. Is this not correct?

 

If a histogram reading at 50% is indeed too much light, what exactly happens to an image when it is overexposed? Does it degrade somehow to produce a noisy image?

 

Thanks
Rudy


Edited by Rudy Pohl, 16 October 2019 - 11:30 AM.


#2 Samir Kharusi

Samir Kharusi

    Mercury-Atlas

  • *****
  • Posts: 2588
  • Joined: 14 Jun 2005
  • Loc: Oman

Posted 16 October 2019 - 11:51 AM

California Nebula should be OK using a modded camera. M45 would be challenging in such a polluted sky because it is bluish white and very dim. From what you say: Bortle 4 to 4.5 = Mag 20/sq arc-sec, roughly. 6 minutes to mid histogram on the Back of Camera Histogram (BoC)? That would imply an f4 OTA? 3-hours will be satisfactory at a dark site even on M45, even one hour will be reasonably nice but your Mag 20/sq arc-sec is around 3 to 5 times worse hence 3 hours is barely enough for M45. An Ha filter would work wonders on the California Nebula though. Assuming modded camera of course.

 

I am using this to try to work things out in reverse but you did not mention the iso used. A BoC Histogram at 50% is not excessive but a Linear Converted Histogram at 50% would be excessive (too little headroom left). More explanation here.



#3 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 7494
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 16 October 2019 - 12:16 PM

I don't know, but I've gotten reasonably good images of the Pleiades in just over 1.5 hours from a red/orange zone (measured SQM ≈ 18.9) using an unmodified Sony NEX-5R APS-C camera. However, I used lots of very short exposures (15 seconds on an f/4.2 refractor). But, longer exposures should still be okay other than that you might saturate most of the stars (depending upon your aperture, f-ratio, and ISO setting).

 

Here is a link to that image on Flickr (taken almost six years ago, with more information about the capture):  https://www.flickr.c.../in/dateposted/


Edited by james7ca, 16 October 2019 - 12:18 PM.


#4 t_image

t_image

    Gemini

  • -----
  • Posts: 3294
  • Joined: 22 Jul 2015

Posted 16 October 2019 - 02:15 PM

Hi there,

 

I am working on processing a 3-hour stack of images (not my own) of M45 to California nebula that should be producing a very nice image showing a good level of dust. The sky was a Bortle 4 to 4.5, the guiding was well under 1 arcsecond for the entire time, and yet the final stacked image was incredible noisy and impossible to process into a good-quality image,

 

Questions:

Could at least part of the reason be that the subs were 6 minutes in length with the resulting historgram was at the 50% mark producing very bright subs?

 

I have always under the impression that the optimal exposure for the subs has been reached when the histogram is at about 1/4 to 1/3 from the left edge. Is this not correct?

 

If a histogram reading at 50% is indeed too much light, what exactly happens to an image when it is overexposed? Does it degrade somehow to produce a noisy image?

 

Thanks
Rudy

A few things.

[if you know such, disregard, maybe some readers may find it helpful]:

#1 Samir is spot on to differentiate the difference between histogram from a consumer camera showing a histogram based on the (jpg-like) image that is displayed, v. the linear data of the image itself.

#2 keep in mind what a histogram shows:a distribution graph of the number of pixels(y-axis height) of a given luminance(x-axis left to right).

It will tell you the span and whether you are crushing the blacks or clipping the highlights, but it also very much depends on the subject matter. It is helpful you described the image (16 deg FOV?)-so lots of sky and not filled with bright nebula as if you were capturing CA nebula alone......IMO the histogram alone doesn't give you the full scope of the situation, though..

#3 visual noise is interesting, because sometimes it means you are seeing noise in the shadows due to the exposure/color correction,

which otherwise is still there, but would be masked/hidden with a lower exposure/color correction....

When an image is over-exposed (in areas of the image), you will lose details/color in the highlights/brightest parts of the capture.

In the shadows, if you haven't adjusted the image up, you would not have increased noise compared with a capture with all same except exposure time. [assuming no additional noise is added by the system]

^ of course depending on the types of noise affecting the system-say upper atmosphere moisture one wasn't aware of, thermal noise with the longer exposures, etc....

 

So what exactly does the noise look like in the stack? Is it in the highlights (nebulosity of pleiades and CA neb?) Is it in the sky and among the dust/dark nebula?

 

Here's the most difficult problem:

What might have happened with the individual subs that were exposed is that the darkest pixels might not be so dark and your lightest pixels are over-exposed, compared to the detail one was aiming for:

expose so dark space =pixels at low values, dust =mid, CA &M45 nebulosity bright.

From this point the capturing dynamic range of collection (was crushed)  compare with the scene dynamic range (especially within the range one desired to capture)......

In so doing,

one will find the noise (grain) will have the same luminance values as the desired shadow detail (dust),

so one cannot just adjust the (offset) or other color correction tools to drop the noise down to the noise floor,

because in so doing,

you will be dropping the shadow detail desired (dust) down as well.....

your noise (grain) with the stack shares the same luminance values.....

 

Additionally, if you think of all the subs as if they were frames of a motion video,

[something that could be a thought-open about 30 subs in LR/Bridge, color correct them for decent subject visibility, and then open them in Photoshop (script>load files into stack) and make them into a short image sequence and look at the movie...Where is the dancing noise?]

consider the 'noise' is temporal and spatially dynamic,

while the shadow details of the target (like dust) should be static.........

This is another way in which one can tackle noise reduction.

 

But since you are looking at the stack,

what would happen if you went through and color corrected the individual subs so the darkest pixels are lowered?

Maybe you could find in the subs a differentiation between the shadow detail you want and the the noise you don't want in luminance values....

If so you can crush the noise out, keep the shadow detail,

and then the stack won't have the noise in the shadows you have now.......

But this is assuming the scenario where the noise is in the shadows, not the highlights,

and  "bright" subs means you may have lost detail in the highlights=not recoverable.....

 

I intentionally refrained from using specific terms in the above,

as maybe some of the processing heavyweights will add in with troubleshoots they use with the particular AP software packages....



#5 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2417
  • Joined: 19 Feb 2013

Posted 16 October 2019 - 02:23 PM

Hi Rudy,

 

Back-of-camera histogram at 50% is absolutely fine.  I operate both my Sony A7S and Nikon Z6 with the histogram at 50% because of limitations in both cameras.

 

The image noise you are seeing will be certainly caused by the level of light pollution - this will be the major source of noise.  You can make the subs shorter but it won't help.  The only answer is to have a longer total integration time.

 

Mark


  • t_image, 2ghouls and ChristopherBeere like this

#6 Rudy Pohl

Rudy Pohl

    Ranger 4

  • -----
  • topic starter
  • Posts: 350
  • Joined: 27 Dec 2016
  • Loc: Ottawa, Ontario Canada

Posted 16 October 2019 - 03:57 PM

Hi Rudy,

 

Back-of-camera histogram at 50% is absolutely fine.  I operate both my Sony A7S and Nikon Z6 with the histogram at 50% because of limitations in both cameras.

 

The image noise you are seeing will be certainly caused by the level of light pollution - this will be the major source of noise.  You can make the subs shorter but it won't help.  The only answer is to have a longer total integration time.

 

Mark

Hi Mark,

 

Thanks for your reply. I will need a little more time to effectively digest some of the other more complex posts above, but in the meantime I can manage yours. So what you're saying is the length of sub really doesn't matter, but it's the total integration time that counts, right.

 

Given that fact, would you guess that a 3-hour stack under the conditions I described above, where the dust is visible but really noisy, would be significantly improved with 3 hours of integration? Also, would using a faster lens and opening up the aperture help reduce the overall time needed?

 

Thanks,

Rudy 



#7 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2417
  • Joined: 19 Feb 2013

Posted 16 October 2019 - 04:08 PM

Hi Mark,

 

Thanks for your reply. I will need a little more time to effectively digest some of the other more complex posts above, but in the meantime I can manage yours. So what you're saying is the length of sub really doesn't matter, but it's the total integration time that counts, right.

 

Given that fact, would you guess that a 3-hour stack under the conditions I described above, where the dust is visible but really noisy, would be significantly improved with 3 hours of integration? Also, would using a faster lens and opening up the aperture help reduce the overall time needed?

 

Thanks,

Rudy 

Every doubling of the total integration time will have a noticeable effect on noise in the stack.  I should have mentioned that changing the lens aperture to be one stop faster will have the same effect as doubling the total integration time.  Both have the effect of collecting twice as many photons and this improves the signal-to-noise ratio in the stack.

 

Mark


Edited by sharkmelley, 16 October 2019 - 04:09 PM.


#8 rkinnett

rkinnett

    Mariner 2

  • *****
  • Posts: 246
  • Joined: 08 Aug 2018
  • Loc: La Crescenta, CA

Posted 16 October 2019 - 06:28 PM

I respectfully disagree with most of Mark's reply except his final point that longer total stacked integration time would help.

 

The short answer is yes, source frames with histogram peaks around 50% will result in higher "noise" (using the term loosely) after you stretch, all else being equal.

 

The problem is that your usable dynamic range has been cut by 50% and the DSO signal you're ultimately trying to recover is spread across half as many discrete values as it would be if you had the full dynamic range available to you.  You will need a much stronger stretch to spread the DSO signal across the mid-tone range in your stretched image, and this stretching more severely amplifies not only background noise but also quantization noise.

 

Consider the light you've captured in your image is composed of background light which dominates the histogram peak, foreground light (stars), and light from the target DSO.  Your preview histogram reflects all three of these sources combined.  If you could magically differentiate between foreground, background, and target signal, this is what your separated component histograms might look like, relative to your original total light histogram.  This example illustrates histograms resulting from ideal exposure and gain:

hist1_ideal_raw.jpg

 

Your goal in post-processing is to enhance detail in your target DSO without saturating your darks or lights and avoiding over-amplifying pixel-scale noise.  This typically involves pushing the background peak down to the edge of saturation, then applying an s-curve stretch to compress both the highs and lows and spread the mid-range DSO signal across the image dynamic range:

hist2_nonlinear_stretch.jpg

 

Notice that the DSO signal now spans a broader portion of the image dynamic range.  This width is what you want.  It's contrast.

 

However, stretching non-linearly like this inherently amplifies underlying shot noise, discretization noise, etc. in your image.  The deeper you have to stretch to recover detail in your DSO, the more you amplify noise.

 

Now consider what limitations arise from overexposed source frames.  Overexposing pushes the source frame histograms to the right.  If your source frames are so overexposed that the central peak is near 50% of your dynamic range, then you would either have to sacrifice your brighter stars:

hist3_overexposed_clipped.jpg

 

..or use lower gain value to compress the foreground signal to fit within the remaining 50% of your available dynamic range:

hist4_overexposed_compressed.jpg

 

But using lower gain compresses your DSO signal more than you would need to otherwise.  Note how narrow the DSO histogram peak is here compared to the first histogram showing ideal gain and exposure.  With the DSO signal so compressed, you can see how it would take a much deeper curve stretch to reveal DSO structure, approaching the histogram shown above for an ideal non-linear stretch.

 

Point being, balancing gain and per-frame exposure to spread your lights and darks across as much of your sensor's dynamic range as possible without clipping at either end will yield better inherent contrast, requiring less stretching and ultimately less noise amplification.

 

Stacking longer total exposure time reduces underlying pixel-scale noise, giving you more headroom to stretch your image to boost DSO contrast before the amplified noise becomes unpleasant.


  • Arthur NY likes this

#9 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2899
  • Joined: 20 Mar 2013

Posted 16 October 2019 - 08:59 PM

Hi Rudy,

 

Mark is right.  Aiming at 50% histogram is the right thing to do.  If you can achieve this and your final result is still noisy, then the most probable explanation is that the total integration is too short under the brightness of the sky.  Bortle 4-4.5 isn't that dark, and 3 hr isn't that long.  I wouldn't be surprised by the noisiness in the final stack even if you do everything right in the single subs.

 

Cheers,

Wei-Hao


  • zxx and sharkmelley like this

#10 rkinnett

rkinnett

    Mariner 2

  • *****
  • Posts: 246
  • Joined: 08 Aug 2018
  • Loc: La Crescenta, CA

Posted 16 October 2019 - 09:39 PM

Mark is right.  Aiming at 50% histogram is the right thing to do. 

I can't fathom how this could be a good recommendation, but you sir make amazing images, so you must be doing it right!  :)



#11 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2899
  • Joined: 20 Mar 2013

Posted 16 October 2019 - 10:24 PM

The point here is read noise, not dynamical range.  If you don't expose sufficiently for the image background to be much higher than read noise, the effect of read noise will keep accumulating in the deep stack, leading to much noisier results.  It there is no read noise at all, then stacking 3600 1-second exposures will be just like taking one single 3600-second exposures.


  • zxx, 2ghouls and rkinnett like this

#12 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2417
  • Joined: 19 Feb 2013

Posted 17 October 2019 - 02:01 AM

I respectfully disagree with most of Mark's reply except his final point that longer total stacked integration time would help.

 

In your reply you are not making a distinction between the back-of-camera histogram and the histogram of actual data values in the raw file.  The BoC histogram is the histogram of the internally calculated JPG i.e. the data has had its exposure pushed (to make headroom for highlights in post-processing) and has been non-linearly stretched (by a gamma of approx. 2.2) and has been saturated, contrast enhanced etc. - just like Photoshop/Lightroom does when it opens a raw file.

 

You can try a very quick and simple experiment.  Expose a DSLR flat frame so the BoC histogram is at 50%.  Look at the pixel values in the raw file and subtract the bias level.  The result will vary from DSLR to DSLR but you will find they are surprisingly small and certainly nowhere near halfway to saturation.

 

Of course, exposing at 50% does reduce dynamic range in the final image and it does saturate a greater number of stars, so it's not a blanket policy that I would recommend.  But the size of the effect is nowhere near what you illustrated in your graphs. 

 

In any case, exposing at 50% is not the cause of "noisy poor-quality images" in the original post.

 

Mark


Edited by sharkmelley, 17 October 2019 - 04:49 AM.

  • RedLionNJ and 2ghouls like this

#13 mic1970

mic1970

    Viking 1

  • -----
  • Posts: 885
  • Joined: 09 Aug 2018
  • Loc: St. Louis Metro Area

Posted 17 October 2019 - 10:06 AM

If you covered this, apologies.  Did you use calibration frames?



#14 rkinnett

rkinnett

    Mariner 2

  • *****
  • Posts: 246
  • Joined: 08 Aug 2018
  • Loc: La Crescenta, CA

Posted 18 October 2019 - 09:19 PM

In your reply you are not making a distinction between the back-of-camera histogram and the histogram of actual data values in the raw file.  The BoC histogram is the histogram of the internally calculated JPG i.e. the data has had its exposure pushed (to make headroom for highlights in post-processing) and has been non-linearly stretched (by a gamma of approx. 2.2) and has been saturated, contrast enhanced etc. - just like Photoshop/Lightroom does when it opens a raw file.

 

You can try a very quick and simple experiment.  Expose a DSLR flat frame so the BoC histogram is at 50%.  Look at the pixel values in the raw file and subtract the bias level.  The result will vary from DSLR to DSLR but you will find they are surprisingly small and certainly nowhere near halfway to saturation.

 

Of course, exposing at 50% does reduce dynamic range in the final image and it does saturate a greater number of stars, so it's not a blanket policy that I would recommend.  But the size of the effect is nowhere near what you illustrated in your graphs. 

 

In any case, exposing at 50% is not the cause of "noisy poor-quality images" in the original post.

 

Mark

How is this relevant?  The OP didn't say he was reading 50% on a back-of-camera histogram.

 

I'm not saying that exposing to 50% by itself is a source of noise, although one interpretation of the elevated histogram is that his gain was too high, and higher gain unquestionably amplifies shot noise.  My main point was that when you expose to 50% you need a much stronger non-linear stretch to bring the darks back down in your final composition, and, generally speaking, the stronger the stretch, the more you amplify underlying noise from whatever source.

 

 

The point here is read noise, not dynamical range.  If you don't expose sufficiently for the image background to be much higher than read noise, the effect of read noise will keep accumulating in the deep stack, leading to much noisier results.  It there is no read noise at all, then stacking 3600 1-second exposures will be just like taking one single 3600-second exposures.

The point here is both read noise and dynamic range.  Of course there's read noise in his subs.  The question is how his final stack got to be so noisy.  He either started with inordinately high read noise, or he may have had relatively normal noise levels in his initial stack but severely amplified the noise in post processing.  I'll go out on a limb and speculate that his subexposure noise levels were perfectly normal.  His subs are 6 minutes long.  If they were only 1 minute long and his histograms were 50% biased then I would suspect his gain was too high, possibly outside of the linear range of his camera, but in this case I'll bet it was close to unity if not lower.  What might have started as manageable noise levels in his initial biased-high stack was amplified when he compensated for dynamic range in his stretched composition.  Dynamic range compensation did not cause noise.  It amplified it.



#15 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2417
  • Joined: 19 Feb 2013

Posted 19 October 2019 - 02:32 AM

How is this relevant?  The OP didn't say he was reading 50% on a back-of-camera histogram.

 

I'm not saying that exposing to 50% by itself is a source of noise, although one interpretation of the elevated histogram is that his gain was too high, and higher gain unquestionably amplifies shot noise.  My main point was that when you expose to 50% you need a much stronger non-linear stretch to bring the darks back down in your final composition, and, generally speaking, the stronger the stretch, the more you amplify underlying noise from whatever source.

 

True, the OP didn't explicitly say he was using the BoC histogram  But almost everyone using DSLR cameras does use the BoC histogram as a guide (typically aiming for a level 1/4 to 1/3) and when I replied saying that BoC at 50% is fine, the OP didn't contradict me.  Besides, I know Rudy to be an experienced DSLR imager producing excellent work over the years and he would never have made the fundamental error you are implying.

 

Your reply shows some other misconceptions or maybe you have just written it in a confusing manner. To clarify for the benefit of other readers:

  • High gain does not contribute to shot noise. By definition shot noise is caused purely by the number of photons recorded, which is not affected by gain.  Instead, high gain (i.e. high ISO) typically has the effect of reducing read noise.
  • In processing we do not perform a nonlinear stretch to adjust the black point - we simply subtract the light pollution.  This is a linear operation on the data.
  • In general, if the total exposure and length of subs is kept the same, a stack shot with a BoC histogram at 50% will contain the same signal to noise as a stack shot with the BoC histogram at 1/4. If anything it will have lower noise (by a tiny amount) because the higher ISO used will have lower read noise.

Mark


Edited by sharkmelley, 19 October 2019 - 06:53 AM.


#16 rkinnett

rkinnett

    Mariner 2

  • *****
  • Posts: 246
  • Joined: 08 Aug 2018
  • Loc: La Crescenta, CA

Posted 19 October 2019 - 12:57 PM

BoC histogram is still irrelevant. OP indicated these are someone else’s images. Surely he’s referring to histograms generated by processing software, not displayed on a camera.

You’re right, I noted the wrong noise term affected by gain. That’s a good catch.

You’re also right that most of us apply a linear stretch to pull the darks down. Most of us eventually follow that with a non-linear stretch to bring out mid-range detail, but that’s beside the point. Stretching of any kind inherently boosts noise, and with such overexposed subs, he needs alot of stretching. But you’re right, the phrase “non-linear” added no value to my reply, and was distracting if not misleading. Sorry for that.

#17 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2417
  • Joined: 19 Feb 2013

Posted 20 October 2019 - 01:54 AM

You’re also right that most of us apply a linear stretch to pull the darks down. Most of us eventually follow that with a non-linear stretch to bring out mid-range detail, but that’s beside the point. Stretching of any kind inherently boosts noise, and with such overexposed subs, he needs alot of stretching.

Your phrase "linear stretch to pull the darks down" is confusing.  Not many people would describe it in those terms. To clarify, we perform a subtraction operation to remove the background light pollution i.e. to adjust its black point. 

 

It's true that you might also subsequently or simultaneously adjust the white point, effectively stretching the data.  You appear to view these operations in terms of their effect on the histogram and that's a good way of understanding them.

 

However, the root of your misunderstanding is where you say "stretching of any kind inherently boosts noise".  Technically that statement is correct but taken in isolation it is completely misleading. The truth is that a linear stretch scales both the noise and the faint signals of interest by the same factor.  To put it another way, linear operations do not alter the signal to noise ratio (SNR) in the data.  By signal I mean the faint image structures we are trying to reveal.

 

Another point to understand is that identical long exposures under identical sky conditions but with different ISO will result in the identical SNR (as long as the read noise is sufficiently swamped).  This is because signal and noise increase (or decrease) together when you change ISO. In other words, the only effect of altering the position of the histogram peak is to change the level at which stars begin to saturate.

 

In summary, the signal to noise ratio is a key concept which is worth taking time to understand better.

 

Mark


Edited by sharkmelley, 20 October 2019 - 06:57 AM.


#18 RedLionNJ

RedLionNJ

    Soyuz

  • *****
  • Posts: 3739
  • Joined: 29 Dec 2009
  • Loc: Red Lion, NJ, USA

Posted 20 October 2019 - 11:01 AM

Great (and quite respectful) discussion, Ryan & Mark.  Good to see this sort of thing on CN. Much-appreciated in this day and age.

 

And nice to meet you in person in Europe earlier this month, Mark!

 

Grant


  • sharkmelley likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics