58 replies to this topic

[schmeah]

I'm interested in why this worked at all.  

 

...when an offset is added, it brings those pixels away from dead black, and our eyes see this as less of a contrast difference?

I would think that's it. But not even dead black. The background doesn't have to be clipped.

 

 

Derek

[pbkoden]

You're just trying to fill in some of the valleys in the background signal to smooth it out. The valleys don't have to be pure black, they just have to be below the brighter parts of the background signal. Once you fill the valleys, the smoothness of the background profile is much better.

 

 

[Thirteen]

That's a good visual. Generally when I've attacked this issue, I've approached it differently. Since these holes contain signal, merely prepare an inverted luminance mask with a slightly fuzzy and soft range selection that allows adjustment to just the valleys. Then I apply a slight histogram boost to these areas. You may think that stretching these areas would alter the noise profile throughout the background, but in practice the necessary adjustment is so slight that it's not noticeable.

These tidal streams of NGC5907 .... beware. It's a humbling experience.

Edited by Thirteen, 29 April 2017 - 05:39 PM.

[pbkoden]

I've given up on the tidal loops for now, but I'll be back to get them sometime in the future when I can get some darker skies.

[dkeller_nc]

That's a good visual. Generally when I've attacked this issue, I've approached it differently. Since these holes contain signal, merely prepare an inverted luminance mask with a slightly fuzzy and soft range selection that allows adjustment to just the valleys. Then I apply a slight histogram boost to these areas. You may think that stretching these areas would alter the noise profile throughout the background, but in practice the necessary adjustment is so slight that it's not noticeable.

These tidal streams of NGC5907 .... beware. It's a humbling experience.

OK, that part I get.  But I don't see why adding a uniform background, which is essentially just adding a pedestal, should smooth those valleys out.  I would think that you'd simply be boosting every pixel by a certain amount, which would still result in non-uniform backgrounds.  Unless something else is going on with the combination algorithm.

[Thirteen]

By the diagram he drew, it looks like he's taking the maximum of either the pedestal or the image. So the valleys are replaced by the pedestal and the peaks are still original data.

Yes, rereading the original post confirms this.

Edited by Thirteen, 29 April 2017 - 09:26 PM.

[Jon Rista]

 

That's a good visual. Generally when I've attacked this issue, I've approached it differently. Since these holes contain signal, merely prepare an inverted luminance mask with a slightly fuzzy and soft range selection that allows adjustment to just the valleys. Then I apply a slight histogram boost to these areas. You may think that stretching these areas would alter the noise profile throughout the background, but in practice the necessary adjustment is so slight that it's not noticeable.

These tidal streams of NGC5907 .... beware. It's a humbling experience.

OK, that part I get.  But I don't see why adding a uniform background, which is essentially just adding a pedestal, should smooth those valleys out.  I would think that you'd simply be boosting every pixel by a certain amount, which would still result in non-uniform backgrounds.  Unless something else is going on with the combination algorithm.

 

It's a uniform level added through a mask...so in the end, it's not a uniform add. It adds more to darker areas and less to brighter areas, and none to the brightest areas. 

[dkeller_nc]

Well, duh, that's what I get from not reading carefully.  The short phrase "Use PixelMath with the Max() operator to add the two together." was the key.

 

So this brings up another question.  Is this a better operation than simply using TGVDenoise and MMT to deal with background noise (which is what I typically do, not always all that successfully)?  Specifically, it's often the "salt and pepper" background that I'm trying to tamp down.  Dithering helps some, but I think most of it is simply shot noise variation in a heavily sky-fogged frame from LP, which dithering doesn't do much for.

 

Thoughts?

[Jon Rista]

Well, duh, that's what I get from not reading carefully.  The short phrase "Use PixelMath with the Max() operator to add the two together." was the key.

 

So this brings up another question.  Is this a better operation than simply using TGVDenoise and MMT to deal with background noise (which is what I typically do, not always all that successfully)?  Specifically, it's often the "salt and pepper" background that I'm trying to tamp down.  Dithering helps some, but I think most of it is simply shot noise variation in a heavily sky-fogged frame from LP, which dithering doesn't do much for.

 

Thoughts?

For moderate to smaller scale mottling, 4-8 pixels or so, MMT will work. For larger scale mottling, techniques as described in this thread might be better.

 

Regarding salt and pepper noise...do you have a screenshot of what you are trying to correct? If it is what I think of as salt and pepper, then dithering, cosmetic correction and rejection during integration are probably better tools. 

[dkeller_nc]

Well, here's one example.  This is an excerpt from a stack of about 8 hours of Ha data, with dithering and drizzle-integrated.

[schmeah]

I know the question pertained to processing in PI, but the principle is the same in PS (ie filling in the valleys without affecting the peaks) so I hope I'm not derailing the thread with this, but I have used the flat background method also for the salt and pepper. 

 

 

You can remove just the pepper:

 

 

Or some of the salt:

 

 

As the level of the flat background continues to increase, the brightness of the image begins to become objectionable. The last version is with the combined levels reduced. Although the brighter stars begin to look a bit harsh, but these can be easily touched up and overall it does a fair job neutralizing the salt and pepper. I don't know how it compares to the typical denoise protocol in PI.

 

 

Derek

 

 

 

 

 

[Jon Rista]

Well, here's one example.  This is an excerpt from a stack of about 8 hours of Ha data, with dithering and drizzle-integrated.Salt and Pepper Extract.jpg

Gocha. Since that is mostly pepper, I call that pitting. Personally, I have resorted to using CosmeticCorrection and ImageIntegration to primarily deal with that. With cosmetic correction, you bring in the cold sigma to catch as many of those dark pixels as you can, without obviously affecting structure or having a large impact on SNR. I will then also do the same with ImageIntegration and sigma clipping...I'll bring in the low sigma to 3, maybe 2, to reject more outlier dark pixels. 

 

There really isn't much salt in you image, none at all, really, as far as I can tell. If there was, however, I would primarily deal with it via CosmeticCorrection. Bring in the cold sigma to clean up more of the moderately brighter outliers.

 

There is a double benefit to cleaning up such noise with CosmeticCorrection. It gets rid of them before the image data has been transformed by registration. That means you are cleaning up the data in it's pure form, original save for calibration. This can eliminate the possibility that any of that salt and pepper noise or pitting could become a locus for artifacts during registration as well. It also helps with sigma rejection in ImageIntegration as well, as it tightens up the standard deviation (since numerous moderate to large outliers cannot overly influence the median calculations). You can get better rejection that way, which helps clean up anything that CosmeticCorrection couldn't identify itself. 

[dkeller_nc]

Very interesting, guys.  I generally have ignored Cosmetic Correction for stuff like this, since the way it was cast in Warren's book was cleaning up just a few hot and cold pixels that dark and bias frame calibration might've missed.  Looks like I've some electricity to burn this afternoon playing around with settings.

 

Derek - That's a fantastic result, and no, your post was spot-on.  Many of us would like to know the specific steps you took in PS to do that - some of us because we use both PI and PS and your technique could be used to squeeze the last little bit out a given image, and others because they use PS exclusively after stacking in something like DSS.  I'm thinking you're using a layer that's been filled with a neutral gray that you then adjust the intensity of to match the average background of the astrophoto, and then one of the blending modes to accomplish filling in the "holes" or subtracting out the "hills".

[schmeah]

 

 

Derek - That's a fantastic result, and no, your post was spot-on.  Many of us would like to know the specific steps you took in PS to do that - some of us because we use both PI and PS and your technique could be used to squeeze the last little bit out a given image, and others because they use PS exclusively after stacking in something like DSS.  I'm thinking you're using a layer that's been filled with a neutral gray that you then adjust the intensity of to match the average background of the astrophoto, and then one of the blending modes to accomplish filling in the "holes" or subtracting out the "hills".

Thanks. I've had to do a lot of experimenting, imaging from a white zone, and your guess is correct. I created a flat background a few years ago, basically by taking one of my best color images that had a very sparce starfield and a small PN that were easy to remove, then with a combination of gaussian blur and dust and scratches filter, similar to JPMs tone mapping techniques came up with this background, which I adjust accordingly with color balance to taste and levels to achieve my desired background with most subsequent images.

 

With images in general I do the following:

1) If it is monochrome image I desaturated the flat background. If it is a color image, I adjust the color balance to roughly match that of the target image (or leave it if I prefer the color of the flat background). 

2) My flat background has a "k value" of 80%. You find this in the window /info tab /CMYK levels. In the reference image, you find a uniform area of background devoid of stars and nebulosity, and run the cursor across this area watching the k value change as you move the curser.

3) In an image with splotchy but otherwise uniform target background, I'll determine the lowest k value (say 70%) and adjust the levels in the flat backround to perhaps 1-2% lower (brighter, say 69%) than that of the target background.

4) Copy the adjusted flat background and paste it over the target background.

5) Change the blending in layers to "lighten"

6) Adjust the opacity % to taste and flatten image.

 

With your image, because it was more a salt and pepper issue it was more guesswork and trial and error. I checked the k value of individual pixels in a magnified segment. The stars had a k value from 0 in the center to 5-10% at the edges and the darkest pixels (pepper) had a k value of 87%. So getting a general sense of the target background level and with successive trial and error, I adjusted the k value of the flat background to 35%, then 25%, then finally 15% which is what you see in the  three examples in the previous post, again layering with a lighten blend. At 15% the image overall was ridiculously bright, so I adjusted the resulting background levels to a k of 50%. Normally, this method has very little effect on star quality and nebulosity, and even on fine nebulosity if done carefully. But in your example, because of the extreme nature of the background neutralization required (wide salt and pepper variation), the brighter stars took on a harsh edge. So a little "less crunchy more fuzzy" application with Carboni's Action Set would fix those up nicely. Anyway, that's about it.

 

Derek 

Edited by schmeah, 30 April 2017 - 08:33 PM.

[dkeller_nc]

Ah - I'd forgotten about the "lighten" blend mode.  I almost never use it in PS, so knew very little about it.  But it makes sense that it would operate much like the "max()" operator in PI, which I'm assuming will use the reference image's pixel if it's lighter than the layer that's being blended, but the blend layer's pixel if the reference image's pixel is darker.  Forgive me if I have that backwards - I suspect that will become immediately apparent when I start messing with it.

 

By the way, is there any particular reason one couldn't simply fill a new layer with neutral gray, and then adjust its intensity to suit?  It would seem considerably less labor intensive than taking a clean image and removing all of the structures from it.

 

Finally, one reason there's such an extreme "salt 'n pepper" effect from the example that I posted is that it's a Histogram stretch copied from the Screen Transfer Function, which generally blows out the background in an astrophoto so one can see fine structure.  I suppose some folks copy the STF and call it a day, but it's generally a little too blown out for my tastes.

[schmeah]

You certainly could fill a new layer simply with neutral gray. I sometimes struggle with background color, so when I came up with this one which I liked, I figured I would just save it to use a reference for future LRGB images. Regarding the salt and pepper, I'm quite familiar with it even without the STF. But because the above technique handles it fairly well, I've been able to get by without dithering.

 

Derek

[dkeller_nc]

Well, that's sort of the thing - dithering doesn't take care of this issue, at least I've been unable to do so.

 

I commented to another member (we've been discussing this thread/technique by email) that the statistical elimination of outliers with the Cosmetic Correction and Image Integration tools in PI is the "kinder gentler" approach.  The flat processing approach that you and the OP detailed is a sledgehammer.  

 

While I doubt Juan Conejero would approve, I was very much in need of a sledgehammer!

 

Thanks greatly for your (and the OPs) input. 

[macnmotion]

PBKoden, your solution for mottling was wonderful to see. I often run into this issue with images from our light polluted skies just outside of Bangkok, Thailand. Today while I was searching for improved DBE techniques I came across this post. Based on your comments I was able to get a very nice result, filling in the darker mottled areas after DBE. However, I'm not entirely sure I made my noise floor image the best way. I'm hoping that you can provide more details of exactly how you created the image (steps, settings). 

 

Here is my before/after on a Luminance stack of NGC 660:

 

 

This allowed me to push my stretch further, with a much better result than I would have had without using this technique:

 

[calypsob]

When I go after the faint stuff I typically reference the world wide telesope and use the plank thermal dust survey.  It includes some Ha data in the dense milkyway regions, but in the galactic cirrus and ifn areas, there is a nice representation of what actually exists.  Its often hard to find actual images of ifn because it is difficult for some to process,  and the necessary conditions are also difficult and rare.  Unless the sky is crystal clear, which is rare for me there always seem to be thermal high level cirrus clouds at some point every night, it is very hard to gather any signal in the low signal areas, thin clouds still pass the brighter regions but the opacity causes us to loose several stops of light, transparency is extremely critical.   You can find astrophotographer Hisayoshi Kato on Flickr, his work is incredible, he has alot of data from high 13,000ft altitudes of mauna kea and has eh perfect combo of fast optics, deep integration, camera sensitivity, and processing.  Often times I reference his work because he can so crisply reproduce super faint data that I cannot find anywhere else.   

[Astro_BC]

I wanted to try this very interesting process on a DSLR/OSC image.  I tried unsuccessfully trying it 2 ways:

1)      With generated noise image, apply PixelMath expression on post-DBE image using the combined RGB/K line.  The end result didn’t remove much/any of the image’s remaining black blotchiness and seemed to make the color noise worse.

2)      With generated noise image, first split post-DBE image into separate R, G, B images.  Then, apply PixelMath expressions on individual R, G, B images. After doing this, the individual R, G, B images look great – the dark blotchiness is gone, leaving an even background.  However, when recombined RGB with Channel combination, the blotchiness returned and I’m not sure why.

Has anyone done this process on RGB images?  What is the right approach?

[miwitte]

yes I would love to know if someone has a more detailed right up for us beginners. I would say that every one of us beginners suffer with this and while TGV and MMT can reduce the effect, if its applied to liberally it will create motteling once you stretch then its a real mess. I would love to know about the CC method as I thought that was just for hot and cold pixles not to get rid of "pepper" which all of us typically have. Also a little write up on how to do a pedestal would be great as well. I finally got some data the last couple nights and would like to try this out myself...

[pbkoden]

Here's a tutorial using PixInsight.

 

1) Start with a noise reduced  monochrome image with no stretch applied. Here is my sample image. You can see some fine mottling in the background data.

 

 

2) Measure the nominal background level using the probe tool. Set the probe size to 11 pixels and readout type to normalized integer range (1e-05). Mine is reading about .00058.

 

 

3) Now set the probe tool to single pixel and measure your darkest and lightest background noise pixels. Mine range from .00048 to .00070.

 

4) Using NewImage, create a new image the same size as your target image, with the background level set to your nominal background.

 

 

5) Use the NoiseGenerator tool to generate uniform noise with an amplitude equal to your max background minus your min background. Mine is .00022.

 

 

 

6) If you zoom in on the new image and compare it to your original, you will see that the noise is much sharper than the original. Use a mild convolution to soften them so that they look similar. Use a small preview window and real-time preview to get a close look at the noise. In my case a StdDev of about 0.7 worked well.

 

 

7) Now in theory, we have two images with a roughly equal background noise in both nominal value and amplitude. Now you can play with the Pixelmath tool. Change your output to new image, and for the expression, use max(image1,image2). You can fine-tune the operation with a multipler on the noisefloor image. For instance, max(image1,image2*1.02). Here is my original image, the pixelmath with a multipler of 1.0, and a multiplier of 1.02 (with a boosted STF applied). You can see the mottling gets lower with higher multipliers. But you need to be sure not to over-do it.

 

 

Here are the original and my new image (1.02 multiplier) with a more reasonable screen transfer function. I've eliminated the mottling, but have maintained faint details and the grain structure of the background noise. Now I've probably  blown away any IFN or REALLY faint data, but my data wasn't good enough for capturing that kind of detail in the first place.

 

Edited by pbkoden, 07 January 2018 - 04:27 PM.

[miwitte]

That is fantastic tutorial I cannot wait to try it on my new data as well as some of my reprocessing. I have struggled like others using DBE and getting the mottled issue. Hopefully this will help. The other thing I'm hoping other will chime in on is using cosmetic correction during pre processing to eliminate some of this "salt and pepper" look that I think most of us have.


I assume I just work on 1 sub to see what hot and cold pixel settings reduce the issue then run it on the whole stack.

Thanks so much for posting that this is probably my biggest roadblock right now everything else is going real well I just am lost after DBE and it looks horrific.

[pbkoden]

Thanks so much for posting that this is probably my biggest roadblock right now everything else is going real well I just am lost after DBE and it looks horrific.

 

I hope it helps you out. I'd be interested in seeing what it does for you.

 

DBE is a strong tool, but you need to be very careful when placing samples. If you end up with dark splotches after running DBE, it usually means you placed a sample too close to a star. If you save the process icon, you can go back and see which sample caused the dark splotch and move it to a safer area and then re-run.

[miwitte]

Also not so great with pixel math could you spare a screen shot with how to do it? Something like this?

Also where in the workflow to do this, when its still linear? Typically after I get my masters, I crop, TGV, MMT, then DBE(which typically results in my mottled mess). Would this be before or after DBE or right after crop?