Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Removing noise from an Image

astrophotography
  • Please log in to reply
17 replies to this topic

#1 BobinBend

BobinBend

    Mariner 2

  • -----
  • topic starter
  • Posts: 205
  • Joined: 02 Mar 2018
  • Loc: Bend, Oregon

Posted 26 June 2019 - 09:54 PM

The attached image NGC 6543 (in JPG format) was taken under less than best conditions. Consequently it contains a lot of noise in the background (primarily red dots). I've attempted to use a number of noise reduction processes to remove the noise but to no success. I calibrated it with about 40 darks, 20 fresh flats, and 40 bias frames using DSS and PI's BatchProcessing script. Both produce similar noise issues. Is there a process in PI that can remove the red dots of noise without affecting the nebula? Or do I have to retake the image which I intend to do anyway.
Thanks for your suggestions, Bob

Attached Thumbnails

  • NGC6543_Sample.jpg

Edited by BobinBend, 26 June 2019 - 09:56 PM.

  • calypsob likes this

#2 mistateo

mistateo

    Apollo

  • *****
  • Posts: 1233
  • Joined: 02 Feb 2017
  • Loc: San Diego, CA

Posted 26 June 2019 - 11:14 PM

Simply put, you need more exposure, and also to shoot in raw format, not jpg.  It's good that you have darks, flats and bias integrated, as it probably cleaned up the data you do have a bit, but you need a lot more "light" frames.  It is critical to shoot in raw format to preserve the dynamic range of your camera, and to not destroy information before stacking/processing by converting to jpg.  Noise in your light frames is pretty random, so by taking many frames and stacking them, the noise gets averaged out.  The more light frames you have, the smoother your noise profile should be (to a point, there are certainly diminishing returns at play here).  Also, the brighter your sky is (measured on the bortle scale) the noisier your data will be, and the more integration time you will need to overcome the noise.


  • ac4lt likes this

#3 scadvice

scadvice

    Apollo

  • *****
  • Vendors
  • Posts: 1126
  • Joined: 20 Feb 2018
  • Loc: Lodi, California

Posted 27 June 2019 - 12:19 AM

First your never going to get rid of all the noise but you can reduce it. The Canon is a noisy camera and the warmer it gets the nosier.

 

Two things I have heard about to help and I do the first.

 

Leave the flip open screen open. This allows warm air to escape easier so the camera runs a little cooler. So less noise.

 

I heard of keeping a small fan pointed toward the camera on low speed to help also to move heat away. I never tried that one.

 

 

PI.... Here is a video by Astro Dave on noise removal (Pixinsight Part 8 noise reduction and Full stretch). Worth watching if you have not seen it.

 

https://www.youtube....Hy531xgLUw&t=5s



#4 kathyastro

kathyastro

    Messenger

  • *****
  • Posts: 418
  • Joined: 23 Dec 2016
  • Loc: Nova Scotia

Posted 27 June 2019 - 07:29 AM

I agree with the suggestions above to avoid getting noise in the first place: many frames, save in Raw format, try to keep the camera cool, etc.

 

But once you are into processing, PI has many noise reduction processes.  I usually do a noise reduction using the ATrous Wavelet Transform as the very first thing I do to a master file fresh out of the BPP script.  At later stages of processing, I use the ACDNR process, which is good at removing the type of chromatic noise seen in your image.  If I am having trouble with a noisy image, I will try some of the other noise reduction processes, but those two ar my "standard" ones.


  • epdreher likes this

#5 WadeH237

WadeH237

    Skylab

  • *****
  • Posts: 4416
  • Joined: 24 Feb 2007
  • Loc: Snohomish, WA

Posted 27 June 2019 - 07:44 AM

Taking many exposures and dithering will greatly reduce the chrominance noise that is prominent in Canon DSLRs.



#6 BobinBend

BobinBend

    Mariner 2

  • -----
  • topic starter
  • Posts: 205
  • Joined: 02 Mar 2018
  • Loc: Bend, Oregon

Posted 27 June 2019 - 07:19 PM

First of all thanks for all your inputs. I should have been more clear about the JPG file, which was converted from a raw stacked image for the purpose of posting it on this forum. Also I failed to mention that I took 41 lights at 300 sec exposures, which is about 3.4 hrs. I also took the images on June 3rd which was a new moon condition. The temperature that night was about 17C or 62F here in Bend OR, so it wasn't too warm. 

 

Scadvice, I will try your suggestions to see if I can reduce the noise, and check out the PI video.

Kathyastro, I never thought of applying two different noise reduction processes to the image, so I will attempt that as well. 

WadeH237, I've never used dithering but will check it out as well. It seemed like one more layer of complexity to an already complex series of steps in capturing images so I've stayed away from it in the past.

 

Normally I don't have the kind of noise I experienced with this image, even though as some of you mentioned the Canon 60D can be noisy when it gets warm. But I'd like to salvage this nebula if at all possible.

 

Thanks for any further suggestions

Bob



#7 pscm

pscm

    Mariner 2

  • *****
  • Posts: 273
  • Joined: 27 Jun 2009
  • Loc: Maine

Posted 27 June 2019 - 08:17 PM

If you can share the files some will gladly take a crack at it.  I personally use star tools.  I started there as the .earni g curve is reduced substantially as I understand.  I used a 3TI and what you posted is about normal from my experience before stretching and noise reduction.



#8 BobinBend

BobinBend

    Mariner 2

  • -----
  • topic starter
  • Posts: 205
  • Joined: 02 Mar 2018
  • Loc: Bend, Oregon

Posted 29 June 2019 - 05:54 PM

I thought I would take a closer look at my capture procedure and discovered (I should have caught this sooner) a Meridian flip about 30 subs into the capture sequences. Although all the images on both sides of the flip appeared to look ok both DSS or PI's BatchProcessing seemed to add a noise into the background of the image. I'm not sure why. When I used DSS to stack the largest set of images before the flip the background noise was greatly reduced. I then used PI to process the image through completion. I'm not satisfied with the image but the seeing and transparency conditions were far from ideal.
I've attached a copy of the final image.
Bob

Attached Thumbnails

  • NGC6543_DSS_Final_Small.jpg


#9 scadvice

scadvice

    Apollo

  • *****
  • Vendors
  • Posts: 1126
  • Joined: 20 Feb 2018
  • Loc: Lodi, California

Posted 29 June 2019 - 06:10 PM

I suspect your issue first to address is focus. I know for me it was. I picked up a Bahtinov mask and along with watching this video have improved my focus dramatically. 

 

https://www.youtube....h?v=k0FIluj9ndQ

 

Have any of us left the Bahtinov mask on after focusing and tried to image with it on?///Naaaa.  coolnod.gif



#10 Sven_Bortle5

Sven_Bortle5

    Explorer 1

  • -----
  • Posts: 58
  • Joined: 08 Feb 2019
  • Loc: Düsseldorf, Germany

Posted 30 June 2019 - 04:12 AM

Hi use a Canon 70D and tried to figure out when noise reduction really gets visible. Following the advice above, here are my experiences of the last 9 months:

 

  • I need 5-10 hours of total exposure (in lightframes) in order to see a significant reduction of noise after stacking.
  • For each physical setup of the telescope (or project), I produce a set of 120 biasframes at the average temperature of the surrounding (I know temperature shouldn’t have an impact on bias frames, but I saw an Australian photographer making excellent shots of the Orion Nebula which made adopt this easy step).
  • For each physical setup of the telescope, I produce 60-80 flat frames. I bought a cheap lightbox for this, because fiddling around with a t-shirt and a rubber band and carrying the rig around was way more stressful.
  • A you have to shoot multiple nights for getting 5-10 hours, I shoot dark frames before and after every session - the lights at the beginning and the end will be the brightest anyway. Though I tend to get greedy with lightframes, I have quickly realized that the darks are as important. I take 20 darks per session now, in the summer up here in Germany. The remaining time is spent on lights.
  • And finally... just found this out last week, by reading CN (thanks to CN!)... one has to use groups in DSS. I throw all flat and bias frames into the master group and lightframes and dark frames (i.e. night) in a separate group. Didn’t know this feature before, but it’s only logical for removing exactly that noise which has been produced in a night’s session.

And outside temperatures above 30°C appear to be a no-go for my 70D.

 

Regards,

Sven


Edited by Sven_Bortle5, 30 June 2019 - 04:13 AM.


#11 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 6965
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 30 June 2019 - 08:21 AM

You've got a lot of very high pixel values in the red channel. I'm not sure where this noise originated (hot pixels or just a bad calibration), but you can remove most of those defects with a simple PixelMath expression. I wrote a quick example, where I check for each red pixel in the image and if it is greater than the average of the green and blue pixel by more than 0.1 then the red pixel is replaced by the green and blue average (giving kind of a neutral value).

 

Below is a screen shot of the PixelMath expression editor showing the math needed to do this fix. But, this should really be done before the image is converted to JPEG and perhaps even while the image is in linear format. However, you'll need to change the 0.1 offset to something lower when the file is still in a linear format.

 

With some further tweaks to the expression you could do a better job, like checking to see if the green and blue average is above a certain level (so that you don't affect any bright features that are actually okay).

 

Also, a copy of the corrected image (after the PixelMath fix). Note, after removing a large part of the red noise I notice that there are also a fair number of bright greenish pixels. Well, you could then process the image a second time to remove those defects (with probably a different offset factor, probably not the 0.1 I used for the red correction). Note, I just guessed at that offset value, although I did measure a few pixels and the mean of the background image to get some idea of the general brightness of the image.

 

You may also want to look at PixInsight's CosmeticCorrection tool, since that too can replace outlier pixels.

Attached Thumbnails

  • PixelMath to Fix Red Pixels.jpg
  • PixelMath Corrected NGC_6543.jpg

Edited by james7ca, 30 June 2019 - 12:44 PM.

  • leemr likes this

#12 BobinBend

BobinBend

    Mariner 2

  • -----
  • topic starter
  • Posts: 205
  • Joined: 02 Mar 2018
  • Loc: Bend, Oregon

Posted 01 July 2019 - 11:49 AM

Thanks to all for your comments!

 

Scadvice, I played around with PI's noise reduction settings too much and sacrificed clarity for added noise reduction. I don't like it either and will fix it.

 

Sven, Thanks for you comments and I'll look into using your steps for my Canon 60D.

 

James7ca, Wow, your suggestion and the results are amazing! I've never used the PixelMath process and wasn't even sure what it's used for. Can you explain the "IF" expression in more detail? I'm familiar with Basic and C++ but I don't recognize this expression's syntax.

 

How did you determine to use "0.1"? Does that represent "10% or greater than the blue and green pixels?

 

Can you recommend a website or book I should use to get more familiar with PixelMath in the PI environment?

Thanks for your help James!

 

Bob



#13 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 6965
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 01 July 2019 - 09:12 PM

The PixelMath expression "iif" is similar to the "if-else" construct. It will do one thing or the other based upon the evaluation of the first statement enclosed in the parenthesis.

 

The "0.1" does NOT mean 10% of the blue or green, it's just a literal value of 0.1. PixelMath uses floating point numbers to represent pixel values, so that values can be anywhere from 0 to 1. I more or less guessed at the 0.1 value, since the red pixel defects were generally several tenths higher than the green or blue values. Also, as I noted originally if you tried to run this on the linear, RAW image you'd have to change that value, since in that case the differences would likely be much lower in numeric value (maybe 0.01, but since I don't have the RAW subs I don't know for sure).

 

Note that the NGC_6543[0], NGC_6543[1], and NGC_6543[2] are just the red, green, and blue channels respectively of the target image, which I renamed as NGC_6543 (actually resolves to a single pixel, as PixelMath automatically loops through each pixel in the image). These files need to opened within the PixInsight workspace. Just switch to the PixelMath Expression Editor and you will see a list of the available images over on the top right of the window.

 

You can find tutorials on PixelMath on the internet, just do a search. Here is one source to get you started (available in both German AND English).

 

  http://www.werbeagen...PixInsight.html


Edited by james7ca, 01 July 2019 - 09:22 PM.


#14 BobinBend

BobinBend

    Mariner 2

  • -----
  • topic starter
  • Posts: 205
  • Joined: 02 Mar 2018
  • Loc: Bend, Oregon

Posted 02 July 2019 - 10:03 AM

James7ca, 

Thanks for the information. I had a chance to play around with PixMath last night and was able to remove the red noise from my stacked raw image. I do have one question: What is the best way to determine what (0-1) value to use in the expression? Trial and error? This appears to be a very powerful tool! I also watched the video you suggested and will checkout other videos on the topic on YouTube.

Thanks again for your help.

Bob



#15 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 6965
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 02 July 2019 - 10:40 AM

PixelMath can do some very useful things, but in this particular case I'd say that it was kind of a tool of last resort. Ideally, you shouldn't have defects like you saw and a better "solution" would be to try and find out what caused this problem. Was it from poorly matched calibration files or something else?

 

As for the range value I used (0.1) it was just a guess, but I did measure a few of the red pixel defects to see how much higher they were than the green and blue.

 

As I mentioned earlier, you should also check out the CosmeticCorrection tool in PixInsight, it might do an even better job than PixelMath.


Edited by james7ca, 03 July 2019 - 01:44 AM.


#16 calypsob

calypsob

    Aurora

  • *****
  • Posts: 4603
  • Joined: 20 Apr 2013

Posted 02 July 2019 - 10:56 AM

The attached image NGC 6543 (in JPG format) was taken under less than best conditions. Consequently it contains a lot of noise in the background (primarily red dots). I've attempted to use a number of noise reduction processes to remove the noise but to no success. I calibrated it with about 40 darks, 20 fresh flats, and 40 bias frames using DSS and PI's BatchProcessing script. Both produce similar noise issues. Is there a process in PI that can remove the red dots of noise without affecting the nebula? Or do I have to retake the image which I intend to do anyway.
Thanks for your suggestions, Bob

Did you dither? I can see alot of horizontal banding noise. In addition there is alot of red, green, blue background noise. How many lights did you take?

 

I can see a satellite trail crossing the image. This should have been rejected during integration.  I suggest using BPP up to image registration. Then manually integrate the calibrated and registered subs.

 

This way you can adjust teh rejection parameters in the roi box until they are effectively eliminating the noise in the background. For NR you probably want to use a star mask and do some chroma cleanup on the background.

 

I like the first image alot even though its noisy, it looks much better than clipping the background to black to hide the defects. Let the cameras characteristics be what they are, if you get around 60-120 dithered light frames on a 60d then you wont see noise in the background appear like this until you have stretched the data extremely hard.

 

IMO you need more subs and some fine tuning, NR is probably not going to be that big of a contender.


Edited by calypsob, 02 July 2019 - 10:57 AM.


#17 vidrazor

vidrazor

    Vostok 1

  • -----
  • Posts: 169
  • Joined: 31 Oct 2017

Posted 02 July 2019 - 08:28 PM

James7ca, 

Thanks for the information. I had a chance to play around with PixMath last night and was able to remove the red noise from my stacked raw image. I do have one question: What is the best way to determine what (0-1) value to use in the expression? Trial and error? This appears to be a very powerful tool! I also watched the video you suggested and will checkout other videos on the topic on YouTube.

Thanks again for your help.

Bob

Although your subsequent attempt looks much better, if you can get only so far in PI, you can finish it off in an image processor using alpha channels. I took your original output and processed it in Photoshop (you can see it below), and got pretty near your second attempt, so if you're hitting a brick wall in the data processing, try and app like Photoshop or GIMP to finish it off.

Attached Thumbnails

  • noise.jpg


#18 bobzeq25

bobzeq25

    Hubble

  • *****
  • Posts: 16009
  • Joined: 27 Oct 2014

Posted 03 July 2019 - 12:57 AM

This book has an excellent discussion of how to use the various noise reduction techniques in PI.

 

https://www.amazon.c...h/dp/1138055360

 

The Deep Sky Imaging Primer also has some useful information.

 

This is a somewhat idiosyncratic approach to noise reduction in PI, but it seems to work well.  Pretty complicated.  Still, it's a good read.

 

https://jonrista.com...duction-part-1/

 

Key philosophical point.   Processing is like playing chess.  You build to a final result, don't try to maximize the result in each step.  For noise reduction that means doing a little in the linear phase, and some more when the image is non-linear.  Sneak up on it.  <smile>

 

Did you do gradient reduction?  That helps, also.


Edited by bobzeq25, 03 July 2019 - 12:59 AM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics





Also tagged with one or more of these keywords: astrophotography



Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics