56 replies to this topic

[Higgsfield]

I've developed a method to remove satellite trails from images. It's taken five months or so, on and off. This is a non-AI-based method that uses classical math to detect and replace the trail. No sigma clipping is used, so only two frames are needed: the image with the trail and the previous image. Using the previous image ensures the statistics of the two frames are closely related. Moreover, large trails caused by airplane lights can also be removed.

 

The mask that covers the trail has a speckled edge which helps knit the frames together seamlessly, where pixels from the previous frame replace those in the current frame to remove the trails. It sounds simple, but it's actually a very difficult problem to solve, especially if you want to remove very faint trails, or pieces of a trail across the image corner, or as often happens when the satellite passes across two or three image frames. The motivation for this was "lucky imaging," where I grew tired of blinking through hundreds, and sometimes thousands, of frames. This example image is a 15-second image of the Angle Nebula (Sh2-106) in Sii.

 

There seems to be a trend toward giving away software, but I'm not in that camp. RC didn't do that, nor are PixInsight or APP free. So my question to the group: would you pay a modest amount for an easy-to-use program that removes satellite trails and quickly pre-sorts images based on other criteria like FWHM, shape (elongation), and SNR combined with star count to detect double stars and cloudy cover in real-time as images are captured, moving the suspect images to a separate folder? It might be the case that the most folks feel PI does an good enough job, so really no need for a standalone program. 

 

Your feedback would be appreciated. 

 

Edited by Higgsfield, 17 June 2025 - 09:19 AM.

[BlueMoon]

If it runs on Linux, I'd consider it. 

[djbridges]

It would all depend on what you mean by "modest amount".  I agree that your time and effort are worth money, but now you're running into Adam Smith's invisible hand.

Edited by djbridges, 17 June 2025 - 09:35 AM.

[RetiredDave]

If given an adequate "free" trial period to assess how well it works and how useful it is, a program like you describe might be worth a one-time purchase, depending on the cost of course.

[afd33]

Personally I wouldn't pay for it. I'm fine with the rejection algorithms in stacking programs. If nothing else losing a couple subs with troublesome trails doesn't bother me that much anyway. Then there's Seti's Comic Clarity which can remove trails if I really wanted to.

[Higgsfield]

Personally I wouldn't pay for it. I'm fine with the rejection algorithms in stacking programs. If nothing else losing a couple subs with troublesome trails doesn't bother me that much anyway. Then there's Seti's Comic Clarity which can remove trails if I really wanted to.

I'm a great fan of Frank's work and contributions to the field, but as you can see in this particular example, the removal process doesn't quite achieve the results I'm looking for. The image shows the use of Cosmic Clarity to remove the same trail as above. The right-hand image clips the satellite trail to black, leaving an obvious artifact. Neither of these processed images would meet my standards for inclusion in a final stack.

 

It turns out that satellite trail removal is extremely challenging from a technical standpoint, at least based on what I've found in the scientific literature. Most published methods rely on sigma clipping or median filtering approaches, and or the Hough transform (in combination with AI) which can work sometimes for bright, obvious trails but struggle with faint streaks or partial trails that only affect a portion of the frame.

 

 

Edited by Higgsfield, 17 June 2025 - 11:03 AM.

[Higgsfield]

It would all depend on what you mean by "modest amount".  I agree that your time and effort are worth money, but now you're running into Adam Smith's invisible hand.

 

Not sure who is Adam Smith, did he have a robot? Reasonable to me is $50, but we are taking about more than Satllite removal.  Perhaps that is too much?

 

If given an adequate "free" trial period to assess how well it works and how useful it is, a program like you describe might be worth a one-time purchase, depending on the cost of course.

 

Yes free trial for sure. 

 

If it runs on Linux, I'd consider it. 

 

Absolutly, and MacOS maybe. 

[WadeH237]

The right-hand image clips the satellite trail to black, leaving an obvious artifact. Neither of these processed images would meet my standards for inclusion in a final stack.

I think that you are missing something here.

 

The reason that the satellite trail is clipped to black is that ImageIntegration (by default) will exclude pure black pixels from the stack.

 

So for example, if you have 24 subs, the pixels in the satellite trail will be the average of 23 frames, specifically excluding the black clipped ones.  This is an excellent solution to removing tough satellite trails (or other aberrations that exist only in a subset of subs).

 

That said, I have yet to encounter a situation (outside of testing corner cases) where I could not fully remove satellite trails in PixInsight without resorting to essentially painting them out.
 

[pfile]

might pay 10-15 for this but not 50, just my opinion. IMO there are only a few places in the sky (like M42) where deleting trails in subs would be advantageous. beyond that usually with enough subs and the proper rejection algo the trails are gone from most of my integrations.

 

maybe starlink will change all this though.

 

rob

Edited by pfile, 17 June 2025 - 11:40 AM.

[Higgsfield]

I think that you are missing something here.

 

The reason that the satellite trail is clipped to black is that ImageIntegration (by default) will exclude pure black pixels from the stack. 

 

I should have been a bit more precise. The issue is not the clipping to black, it's the incomplete removal of the trail and the visable boundary. Anytime you rely on thresholding or AI you will have this type of problem. The brigthest part of the trail will be included in the final image in this case. 

 

[WadeH237]

With a stack of real data, I suspect that I could remove the trail completely without black clipping anything.

 

If I had to resort to this solution, I would probably clone stamp black over the satellite trail and ensure that I cover all of the trail's pixels, assuming that I really needed to keep the sub.

 

If the above trail was painted over by some kind of process or script, it would benefit from a "grow" option to cover the faint edges of the trail that were not blacked out.  Also, if I were to write it, I would have it find the start and end of the trail and cover the entire line from end to end.  Those two things would completely eliminate the trail, leaving only a slightly lower S/N in the replaced pixels (owing to one less sub in the stack for the black pixels).

 

All that said, I have personally not found satellite trail rejection to be a problem using just the tools in PixInsight.  I know that others have different experiences, and other editing packages probably don't have the equivalent rejection options so I'm probably just not part of your market.

[dghent]

Interesting statement. Aren’t you a user of NINA?

[Higgsfield]

Interesting statement. Aren’t you a user of NINA?

Hi Dale! Yes, hee hee. 

[AstroFromHome]

I rely on the rejection algorithms to get rid of satellite trails.

In all the time I only had one image that did not fully average them out. It's an image of Lowers Nebula which I took using 900 sec subs. The number of subs has just been too little to get rid of the trails. In order to see them going to 100% image size is needed.

Next winter I intend to image the DSO again and once that will have happened the trails will disappear. 

[Higgsfield]

I rely on the rejection algorithms to get rid of satellite trails.

In all the time I only had one image that did not fully average them out. It's an image of Lowers Nebula which I took using 900 sec subs. The number of subs has just been too little to get rid of the trails. In order to see them going to 100% image size is needed.

Next winter I intend to image the DSO again and once that will have happened the trails will disappear. 

The reason for developing this algorithm is for imagers like yourself, taking very long exposures where every sub is precious. Rejection algorithms cannot completely remove a satellite trail - there will always be some residual at the trail edges. Moreover, faint trails are likely to go undetected. That said, if you have many frames, then trails are unlikely to be problematic, but if you have relatively few frames, then trails may still be visible in the final stack, especially the fainter one that were not clipped. 

 

The problem is amplified when you have a very small image scale at long focal lengths. The residual portions of the trail become particularly problematic, as you are pixel peeping by virue of the image scale. I'm thinking of combining satellite removal with classical deconvolution. This would aid greatly in the stacking process, especially where PixInsight and other similar programs can struggle to find enough stars for proper alignment. Perhaps that additional functionality would add appeal to a standalone program.

 

Here is a comparison between sigma clipping in Pixinsght and my method. If you look really, really closely at image on the left, you will see the residual outer edge. Mine is on the right. In the UK we had spot the ball, it's kind of like that! This is close to the crop size of the final image.

 

Edited by Higgsfield, 17 June 2025 - 07:49 PM.

[rjkrejci]

I currently use ASIFITSview to quickly go through my images, deleting the outliers before loading in WBPP.   If there was a program that was very fast, autostretched the images, let me sort by FWHM, elongation, detects double stars from wind or bad tracking, easily right there allows me to delete/move those frames with a click I deem unfit and fixed satellite trails, I'd pay money for it for sure

[Higgsfield]

I currently use ASIFITSview to quickly go through my images, deleting the outliers before loading in WBPP.   If there was a program that was very fast, autostretched the images, let me sort by FWHM, elongation, detects double stars from wind or bad tracking, easily right there allows me to delete/move those frames with a click I deem unfit and fixed satellite trails, I'd pay money for it for sure

Thanks for the feedback. 

[Greg M]

The reason for developing this algorithm is for imagers like yourself, taking very long exposures where every sub is precious. Rejection algorithms cannot completely remove a satellite trail 

But for these users that are shooting 900sec or so they are going to only have a few subs. If that's the case, then using Skill (Satellite Killer) in PI will do just fine. Sure, not an automatic function but not that big a deal if you only have 15-20 subs to check.

[Greg M]

I currently use ASIFITSview to quickly go through my images, deleting the outliers before loading in WBPP.   If there was a program that was very fast, autostretched the images, let me sort by FWHM, elongation, detects double stars from wind or bad tracking, easily right there allows me to delete/move those frames with a click I deem unfit and fixed satellite trails, I'd pay money for it for sure

SFS in PI with an Approval Expression like this will do the trick.

 

FWHMSigma <=2 &&
EccentricitySigma <=2 &&
MedianSigma <=2 &&
StarsSigma >= -1.5

[Higgsfield]

But for these users that are shooting 900sec or so they are going to only have a few subs. If that's the case, then using Skill (Satellite Killer) in PI will do just fine. Sure, not an automatic function but not that big a deal if you only have 15-20 subs to check.

 

I really didn't know this tool existed, so thanks for bringing it to my attention and others here. However, it's really a legacy tool, and certainly wouldn't provide an acceptable result in the final stack. For my purposes, it's totally impractical to use even if it yielded good results.

 

My motivation for developing this algorithm stems from the fact that there are no adequate solutions to this problem outside of sigma clipping, which can work but will leave residual artifacts from bright trails due to how it functions. AI has thus far failed to deliver viable results, though it may improve and prove workable in the future. The solutions outlined in the professional literature also fail to deliver effective results, in my opinion. None of the described techniques work in practice. Reading some of the posts in this thread, I'm not sure people fully appreciate the magnitude of this problem.  

 

When shooting after dusk and before dawn, I see trails in nearly every image—sometimes multiple trails per frame—and this problem is only getting worse, much worse. Even at 2:00 AM there are faint trails - I am shorting from a Bortle 1 site. Here's one such example taken from a stack of 353-60s frames, which creates a tedious sorting problem, only to have to make the decision to keep or dump the frame. It's not practical to discard large amounts of data when shooting in certain parts of the sky at certain times. Moreover, PixInsight takes many hours to process this number of frames, so if trails do become visible in the final stack, the only solution is to identify the problematic frames and start the entire process over again.

 

This example shows a faint 2:00am trail where the satellite enters at the top right corner and is picked up in two frames:

 

Edited by Higgsfield, 18 June 2025 - 10:12 PM.

[WadeH237]

My motivation for developing this algorithm stems from the fact that there are no adequate solutions to this problem outside of sigma clipping, which can work but will leave residual artifacts from bright trails due to how it functions. AI has thus far failed to deliver viable results, though it may improve and prove workable in the future. The solutions outlined in the professional literature also fail to deliver effective results, in my opinion. None of the described techniques work in practice. Reading some of the posts in this thread, I'm not sure people fully appreciate the magnitude of this problem. 

We are definitely not in agreement about this.  And yes, I certainly understand the magnitude of the problem.

 

I think that there is a prevalence of people who are looking for a "one button" solution.  As far as I know, there is no such thing.  In order to help educate people on ways to handle satellite trails, I posted this topic in the Beginning Deep Sky Imaging forum a few months ago.  It shows some practical ways to handle the problem on a difficult stack (I rejected all the clean subs and kept only those with trails).
 

[Higgsfield]

We are definitely not in agreement about this.  And yes, I certainly understand the magnitude of the problem.

 

I think that there is a prevalence of people who are looking for a "one button" solution.  As far as I know, there is no such thing.  In order to help educate people on ways to handle satellite trails, I posted this topic in the Beginning Deep Sky Imaging forum a few months ago.  It shows some practical ways to handle the problem on a difficult stack (I rejected all the clean subs and kept only those with trails).
 

I hope those trails were not in the same image?  Be an excellent test of my software! Are you up for a challenge? Clearly have a deep interest in trail elimination. If you upload to the cloud I'll run the images through my software. There needs to be at least one clean image. Here's the thing, the modified image that has the trail(s) removed becomes the reference for the next image. Ideally the images need to be concurrent in time so the stats match, else the infill may be inperfect.

 

I think you proved my point with your sigma clipping examples. You had to dial it in to get this result, and even go to the nuclear option for one of the trails. In the case example I just posted, it will take many hours to run the subs through PI given the sizes of the data files. Then you are going to try to dial in the right setting and run again, and perhaps again. 

Edited by Higgsfield, 18 June 2025 - 10:55 PM.

[WadeH237]

I hope those trails were not in the same image?  Be an excellent test of my software! Are you up for a challenge? Clearly have a deep interest in trail elimination. If you upload to the cloud I'll run the images through my software. There needs to be at least one clean image. Here's the thing, the modified image that has the trail(s) removed becomes the reference for the next image. Ideally the images need to be concurrent in time so the stats match, else the infill may be inperfect.

 

I think you proved my point with your sigma clipping examples. You had to dial it in to get this result, and even go to the nuclear option for one of the trails. In the case example I just posted, it will take many hours to run the subs through PI given the sizes of the data files. Then you are going to try to dial in the right setting and run again, and perhaps again. 

Sure, I am happy to help.

 

The image in the first post in my topic was created by going through the luminance subs from a 3 night project.  I hand picked every sub with a satellite trail and rejected all the "good" subs.  The idea was to create a torture test for satellite removal.  I started out synthesizing trails, but they were too easy to remove.  I found that live data gave a good distribution of the kinds of problems that make it difficult.

 

It will have to wait for a couple of weeks, though.  I am set up at the Oregon Star Party site for the next two weeks.  I don't have access to my archive of older data, including the M63 project.  When I get back, I can share whatever would be helpful for you.

[calypsob]

Ok so I have a population of 300 light frames each 600s in length. From those calibrated frames I select and integrate 25 perfect images with the best possible fwhm. I now have an exceptional registration frame. I calibrate and integrate the entire population of data - 300 frames - against my highly refined sample and in pixinsight I can not only choose to select the average or median, I can weight all of the images, I can use MAD to evaluate the spread of volatility across sll of my frames, and finally I can pick from 10 extremely powerful rejection models” i pulled them from gpt” and integrate the data. So with those tools at my disposal, why would painting a mask over the raw data be a superior solution to mathematically rejecting outliers to achieve optimal results in the final integration?
What artifacts are created by your tool? Maybe the artifacts your tool makes could be rejected yielding a superior image to one with trails. Idk, but if im going to pay for something like this, it needs to be backed by hard evidence that anyone can comprehend.

1. No Rejection
• Description: No pixels are rejected.
• Formula:
I(x, y) = (1/N) * Σ I_i(x, y)

⸻

2. Min/Max Clipping
• Description: Rejects the minimum and/or maximum pixel values.
• Formula:
Reject min(I_1, ..., I_N) and/or max(I_1, ..., I_N)

⸻

3. Percentile Clipping
• Description: Rejects values below/above specified percentiles.
• Formula:
Reject I_i(x, y) not in [P_low, P_high]

⸻

4. Sigma Clipping
• Description: Rejects values beyond k standard deviations from the mean.
• Formula:
μ = mean, σ = standard deviation
Reject |I_i - μ| > kσ

⸻

5. Winsorized Sigma Clipping
• Description: Replaces extreme values before calculating σ, then applies sigma clipping.
• Formula:
Same as Sigma Clipping, but after Winsorization

⸻

6. Averaged Sigma Clipping
• Description: Averages results of multiple sigma clipping passes.
• Formula:
Iteratively refine μ and σ, then apply: Reject |I_i - μ| > kσ

⸻

7. Linear Fit Clipping
• Description: Fits each image to a reference, then clips based on residuals.
• Formula:
I'_i = a_i * I_i + b_i
Reject |I'_i - I_ref| > kσ

⸻

8. ESD (Extreme Studentized Deviate)
• Description: Rejects outliers based on largest normalized residual.
• Formula:
R_i = |I_i - mean| / stddev
Reject R_i > critical value

⸻

9. CCDR (Chauvenet’s Criterion Distribution Rejection)
• Description: Rejects values with low probability under normal distribution.
• Formula:
P = 1 - erf(|I_i - μ| / (√2 * σ))
Reject if P < 1 / (2N)

⸻

10. MRS (Median Range Sigma)
• Description: Uses median and IQR for robust rejection.
• Formula:
Reject |I_i - median| > k * IQR

[WadeH237]

So with those tools at my disposal, why would painting a mask over the raw data be a superior solution to mathematically rejecting outliers to achieve optimal results in the final integration?

Just as there is a brightness profile to stars, there is a brightness profile to satellite trails.  They are brightest on the center line, and they fade into the background sky as you get farther from the center.

 

There are inevitably some pixels in the satellite trail that are so close to the background sky level that they are not identified as outliers.  These pixels will end up included in the stack.  If you have enough subs, the averaging will diminish them, but not remove them completely.

 

You can crank the sigma factor to try and catch more of those fainter pixels, but if you get aggressive enough, you end up with so many pixels as outliers that the S/N of the stack is noticeably affected.  If you read through the thread that I linked above, I demonstrated this.

 

Coming up with a non-statistical way of identifying satellite trails and black clipping them gives you complete removal of the trail, while not forcing you to increase the statistical rejection aggressiveness.

 

The tricky part is how you actually identify the trails, and I believe that is what the proposed software is addressing.