Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Odd Stars Batch Preprocessing PixInsight

  • Please log in to reply
26 replies to this topic

#1 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 19 October 2019 - 11:50 PM

I ran my Ha through Batch Preprocessing and some of my stars around the edges look jagged. Does anyone know what this is. I thought about running the script again without Generate drizzle. I have no idea though. Always a lot to keep an eye on. Here is a sample.You might have to click on the image to fully see it. Thanks!

Edit: look at the larger stars to see it. 

 

light-FILTER_Ha-BINNING_1.jpg


Edited by Moreflying, 19 October 2019 - 11:51 PM.

  • artem2 likes this

#2 Jim Waters

Jim Waters

    Gemini

  • *****
  • Posts: 3086
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 20 October 2019 - 12:14 AM

Did you Blink your images first?  What was the average FWHM of the subs - focus...?

 

Did you pick the sub with the highest SNR?

 

You can try rerunning with Calibrate Only checked.  Run StarAlignment manually and play around with Star Detection and Star Matching values.  


Edited by Jim Waters, 20 October 2019 - 12:17 AM.

  • elmiko likes this

#3 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 01:30 AM

Going to run subframe selector and see what I've got...Majority are in the 3.3 to 3.8 FWHM. I picked a sub that was 3.38 FWHM. I did blink and go one with aircraft lights in it. Using Optec focusing. I may have to try manual. So weird that some of those large stars are getting that.  



#4 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 03:29 AM

I tried Sigma Clipping for Lights integration instead of Winsorized and the brighter stars looked better. So, what I might try in the morning is looking at Sigma high and low settings and put it back to Winsorized. Trial and error it seems. 



#5 Jim Waters

Jim Waters

    Gemini

  • *****
  • Posts: 3086
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 20 October 2019 - 01:25 PM

Let us know how it turns out.



#6 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2145
  • Joined: 08 Apr 2015

Posted 20 October 2019 - 02:00 PM

That's an artifact of not setting the "high" rejection in Winsorized Sigma Clipping high enough. Experiment with raising that setting and those artifacts will go away.

 

Tim


  • Francopoli and Moreflying like this

#7 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 02:16 PM

I went back into settings for batch preprocessing and under (Image Integration) changed back to Winsorized Sigma Clipping for lights only.  From there I changed sigma high to 2.00 rather than 3.00 and ran the script again. That appears to have helped the bright stars from looking so jagged on the outlying circumference. My issue now is the bright star cores are blown out before I get to stretching. I need to figure that out next. If any ideas please let me know. 

 

 Again you may need to click on the attached image to see the detail. 

light-FILTER_Ha-BINNING_1.jpg


  • elmiko likes this

#8 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 02:20 PM

That's an artifact of not setting the "high" rejection in Winsorized Sigma Clipping high enough. Experiment with raising that setting and those artifacts will go away.

 

Tim

Okay, I'll run it at 3.5 to 4 and see what I get. I went down to 2.0 from 3.0 and it looks like it went away but I'm up for trying it higher.



#9 Jim Waters

Jim Waters

    Gemini

  • *****
  • Posts: 3086
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 20 October 2019 - 04:07 PM

For BPP I use the following... (OSC and DSLR)

 

Winsorized Sigma Clipping - 15 Plus Subs

Sigma Clipping - 8 to 10 Plus Subs

 

Bias

  • Winsorized Sigma Clipping -or- Sigma Clipping
  • Sigma Low - 3.0
  • Sigma High - 3.0

Darks

  • Winsorized Sigma Clipping -or- Sigma Clipping
  • Sigma Low - 3.0
  • Sigma High - 3.0

Flats

  • Winsorized Sigma Clipping -or- Sigma Clipping
  • Sigma Low - 3.0
  • Sigma High - 3.0

Lights

  • Winsorized Sigma Clipping -or- Sigma Clipping
  • Sigma Low - 4.0 (My Range 3.75 to 4.25)
  • Sigma High - 3.0 (My Range 2.75 to 3.25)

http://pixinsight.co...description_003


  • Moreflying likes this

#10 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 10:17 PM

I took sigma high up to 6 and it got rid of all but a few pixels that I was able to repair. So, I was mistaken about lowering it before. Thanks guys. Is there a downside to raising the value to 6 as far as degrading the data at all?

 

light-FILTER_Ha-BINNING_1-1.jpg



#11 Jim Waters

Jim Waters

    Gemini

  • *****
  • Posts: 3086
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 20 October 2019 - 10:45 PM

For Lights? Sigma High of 6 is kind of high. What was Sigma Low? The default values are above.

 

If they are too high or low you will not eliminate noise or eliminate valuable data.



#12 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 20 October 2019 - 11:57 PM

Sigma low was default 4 for lights. I sure wish I knew why those pixels around some bright stars looked that way. Maybe I’ll try  sigma clip again and decrease sig high tomorrow. It’s pretty time consuming each fresh run.  If I knew the source of the issue and understood what I was seeing, I’d go after it. Everything else was set to default on the calibration frames. 



#13 Monkeybird747

Monkeybird747

    Viking 1

  • *****
  • Posts: 635
  • Joined: 31 Aug 2017
  • Loc: Louisville, KY

Posted 21 October 2019 - 11:37 AM

If your trying to experiment with integration pixel rejections settings just use the stand-alone process and do it manually on your calibrated frames. It would be faster than running a full bpp each time. Plus, when you get to the image integration phase open one of your images and define a small preview around one of the jagged stars. In the integration window select ROI and then select your preview. It will run integration on that tiny area and move very fast. This is great for finding settings that will eliminate satellites, airplanes, etc.

 

Sounds like you're on the right track, but in the future consider running a full manual calibration routine and leave BPP out of it as a troubleshooting step. Evaluate your frames at the end of each process to see if you can identify where the anomaly lives. As Jim said make sure to blink your data and see if there are a few frames that might be skewing the integration, like stars with tree spikes or the moth man on your objective.


  • elmiko and Moreflying like this

#14 Monkeybird747

Monkeybird747

    Viking 1

  • *****
  • Posts: 635
  • Joined: 31 Aug 2017
  • Loc: Louisville, KY

Posted 21 October 2019 - 11:52 AM

It's really hard to see the artifacts in that stack because of the small jpg. It almost looks as if some of the stars are out of alignment, and pixel rejection is trying to reject the misaligned portions? Or maybe its trying to reject some of the halo from a few frames with bloated stars? If you have a way of posting your full data set I can run a few integration scenarios for you.



#15 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2145
  • Joined: 08 Apr 2015

Posted 21 October 2019 - 05:25 PM

For Lights? Sigma High of 6 is kind of high. What was Sigma Low? The default values are above.

 

If they are too high or low you will not eliminate noise or eliminate valuable data.

That's not particularly high for Winsorized sigma clipping. I typically start at 7. WSC works much differently than regular sigma clipping. In regular sigma clipping, the outliers are included in the calculation of the standard deviation (sigma). If there are bright pixels in the pixel stack, the resulting sigma will be much larger than that of the underlying no-outlier distribution. As a consequence, it takes a small sigma multiplier to reject the outliers but not the underlying distribution. With Winsorized sigma clipping, the data set is Winsorized prior to calculating sigma. Basically what Winsorization does is replace the outliers with the closest inlier. The resulting sigma value is much, much smaller. As a result, it takes a significantly greater sigma multiplier to reject outliers yet still retain the underlying distribution.

 

The result of choosing a sigma multiplier with WSC that is too small is that too much of the underlying distribution gets rejected. This is particularly noticeable around bright stars. It is the result of star sizes in the subs that vary in size due to focus, seeing, guiding, etc. In that case, the regions surrounding the star core will have pixel stacks that are bimodal - some are dark background, and others are much brighter, with star stuff in them. When you apply a too-small sigma multiplier to a bimodal distribution like that, a lot of the bright pixels will be rejected, making the pixel much darker than a plain mean of the pixel stack would produce. The result is dark splotches around the cores of stars. Those are the results that are obvious. The less obvious result of a too-small sigma multiplier is increased noise in the darker areas as too much of the true underlying distribution in the pixel stacks gets rejected.

 

Tim


  • elmiko and Moreflying like this

#16 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 21 October 2019 - 05:34 PM

Thank you for the information and taking the time. I'm still working my way around Pix. I'll have to get my workflow shifted into a stand alone/manual process. It seems like this a better way to go, so I will have to do some self training. I'll get on astrobin too with a full resolution image so it's easier to see this artifact and link it.



#17 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 21 October 2019 - 05:36 PM

When I get home. I'll take a better read through and respond back with more details and a link.


  • elmiko likes this

#18 Monkeybird747

Monkeybird747

    Viking 1

  • *****
  • Posts: 635
  • Joined: 31 Aug 2017
  • Loc: Louisville, KY

Posted 21 October 2019 - 07:19 PM

So I just realized you appear to be using the BPP script for integration as well? Just FYI, the creator of PI (last time I checked) recommends not using the BPP script for the integration process, except for running a test stack. Using it for the calibration routine is fine, but he says you should stop there and do the final integration with the stand alone integration process. Much of the reasoning is for what you are experiencing, and having a need to test different rejection operators and settings. That's not a diss on the BPP script. YMMV.



#19 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 21 October 2019 - 10:06 PM

That's not particularly high for Winsorized sigma clipping. I typically start at 7. WSC works much differently than regular sigma clipping. In regular sigma clipping, the outliers are included in the calculation of the standard deviation (sigma). If there are bright pixels in the pixel stack, the resulting sigma will be much larger than that of the underlying no-outlier distribution. As a consequence, it takes a small sigma multiplier to reject the outliers but not the underlying distribution. With Winsorized sigma clipping, the data set is Winsorized prior to calculating sigma. Basically what Winsorization does is replace the outliers with the closest inlier. The resulting sigma value is much, much smaller. As a result, it takes a significantly greater sigma multiplier to reject outliers yet still retain the underlying distribution.

 

The result of choosing a sigma multiplier with WSC that is too small is that too much of the underlying distribution gets rejected. This is particularly noticeable around bright stars. It is the result of star sizes in the subs that vary in size due to focus, seeing, guiding, etc. In that case, the regions surrounding the star core will have pixel stacks that are bimodal - some are dark background, and others are much brighter, with star stuff in them. When you apply a too-small sigma multiplier to a bimodal distribution like that, a lot of the bright pixels will be rejected, making the pixel much darker than a plain mean of the pixel stack would produce. The result is dark splotches around the cores of stars. Those are the results that are obvious. The less obvious result of a too-small sigma multiplier is increased noise in the darker areas as too much of the true underlying distribution in the pixel stacks gets rejected.

 

Tim

You seem to have a real understanding of the different modes of integration. I don't have enough experience to comment on the finer points and how the mathematics/statistics apply here. I have a lot to learn. Where did you study up to properly use these tools? I need to get reading. I really think PixInsight has been great so far. All that said, I think I'm reading you correctly. Too small a multiplier can be more detrimental, in certain conditions, by increased the noise in the darker areas. So, in my case the increase to 6 is better overall to include getting rid of the artifact around the stars. Is that right? Thank much.



#20 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 21 October 2019 - 10:17 PM

So I just realized you appear to be using the BPP script for integration as well? Just FYI, the creator of PI (last time I checked) recommends not using the BPP script for the integration process, except for running a test stack. Using it for the calibration routine is fine, but he says you should stop there and do the final integration with the stand alone integration process. Much of the reasoning is for what you are experiencing, and having a need to test different rejection operators and settings. That's not a diss on the BPP script. YMMV.

Thanks a lot for your comments and help. I've been using BPP for my work. I'm surprised just how great a result I get with it compared to other software I've used. Pix did give me that warning and suggested to a manual process for better results. So, I need to head in that direction with manual calibration to fine tune. Thanks too for the info on how quickly check results rather that run BPP full each time.



#21 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2145
  • Joined: 08 Apr 2015

Posted 22 October 2019 - 02:45 PM

You seem to have a real understanding of the different modes of integration. I don't have enough experience to comment on the finer points and how the mathematics/statistics apply here. I have a lot to learn. Where did you study up to properly use these tools? I need to get reading. I really think PixInsight has been great so far. All that said, I think I'm reading you correctly. Too small a multiplier can be more detrimental, in certain conditions, by increased the noise in the darker areas. So, in my case the increase to 6 is better overall to include getting rid of the artifact around the stars. Is that right? Thank much.

There are multiple resources for learning more about PI. Here is the reference documentation on image integration:

 

https://pixinsight.c...ntegration.html

 

It is a bit arcane in the way it is written but there is a lot of good info there. There are also multiple good tutorials both in the form of web pages and videos. Some searching on "Pixinsight tutorial" will turn up hundreds of hits.

 

Tim


  • Moreflying likes this

#22 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 22 October 2019 - 05:38 PM

There are multiple resources for learning more about PI. Here is the reference documentation on image integration:

 

https://pixinsight.c...ntegration.html

 

It is a bit arcane in the way it is written but there is a lot of good info there. There are also multiple good tutorials both in the form of web pages and videos. Some searching on "Pixinsight tutorial" will turn up hundreds of hits.

 

Tim

Thanks for the info. I'm running through the manual integration now. I've made my master bias and master dark. Tonight I'll make the master flat and see about a new result. 



#23 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 22 October 2019 - 11:43 PM

I worked through calibration and star align...stacking manually. I get the same result this way with Winsorized. In fact it almost looked better with the batch script overall. There was less overall noise. 

 

Winsorized sig. high 4

integration-win.jpg

 

 

Winsorized sig high 7 left a few dark pixel in some of the brighter stars. 

integration-sighigh-7.jpg

 

 

Standard sigma clip sig high 3 No artifacts.

integrationcigclip-sighigh3.jpg

 

 

 

 



#24 Monkeybird747

Monkeybird747

    Viking 1

  • *****
  • Posts: 635
  • Joined: 31 Aug 2017
  • Loc: Louisville, KY

Posted 22 October 2019 - 11:47 PM

Interesting. Are you able to link up the data set?

These pics are more clear. Since the problem is at the edges I suspect there is possibly some distortion. It looks like part of the misaligned star is getting rejected, but hard to tell without full res data.

Can you run star alignment again using distortion correction checkbox enabled, and change the Registration Model to 2-D Surface Splines. Then run a new integration using windsorized sigma with defaults, and try one with linear fit clipping and defaults.

Edited by Monkeybird747, 22 October 2019 - 11:56 PM.

  • Moreflying likes this

#25 Moreflying

Moreflying

    Vostok 1

  • -----
  • topic starter
  • Posts: 111
  • Joined: 12 Jul 2016

Posted 23 October 2019 - 10:26 AM

Interesting. Are you able to link up the data set?

These pics are more clear. Since the problem is at the edges I suspect there is possibly some distortion. It looks like part of the misaligned star is getting rejected, but hard to tell without full res data.

Can you run star alignment again using distortion correction checkbox enabled, and change the Registration Model to 2-D Surface Splines. Then run a new integration using windsorized sigma with defaults, and try one with linear fit clipping and defaults.

Thank you..I will try that tonight and see what happens. I'm not sure how to link up the data set. I will go back and see if the star alignment is off with a sub or two. I would think PixInsight would do a pretty good job with this but not certain.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics