Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

PixInsight Batch Preprocessing Script - Re-do Image Integration manually?

  • Please log in to reply
40 replies to this topic

#1 MCovington

MCovington

    Gemini

  • *****
  • topic starter
  • Posts: 3404
  • Joined: 13 May 2014
  • Loc: Michael Covington - Athens, Georgia

Posted 20 October 2017 - 09:42 PM

I use the Batch Preprocessing script to calibrate and stack images in PixInsight.

Of course, I get a warning that the version of Image Integration done within that script is just preliminary, and I can (supposedly) get much better results by re-doing the Image Integration and making more settings manually.

 

To what extent is that true?  How much am I losing, really?  Or what do I stand to gain?


Edited by MCovington, 20 October 2017 - 10:13 PM.

  • calypsob likes this

#2 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 21 October 2017 - 01:25 AM

I use PixInsight and Batch Preprocessing.  I don't know the answer.  It may be best to post this in the PixInsight Forum.



#3 HockeyGuy

HockeyGuy

    Ranger 4

  • -----
  • Posts: 376
  • Joined: 11 Nov 2015
  • Loc: Vaughan, ON, Canada

Posted 21 October 2017 - 12:05 PM

I do it manually in PixInsight. I use CosmeticCalibration to remove remaining hot pixels after dark calibration, and SubframeSelector to weigh my individual light frames before aligning and stacking. I also identify the reference light frame at this point (the one with the highest weight). I integrate using DrizzleIntegration and LocalNormalization. The results are a little different compared to a quick and dirty batch pre-processing approach - enough for me to go the extra mile and squeeze more out of the same data set.


  • G. Hatfield, calypsob and MCovington like this

#4 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 21 October 2017 - 12:30 PM

Is the manual method that much better?  Is it that noticeable?  


  • MCovington likes this

#5 ChrisWhite

ChrisWhite

    Mercury-Atlas

  • *****
  • Posts: 2725
  • Joined: 28 Feb 2015
  • Loc: Colchester, VT

Posted 21 October 2017 - 01:39 PM

Is the manual method that much better?  Is it that noticeable?  

 

Depends.  I have had BPP do a less than perfect job with flats calibration.  I have also had problems with some interpolation methods with some data.  I find that manually calibrating, registering and integrating does get me better results, and if there are issues with the data its easy to see where the problems occurred when you break apart the steps. 

 

I definitely get better integrations when I do it manually.  I never use BPP anymore.



#6 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 21 October 2017 - 04:06 PM

I have always used the BatchPreporcessing script.  Is their a video that shows how to to it manually?



#7 ismosi

ismosi

    Viking 1

  • -----
  • Posts: 600
  • Joined: 19 Apr 2014
  • Loc: New London, Pennsylvania, USA

Posted 21 October 2017 - 04:15 PM

I have always used the BatchPreporcessing script.  Is their a video that shows how to to it manually?

 

This isn't a video, but:

http://www.lightvort...t.html#Section2

 

I don't use BPP, I'm too much of a control freak I guess. 


  • G. Hatfield likes this

#8 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 21 October 2017 - 04:25 PM

Thanks - I am too.



#9 ChrisWhite

ChrisWhite

    Mercury-Atlas

  • *****
  • Posts: 2725
  • Joined: 28 Feb 2015
  • Loc: Colchester, VT

Posted 21 October 2017 - 07:06 PM

David Ault has a nice tutorial on manual processing:

http://trappedphotons.com/blog/?p=693


  • Tim likes this

#10 calypsob

calypsob

    Mercury-Atlas

  • *****
  • Posts: 2789
  • Joined: 20 Apr 2013
  • Loc: Roanoke, Virginia

Posted 21 October 2017 - 11:06 PM

So the warning message as you can see from some things already pointed out by others, occurs because you can fine tune every step of calibration, registration, and stacking to a maddening extent.  I typically use BPP for bias, darks, and flats. With lights I do not integrate or register in BPP.  I have manually gone through each step before because I had some hot pixels that I was determined to eliminate.  When I went this route I began to learn about using things like the pedestal tool in image calibration, and the region of interest tool in image integration- which is huge if you want to go beyond internet suggested settings for using something like linear fits high and low adjustment points. You can also tweak the buffer and stack size in image integration which can help you avoid bottlenecking your computer if you have the hardware resources to process higher data throughput.  I am typically pretty happy with the BPP quality with my generic settings loaded but I do not like how the final image is cropped, so I typically make a mosaic registration frame an stack everything against that.  Typically this is not a necessary process, but if you are using two cameras side by side, or even if you re frame over several nights and do not want to have perfectly good data cropped out, then BPP cannot really help you there.   


Edited by calypsob, 21 October 2017 - 11:06 PM.


#11 MCovington

MCovington

    Gemini

  • *****
  • topic starter
  • Posts: 3404
  • Joined: 13 May 2014
  • Loc: Michael Covington - Athens, Georgia

Posted 22 October 2017 - 12:02 AM

"To a maddening extent" sums it up!


  • calypsob likes this

#12 cfosterstars

cfosterstars

    Viking 1

  • *****
  • Posts: 545
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 22 October 2017 - 11:00 AM

Richard Block has a nice YouTube video on the manual calibration and stacking method. I have watched it about a dozen times:

 

https://www.youtube....JgjKuQQ&t=1224s



#13 pfile

pfile

    Skylab

  • -----
  • Posts: 4250
  • Joined: 14 Jun 2009

Posted 22 October 2017 - 07:30 PM

i think the crux of this warning message is that when you set up the ImageIntegration pixel rejection parameters in BPP, you set it up only once and then you get a light master. the light master may look fine, but it could be that you have configured the pixel rejection parameters too aggressively, thus inadvertently lowering the SNR of the result.

 

juan is trying to say that you may need to iterate manually over the pixel rejection parameters, and inspect the rejection maps, to make sure get the highest SNR result that still has all outlying pixels rejected. thus, the integrated result should just be considered a preview.

 

rob


  • Astrocava, calypsob and MCovington like this

#14 cfosterstars

cfosterstars

    Viking 1

  • *****
  • Posts: 545
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 23 October 2017 - 10:04 AM

By the way, I had terrible results using the SuperBias for light frame calibration for my DSLR data. When I used just the normal bias master, the data was fine. I dont know what I did wrong, but it cost  me a lot of time. My advice is to check you images after each step in screen stretch before going to the next step in the process. 



#15 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 23 October 2017 - 11:14 AM

I had the same problem.



#16 cfosterstars

cfosterstars

    Viking 1

  • *****
  • Posts: 545
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 23 October 2017 - 12:14 PM

I had the same problem.

For me the SuperBias results looked like the offset and scaling were way too much and were seriously over-correcting the images. There were lots of zeroed pixels and the dust motes and vignetting were exaggerated rather than eliminated. I deleted the files, but should have saved some to post. What did your's look like? Similar?

 

I was Following Warren Keller's book for the process, BTW.



#17 Jim Waters

Jim Waters

    Viking 1

  • *****
  • Posts: 575
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 23 October 2017 - 12:27 PM

Yes - I saw the same problem.  I don't see the benefit in Super BIAS.  It over corrects the images. 

 

For my T3i I take ~30 Bias's.  I only take ~15 for my 6D.



#18 cfosterstars

cfosterstars

    Viking 1

  • *****
  • Posts: 545
  • Joined: 05 Sep 2014
  • Loc: Austin, Texas

Posted 23 October 2017 - 01:05 PM

I guess I am going way overboard. I have been taking 50 flats for each filter and 50 flat darks now for my ASI1600MM-C since I have had no success with BIAS frame calibration at all with that camera. 



#19 Astrocava

Astrocava

    Sputnik

  • -----
  • Posts: 47
  • Joined: 21 Dec 2009

Posted 24 October 2017 - 01:07 PM

i think the crux of this warning message is that when you set up the ImageIntegration pixel rejection parameters in BPP, you set it up only once and then you get a light master. the light master may look fine, but it could be that you have configured the pixel rejection parameters too aggressively, thus inadvertently lowering the SNR of the result.

 

juan is trying to say that you may need to iterate manually over the pixel rejection parameters, and inspect the rejection maps, to make sure get the highest SNR result that still has all outlying pixels rejected. thus, the integrated result should just be considered a preview.

 

rob

That's exactly what I do. As all of you surely know a simple average integration of the lights gives the maximum SNR, so I run the ImageIntegration process with the registered files from BPP script in a first run without pixel rejection. I write down the final number about noise reduction from the processing console. Then I run several integrations with the pixel rejection increasing the sigmas (low and high) to achieve almost the same noise reduction from average alone but killing all the hot pixels I have. 

I am ending using BPP in "calibrate only" mode, registering in manual mode and filtering with subframeselector to feed the integration process.

 

Sergio


  • pfile likes this

#20 MCovington

MCovington

    Gemini

  • *****
  • topic starter
  • Posts: 3404
  • Joined: 13 May 2014
  • Loc: Michael Covington - Athens, Georgia

Posted 24 October 2017 - 02:11 PM

Could this manual iteration be automated?  I'm a computer scientist -- I do not view a computer as a hand tool...



#21 NMCN

NMCN

    Explorer 1

  • -----
  • Posts: 74
  • Joined: 20 Sep 2006

Posted 24 October 2017 - 02:59 PM

Could this manual iteration be automated?  I'm a computer scientist -- I do not view a computer as a hand tool...

Ha ha, the computer scientist asking the non-computer scientists if a process could be automated.  Seriously, I think you should accept the challenge. :) 



#22 MCovington

MCovington

    Gemini

  • *****
  • topic starter
  • Posts: 3404
  • Joined: 13 May 2014
  • Loc: Michael Covington - Athens, Georgia

Posted 24 October 2017 - 08:18 PM

 

Could this manual iteration be automated?  I'm a computer scientist -- I do not view a computer as a hand tool...

Ha ha, the computer scientist asking the non-computer scientists if a process could be automated.  Seriously, I think you should accept the challenge. smile.gif

 

One day, one day!  I have other big projects in hand right now...

 

The other thing I would like to see is a DSLR version of BatchPreprocessing.  The main differences would be:
(1) default to DSLR raw file types rather than all file types;
(2) use flat darks rather than bias frames;
and perhaps other settings more adapted to DSLR image processing, I'm not sure exactly what.

I did contribute a minor fix to BatchChannelExtraction a while back -- you will see my name on it now as a contributor.



#23 whwang

whwang

    Surveyor 1

  • *****
  • Posts: 1877
  • Joined: 20 Mar 2013

Posted 24 October 2017 - 10:20 PM

Hi Michael,

 

I suggest you to go to the PixInsight forum to express your wishes (1) and (2).  For (1), I think the current BPP covers DSLR very well and nothing has been compromised.  On the other hand, I do agree with you on (2).  I once told them to allow flats to have a different set of bias.  This can be used as bias, if flats are taken with different ISO, or flat dark, as in your case.  My request was discussed in the forum, but never implemented.  If more people go there and make a similar request, they will consider it more seriously.

 

For your original question, I think one key is in the "Image Integration" tab in the ImageIntegration function.  Many of the options there are not in the BPP integration.  For example, if your images are taken under varying sky brightness or varying transparency, ImageIntegration can match their brightness/contrast first before integration.  This is essential for drizzle integration.  Also, ImageIntegration can measure the noise property of each image and weight the images differently according to the noise.  This can lead to optimized S/N in the stacked image when the images are taken under very different conditions.  For example, if we track an object from zenith to 30-deg elevation, we would want to weight the images taken near zenith more, as there the sky is darker and more transparent and seeing is better etc. Also, if we have two sets of images, one taken in a dark place, the other taken in light polluted area, ImageIntegration should be able to automatically down-weight those taken under light pollution and give an optimized stack.

 

In the principle level, what ImageIntegration does is more correct and optimized.  In practice, I haven't verified that the end result is better than the simple integration in BPP.  However, a couple of years ago here on this forum, someone demonstrated that PI's ImageIntegration gives much better results than that from DSS, when images were taken under different conditions.  I think the same will apply to PI ImageIntegration vs integration in BPP.

 

I think if your images come from a short imaging session (an hour or two) when the condition is very stable, the BPP stacking should be sufficient.  Otherwise it should be better to go to ImageIntegration and let it handle the images taken under variable conditions.

 

Cheers,

Wei-Hao

Attached Thumbnails

  • Screen Shot 2017-10-25 at 11.03.06 AM.jpg


#24 MCovington

MCovington

    Gemini

  • *****
  • topic starter
  • Posts: 3404
  • Joined: 13 May 2014
  • Loc: Michael Covington - Athens, Georgia

Posted 24 October 2017 - 10:59 PM

Thanks!  So the key to it is that BPP's integration assumes the images are all very much alike, and different options should be used if the images being stacked are different from each other in any significant way.  Right?



#25 whwang

whwang

    Surveyor 1

  • *****
  • Posts: 1877
  • Joined: 20 Mar 2013

Posted 24 October 2017 - 11:12 PM

Thanks!  So the key to it is that BPP's integration assumes the images are all very much alike, and different options should be used if the images being stacked are different from each other in any significant way.  Right?

 

 

I think so.  But I won't be surprised if someone tells me there is more to that.  PixInsight is so complicated.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.







Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics