Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Integrating images of already stacked lights

  • Please log in to reply
14 replies to this topic

#1 Startex

Startex

    Vostok 1

  • -----
  • topic starter
  • Posts: 117
  • Joined: 23 Dec 2018
  • Loc: Texas

Posted 21 September 2019 - 06:55 AM

So I have 250 lights (37s) of M81 & M82 that my laptop just can't do all at once, I tried and after more than 10 hours my system gave up I guess. So I tried to integrate 36 lights along with 7 darks, 7 flats and 7 bias. After 2 hours 15 minutes, SUCCESS!  So can I calibrate 30-35 lights at a time then try to stack the 7-8 masters at the end? What is this process called? Do I stack the 7 masters with more flats and darks etc? 

 

For the insane images I see that are total integrations of 10-15-20 hours, heck I've seen some that are 40+ hours. Are these done this way or are these guys simply loading PI with literally hundreds of 3-4-5 minute subs?

 

If this is still too much for my laptop I'll just combine everything down the road when I get my more powerful desktop.


Edited by Startex, 21 September 2019 - 07:26 AM.


#2 kyle528

kyle528

    Messenger

  • *****
  • Posts: 472
  • Joined: 02 Feb 2019
  • Loc: SE Ohio

Posted 21 September 2019 - 08:58 AM

It comes down to the power of your machine. I have a modest budget build using an AMD Ryzen 7 1700x and 32 GB of RAM and my PC calibrated, registered and integrated 510 3 minute subs (25.5 hours) in just under an hour last night. Not lightning fast, but it's a mid-range desktop processor. It's not ideal to stack "sub integrations" into a master one. You are taking an average of averages when you do that, and it is NOT the same as taking one average of an entire data set. I've read somewhere that someone had luck doing that, but I think he was combining the "sub integrations" using PixelMath, not ImageIntegration, IIRC. 



#3 schmeah

schmeah

    Fly Me to the Moon

  • *****
  • Posts: 5239
  • Joined: 26 Jul 2005
  • Loc: Morristown, NJ

Posted 21 September 2019 - 09:34 AM

It's not ideal to stack "sub integrations" into a master one. You are taking an average of averages when you do that, and it is NOT the same as taking one average of an entire data set. 

I’m curious about this as well. Why can you not sum substacks? Do you have examples of how the results are not equivalent to summing the individual subs? There must be some definitive discussion of this in the archives.

 

Derek



#4 kyle528

kyle528

    Messenger

  • *****
  • Posts: 472
  • Joined: 02 Feb 2019
  • Loc: SE Ohio

Posted 21 September 2019 - 09:59 AM

I’m curious about this as well. Why can you not sum substacks? Do you have examples of how the results are not equivalent to summing the individual subs? There must be some definitive discussion of this in the archives.

 

Derek

Take this for what you paid for it, because I may not understand this the best either grin.gif , but the way I understand this is that combining the data set as a whole is better than the substacks because you are getting a better defined average that represents the data set as a whole. Say you are imaging with the moon near last quarter like now, you may start your run with no moon, but the moon is up and shining by the time your session is over. The subs at the beginning of the run will have better SNR (talking about target signal here) than with the moon up. Or maybe you start imaging low in the sky and end near zenith. Maybe you have intermittent high clouds. Taking an average of the entire set will likely give you a different value than taking an average every 50 frames, and the averaging those together. Of course you could weight the stacks when integrating them together, or just integrate them without weighting and you could have good results that way. Pixel rejection works better the more subs you have as well. Keep in mind that I didn't say it *couldn't* be done with good results, just that it isn't ideal. In one of my posts about this very question, I believe it was Alex McConahay that gave the example: Taking the average of the average GPA of the 3 speakers at a high school graduation and the average of the rest of the students would not give the same results as taking the average of the student body as a whole. 


  • schmeah likes this

#5 nimitz69

nimitz69

    Apollo

  • *****
  • Posts: 1464
  • Joined: 21 Apr 2017
  • Loc: A barrier island 18 miles south of Cocoa Beach

Posted 21 September 2019 - 11:05 AM

Do yourself a huge favor if you plan to stick with AP and upgrade your Computer.  Trying to cobble together some way around simply stacking all your images will not do any favors for your images.

 

you can get a refurbished laptop from many places on-line.  I routinely stack 200-300 images plus master cal files in DSS with a refurbed i5 laptop and it never takes more than about a  hr total.  10 hrs to stack & then it fails?  I’d be doing another hobby ....dalek12.gif



#6 AhBok

AhBok

    Vanguard

  • *****
  • Posts: 2296
  • Joined: 02 Dec 2010
  • Loc: Lakeland, TN

Posted 22 September 2019 - 08:02 AM

While I more or less agree with the others, I’ll add that lots of guys stack in sub-stacks as you have asked. I prefer to stack all at once, but occasionally stack new data with an older stacked fits image. You should try this while you are saving for a more capable laptop.

#7 Charlie B

Charlie B

    Apollo

  • -----
  • Posts: 1448
  • Joined: 22 Mar 2008
  • Loc: Sterling, Virginia

Posted 22 September 2019 - 09:40 AM

So I have 250 lights (37s) of M81 & M82 that my laptop just can't do all at once, I tried and after more than 10 hours my system gave up I guess. So I tried to integrate 36 lights along with 7 darks, 7 flats and 7 bias. After 2 hours 15 minutes, SUCCESS!  So can I calibrate 30-35 lights at a time then try to stack the 7-8 masters at the end? What is this process called? Do I stack the 7 masters with more flats and darks etc? 

 

For the insane images I see that are total integrations of 10-15-20 hours, heck I've seen some that are 40+ hours. Are these done this way or are these guys simply loading PI with literally hundreds of 3-4-5 minute subs?

 

If this is still too much for my laptop I'll just combine everything down the road when I get my more powerful desktop.

What machine do you have and how much ram? Image integration has parameters that define your stack size (buffer size and stack size in Mbytes).  If you are short of ram, the program will slow down a lot, because it will be using hard disk as a ram buffer.  You may be able to speed up the processing a lot by increasing memory to the maximum.  I have 16 GB of ram and have no speed issues.  When I only had 4 GB, I did.  Also, mathematically, stacking pre-stacked lights should be mathematically equivalent, as long as the stacking is average.  Weighting of the stacked subs should carry over.

 

Regards,

 

Charlie B 


  • md11spotter98 likes this

#8 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5172
  • Joined: 14 Jun 2009

Posted 22 September 2019 - 09:55 AM

you totally can stack stacks. there's no difference between stacking a bunch of masters and stacking all the subs in one go, provided that there are enough subexposures in each stack to do proper outlier rejection. if so, each stack should be free of hot pixels and other outliers and then there's just no issue with 'averaging the averages'.

 

are you using Pixinsight? are you calibrating your images first? if you are trying to directly integrate CR2 files in pixinisight this puts a lot of pressure on your system's memory. under pixinsight, CR2 files have to be read into the computer's memory wholesale during integration, so you can imagine how much memory you're asking for when you stack a bunch of CR2s. but if you calibrate your images first, PI converts the CR2 files to XISF files, which support incremental reading. the upshot is that all of the files being integrated don't need to be stored in memory the whole time ImageIntegration is running, so the pressure on the memory system is much less.

 

rob


  • md11spotter98 likes this

#9 md11spotter98

md11spotter98

    Vostok 1

  • *****
  • Posts: 137
  • Joined: 23 Jun 2018
  • Loc: Coral Springs, FL (home) Charlottesville, VA (college)

Posted 22 September 2019 - 04:10 PM

I started stacking sub-stacks when I was getting started, and it has saved me so much time. My longest integrations are made up of 1000+ subs since I'm limited to 60 seconds without guiding. With PixInsight it's easy to weight the subs and sub-stacks for better SNR so that the final image is not negatively affected by any changing light pollution from the moon, transparency, etc.

 

Although my sub-stacks are usually 100 to 200 images, and it's done in about 2 hours. A better computer wouldn't hurt.


Edited by md11spotter98, 22 September 2019 - 04:15 PM.


#10 kathyastro

kathyastro

    Viking 1

  • *****
  • Posts: 749
  • Joined: 23 Dec 2016
  • Loc: Nova Scotia

Posted 22 September 2019 - 04:29 PM

The difference between stacking all the subs in one operation and integrating two (or more) stacks is a matter of weighted averages.

 

Suppose you have 100 frames in one stack, and 50 frames in another stack.  If you just average the two stacks, each frame of the 100-frame stack contributes 0.5% of the final image, whereas each frame of the 50-frame stack contributes 1%, twice as much.  Perhaps you don't care (for example if the 50 frames were better than the 100 frames), in which case, go right ahead and do it.  But if you want to get the most out of each frame, you will want to weight each frame equally.  You can do this in PixelMath, with a function like 0.67*A + 0.33*B.


Edited by kathyastro, 22 September 2019 - 04:30 PM.


#11 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5172
  • Joined: 14 Jun 2009

Posted 22 September 2019 - 05:11 PM

that's true - but by default ImageIntegration weights images by SNR so if you just use II it will naturally weight your 100 frame stack at 0.66 and your 50-frame stack at 0.33, or thereabouts.

 

so all you'd have to do is reset the II process and load your images and run. if you only have 2 stacks, you need to load each one twice since II requires a minimum of 3 images to run. if you want custom weights then you need to do as kathy says.

 

rob


  • jdupton and kathyastro like this

#12 jdupton

jdupton

    Surveyor 1

  • *****
  • Posts: 1926
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 22 September 2019 - 05:46 PM

Rob,

 

   This brings up a question I've asked myself a few times and found I cannot think my way through it.

 

   If you are stacking stacks of stacks from different imaging nights, for instance, would it make any sense to take the nightly stacks back through SubFrameSelection and reassign new weights?

 

   In my case, I would have to fudge the process a little because of the "ragged edges" of my individual stacked nights and the inevitable inexact alignments. I would probably need to crop each night's stack to clean data "using a least common denominator" crop and then manually copy the weight keyword back the real stack that would be used. It would be complicated and labor intensive if you have 20 nights of data but if it helps, I'd be willing to give it a try. (Such cropping and manual manipulation would not be required if your nightly stacks were all well centered and matched in rotations.)

 

 

John


Edited by jdupton, 22 September 2019 - 05:50 PM.


#13 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5172
  • Joined: 14 Jun 2009

Posted 22 September 2019 - 07:01 PM

i think if you wanted to do that you could use ImageContainer and Crop so that you don't need to fix each image individually. then if you put the cropped images into SubFrameSelector there wouldn't be any 'ragged' stuff to throw off the statistics. i don't think you'd have to copy any keywords around since SFS would re-write them, right?

 

but i do wonder if it makes sense, since i assume the FHWM/eccentricity of each stack will be pretty good, possibly leaving SNR as the only candidate for weighting... which ImageIntegration will do anyway.

 

rob



#14 jdupton

jdupton

    Surveyor 1

  • *****
  • Posts: 1926
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 22 September 2019 - 07:45 PM

Rob,

 

   Thanks. This sounds like an experiment to put on the ToDo list for someday.

 

   Yes, I could use an ImageContainer to apply the common crop to all frame stacks. However, I would like to end up actually integrating all of the non-cropped images as I do now. That means that the Weighting Keyword would be written to the wrong files. That is where I would have to copy the Weight Keyword from the cropped frames back to the corresponding full un-cropped frame.

 

   I think you may be right about ImageIntegration selecting an SNR Weight that may be as good as what SS could do. Still, I have a nagging feeling it might yield a slightly better result -- I just don't know.

 

 

John


Edited by jdupton, 22 September 2019 - 07:46 PM.


#15 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5172
  • Joined: 14 Jun 2009

Posted 22 September 2019 - 07:50 PM

oh i see what you mean... yeah. not sure how to copy back FITS keywords in an automated fashion.

 

yeah, i hear that... there's always some experiment or other that i want to try.

 

rob




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics