Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

PixInsight file archiving, for multi-year projects

  • Please log in to reply
13 replies to this topic

#1 JDAstrophoto

JDAstrophoto

    Lift Off

  • -----
  • topic starter
  • Posts: 12
  • Joined: 06 Jan 2022

Posted 22 September 2022 - 06:57 PM

PixInsight experts, lend me your expertise.  So far, I've been saving everything, dark flats, flats, calibrated directory, cosmetized directory, etc. and the PixInsight project itself. I have a QHY600 60Mp camera, so these files and directories along with the project can be 6.5 GIG for each object.  This doesn't include the lights or the darks.  

 

What is really required for a multi-year project.  Year to year I would like to just keep adding data to an object.  I would think that only the Master Lights are required, is that correct?  Therefore you would only need to keep the Master Lights for each filter from year to year and then combine Master Lights each year you wanted to add data to the object.  



#2 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 29,853
  • Joined: 27 Oct 2014

Posted 22 September 2022 - 07:05 PM

If you mean the calibrated, corrected, debayered (if necessary), aligned lights, that is correct.

 

Some people like to also save the master bias, flat, dark.  Can maybe be handy sometime, take up little space.

 

I have about 20 terrabytes of spinning drives for archiving things.


Edited by bobzeq25, 22 September 2022 - 07:05 PM.


#3 JDAstrophoto

JDAstrophoto

    Lift Off

  • -----
  • topic starter
  • Posts: 12
  • Joined: 06 Jan 2022

Posted 22 September 2022 - 07:11 PM

Thanks for the response.  Yes, calibrated, corrected, aligned lights.  Master Light in the Master directory after Weighted Batch PreProcessing (WBPP).  

So if I had master lights from 2021 and I wanted to combine them them with 2022 data.  I use WBPP to create Master lights for 2022, then combine the 2022 Master Lights with the 2021 Master Lights?

Does that work in WBPP?



#4 choward94002

choward94002

    Apollo

  • *****
  • Posts: 1,303
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 22 September 2022 - 08:39 PM

I too have nearly 175 TB's of images over the last 8 years ... simple answer is, keep everything including the individual subs; 7Zip does well on the images, and drives are really cheap (I can get a 6TB drive for around $200 these days).  Duplicate the drive (so if one dies you have a backup) and just keep track of what targets is on what drive ...

 

The reason I keep the individual subs, rather than a master light, is that as time goes on the noise reduction algorithms get better ... so subs that before had a pesky walking pattern noise now clean up very nicely, or a sub that lost a bunch of dynamic range for some reason can be restored.  By keeping the individual subs you can do that kind of reprocessing, while if you've got only a master light you can't do that anymore ... 


  • jdupton, 17.5Dob, Linwood and 1 other like this

#5 Linwood

Linwood

    Gemini

  • *****
  • Posts: 3,170
  • Joined: 09 Jul 2020
  • Loc: Ft Myers, Florida, USA

Posted 22 September 2022 - 08:40 PM

Some of the last year's changes in Pixinsight has convinced me to go the extra step, and now I save ONLY the raw images captured (flats, lights), and have a library of master darks which I will at some point need to rebuild, and I would rerun WBPP to do the overall process again. 

This way I get the benefit of the new weighting algorithms during integration, and if they come about same for calibration.

This is also what Juan recommends (more precisely he recommends you save the individual darks also in case you need to rebuild the dark library, I haven't done that yet, but might next time I do darks). 

 

I save the raws from each session in a subfolder with the morning's date (specifically MORNING_YYYYMMDD) and use a keyword of MORNING in WBPP to cause it to correctly match lights with flats.

 

At some point if I have multi-year subjects as opposed to many nights, I'll need something like a DARKLIB_1, DARKLIB_2, etc and use DARKLIB as a keyword and put that above the MORNING folders to associate those with the corresponding dark library (or raw darks as the case may be). 

This is a LOT of stuff; I bought a couple new offline drives to save all this raw data; the final copies are online and backed up. I'm not bothering to backup (other than on two on-site drives) all the raw data, it is just getting too large to do a cloud backup of the raw images. 


  • nyairman likes this

#6 ngatel

ngatel

    Viking 1

  • -----
  • Posts: 792
  • Joined: 10 Dec 2006
  • Loc: Palm Springs, CA

Posted 22 September 2022 - 08:43 PM

PixInsight experts, lend me your expertise.  

Not an expert at all. But I save everything including the PixInsight Project on two 12TB external hard drives (same files on each). Back up media is very inexpensive these days.



#7 JDAstrophoto

JDAstrophoto

    Lift Off

  • -----
  • topic starter
  • Posts: 12
  • Joined: 06 Jan 2022

Posted 22 September 2022 - 08:57 PM

"You are going to need a bigger boat"

https://www.youtube....h?v=QT9BeGNnCqw



#8 choward94002

choward94002

    Apollo

  • *****
  • Posts: 1,303
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 22 September 2022 - 09:04 PM

I used to have a very byzantine folder structure for targets and such, that became untenable with projects spanning drives and years ... so these days I have a very simple structure <target><lights/darks/bias/flats> and let Voyager fill in all the pertinent info into the filename (up to 256 characters)

 

I use this utility [https://www.wincatal...cataloging.html] to quickly find thing across drives, and then the new WBPP makes it very easy to mount the appropriate drive and pull the needed file ...



#9 jesco_t

jesco_t

    Mariner 2

  • -----
  • Posts: 272
  • Joined: 23 Jan 2020
  • Loc: Hamburg, Germany

Posted 23 September 2022 - 03:31 AM

I don’t understand what the benefit is in saving raw files plus calibration frames. All it does is making the archive more complex.

There haven’t been any new paradigms and developments in image calibration in the last years. The new stuff in stacking are the improvement in normalization, outlier rejection and drizzling, but not the pre-processing.

I just make my life easier and save only calibrated lights in my archive. WBPP and APP work equally nice if you throw calibrated frames at them. It’s actually less work that way if you want to restack a set of lights or add new data.

Just my 0.02$.

Jesco

#10 Linwood

Linwood

    Gemini

  • *****
  • Posts: 3,170
  • Joined: 09 Jul 2020
  • Loc: Ft Myers, Florida, USA

Posted 23 September 2022 - 06:35 AM

I don’t understand what the benefit is in saving raw files plus calibration frames. All it does is making the archive more complex.

There haven’t been any new paradigms and developments in image calibration in the last years. The new stuff in stacking are the improvement in normalization, outlier rejection and drizzling, but not the pre-processing.

 

That's not exactly true, the new integration weighting algorithms (maybe others) depend now on data calculated during calibration, and do not work with previously calibrated images. 


  • jesco_t likes this

#11 jesco_t

jesco_t

    Mariner 2

  • -----
  • Posts: 272
  • Joined: 23 Jan 2020
  • Loc: Hamburg, Germany

Posted 23 September 2022 - 02:14 PM

That's not exactly true, the new integration weighting algorithms (maybe others) depend now on data calculated during calibration, and do not work with previously calibrated images. 

Good point. I think you're referring to the new PSF*Weights in PixInsight.

 

For OSC images the weight calculation is done during Debayer. So you can just feed calibrated OSC lights into the WBPP pipeline and it all works with the newest integration weighting. For monochrome frames the weights are indeed calculated during image calibration. I just tried to run ImageCalibration without any calibration frames just to get the noise & signal weights calculated and it doesn't work that way. So you're right, for monochrome you'll need to keep raw and calibration frames. I don't think there's a technical reason, though - the weight calculation runs on calibrated data (it says so in all documentation and forum posts over at PixInsight).

 

I shoot OSC, so that point escaped me smile.gif
 



#12 Linwood

Linwood

    Gemini

  • *****
  • Posts: 3,170
  • Joined: 09 Jul 2020
  • Loc: Ft Myers, Florida, USA

Posted 23 September 2022 - 05:04 PM

I shoot OSC, so that point escaped me smile.gif

And for opposite reasons I hadn't realized it was different for OSC.  I live in the B&W era still, where Lucy Ball had grey hair not red.  lol.gif

 

I actually wish PI would spend some time on lifecycle issues, there's some serious good work in WBPP already, maybe one day we just feed in files and they'll build and archiving system to save off only what we need, and be able to rebuild with just a few buttons. 



#13 choward94002

choward94002

    Apollo

  • *****
  • Posts: 1,303
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 23 September 2022 - 05:46 PM

And for opposite reasons I hadn't realized it was different for OSC.  I live in the B&W era still, where Lucy Ball had grey hair not red.  lol.gif

 

I actually wish PI would spend some time on lifecycle issues, there's some serious good work in WBPP already, maybe one day we just feed in files and they'll build and archiving system to save off only what we need, and be able to rebuild with just a few buttons. 

Actually apparently WBPP now does really good with incremental stacks (there's another recent thread here) where I can take a WBPP stack and just "add in" the new pix without needing to make an entirely new stack ... got to try that this weekend!

 

Personally, drive space is cheap and data is hard to come by ... and once you've deleted it then it's gone forever.  A good case in point is that I used to discard pix that had satellite trails in them because they would goof up the stacking ... then a better WBPP algorithm came out such that I can salvage signal even from pix that have trails in them.  Who knows how much slightly better some target stacks might be had I been able to add those discarded pix into the stack ... but again, once deleted always gone

 

So, I have about 175TB of raw data (to be fair, it's 2x duplicated so about 60TB of unique data) that maybe someday I'll reprocess when WBPP v2 or similar comes out ...



#14 jpengstrom

jpengstrom

    Apollo

  • *****
  • Posts: 1,141
  • Joined: 08 Feb 2021
  • Loc: North Texas

Posted 23 September 2022 - 07:26 PM

Thanks for the response.  Yes, calibrated, corrected, aligned lights.  Master Light in the Master directory after Weighted Batch PreProcessing (WBPP).  

So if I had master lights from 2021 and I wanted to combine them them with 2022 data.  I use WBPP to create Master lights for 2022, then combine the 2022 Master Lights with the 2021 Master Lights?

Does that work in WBPP?

If you're talking about the integrated lights that are in the "master" directory, then no, you should not do as you describe. In order to integrate new lights (after they've been calibrated with their appropriate flats and darks), you need to have the old calibrated lights from the previous year.  I'm not at my processing computer, but I believe they are in the "calibrated" directory.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics