For those of you that image a lot, how do you manage all your data?
I am fortunate that I have clear skies close to 200 days a year and I have a semi-permanent setup so I am able to image a lot. I have 1TB on my NUC on the scope and I generally manually copy files in the morning (drag/drop with file explorer). Recently I have started imaging unbinned with my ASI294mm and the files are pretty big, and with broadband imaging I do 60s subs, so this generates a good bit of data (35G/night).
In terms of backup, there is too much data to back it all up to cloud storage (especially when you consider all the pre-processed interim files) but I want to back up more than just my final image so I am starting to think about how to organize my file system to make backup easier.
Here is what I am thinking:
Tier 1: Master stacks and processed final (backup to cloud)
So master stacks and processed final (and steps) should be backed up to the cloud.
Tier 2: Source data: Subs and flats (backup to removable HD and keep a copy on my PC)
Source data should be saved, but maybe just to local backup (archievied)
Tier 3: Calibration and preprocessed subs (registered, etc.) (backup to removable HD or discard?)
I guess I’ll hold on to these too until maybe one day if it is problematic?
Want to keep them separate though.
Calibration Data Organization
Want to have flat masters and dark masters organized somewhat.
Want to ensure that it is clear which image train/dates/gain, etc.
Separate out flat/dark subs from masters
Folder Structure (draft?)
So here is what I am thinking:
- LIGHTS (this is just the raw LIGHT subs)
- DARKS (this is just the raw DARK subs)
- Gain 121
- Gain 300
- FLATS (this is just the raw FLAT subs)
- PREPROCESSED (this is calibrated/regstered, etc.)
- PROCESSED (this is the master stacks and post-processing)
- FLATMASTERS (this is the stacked master flats, by image train, dated)
- DARKMASTERS (stacked master darks by camera/bin/gain)
- Gain 121
- Gain 300
Interested in anyone's thoughts or input into this!