Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

What Files To Keep / Cloud Storage

  • Please log in to reply
6 replies to this topic

#1 MadEquinox

MadEquinox

    Explorer 1

  • *****
  • topic starter
  • Posts: 84
  • Joined: 31 Jul 2009
  • Loc: Northern Virginia

Posted 10 July 2021 - 03:09 PM

As I am starting to amass a lot of images files, I am trying to understand which files I should be keeping in case I want to return to the target sometime in the future.

  • Do I store all lights?
  • All calibration frames?
  • Master integration files from WBPP only?
  • PI projects?

I do not want to store locally, it will all be in the cloud. Its not really feasible to store everything (storage/cost).

 

Thoughts/ideas/suggestions?

 

Thank you.


  • mariemarie likes this

#2 maxsid

maxsid

    Viking 1

  • *****
  • Posts: 539
  • Joined: 11 Sep 2018
  • Loc: Sunnyvale, CA

Posted 10 July 2021 - 04:36 PM

Why wouldn't you store everything locally?

It's easy and cheap.

?



#3 RazvanUnderStars

RazvanUnderStars

    Apollo

  • *****
  • Posts: 1,394
  • Joined: 15 Jul 2014
  • Loc: Toronto, Canada

Posted 10 July 2021 - 04:48 PM

Absolutely keep all the lights. As seen recently with the development of the NormalizedScaleGradient script in PixInsight, improved ways to do even the initial steps can come up at any time.

 

The lights will be most of the data so how much you keep from the other items you listed is a matter of taste. I don't normally keep individual darks, flat darks and flats, just the master dark and master flat. You can keep the master light just after WBPP, why not, it's just one file and it will save you some time, should you want to re-process starting with it. 

 

You can keep the projects as well, especially initially. I would expect that after a few years, one would start from fresh rather than jumping in the middle of an older project so you can prune them then, if you want, but it's up to you. 

 

 

As I am starting to amass a lot of images files, I am trying to understand which files I should be keeping in case I want to return to the target sometime in the future.

  • Do I store all lights?
  • All calibration frames?
  • Master integration files from WBPP only?
  • PI projects?

I do not want to store locally, it will all be in the cloud. Its not really feasible to store everything (storage/cost).

 

Thoughts/ideas/suggestions?

 

Thank you.



#4 RazvanUnderStars

RazvanUnderStars

    Apollo

  • *****
  • Posts: 1,394
  • Joined: 15 Jul 2014
  • Loc: Toronto, Canada

Posted 10 July 2021 - 04:54 PM

I can't answer for the OP, but one reason would be disaster recovery. A frequent practice is the "3-2-1 rule": 3 copies, 2 different media, 1 copy offsite).

 

Even with RAID NAS units (I use one at home), it's possible to have a flood or a fire that destroys the unit (in the case of a fire, I guess the astro images are the least concern but a flood is a more localized disaster.) Or ransomware.

 

 

Why wouldn't you store everything locally?

It's easy and cheap.

?



#5 Dan Crowson

Dan Crowson

    Gemini

  • *****
  • Moderators
  • Posts: 3,104
  • Joined: 08 Oct 2010
  • Loc: Dardenne Prairie, MO

Posted 10 July 2021 - 06:03 PM

I would keep your calibrated files. Your bias, darks and flats won't get better with time.
 
Local storage is cheap. Few image more than I do and my astronomy folder over 10+ years contains about 2.5 TB of data over 120,000 files. USB drives are cheap and you can keep it off-line when not in use.

Dan



#6 Chris Ryan

Chris Ryan

    Mariner 2

  • -----
  • Posts: 225
  • Joined: 08 Mar 2014
  • Loc: Flaxton, Qld, Australia

Posted 10 July 2021 - 06:22 PM

I would keep your calibrated files. Your bias, darks and flats won't get better with time.
 
Local storage is cheap. Few image more than I do and my astronomy folder over 10+ years contains about 2.5 TB of data over 120,000 files. USB drives are cheap and you can keep it off-line when not in use.

Dan

I think you'll find that with today's CMOS cameras, generating 2.5TB of data will take a lot less time than 10 years. 

 

However, your point about local storage still stands.  6, 8, etc TB HDD are relatively cheap - even if you bought 1 / year.

 

Cloud backups are worth it for the offsite aspect.  But as mentioned above, I'd be using it in combination with the local backups as well.  If you have alternate physical sites, it could also be worthwhile periodically copying a backup disk and taking it to the other site(s).



#7 MadEquinox

MadEquinox

    Explorer 1

  • *****
  • topic starter
  • Posts: 84
  • Joined: 31 Jul 2009
  • Loc: Northern Virginia

Posted 13 July 2021 - 03:00 PM

Good call on the local storage. After reading the responses I remembered I had an old Synology DS213 with 3GB HDD (RAID1). Looking at these drives online, they seem to be fairly cheap these days. Configured the NAS and moved my stuff out of the cloud. Thank you.

 

Based upon responses, going to keep all lights and calibration masters. Will address storage constraints when I get to them.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics