Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Storage issue?

  • Please log in to reply
5 replies to this topic

#1 George P Dunham

George P Dunham

    Viking 1

  • -----
  • topic starter
  • Posts: 836
  • Joined: 24 Jun 2012
  • Loc: Crawford,CO USA

Posted 16 January 2022 - 11:59 PM

I store originals on an external 2tb ssd attached to my telescope computer and down load them to my processing computer which stores them on another external 2tb ssd drive.  I save all of my imaging files on this second drive.  I am running out of storage and need to revise my strategy.  Anybody care to share how they are doing it?  I need to reinvent my file structures and calibration library storage.  Losing track of flats.



#2 JCDAstro

JCDAstro

    Mariner 2

  • -----
  • Posts: 279
  • Joined: 25 Mar 2021

Posted 17 January 2022 - 12:09 AM

I have a synology NAS, 4 bays, as I run out of space I upgrade the drives and it figures it out if you change them one at a time or even if you only use one or two bays. I currently have 3 of 4 bays populated. Well worth the investment. My scope computer can also reach the NAS, makes transfers for processing easier as a bonus. I started out with two 2 TB drives, then added a 4 TB, now have 3 4TB drives in the NAS.

Based on others advice here on CN, I now only save the calibrated lights, that way I don't need to save flats and dark flats after they have been used.
  • CharLakeAstro likes this

#3 CharLakeAstro

CharLakeAstro

    Viking 1

  • *****
  • Posts: 915
  • Joined: 12 Jan 2015
  • Loc: 44.53,-76.01

Posted 17 January 2022 - 12:09 AM

I resolved my storage in the simplest manner possible.

 

For my CCD camera, I always kept sub frames, but since I have started CMOS imaging, with more frames per target, I now only keep the calibrated stacks, I no longer keep the individual frames.



#4 Sean1980

Sean1980

    Vostok 1

  • -----
  • Posts: 109
  • Joined: 10 Aug 2020

Posted 17 January 2022 - 12:15 AM

Keep the uncalibrated lights, master flat, master dark. Have a 4-bay NAS with 4TB drives in raid5 giving me 12TB storage which for now seems enough. I do cull older data after major image train changes since it will not be compatible with new data

#5 galacticinsomnia

galacticinsomnia

    Vanguard

  • *****
  • Posts: 2,432
  • Joined: 14 Aug 2020
  • Loc: Pacific Northwest - Oregon

Posted 17 January 2022 - 03:35 PM

I use old school raid with this kind of organization.  My calibration file library is done by Camer/Scope/Date/(flats, darks, bias)

2021-10-23_160435.jpg

 

Clear Skies !!
 



#6 whwang

whwang

    Soyuz

  • *****
  • Posts: 3,987
  • Joined: 20 Mar 2013

Posted 17 January 2022 - 08:46 PM

I never delete raw files.  They are the direct products of our hard works at night.  Your processing skills can evolve over time so your processed images can be replaced in the future.  However, unless you substantially improve your skills or equipment, the raw files should stay there.

 

That being said, you don't need a fast SSD for storing raw data. Hard disks are cheap. You can easily purchase a single 12 to 16 TB HD to archive the raw data, and some processed data. You can get a second one for backup. They won't cost as much as your camera or telescope. So this is an absolutely necessary investment.

 

Then for imaging, a fast SSD can be useful, but it doesn't need to be huge. I believe a 512 GB SSD would be sufficient.

 

Finally, for data processing, large and fast SSD will be useful. You can probably get two SATA SSD and use them in RAID0 mode to further speed the I/O up. (If you can afford PCIe SSD (3rd or 4th generation), then you won't need RAID0.) Here, storage space can be precious. So you may delete those intermediate files produced during the process, like calibrated and/or registered subs. You can keep the stacked images and the final processed images here in the fast SSD or move them to the HD archive, depending on how frequent you are going to use them.

 

I do the above for not only my astrophotography images, but also my data taken from large observatories.

 

Hope this helps.

 

Cheers,

Wei-Hao




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics