32 replies to this topic

[fewayne]

My 4 TB NAS is knocking on the door marked "WE'RE FULL, GO HOME", and my reasonably-priced iDrive cloud-backup plan is about to get a LOT less so at the same time.

 

I'm probably never going to go back and reprocess six-year-old subexposures...yeah, I see a lot of you cringing by reflex as you read that. You're right, I don't want to just pitch 'em.

 

But archival storage is a real conundrum. The lifespan of the medium itself is one thing, but the technological lifespan is something entirely else again. (I mean, suppose I'd stored a bunch of stuff on Zip drives. I'd be screwed now.)

 

If I had a Blu-Ray writer I might use that, pretty good balance of media cost, archival permanence, and reasonably predictable tech lifespan.

 

What I've got are DVD-Rs and an Apple SuperDrive. Any better suggestions? I guess $100 for a Blu-Ray writer and 100 GB per $0.50 disk is not at all unreasonable, esp. since it will read/write DVD-Rs too.

[maxsid]

OMG.. just get a couple of 8Tb hard drives or SSD. And an external enclosure.

[rickmurray1989]

As a professional photographer, I have found all forms of "home burned" optical media  (CD, DVD, Blu-Ray) to become vulnerable to degradation over the long haul, unlike their commercially manufactured peers (commercial media burned with physical pits for data as opposed to home burned media burning dye spots). Due to the low cost of hard drives, both moving disc and SSDD, I just recopy critical data every so often to a new removable disc drive and set it aside in a climate-controlled space.

[rj144]

Agree with suggestions to get a couple of larger drives and mirror RAID them possibly.  Do not use optical media.

[imtl]

NO DVD. External SSD.

[dayglow]

I use BLU-RAY disks for my archives. 

 

Although I have not yet used them, the M-DISK type are some made for long duration storage and many drives already have burning capability for them.  They sound costly but in reality one would not use very many of them and justification is pretty easy.

[Mark Lovik]

The old writeable CDs were reasonably stable, writeable DVD's were not.

If the medium is not as reliable - you need duplicate copies

 

My choices

- SSD external drives (fairly reliable if you do not use these continually)

- Larger USB thumbdrives.  They are getting cheap for the 256-512Gb versions.  These can be archived over time periods.  Incredible I/O speeds are not critical ... this is offline storage and can be cheaper than the SSD external drives.

- Large external hard drives as a backup (2nd copy)

 

Then a reasonable question - what about paring down your old data.  Just keep the final images for views that are older, or less successful early imaging attempts.  Summarizing and reorganizing your data can have more value than providing more image space.

Edited by Mark Lovik, 24 January 2022 - 05:06 PM.

[Psychlist1972]

Good write-once optical media is considered to be really good for long-term backup. A properly stored (away from UV light, sealed, etc.), quality write-once disk will last much longer than most of us.

 

Other than that, the most reliable long-term consumer storage is the classic spinning rust drive. You can get 10-15 years or so out of them before there's a concern, assuming you store them properly.

 

I wouldn't go with SSDs are they will degrade faster in most cases. Plus, they are more expensive. Similar issues with USB thumb drives and similar.

 

There's also backup tape, but I don't think any of us want to deal with that anymore.

 

But really, ask yourself how long you want to keep this stuff around, and what you really need. IMO, the calibrated lights are the only things worth keeping. I wouldn't keep original lights/flats/darks/bias/etc.

 

Pete

[Psychlist1972]

BTW, there's a reliability and longevity difference between archival-quality media and the bulk stuff you get on ebay. 

 

Pete

[rj144]

I have had more than one optical storage medium fail over the years.  I haven't used one in 10 years since flash media and HD became quite a bit cheaper.

[Psychlist1972]

I have had more than one optical storage medium fail over the years.  I haven't used one in 10 years since flash media and HD became quite a bit cheaper.

Again, big difference between archival media and the stuff you got in boxes of 10 at Radio Shack.

 

HUGE archival systems by companies like Panasonic are based entirely off of cartridges of optical media, and offer 50+ years of storage longevity.

 

So if you get archival blu ray disks instead of the cheap DVD RW stuff, you'll be well on your way. The rest is about storing them in controlled conditions, away from heat and UV.

 

Pete

[jdupton]

fewayne,

 

   I am something of a packrat and try to keep all of my original image capture data. I know many do not do that but I find it fun from time to time to go back and reprocess the older data using newly available tools and techniques. I also keep all of my raw Library calibration frame originals. I do not keep any intermediate data at all.

 

   When I save such data, I find that the raw FITs frames compress very well. After working on a project, I keep the raw captured FITs and compress them into a dated compressed ZIP archive folder. They usually shrink down by 35% to 40% or more saving storage space. I also keep the final couple of versions of my processed images from that target with the original raw capture frames. Everything in between gets deleted as it can recreated at will using newer tools and techniques; often giving a better image than the original processing.

 

   By doing this, I find that my archived (ZIP'ed) data takes much less space. I keep two copies of that all that data. My image processing system has a 6 TB hard drive for all archived data. I also keep a separate copy on an external hard drive in a multiple drive unit attached via USB-3.2 connection. The external hard drive (and a smaller 1 TB SSD in the same external device) are not left powered and are only used for archiving copies of data I want to keep.

 

   My actual data organization is not so clean in terms of folder structures. I do need to do some reorganization but at least I have copies of nearly all my astrophotography data back to about 2007. Modern hard disk technology is cheap enough to just use offline hard disk storage for my archival. It probably takes less physical space than if I had burned everything to Blu-ray or DVD.

 

 

John

[fewayne]

I guess if I use a very mainstream file system and ensure that it's accessible over a pretty long-lived tech like USB, spinning drives are safe enough. I worry some about SSDs and flash. Now that I've burned half a dozen DVDs I'm pretty sure I don't want to do this hundreds of times more! But I started with the oldest stuff anyway, I won't have to copy it back to archive onto hard drives.

 

I have been suspicious of spinning-drive storage primarily because interface tech morphs relatively rapidly. Got a machine that can read a SCSI disk? Yeah, me neither. I could probably find IDE if I had to, SATA probably still has some life left in it. But USB-C will be around long enough that I'll be senile before it goes.

 

Thanks.

[unimatrix0]

For backing up data, you can get by HDD hard drives also. The cool thing with HDD, that data can be recovered even if you delete them accidentally, not so much with SSD. 

[rj144]

My 4 TB NAS is knocking on the door marked "WE'RE FULL, GO HOME", and my reasonably-priced iDrive cloud-backup plan is about to get a LOT less so at the same time.

 

I'm probably never going to go back and reprocess six-year-old subexposures...yeah, I see a lot of you cringing by reflex as you read that. You're right, I don't want to just pitch 'em.

 

But archival storage is a real conundrum. The lifespan of the medium itself is one thing, but the technological lifespan is something entirely else again. (I mean, suppose I'd stored a bunch of stuff on Zip drives. I'd be screwed now.)

 

If I had a Blu-Ray writer I might use that, pretty good balance of media cost, archival permanence, and reasonably predictable tech lifespan.

 

What I've got are DVD-Rs and an Apple SuperDrive. Any better suggestions? I guess $100 for a Blu-Ray writer and 100 GB per $0.50 disk is not at all unreasonable, esp. since it will read/write DVD-Rs too.

I think 100GB disks are way more than $0.50.  Those are the 25 GB size discs.  

[rj144]

Again, big difference between archival media and the stuff you got in boxes of 10 at Radio Shack.

 

HUGE archival systems by companies like Panasonic are based entirely off of cartridges of optical media, and offer 50+ years of storage longevity.

 

So if you get archival blu ray disks instead of the cheap DVD RW stuff, you'll be well on your way. The rest is about storing them in controlled conditions, away from heat and UV.

 

Pete

To me it doesn't make any sense though.  If you want to back up a TB of data, that's 40 regular sized Blu-rays you need to burn and keep track of or >$100 for 10 of the 100 GB sized variety.  For about $120 you get 2, 2 TB mechanical portable USB drives.  You can make redundant copies of your data in different locations.  And, if you don't use them often, they'll last forever.

Edited by rj144, 24 January 2022 - 09:19 PM.

[Psychlist1972]

To me it doesn't make any sense though.  If you want to back up a TB of data, that's 40 regular sized Blu-rays you need to burn and keep track of or >$100 for 10 of the 100 GB sized variety.  For about $120 you get 2, 2 TB mechanical portable USB drives.  You can make redundant copies of your data in different locations.  And, if you don't use them often, they'll last forever.

Actually, spinning rust drives don't last forever if unused. But they do last 10+ years if kept in an ideal environment, which may be long enough. The data on them does degrade over time.

 

If you actually spin them back up and refresh things yearly, they'll last longer, but mechanical wear and tear is a real thing, limiting drives to around 5 years of active service.

 

Pete

[rj144]

Actually, spinning rust drives don't last forever if unused. But they do last 10+ years if kept in an ideal environment, which may be long enough. The data on them does degrade over time.

 

If you actually spin them back up and refresh things yearly, they'll last longer, but mechanical wear and tear is a real thing, limiting drives to around 5 years of active service.

 

Pete

Thanks.  I have a lot of portable USB drives and only one has kind of failed.  It's not really dead... it just doesn't spin up all the time, but when it does, I can see the data fine.  My oldest portable drive is 10+ years old and still works completely fine.  I have at least 6 more than 5 years old too that all work fine.

[ChrisWhite]

If its taken 7+ years to fill up 4tb of hard drive that's not too much data. If I were in your shoes, I'd park that drive somewhere (back it up on optical disks for offsite backup) and invest in a nice 2tb external ssd for going forward. I don't use spinning drives anymore, as they are too susceptible to failure when bumped or dropped. I've had a couple go this way.

[Psychlist1972]

Thanks.  I have a lot of portable USB drives and only one has kind of failed.  It's not really dead... it just doesn't spin up all the time, but when it does, I can see the data fine.  My oldest portable drive is 10+ years old and still works completely fine.  I have at least 6 more than 5 years old too that all work fine.

That's great. You've been really lucky or have taken great care of your hardware.

 

In active use, this is what spinning drives look like (there are other similar graphs, with similar numbers)

 

 

https://www.backblaz...sk-drives-last/

 

Note: that's for drives in use (internal, USB, etc.), not for those in storage. Stored drives have a different failure mode based on degradation of the magnetics.

 

So, it's a lottery. If you're lucky, awesome. I've had plenty of spinning drives fail over time. I've also had a full slot PCIe NVME drive last well beyond what it should have (but it was originally marketed as an enterprise-grade drive as I recall).

 

Pete

[fewayne]

Drive life is one area where personal experience is not a good guide -- ya gotta go with the industry-wide statistics.

 

Thanks for the thoughtful and informed advice, folks. 

[IR656nm]

Hello Fewayne,

Very interesting topic you have raised. I am going through the same problem.

I have started digital dSLR work since around 2009.

I have kept old files (all) in either a combination of hard drives and optical (DVD-R, BD-R, BD-RE). I looked at the options of NAS-RAID and other stacked drive storage.

I decided that the answer depends on a number of things.

Ultimately, it will be personal decision.

How much time will you have to analyse/re process old data ?

How much data can you process to your satisfaction ?

 

I still have old data from a milky way shoot in a dark site about 4 years ago. I do not expect to shoot these conditions very often.

So for special data, you could do multiple back ups e.g. BD-R for archive. and also keep a copy in a portable hard drive. Go to the hard drive for another look or further processing, and use the archive in case the hard drive does not work.

 

Ultimately, how much time do you have to do justice to all the files.

Maybe just keep the worthwhile data. The early ones where you did not use good enough equipment or techniques may not be worth keeping. It depends on your finances.

 

I can imagine Bill Gates could afford limitless space and hard drives, and keep every single thing he needs to forever, in climate controlled conditions in a climate controlled room with people looking after it for him ?

 

Best wishes.

Edited by IR656nm, 27 January 2022 - 03:57 AM.

[IR656nm]

Hi Fewayne,

I recently purchased a 8 TB portable thumb drive, I think it is SSD mobile solid state drive with USB3.1 connection. Only 40 pounds sterling (including postage). Maybe it is cheaper in the USA. I have not tested large data transfers or how long the data could last. But buy 2 or 3 and you could be covered for 10 to 20 years depending on how much data you acquire.

 

How much data do you acquire per year on average ??

 

Hope this helps.

[rj144]

Hi Fewayne,

I recently purchased a 8 TB portable thumb drive, I think it is SSD mobile solid state drive with USB3.1 connection. Only 40 pounds sterling (including postage). Maybe it is cheaper in the USA. I have not tested large data transfers or how long the data could last. But buy 2 or 3 and you could be covered for 10 to 20 years depending on how much data you acquire.

 

How much data do you acquire per year on average ??

 

Hope this helps.

That's quite the "thumb" drive.  Are you sure it's a thumb drive?  

[rj144]

If its taken 7+ years to fill up 4tb of hard drive that's not too much data. If I were in your shoes, I'd park that drive somewhere (back it up on optical disks for offsite backup) and invest in a nice 2tb external ssd for going forward. I don't use spinning drives anymore, as they are too susceptible to failure when bumped or dropped. I've had a couple go this way.

Yes, but ssd/flash drives will fail after a certain amount of writes/reads and when they fail, it's almost impossible to recover the data.  Just be careful