21 replies to this topic

[DanMiller]

I have been writing a couple of DOS scripts for various part of my processes.  Clearing out the folder on my session laptop of previous session image files, put it at the beginning of the NINA session.  Have one at the end of the session to copy everything I took that night to a thumb drive. And I just was playing around with a script for backing up all of my images.

 

I had previously used XCOPY command to copy the files over to the backup external drive, so I had probably over 100 gig already backed up there.  I decided to update that backup folder with any new or changed files.  This time I decided to see how ROBOCOPY would perform.   My current folder for AP images is at 146 gig, I know not that big.   When I ran robocopy with the correct flags to only copy new or changed files/folders.  That bad boy ran in like 5 minutes. 

 

I only post this because I was thinking of either buying backup software or looking into some open source.  Not anymore.  robocopy, which is already installed on Windows 10/11 is all that I need.  Just something for people to consider when creating backups.  Why buy when you already have a good backup utility if you use windows.  I don't have any unix/lynix machines anymore, so I can't test anything with them.

[gmiller123456]

Robocopy is tough to beat, I've been using it and rsync (for Linux) pretty much exclusively for decades.

[Skywatchr]

In case anyone wants to know the command line options for robocopy...

 

https://adamtheautomator.com/robocopy/

[DanMiller]

In case anyone wants to know the command line options for robocopy...

 

https://adamtheautomator.com/robocopy/

Thank you for posting that, I should have

[arbit]

Since you are using NINA, there's a robocopy plugin.

It backs up the subs as they come in to another folder. In my setup, the capture minipc uses the plugin to backup over WiFi to the processing PC in real time.

Saves time and is real time backup


Sent from my SM-S908E using Tapatalk

[DanMiller]

Since you are using NINA, there's a robocopy plugin.

It backs up the subs as they come in to another folder. In my setup, the capture minipc uses the plugin to backup over WiFi to the processing PC in real time.

Saves time and is real time backup


Sent from my SM-S908E using Tapatalk

I did not know that.  I have been saving everythig to the session laptop.  At the end of the session, I copy everything to a thumb drive so I can just transfer it easier. I clear out the tumb drive before I copy to it.  And at the beginning before I start actual imaging, I clear that folder out.  Right now, those scripts are using the XCopy DOS command. But, I will definately check out that plugin.

 

Thank you!

 

Dan

[johnpd]

Robocopy is marvelous. I use it regularly. Beats XCopy in that it copies multiple files concurrently.

 

JohnD

[F.Meiresonne]

Robocopy is the way to go. No cost and does the job. Xcopy never worked as areal copy program. At the slightest event..it stops.

 

I use robocopy too. Configured to mirror my source data, so i have at a identical copy of my source data.

[SteveL42]

Just be aware that programs like Robocopy create a copy - there is no versioning.  So if you corrupt a file and copy it over your good version... you "backed up" the bad version and destroyed the good.

 

A true backup scheme will capture the changes AND keep the previous versions.  

[DanMiller]

Just be aware that programs like Robocopy create a copy - there is no versioning.  So if you corrupt a file and copy it over your good version... you "backed up" the bad version and destroyed the good.

 

A true backup scheme will capture the changes AND keep the previous versions.  

well, you could  probably code around that.  I seen that robocopy has a flag where it only produces a list of what it is going to copy. You could run that first, and then using that list create your own backups. I doubt very much I am going to go through all of that, but it is an option.  

[Paul Romero]

hi there,

There are several good freeware graphical user interface programs out there for RoboCopy. The one I picked even includes a scheduler for automated.

Clear skies, Paul.

[DanMiller]

hi there,

There are several good freeware graphical user interface programs out there for RoboCopy. The one I picked even includes a scheduler for automated.

Clear skies, Paul.

Thanks Paul

[gmiller123456]

A true backup scheme will capture the changes AND keep the previous versions.  

A copy is a "true" backup.  Granted there are backup systems with more features than just "copy".  I'm sure a lot of people are different, but for me, the overwhelming number of files will never get changed again after the first backup.  In the instances where I do have a version worth saving I'll just save a copy with a name that describes it, which is a lot easier to deal with than any backup or versioning software I've dealt with.

[SteveL42]

A copy is a "true" backup.  Granted there are backup systems with more features than just "copy".  I'm sure a lot of people are different, but for me, the overwhelming number of files will never get changed again after the first backup.  In the instances where I do have a version worth saving I'll just save a copy with a name that describes it, which is a lot easier to deal with than any backup or versioning software I've dealt with.

 

A copy can certainly be considered a backup.  My point is if a virus/ransomware/user error/etc., corrupts the active file, and you "back it up" by copying it to your backup drive... you just corrupted your backup.

 

I run all sorts of backup systems - Acronis, Synology, Veeam, Cloudberry, etc., etc, and all will allow me to recover previous versions of files, and some a previous version of my entire system.  My backups are also stored both onsite and offsite - another consideration.

[DanMiller]

Watching this, the only thing I save is the raw files.  And they don't change. I upload anything I like to astrobin, so if I want a copy of it.  I just will download it from there.  And if I didn't upload something there, it wasn't worth the trouble to me.  Either because I didn't like my edit or the data wasn't up to par. In either case, not really worried about it.

[Bob Denny]

A copy can certainly be considered a backup.  My point is if a virus/ransomware/user error/etc., corrupts the active file, and you "back it up" by copying it to your backup drive... you just corrupted your backup.

 

I run all sorts of backup systems - Acronis, Synology, Veeam, Cloudberry, etc., etc, and all will allow me to recover previous versions of files, and some a previous version of my entire system.  My backups are also stored both onsite and offsite - another consideration.

Unsure why you would need all those backup systems. I have used one of them for 15+ years. I've replaced hard drives in notebooks in one hour, kept monthly system image backups in my safe box, recovered from hard drive crashes in a couple of hours max with essentially no lossage and no reinstalling of the OS and reinstallation of apps (GACK!), recovered previous versions of files, accidentally deleted files, etc. etc. Given the cost of these imaging tools I don't understand why people don't use one of these. 4Tb portable USB drive (Seagate) $120, Backup app ($50/yr). Freeware? When was the last time you went out to dinner? What's your time worth? Sorry for the snarky reply.

Edited by Bob Denny, 26 May 2023 - 08:44 PM.

[DanMiller]

Unsure why you would need all those backup systems. I have used one of them for 15+ years. I've replaced hard drives in notebooks in one hour, kept monthly system image backups in my safe box, recovered from hard drive crashes in a couple of hours max with essentially no lossage and no reinstalling of the OS and reinstallation of apps (GACK!), recovered previous versions of files, accidentally deleted files, etc. etc. Given the cost of these imaging tools I don't understand why people don't use one of these. 4Tb portable USB drive (Seagate) $120, Backup app ($50/yr). Freeware? When was the last time you went out to dinner? What's your time worth? Sorry for the snarky reply.

I have 1 1t drives whhich is my active drive, and the 4tb for where I backup to.  I probably will just clean off the 1tb when it gets filled after backing it up.  Like I said,I only keep the raw files. The images that I like, I post to astrobin.  I can always download them. As often as PI is updated, I keep the raw files if I ever want to reedit again. I don't even worry about masters.

 

Chuckle, but I only have roughly a year under my belt. Like everything else, my mind may chhange

[SteveL42]

Unsure why you would need all those backup systems. I have used one of them for 15+ years. I've replaced hard drives in notebooks in one hour, kept monthly system image backups in my safe box, recovered from hard drive crashes in a couple of hours max with essentially no lossage and no reinstalling of the OS and reinstallation of apps (GACK!), recovered previous versions of files, accidentally deleted files, etc. etc. Given the cost of these imaging tools I don't understand why people don't use one of these. 4Tb portable USB drive (Seagate) $120, Backup app ($50/yr). Freeware? When was the last time you went out to dinner? What's your time worth? Sorry for the snarky reply.

I spent 20+ years doing Systems Engineering work in the data management space, focused on backup and recovery.  These are the tools I use:

 

• Acronis True Image - runs weekly image based backups of my 3 work machines on my home network - my main desktop, my wifes laptop, and a desktop dedicated to ham radio.
• Synology Active Backup for Business - free on my Synology NAS, where I store my onsite backups. I use this to backup other machines - a Mac, my Astro computer and laptop, etc.  
• Veeam - I use this at work to back up VMWare instances and a few workstations. The Synology solution is also used for workstation backups.  Veeam allows me to replicate VMware machines easily as well.
• Cloudberry - I use this one at home to push archives into Amazon Glacier.
• Backblaze - Forgot to mention perhaps the most import one, and the one I recommend if you're only going got use one.  I use Backblaze on my main desktop, it backs up the machine to Backblaze's offsite servers. I also use Backblaze at work for my offsite Veeam backups.
• Synctoy - a useful data copy or sync tool for moving data between my various machines and the NAS.
• Clonezilla - a free disk image/cloning tool.
• Google Drive - I have 2TB of storage on Google that I use to move files around on to family, friends, etc.  

These are just different tools that allow me to get what I want. What do I want?  Let's look at that...

• Continuous access to all my data wherever I am.
• No hassle fast recovery should a drive fail, or some other issue that stop my computer from functioning. 
• Protection of my data should I lose my house - fire, flood, alien invasion, etc. 
• Long-term archiving of the data that doesn't change - 25 years worth of digital photos, plus scanned film prior to that.
• Depth to protection - no single point of failure.

Let's just look at my Astrophotography data.  The raw data is moved from my mount computer to the NAS during an imaging run.  From there it's synced to my main desktop so I have 2 local copies, plus it's sent to Backblaze automatically in almost real time (within hours).  I have 2 copies onsite, and 1 offsite.  Once I do a Glacier Archive (I do this periodically throughout the year), I'll have 2 offsite copies.  As I process raws and create a finished image, I'll usually have many different versions along the way.  Backblaze will capture these and keep them offsite. 

 

Other than the Glacier backups, everything is automatic and doesn't require user intervention. 

 

I have a lot of experience in this area, and my approach is probably overkill for most!  A NAS is a great data management tool to have, it's the "hub" of all my data at the house, and at work.  Synology has a nice, free backup tool included, so I use it.  But if you were going to do one thing, I'd recommend Backblaze.  Unlimited backup for a single machine for $70/year.  Automatic, and gets your data offsite.  If you have a NAS, and a friend with a NAS, you could back to each others for "free" offsite backup, you'd just have to buy the storage.

 

Good backup protects against everything from drive failures, laptop loss/theft (protecting the stolen data is another topic), ransomware, viruses, user error, fire, flood, etc., etc.  My offsite work backups are stored in immutable storage pools - further protecting our data from aggressive ransomware.

 

YMMV.

[Bob Denny]

Excellent!!! That’s what I call thorough! I have all 5 points of protection via a pair of 4Tb drives that rotate with the bank safe box, Dropbox and Google Drive for instant access but all that stuff is on the 4TBs as well. Acronis runs overnight 1 full + 6 incremental. Me too for 15 years (smaller drives in the old days and loads of archive DVDs in the safe box too). It just takes a few hundred bux and a bit of time for bank visits. WORTH IT!!

Edited by Bob Denny, 27 May 2023 - 01:22 PM.

[gmiller123456]

• Acronis True Image - runs weekly image based backups of my 3 work machines on my home network - my main desktop, my wifes laptop, and a desktop dedicated to ham radio.
• Synology Active Backup for Business - free on my Synology NAS, where I store my onsite backups. I use this to backup other machines - a Mac, my Astro computer and laptop, etc.  
• Veeam - I use this at work to back up VMWare instances and a few workstations. The Synology solution is also used for workstation backups.  Veeam allows me to replicate VMware machines easily as well.
• Cloudberry - I use this one at home to push archives into Amazon Glacier.
• Backblaze - Forgot to mention perhaps the most import one, and the one I recommend if you're only going got use one.  I use Backblaze on my main desktop, it backs up the machine to Backblaze's offsite servers. I also use Backblaze at work for my offsite Veeam backups.
• Synctoy - a useful data copy or sync tool for moving data between my various machines and the NAS.
• Clonezilla - a free disk image/cloning tool.
• Google Drive - I have 2TB of storage on Google that I use to move files around on to family, friends, etc.  

I'm curious how each of those handles against loss of software, registration keys, username/password, or the company going out of business?  I remember being concerned about the fragility of the process back when I used tapes.  It backed up to proprietary media, in a proprietary format, using proprietary hardware, and proprietary software.  It seems quite silly for a backup process to require backing up software and data using some other process.  I actually still have the tapes, but no way to read them, or to even tell if they're readable.  Hence the attractiveness of a simple copy, there's nothing needed for a recovery other than an an operating system.

 

It would be nice to have something that produces a copy like Robocopy, but before copying a changed file, the old one is moved to a folder named with the current date. I doubt I'm the first to think of that, anyone know of software that does anything similar.  Since most people rarely change old files, it wouldn't use up that much more space.

[SteveL42]

I'm curious how each of those handles against loss of software, registration keys, username/password, or the company going out of business?  I remember being concerned about the fragility of the process back when I used tapes.  It backed up to proprietary media, in a proprietary format, using proprietary hardware, and proprietary software.  It seems quite silly for a backup process to require backing up software and data using some other process.  I actually still have the tapes, but no way to read them, or to even tell if they're readable.  Hence the attractiveness of a simple copy, there's nothing needed for a recovery other than an an operating system.

 

It would be nice to have something that produces a copy like Robocopy, but before copying a changed file, the old one is moved to a folder named with the current date. I doubt I'm the first to think of that, anyone know of software that does anything similar.  Since most people rarely change old files, it wouldn't use up that much more space.

They are each going to be different in their abilities - Acronis and Clonezilla are image backups, Acronis runs on a live system, Clonezilla boots from a CD/USB stick and performs a disk image backup while the system is offline.  Both bring back the entire system - OS, software, registration keys, etc.  They are "images" of the actual disk (for practical purposes). Furthermore, Clonezilla is free and opensource.

 

Backups and archives are different things with different requirements. You can mix the two, and that is where the issues you mention come into play.  I have changed my offsite backup provider several times over the years, and I could move tomorrow to another one if I need to.  Same with any backup software - just start using the new one. It's protecting my current set of data (my Backblaze account only stores 30 days of changes).  

 

Archives are long term, and then the issues of permanence and accessibility come into play.  However, all my archives (30+ years of images and documents) are also on my live system.  IF, if I archived and purged old data, like pictures from 30 years ago, then long term accessibility is a concern - like your tapes from years ago.   But as it stands, for the data I am collecting, local disk storage is cheap enough that I can basically keep everything.  In the 80's, I had 2, 20MB drives.  Today, I have 24TB on my NAS (4, 6TB drives), and I can buy a single, 22TB drive on Amazon for a little over $350.  Amazing!

 

Back on subject, I have chosen Amazon Glacier for long term offsite storage.  Amazon probably isn't going out of business anytime soon, but if they did, I'd find another place to store my archives. 

 

The tools I listed (and Synctoy is an fancy, GUI based robocopy) allow me to protect individual files by making copies (what you're doing with robocopy), but also protect machine state from when bad things happen - bad/failed upgrade, virus, corruption, drive failure, etc.  

[Spile]

I have used Veeam (free edition) for many years and can highly recommend it.