Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Does anyone have an "automatic copy" process that works (e.g. from remote PC to processing PC?)

  • Please log in to reply
25 replies to this topic

#1 ryanha

ryanha

    Viking 1

  • *****
  • topic starter
  • Posts: 990
  • Joined: 05 Aug 2020

Posted 27 October 2021 - 11:20 AM

I am trying to further automate my setup.

 

I have a remote PC in the back yard, and an image processing PC in my office.

 

Each morning I copy the files from the remote PC to my imaging PC and go through the images.

 

With a full night of imaging, sometimes this can take a while.

 

Ideally the subs would just be copied over to my imaging PC through the night.

 

Does anyone do this, and what software do you use, etc.?

 

Thanks,

--Ryan



#2 njrusty

njrusty

    Mariner 2

  • -----
  • Posts: 202
  • Joined: 17 Mar 2006

Posted 27 October 2021 - 11:31 AM

Not aware of an automated process.  I do the same as you with my obervatory on my property.  Have remote pc mapped to my main pc.  I just copy the files over (only takes a few seconds) then I load up Pixinsight and do a Blink to check everything.  From that point, the files can wait until I can do the processing.



#3 silios

silios

    Explorer 1

  • -----
  • Posts: 58
  • Joined: 23 Sep 2016
  • Loc: Greece, Athens Red Zone (Bortle 9)

Posted 27 October 2021 - 11:38 AM

Plenty of solutions either via free software (https://freefilesync.org/) or paid ones.

If both machines are part of the same network make a network drive map of the home PC (or just a folder of it) to the obsy one and save there the data in the first place.


Edited by silios, 27 October 2021 - 11:43 AM.

  • psandelle, Astrola72 and santafe retiree like this

#4 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5,775
  • Joined: 14 Jun 2009

Posted 27 October 2021 - 11:39 AM

the way to do this is thru some kind of cloud service, for instance, install google drive on the imaging PC and let it upload the subs to google's cloud. on your processing pc, run the google cloud client and have it automatically synchronize the folder that the imaging PC is uploading to.

 

additionally, there are probably more than one type of NAS that can do a "private cloud" - synology is one. you install their software on at least the imaging PC, which similar to the google cloud software will upload all new files to the NAS. then on your processing PC you can just mount the shared folder from the NAS and all your subs are just there.

 

rob


  • silios likes this

#5 ryanha

ryanha

    Viking 1

  • *****
  • topic starter
  • Posts: 990
  • Joined: 05 Aug 2020

Posted 27 October 2021 - 11:41 AM

I just copy the files over (only takes a few seconds) then I load up Pixinsight and do a Blink to check everything. 

 

How much data are you copying?  When I am doing broadband targets I will do 60s subs and these are 70M each so that is like 70G of data or something through the night.  This takes me several minutes.

 

I wonder if maybe I could get a faster network to make this be less of an issue.  

 

--Ryan



#6 iwannabswiss

iwannabswiss

    Apollo

  • *****
  • Posts: 1,473
  • Joined: 14 Feb 2014
  • Loc: Somewhere in Space

Posted 27 October 2021 - 11:43 AM

You could create a batch or PowerShell script and then use a schedule to having it done throughout the night.  I, however, having a script that runs in the morning.  Using a PowerShell script, it, calls 7-zip to compress the files into a zipped folder and then transfers to another system.



#7 ryanha

ryanha

    Viking 1

  • *****
  • topic starter
  • Posts: 990
  • Joined: 05 Aug 2020

Posted 27 October 2021 - 11:44 AM

Plenty of solutions either via free software (https://freefilesync.org/), paid ones.

If both machines are part of the same network make a network drive map of the home PC (or just a folder of it) to the obsy one and save there the data in the first place.

To clarify, I know there are ways to do it, I am looking for practical experience and the "gotchas".

 

For example, mapping the drive to my other PC I worry that if something glitches in the network, then my imaging software may barf and I don't want to lose data, etc.

 

For the suggestion of cloud, I also think that maybe overkill and I don't want to upload 50 - 100G of data every night and then download it.  

 

--Ryan



#8 Midnight Dan

Midnight Dan

    Voyager 1

  • *****
  • Posts: 14,759
  • Joined: 23 Jan 2008
  • Loc: Hilton, NY, Yellow Zone (Bortle 4.5)

Posted 27 October 2021 - 11:48 AM

Lots of free or cheap software out there that will synchronize two folders.  And of course, one folder can be out there on your network, in the backyard computer.  Most allow you to setup parameter as to when to synchronize, such as once per day, once per hour, or whenever the contents of the folder changes.

 

-Dan


  • psandelle and santafe retiree like this

#9 rgsalinger

rgsalinger

    Cosmos

  • *****
  • Moderators
  • Posts: 9,791
  • Joined: 19 Feb 2007
  • Loc: Carlsbad Ca

Posted 27 October 2021 - 11:49 AM

I use google drive desktop. My imaging systems are 90 miles away. In the morning, by the time I've had my coffee, all of last night's images are sitting on my computer to be processed. I like that solution since it also works in my backyard or anywhere that there's a good internet connection. 



#10 silios

silios

    Explorer 1

  • -----
  • Posts: 58
  • Joined: 23 Sep 2016
  • Loc: Greece, Athens Red Zone (Bortle 9)

Posted 27 October 2021 - 11:59 AM

To clarify, I know there are ways to do it, I am looking for practical experience and the "gotchas".

 

For example, mapping the drive to my other PC I worry that if something glitches in the network, then my imaging software may barf and I don't want to lose data, etc.

 

For the suggestion of cloud, I also think that maybe overkill and I don't want to upload 50 - 100G of data every night and then download it.  

 

--Ryan

I hear ya on the glitch front :)

If you are on cabled network i wouldn't worry about it, been using that method (on a NAS actually but its the same thing) for 15 years without a hiccup, on multiple sites.

If network disconnects from wifi or other reasons is a thing in the current setup, the only reliable/solid solution is good mirroring software such as rsync or some windows equivalent with a gui for easier use.

Fabricating something with powershell and a scheduler should be relatively easy, but YMMV



#11 Midnight Dan

Midnight Dan

    Voyager 1

  • *****
  • Posts: 14,759
  • Joined: 23 Jan 2008
  • Loc: Hilton, NY, Yellow Zone (Bortle 4.5)

Posted 27 October 2021 - 12:05 PM

If you run the synchronizing software on your home computer, there's far less chance of a glitch interfering with anything going on in your imaging computer.  The image transfers may stop, but the imaging computer should keep going with any local tasks.

 

-Dan


  • santafe retiree and silios like this

#12 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5,775
  • Joined: 14 Jun 2009

Posted 27 October 2021 - 12:17 PM

if you use a cloud-type synchronizing software then the server is not mounted on the imaging PC. the files are written directly to the imaging PC's disk and a loss of network won't stop the files from being written, they just won't be synchronized is all.

 

if you map a NAS onto the imaging PC, then yes, a network issue could prevent images from being saved.

 

rob



#13 ngatel

ngatel

    Mariner 2

  • -----
  • Posts: 289
  • Joined: 10 Dec 2006
  • Loc: Palm Springs, CA

Posted 27 October 2021 - 12:50 PM

If you run the synchronizing software on your home computer, there's far less chance of a glitch interfering with anything going on in your imaging computer.  The image transfers may stop, but the imaging computer should keep going with any local tasks.
 
-Dan


+1

Every morning I have to go out to the scope, put the scope dust caps on and cover the scope. I just walk out with my MacBook Pro, plug an external SSD into the mini PC and copy my files. Only takes a minute or two. Obviously won’t work if your equipment is miles away.
  • dswtan likes this

#14 dghent

dghent

    Apollo

  • *****
  • Vendors
  • Posts: 1,274
  • Joined: 10 Jun 2007
  • Loc: Maryland, US

Posted 27 October 2021 - 12:58 PM

there are 3 flavors of options for this:

 

1. Cloud file service sync. OneDrive. Google Drive. Dropbox, etc.

2. Do a network share - share out a folder on your processing PC and have your imaging PC mount it (or vice versa), with your imaging app putting files onto that share

3. Manually syncing with something like Windows' robocopy utility, rsync, or other such app

 

Whichever one works for you depends on your situation. The cloud drive strategy likely isn't great because you're not using a remote PC outside your home network and that would mean the data would just bounce out over your internet connection rather needlessly and slowly. I would explore the network share option. That's what I do.


  • dswtan likes this

#15 raguramm

raguramm

    Vostok 1

  • *****
  • Posts: 146
  • Joined: 15 Jan 2019

Posted 27 October 2021 - 02:02 PM

I used to use resilio sync. I have since moved to batch uploading/downloading files via google drive, since I required asynchronous utility and because the remote observatory wifi is too slow to handle a remote virtual desktop connection AND file uploads after business hours. 

 

Since you're in the backyard honestly just going out with a flash drive would be way faster than uploading/downloading, unless you have blisteringly fast upload speeds and unlimited data.


Edited by raguramm, 27 October 2021 - 02:02 PM.


#16 dghent

dghent

    Apollo

  • *****
  • Vendors
  • Posts: 1,274
  • Joined: 10 Jun 2007
  • Loc: Maryland, US

Posted 27 October 2021 - 02:22 PM

Yeah, that's a good point. Doing things over your local network is fine - as long as whatever connection you have out to your imaging PC is decent. I installed a mesh access point system (I'm a big fan of Ubiquiti Unifi stuff) to ensure I get enough signal at my mount and I rip along at 400Mb/s.

 

I also have NINA save my images as compressed XISF, so they're typically just over half the size they would be as a FITS image, which further cuts down on the transfer time. If your imaging app does not save image data asynchronously, saving to a network share over a slow network link can ultimately slow down your entire session because file saves will block the following exposure from starting. You can fudge this with a file sync app by having your imaging app saving to local disk, then the file sync app copies it to the network share on its own time.



#17 buckeyestargazer

buckeyestargazer

    Vendor - Buckeyestargazer.net

  • *****
  • Vendors
  • Posts: 5,552
  • Joined: 12 Jan 2008
  • Loc: IN, USA

Posted 27 October 2021 - 03:56 PM

I use freefilesync on my home network (desktop PC inside, NUC in the OBS outside).  The install includes another utility called realtimesync that does exactly what the title says.  It's a bit clunky to set up but it works well.  I don't even think about file transfers anymore.  However I should note that I am connected via Ethernet cable and not just home wifi. 



#18 rockstarbill

rockstarbill

    Cosmos

  • *****
  • Posts: 9,871
  • Joined: 16 Jul 2013
  • Loc: Snohomish, WA

Posted 27 October 2021 - 04:00 PM

I use a script at end of my imaging run that runs Robocopy to move all of my data from the mini PC to my desktop.

#19 bugbit

bugbit

    Vendor - Octopi-Astro

  • *****
  • Vendors
  • Posts: 357
  • Joined: 28 Apr 2015
  • Loc: west of seattle

Posted 27 October 2021 - 05:32 PM

I have a mini computer in the ROR about 100 ft out in the yard and use  WD_black p10 armored hd's plugged in to the usb3 socket.

 

(https://www.amazon.c...OR_STORAGE&th=1)

 

This one is 4 terra byte so it also serves to store data on and I just walk out and grab it when I feel like it. They are about $125 each for this size and they come much larger but I prefer to have multiple smaller ones so if one dies I won't lose too much.



#20 avarakin

avarakin

    Vanguard

  • *****
  • Posts: 2,490
  • Joined: 13 Jul 2009
  • Loc: Parsippany NJ, USA

Posted 29 October 2021 - 12:13 AM

I am using syncthing. It is free and available for all widely used OS. It syncs data between my two RPI4 running in backyard observatory and processing workstation. 



#21 Greg M

Greg M

    Ranger 4

  • *****
  • Posts: 381
  • Joined: 03 Sep 2007
  • Loc: Henderson, NV

Posted 29 October 2021 - 10:35 AM

This option wouldn't work for a truly remote telescope location but since your rig is in your backyard, why not just plu an external SSD drive into your telescope computer, then the next morning just walk outside, unplug it and bring it inside to your processing computer.

Maybe not the best way but I'm always looking for the easy and cheap way to do things.



#22 Pauls72

Pauls72

    Mercury-Atlas

  • *****
  • Posts: 2,993
  • Joined: 28 Oct 2007
  • Loc: LaPorte, IN USA

Posted 29 October 2021 - 11:48 AM

Create a task that uses Windows XCOPY command and then set up the task to happen daily in the Windows scheduler.

https://docs.microso...-commands/xcopy

https://www.windowsc...uler-windows-10


Edited by Pauls72, 29 October 2021 - 11:54 AM.


#23 LauraMS

LauraMS

    Ranger 4

  • -----
  • Posts: 378
  • Joined: 29 Mar 2011
  • Loc: Germany

Posted 29 October 2021 - 05:13 PM

I have tried to generate an automated system of file transfer for my solar and deep sky astrophotography setup since I have not much time for the hobby, and not so much patience with unpleasant things like waiting for copy processes to end, and also tend to make errors when shuffling many datablush.gif .

 

I have a network of Windows 10 and Stellarmate (Raspberry Pi 4, Linux) image acquisition computers, and Windows 10, Linux (Ubuntu), and Mac post-processing computers. The two acquisition computers at the telescopes are connected via Mesh network to the rest of the network. If I acquire large data quickly (typically in solar imaging with  1TB or more in a single session) I use an additional CAT6a-network cable.

 

On my central windows 10 post-processing computer I run http://Syncovery.com software. It is available for Windows, Mac, and Linux, and it connects via network (ethernet or wifi) to the acquisition computers' acquisition directory and automatically moves the files from there to a central acquisition directory on the central post-processing PC. Syncovery allows one to set up profiles for a variety of tasks with optional bandwidth limitations (e.g. to not interfere with an existing remote desktop connection ). Image download is initiated automatically every few minutes.

 

I also have SYNCOVERY profiles to automatically copy image and video data during the night from the post-processing PC to a local NAS, as well as results of the post processing directory. After post-processing I can just delete the acquisition files because everything is backed-up already.

 

SYNCOVERY is excellent wrt to the copy-profile flexibility (other software may be as well) but it takes a bit of time to optimize everything before it runs smoothly. This optimization is not so much of an issue with deep sky image data but video data in solar imaging (10GB ser-files with rapid acquisition of images at 165 2megapixel images/sec for typically 20sec intervals, seeing triggered, and sometimes saturating/overheating the 2TB PCie3.0 NVMe SSD at the telescope's NUC because of the high data rate) . Since the NUC only has 1GB ethernet I use a USB3 to 5GB ethernet converter connected to my 10GB switch to accelerate downloads if needed. This converter has a lot of overhead but gets me about 2Gb/sec throughput for download. For deep sky  images the wifi connection (typically somewhere between 54 and 200Mb/sec) is sufficient, however .

 

I hope there are some ideas in this which may be of interest to the OP.

 

Best, Laura 


Edited by LauraMS, 29 October 2021 - 05:18 PM.


#24 AstroTuna

AstroTuna

    Lift Off

  • -----
  • Posts: 2
  • Joined: 15 Apr 2021
  • Loc: Canada

Posted 30 October 2021 - 07:08 PM

I use either a PI4 using Astroberry or a mini PC running Ubuntu as my acquisition computers at my scope. Both use kstars/ekos/indi to run the acquisition sessions. Both have wifi connectivity to my home network where I have a NAS device. Kstars allows you to define and execute a script after each image capture so I use a script to spawn rsync in the background. The rsync takes the last captured image and syncs it to the NAS. The NAS then presents a windows share that I can see from my Windows machine in my house. During the capture sessions I can then run live stacking on the rsync'd images as they sync or I can just grab them later to post process. Since the sync happens after each capture I don't have to wait at the end of the night for everything to copy over.



#25 ryanha

ryanha

    Viking 1

  • *****
  • topic starter
  • Posts: 990
  • Joined: 05 Aug 2020

Posted 30 October 2021 - 09:06 PM

I think I'll try syncovery. I like it as a backup but done want to rely on a network munted drive as the wifi back there is occasionally flakey. I spent a bunch of time this week trying to get Ethernet over power line to work and of course had only frustration :) oh well.

Thanks everyone for the input!!!


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics