Jump to content


Photo

Remote and Robust! How do you do that? Generally.

  • Please log in to reply
33 replies to this topic

#1 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 03 November 2012 - 10:45 AM

"There is more than one way to skin a cat". I work in the software development business and the old adage certainly applies there. I know there was a thread here a while ago about connection software from one computer system to another but I am concerned about much more than that at this point. Ultimately my Star Cruiser will be fully remote, hundreds or perhaps thousands of miles away. It will need to run on solar power with battery storage and have systems to control the physical hardware of the roof, the scope, the cameras, be sensitive to the weather, and have redundant systems to allow for unforeseen software or hardware glitches. I will need at least one back door system that can hopefully reset things in the event of certain system failures etc. The list goes on.

This is where I need guidance. I am sure I could piece together something that would work, but would it be robust? It's the system failures that have me concerned: the single points of failure specifically that I do not have enough experience with. Working with computers all day I know just how unreliable they can be. And the software that they run - well that really has me worried. I've worked for 20 years in software QA. And believe me - I will NEVER be out of work! EVER!

1) What does a robust fully remote observatory look like? I mean the pieces specifically. How do they usually go together? Is it best to have one main control center like a PC running Windows that connects to a main hardware controller or is it best to have discreet systems that work in parallel with one another controlled by a series of hardware limit switches etc?

2) What would the statistical points of failure be? Are these systems usually redundant? What does that look like? I could see one quickly making every system redundant. (On second thought there are quite a few things that can't be - the mount, the roof, the main camera). So how far down the road to redundancy do you go?

3) Are there online resources that discuss this? There are only 1200 messages in the Open Observatory Control System Project so that may be limited.

Money is a concern but time is not. I have a friend who is an electronics wiz so building discreet systems by hand is an option. I work at a software dev shop so working with an open source project is also an option. I just need a good direction to go to get me started at this point. I would much rather learn from others experience before I go headlong into a direction that has proven unreliable in the past.

#2 Alex McConahay

Alex McConahay

    Vanguard

  • -----
  • Posts: 2378
  • Joined: 11 Aug 2008
  • Loc: Moreno Valley, CA

Posted 03 November 2012 - 11:35 AM

One thng that most fully robust remote observatories have is somebody living on the premises who can go out and re-plug something that has gotten loose, somebody who can unwind some cords that got twisted, somebody who can walk out and check what the scope is really doing. So, consider putting your remote observatory in somebody's observatory farm.

Alex

#3 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 03 November 2012 - 02:22 PM

... So, consider putting your remote observatory in somebody's observatory farm.

Alex


Good call Alex. I was thinking that would be a good idea. I am also going to play with it in my driveway for a year or two before moving it any large distance.

#4 Mirzam

Mirzam

    Skylab

  • *****
  • Posts: 4447
  • Joined: 01 Apr 2008
  • Loc: Lovettsville, VA

Posted 03 November 2012 - 02:47 PM

I think that having a webcam as part of the set-up would also be helpful. Then if the roof doesn't close or there is some other glitch, you will at least know about it.

JimC

#5 BPO

BPO

    Ranger 4

  • -----
  • Posts: 367
  • Joined: 23 Feb 2010
  • Loc: South Island, NZ

Posted 03 November 2012 - 03:21 PM

The Achille's Heel of my remote site is the comms link. I don't yet have an optical fibre to the location so I'm reliant on a cellular data connection, and it's not reliable, even with 100% signal strength. It drops randomly and the braindead client software does not include the option to automatically reconnect, which means a long drive.

When it is working, I connect to the PCs on site via LogMeIn. This is the only solution with cellular that has ever worked. I tried a few others, but only LogMeIn works in my case. And it works well.

The site is entirely off-grid, so power is via a PV system. This is very reliable, mainly because I was realistic when it came to sizing. The array is more than large enough to keep the battery bank fully charged in even prolonged periods of poor weather, and the battery bank is large enough to supply far more power than I've ever demanded from it. Capacity is key.

PCs are all Atom-based, so power consumption is ridiculously low. They run 24/7 without problems.

So, you need a reliable power supply, and a reliable data link. I don't know what US cellular comms is like, but if it's anything like that here in NZ, you'd better keep the gas tank full. Satellite would be much better, but in New Zealand the cost is prohibitive and the data plans ludicrous, so not an option. YMMV. An actual telephone line and regular broadband service would be best, and a FO cable even bester! :)

Hope some of that info may be of some use.

#6 JAT Observatory

JAT Observatory

    NOT a Wimp

  • *****
  • Posts: 9459
  • Joined: 20 Feb 2005
  • Loc: In the Primordial Soup

Posted 03 November 2012 - 06:25 PM

One of the best things you can do to be sure that your obs stays up and running is preventive maintenance. Make yourself a PM schedule and stick to it.

As far as the other stuff:
Only let the PC control what it has to..
Be sure you can close your roof/dome if the PC fails.
Be sure the roof/dome will auto close in inclimate weather.
Be sure your roof/dome can be closed if there is a power failure.

Install cameras. If you have dome install a camera on the mount so you can see where to scope is pointed. This will allow you to see if the dome's aperture is where its suppose to be.

Be sure you have a climate control system that will dry the inside of your obs after you close it up so any dew that may be on the inner walls will dry quickly and not get moldy.

Make sure the mount will stop moving on its own before it contacts something that will hurt it or the scopes its carrying.

#7 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 03 November 2012 - 10:26 PM

The Achille's Heel of my remote site is the comms link.

PCs are all Atom-based, so power consumption is ridiculously low.

So, you need a reliable power supply, and a reliable data link.

Hope some of that info may be of some use.


Very useful indeed. I would not have stumbled upon the importance of the comm link while testing in the driveway. User experiences are exactly what I am hoping to hear. I was leaning toward the Atom-based chips myself. Good to know that they can be reliable. And thanks for the suggestion about over sizing the power supply. Isn't that just the way. We end up needing more than what we originally plan for. Every time.

#8 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 03 November 2012 - 10:51 PM

One of the best things you can do to be sure that your obs stays up and running is preventive maintenance. Make yourself a PM schedule and stick to it.


Sounds reasonable. I guess I will get a feeling for how often it is required when in my driveway.

Only let the PC control what it has to..

Software can be so convenient - and yet so dangerous! I suppose there is a fair amount PC's will be controlling anyway (Mount, web cams, CCD camera, etc).

Be sure you can close your roof/dome if the PC fails.
Be sure the roof/dome will auto close in inclimate weather.
Be sure your roof/dome can be closed if there is a power failure.


On battery power alone this becomes that much more important I would think. Although a separate battery or at least a low battery monitor to make sure the roof closes could be a good idea?

Install cameras. If you have dome install a camera on the mount so you can see where to scope is pointed.

Cameras do sound like a good idea. Do you have a USB hub for many cameras? I could see the benefit for having multiple.

Be sure you have a climate control system that will dry the inside of your obs after you close it up so any dew that may be on the inner walls will dry quickly and not get moldy.

This may be a difficult one to fulfill on batteries. I was thinking about skinning the inside with aluminum. At least aluminum foil could prevent water soaking into things. Active heaters sounds like they would draw too much current and dehumidifiers also take way to much juice. Are there any other options here? I will have a couple of fans exchanging the air when required.

Make sure the mount will stop moving on its own before it contacts something that will hurt it or the scopes its carrying.

Also sounds like sound advice. I was planning on installing some limit switches with normally closed contacts that could shut the mount down if it went haywire. But I wasn't sure if that could scramble its brains to the point where homing it would become problematic. Is there a "Cancel" move command that a Meade scope would immediately respond to? I was thinking probably not.

#9 Starhawk

Starhawk

    Space Ranger

  • *****
  • Posts: 5553
  • Joined: 16 Sep 2008
  • Loc: Tucson, Arizona

Posted 03 November 2012 - 11:39 PM

Consider carefully what you're putting in the observatory. It's a lot of trouble to do this trick- make sure it's gear worth giving the opportunity to.

-Rich

#10 JustinO

JustinO

    Explorer 1

  • -----
  • Posts: 68
  • Joined: 08 Oct 2012

Posted 04 November 2012 - 08:27 AM

My experience is not with remote observatory equipment, but remote integrated/converged communications -- A/V, data, etc.

The best systems for remote management of faults had two things that were useful: first, you could see which devices were working or not working and get into any command interface, and second, you could toggle the power on each device.

One of the sweetest things is being able to toggle the power -- they have power strips that have a TCP/IP interface, and you can toggle each socket individually.

#11 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 04 November 2012 - 09:01 AM

My experience is not with remote observatory equipment, but remote integrated/converged communications -- A/V, data, etc.


Nice. I have spent my time with medical PACS systems in hospitals from the integration, service and QA roles. They were remote in the sense that I connected with them over the internet for diagnosis and upgrade but they still had people putting their hands on them regularly.

first, you could see which devices were working or not working and get into any command interface

second, you could toggle the power on each device.


Agreed with the first for sure. We have diagnostic logging that we turn on to trace the exact steps being taken. Hopefully the software pkgs I choose have that ability. Perhaps limit switches or even a simple webcam can suffice for the strickly hardware devices. I have only recently heard of the power bars. They sound like sweet devices. Although I don't seem to remember unix computer systems needing a kick in the pants like modern windows systems :(. Most of our PACS systems have been based on the unix platform and I can only remember one time in 20 years when the drives got so corrupted they would not come back up after a fatal power failure. And running the uptime command on those systems is a treat. You can measure the time in years - not days :).

Good ideas, thanks Justin.

#12 JustinO

JustinO

    Explorer 1

  • -----
  • Posts: 68
  • Joined: 08 Oct 2012

Posted 04 November 2012 - 09:33 AM

You should look at industrial controls.

Programmable Logic Controller (PLC). They are robust and deterministic.
automationdirect.com is the most amateur friendly source.

A couple important things:
---The roof should be closed when it is supposed to be closed. A design that closes by gravity or spring if power is lost would be nice.
---The roof should not be able to collide with the telescope. Geometry.
---The telescope should not be able to collide with anything. Limit switches.

#13 mclewis1

mclewis1

    Thread Killer

  • ****-
  • Posts: 10892
  • Joined: 25 Feb 2006
  • Loc: New Brunswick, Canada

Posted 04 November 2012 - 09:45 AM

I really agree with Marcus's and BPO's approaches and Justin's comments.

Some general thoughts ...

- Do everything you can to simplify the setup. For example the control PC(s) should be as basic as possible, you want it as bullet proof as you can make it. I also really like the ATOM cpus (no CPU cooling fans required). Use SSDs over HDDs, and if possible stay away from any add on expansion boards (less additional connections). A simple PC with an SSD will also reboot in almost no time. An example motherboard would be - http://www.newegg.ca...N82E16813121442 . I like Linux for the OS but that choice depends on what apps you need to have for observatory management, mount control, imaging camera controls, etc.

- Control the power with intelligent TCP/IP connected power strips. Set the PC BIOS up to boot up on power up (no power switch required). Have the internet connection battery powered so it is always available to control the essentials and security cameras. If you don't want to have all the power in the remote observatory on a battery at least ensure that if the main AC power is lost that you can still park the scope and close the roof. You could do this via a UPS which will maintain full power for a short period of time after loosing the AC source. This is usually cheaper than fully battery powering the whole setup.

- Be able to clearly view all the operations. Multiple IP attached low light security cameras (and not webcams attached back to the PC) will give you peace of mind for a) security and b) ongoing operations (is the roof fully open/closed, is the scope tracking properly - no runaways and nothing touching the mount?). These cameras are wired back to your internet router so they don't require any PC horsepower to use and will always be operational - even if you haven't booted up the PC or opened the roof.

- Plan everything out and separate the functionalities into sub groups for testing and function. For example ... 1) Internet connection 2) Power - AC, 12v, battery, UPS 3) Observatory functions (roof movement, etc.) 4) Security cameras 5) Mount control 6) Imaging camera control etc.

- If possible setup a complete running observatory (or at least a full approximation of all the functions) locally so you can more easily work through the inevitable teething issues.

#14 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 04 November 2012 - 02:05 PM

---The roof should be closed when it is supposed to be closed. A design that closes by gravity or spring if power is lost would be nice.
---The telescope should not be able to collide with anything. Limit switches.


Justin, you bring up another question I was wondering about: mainly homing the telescope in the event that the mount fails catastrophically. In my trailer design the roof will definitely collide with the scope unless it is properly "home". Meade mounts are not known to have the most reliable electronics so a complete failure is always a possibility. If that were to happen it could lead to a cascade of failures. Is it overboard to have a secondary mechanism to home the scope? Do people do this mechanically? Hopefully some person would be able to manually move it - but if the location is too remote that may not be possible. I suppose in that situation the mount would need repair anyway - but it would be nice if it were still worthy of being repaired - ie not destroyed by weather. Has anyone ever dealt with this issue before?

#15 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 04 November 2012 - 02:18 PM

- Do everything you can to simplify the setup.


Sounds good.

- Control the power with intelligent TCP/IP connected power strips.


Do all the devices (Intelligent power strips, security camera's) require separate IP's? I guess that will require a client that can access them all in a cohesive way. I imagine I will need a network switch of some sort that can do security and filtering then.

- Plan everything out and separate the functionalities into sub groups for testing and function.


I like this plan.

- If possible setup a complete running observatory (or at least a full approximation of all the functions) locally so you can more easily work through the inevitable teething issues.


I will certainly need to write up some overall design and work through the issues on paper first. Perhaps start a form of risk assessment after I have an idea of the required features of the system. Being as the trailer will be in my driveway for a year of two that will probably accomplish working through the teething issues too.

Thanks Mark

#16 Lorence

Lorence

    Viking 1

  • -----
  • Posts: 844
  • Joined: 15 Sep 2008

Posted 04 November 2012 - 06:06 PM

I have a webcam on it's own mount that can view anywhere inside the observatory. It has two 12V lights on it. They need to be on to see anything but when turned off I can see the individual LEDs on most of the equipment. It's good to see the telescope mount power is actually turned off before closing down. I'm in the habit of watching the telescope whenever it's slewing. I've had it run off on its own once and was able to stop it before it was too late. Although I have never implemented any sort of security in the observatory I could have that camera light up and pan towards the door should a break in be detected. I'm sure that would be enough to discourage most intruders.

I have a microphone in the observatory and can hear all the noises made by the equipment. I know immediately if something is wrong just by the sounds made, like when the roof or telescope are moving. I can have a conversation between the house and someone in the observatory if it were necessary. I can hear the equipment cabinet cooling fans running. I have a small bulb connected to the dew heaters that I can see when they are powered on. It's about as close to being there as you can get.

Power control is critical. I can turn on or off almost every individual piece of equipment. All to often needed when running four windows PC's. That is accomplished with two CPS power switches http://www.bomara.com/cps/cps.htm and a Digital Loggers Web Power Switch http://www.digital-l...rs.com/lpc.html
The Digital Loggers Web Power Switch is an excellent product. I can't say the same for the CPS power switches. I bought them early in the setup and am stuck with them. I wouldn't use them at all if my observatory was more than a few hundred feet away.

Most of the software I use is commercial. The free stuff works to a degree but you get what you pay for and if you use some of those free programs you get what you deserve.

Setting up in the driveway is a good idea. Do one better and set up as much as you can in the house. You probably have no idea how much "Fun" you're in for at this point but you will find out very quickly.

#17 Mirzam

Mirzam

    Skylab

  • *****
  • Posts: 4447
  • Joined: 01 Apr 2008
  • Loc: Lovettsville, VA

Posted 04 November 2012 - 06:37 PM

This is a really interesting thread!

Seems like a lot of redundancy could be provided by a second laptop computer with a solar battery charger. It would normally just read the status of other subsystems, but could issue control commands if necessary (e.g. other computer is locked up.)

Seems to me that the biggest concerns are all about control. If the roof gets stuck that's not going to be fixed remotely.

JimC

#18 JustinO

JustinO

    Explorer 1

  • -----
  • Posts: 68
  • Joined: 08 Oct 2012

Posted 05 November 2012 - 08:42 AM

@Clark
My poor old scope has many broken castings from trying to compete for space with a mechanized ROR. Clamp rings, sector arms, screw jack plates, etc. The welded, brazed, braced, and replaced parts are a constant source of sorrow for me. My telescope shall live it's life out in a dome where it can slew freely. I will not compromise.

You have a typical Meade Alt-Az mount? Maybe you could rig the clamps to be actuated by an electromagnet; if the power is lost, the mount swings free. Maybe you can weight the tube so that it's natural resting place is 'safe', without imbalancing it so much that it strains the drive. If you are super, super clever, you could have balancing weights that are electromagnetically actuated, balancing the tube when actuated, and imbalancing the tube when power is lost, so it returns to a safe position.

If you have lots of money and time, you could put a differential gear somewhere in each drive, and have a backup system for positioning the scope.

ROR are great in many ways, but not in all ways. A little dome would avoid all these bizarre ideas.

--Justin

#19 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 05 November 2012 - 09:52 AM

My poor old scope has many broken castings from trying to compete for space with a mechanized ROR. --Justin


This is one of my fears. I have never experienced a run-away scope before but the very thought of it scares me. This is a small little trailer that houses a large scope. It is just 4 feet wide and 7 feet long and houses the complete fork arms and wedge for the Meade 16" with a C14 inside. When the roof closes there is just 2 inches of room between scope and roof - and that is with the scope "home". The scope must be in the home position or the roof will not close. Period. Limit switches are a must for me. I am looking to put a Hyperstar on the C14. Any errors there and the scope will be toast so I must do this right the first time. Hence my question about preemptive commands given to the Meade mount. Ultimately I need to create a dead man switch and stop the mount cold turkey if it ever does run away. At that point it would not be wise to power it back up in that position for fear it would keep running or attempt to initialize the wrong way. I would need a mechanism to move it back home.

Just thinking - it is the "brains" of the mount at that point that are scrambled (software glitch). Is there a way to use a Meade mount in such a way that you just control the motors directly - like with the hand controls? Just tell the mount to "lay off" from thinking for a while and let something else take it home. That way I would just need to boot the mount up in that mode without initialization and control it from my own "automated" homing routine.

With regard to the PLC's - that is a possibility also. Before my work in software I spent 8 years as a wireman making electrical control enclosures for industry. PLC's are made for this type of thing for sure. At the moment I am leaning toward a Raspberry Pi and extension board however as it provides me with a an OS that I am more familiar with and I can get computer redundancy in a cost effective manner.

#20 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 05 November 2012 - 10:14 AM

Power control is critical. The Digital Loggers Web Power Switch is an excellent product. I can't say the same for the CPS power switches.


Good to know. Thanks Lorence

#21 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 05 November 2012 - 12:39 PM

With all the talk of security cameras - we use this one at work and it is pretty awesome for about $100 - Foscam FI8910W. It allows for remote control with phone apps and is very good in low light conditions if you turn on the built in lights. I am seriously considering at least one of these and mount it somewhere that provides a good field of view.

#22 EricG

EricG

    Lift Off

  • -----
  • Posts: 22
  • Joined: 05 Jun 2011
  • Loc: Gatineau, Qc

Posted 05 November 2012 - 09:46 PM

I am documenting the building of my backyard observatory, so you may be interested in some of my documents. I attached the system architecture document to this post. You can find more documents here: sites.google.com/site/egastrophotography/Equipment

Note that I have not implemented everything that you'll see in the design documents. Also, this is my first time building an observatory, so I am learning as I go even though I did spend a lot of time researching before I started.

Eric

Attached Files



#23 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 05 November 2012 - 10:23 PM

I am documenting the building of my backyard observatory, so you may be interested in some of my documents.

Also, this is my first time building an observatory, so I am learning as I go even though I did spend a lot of time researching before I started.

Eric


Good job Eric. Looks like you have indeed done your homework. I will be lucky to produce a tenth of the documents you have created. On a good day I get a few scribbles on paper :)

#24 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 06 November 2012 - 11:20 AM

oops - The edited code segment got mangled.
Next post below :)

#25 CMacD

CMacD

    Vostok 1

  • -----
  • Posts: 171
  • Joined: 23 Jan 2012
  • Loc: Ontario, Canada

Posted 06 November 2012 - 11:46 AM

I woke up this morning thinking about this stuff. And since I think best in the morning I decided to write it down. Below is a higherarchy of the various systems that I think I will be employing based upon your feedback with risk and nessesity. The higher the number the more basic and reliable the operation. I think this is the way to do this - but please correct me if I am wrong :)
          -------------------------------
          [ Level 1   Atom Based PC #1  ]
	  [           Atom Based PC #2  ]
          [           Primary Control   ]

        [ Level 2     Raspberry Pi (I/O)     ]
        [             System Health          ]
        [             Backdoor control       ]

      [ Level 3      Electrical interlocks       ]
      [              Independant systems         ]

    [ Level 4        Hard physical Stops           ]
	
  [  Level 5         Human Intervention              ]
  
  -------------------------------------------------------
  
  Level 5 - Last fallback for saftey - a warm friendly body
  
  Level 4 - Primary risk (high cost) - Damage to the corrector plate, hyperstar, and CCD
          - Hard physical stops to prevent breakage from a runaway mount
		  - This may reck the mount but save the scope (Lesser of two evils)
		  - Hopefully we will never get here
	  
  Level 3 - Individual system's power will be interconnected to other systems through
            electrical interlocks by way of limit switches with multiple contacts
		  - This system runs independant of higher order systems and takes control
		  - Primary purpose if tripped: Home the scope and close the roof
		  - Roof cannot open unless scope is parked
		  - Scope cannot power-on unless roof is open
		  - Limit switches engage before hard physical stop, rain sensor trips etc.
		  - Returns control to higher order systems after trippage
		  - This system should never fail (Very robust components - relays, switches)
		  
  Level 2 - Raspeberry Pi with I/O board
          - Primary purpose: System health, backdoor into observatory over internet
          - All limit switches auxilliary contacts connect here  
		  - Temperature sensors, weather monitor
		  - Power cycling, environment/fan control (Through normally closed contacts)
		  - Single Foscam robotic security camera
		  - Web Server for observatory health/security with email warnings
		  - This system may fail but not be fatal
  
  Level 1 - Multiple Atom based PC's
          - Primary connectivity device (tightvnc)
          - Primary telescope, CCD, webcam control
		  - Image processing etc.
 

  Other Systems running in parallel and connected
          - Battery backup/Primary power / Solar Panels (Self contained)
		  - tcp/wifi/router/hub/turbo-hub/internet (Maybe one device - but bootable)
		  - backdoor cellphone or other gps device (Location if stolen)







Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics