Thoughts on Architecture: Fault Tolerance, Reliability, etc

Let's turn our thoughts to infrastructure (or Architecture).  Your thousand dollar home monitoring considers "architecture" doesn't it?  You know, security, availability, reliability, those boring things, right?

Home Alone uses SSL to connect from the home base to the "cloud" server.  We don't want people peeking at your data (or capturing plain text passwords). I don't know if the other monitors do secure connections, but *everything* (in the internet-of-things) should.

The home base software is designed with failure in mind. There will be internet outages. They happen. There will be moments (or minutes or hours) that the you could have service interruption. The home base software is designed to log sensor data to "persistent storage" (flash or hard disk) until a good connection is made to the server. If that connection goes away, data is still retained in persistent storage until the connection comes back.

I mentioned "fault tolerance" before. While there is no real recovery from total hardware failure (if the base station hardware dies, it dies -- there is no redundancy there unless you want to supply a backup base station), I do everything I can to make sure that data isn't lost.  Above, I mentioned logging sensor data to "persistent storage" when there isn't internet connectivity. Well, the system actually logs data regardless of connectivity. As soon as sensor data is collected (and filtered) it is written to non-volatile storage. The data is only deleted when it has been "confirmed" successfully received by the server.

Bad connection? No data is lost.  Home base loses power suddenly? Any logged data is not lost (which is probably most if not all sensor data collected up to the moment of power loss).

On the internet side, the data is also persisted. It is never thrown away. It is archived. Pick a day; it can be "played back".

There are plans for server "fault tolerance" too. I plan to have mirrors (US east and US west). Data will be replicated between the servers. A server outage (Amazon East Coast I am looking at you) won't result in total failure.

Architecture matters.

Monitor your "Independent" Grandmother for under $200 (hardware cost)

I'm looking into appropriate ARM based devices for the base station, one with good wi-fi coverage and enough memory/horsepower to be the base station. The current contender is about $75.  Cheaper ones can be had for under $50, but I wanted one with a good antenna.

Pair that with a few sensors and you can get started for under $200. What would this get you?

Let's assume X10 for now (until I get my Bluetooth LE sensor project going).

Let's also assume an apartment or condo...

You'll need

• 1 Wi-fi Base Station ($75)
• 1 X10 USB RF Transceiver CM19a : (ebay *new* ~ $20 each)
• 1 Door sensor, DS-10a or DS-12a : (ebay *new* ~ $13 each)
• 4 Motion sensors MS-14a or MS-16a (ebay *new* ~ $12 each)
• Living/Dining Room
• Bed Room
• Kitchen
• Bath Room
• 1 USB temperature probe (~$15 Amazon, ebay, etc)
• 2 "I need you now" buttons KR-15a (ebay *new* ~ $8 each)
• Mounted Next to bed
• Mounted in Bathroom

So, to monitor Grandma,  you'll need to buy around $188 in hardware.

With this hardware, you can monitor stuff like:

1. Did Grandma go to bed last night?
2. Is Grandma in bed unusually late in the morning?
3. Did Grandma leave the apartment (wandering...)?
4. Did Grandma eat breakfast, lunch and dinner (activity in kitchen followed by dining room)?
5. Did Grandma leave the kitchen stove on unattended?
6. Is Grandma in the bathroom an unusually long time?
7. Does Grandma need assistance? (Pressed Big Red Button)

This isn't a scary amount of money.  What do you think?

Stove Monitoring

Right now, I have a monitor in place for stove usage. This monitor will detect if the stove is "on" but unattended. This is a non-invasive monitor (i.e. no modifications made to the stove). The monitor utilizes a motion sensor (for the whole kitchen) and a USB temperature probe (TEMPerl).

The probe is mounted on the back wall (behind the stove) up near the vent.  It is not in the direct path of heat, but it doesn't need to be.

I am taking rolling/moving averages of sampled temperature (in 30 second intervals) and any "sudden" spike in temperature (say from 73.3F average to 75F -- one or two degrees) will make the monitor believe the stove is on.   While the stove is "on", if there is no motion in the kitchen for N minutes (e.g. 30 minutes) then an alert is emailed.  If there is motion, the timer is reset. If the sampled temperature drops (a degree or two) below the now high average, then the stove is cooling down (turned off?).

There is a lot of tuning to do, but the smoothing caused by the rolling/moving average helps significant rises and declines to stand out while normal warming/cooling of the kitchen (due to the sun or thermostat) is averaged out. It may be worth looking into Bayesian classifiers to see if it can better tune out normal warming/cooling trends.

Is the Internet reliable enough?

Home Alone is based on the premise that you have 24x7 Internet connectivity. The speed of this connection doesn't matter (there isn't a lot of data sent... the system would work well on a mere kilobyte level connection), but the availability of a good connection does.

The Internet connection could be through broadband cable/fiber, ISDN, 3G or even EDGE. If you have cell phone "hot spot", this should be good enough.

An important reminder (mainly for myself): This is not a security system nor is it meant to be a "lifeline".  This system is mainly for "how are they doing" notifications. This is NOT the system for you if you are expecting it to ensure that the monitored person takes life critical medicine on time.

That being said, the system is designed to be reliable. No messages arriving at the base station should be lost and even with a flaky internet connection, the notification will be delivered to the internet server for delivery to the caretaker.

A note on reliability: No TOY technology zone

You can say what you want about X10 (toy technology, crap, unreliable, etc). It is just a placeholder right now (I'm looking to design my own Bluetooth LE -- or perhaps ANT sensors).

The real technology is in the software. There are some provisions for dealing with sensor flakiness, but the software itself is meant to be "industrial grade".  This means, among other things: fault tolerance.  The software must deal with bad sensors, bad internet connections.

What this really means: Reliability and integrity.

The system assumes that anything can fail. I will have automatic failure recovery where possible.
The weakest link will be the sensor base station. (Providing a hot swappable base station is too cost prohibitive, IMHO).

Integrity is very important to me. This system should be trusted. No spaming the caretaker with duplicate emails or losing an important notification because an internet connection flaked out.  I can't control the internet, but my software considers it unreliable. Pull a connection and messages are NOT dropped.  If someone is supposed to be notified of an event, they *will* be.

This is NOT toy technology.  I'm building this system to be rock solid.

Home Alone Status (Lua, X10, AWS, etc)

It's been over 2 years since I began thinking about this project. Now, finally, I've got something working.

Right now, I have the following installed in my house:

• 2 X10 RF door/window sensors (Kitchen Deck door and Front Door)
• 4 X10 RF Motion sensors (kitchen, living room, family room, 1 bedroom)
• 1 X10 RF Panic button
• 1 (temporary) X10 keyfob button for logging medicine taking
• (Soon to be replaced by a X10 sensor mounted in pill box)
• 1 USB temperature probe (mounted over stove)
• 1 Thinkpad T60 laptop discretely placed (closed) over the microwave
• Ubuntu 12.04
• Prototype base station software written in Lua!
• 1 USB X10 CM19 RF adapter
• 1 Amazon AWS instance (Internet host)
• RabbitMQ server for messaging
• Prototype email delivery system written in Lua

Total hardware prototype costs (not including the Thinkpad -- old laptop): < $130 (mostly X10 stuff)

I am notified on such events as

• Deck door has been left open over 5 minutes (kids forget to close door)
• No activity in the house plus Front door was recently closed = Everyone has left house
• Medicine (Pills) were taken (or not taken) 3 times a day (w/ 20 minute grace periods)
• Stove burners are on and no activity is detected in Kitchen for 15 minutes

The event periods/frequencies can be customized.

Lots of code cleanup to do and I've got new sensors coming in this weekend!

Home Alone (Home Monitoring the Independent Elderly)

Home Alone Monitor System

Goals

Home monitoring and automation has so far failed us. It is much too expensive and tend to be used exclusively by wealthy techies (for multimedia, etc) and X10 automation geeks (folks who just like messing around).

While everyone could/should afford a Home monitoring system (automation is a different beast -- I’ll discuss this later), who would benefit the most?  I think that the expanding “independent elderly” demographic is prime here. Not so much for their own purposes but to allow the children or grandchildren to keep an “eye” on their house to make sure that they are okay.  Has the stove been left on?  Is Grandma up and about?  Did she make it to her bed last night?  Has someone been ringing the door? Has the patio door been left open at night? Is the basement flooding? Is the house too warm?  Too cold? Has a window been left open?

Current solutions are too cost prohibitive and most don’t focus on the Independent Elderly.

They also use old, proprietary, non-internet saavy technology.

I propose Bluetooth LE.  It is most likely the future for low power consumption sensor nodes.

The prototype will use X10 (due to immediate availability) and wired USB solutions (temperature monitoring, etc).

For the purpose of this document/post, I refer to the independent elderly person as IE and the recipient of logs and alerts as the PC (primary caretaker).

My goals are:

2. Affordable ($10-20 per sensor, $200 base station)
3. Internet/Cloud ready (collect, log, view and distribute data)
1. Smart phone accessible
2. Notifications via email, custom smart phone app, etc
3. Web browsing of collected data
4. Zero interface once installed/callibrated
5. Non-intrusive (avoid “Big Brother” feel)
6. Open source protocol and software
1. For the tinkerer/hacker it means... FREE!
2. The value is in the backend web service subscription
1. Not open source
3. Get wider adoption and invigorate sensor market
7. Bluetooth LE based wireless sensors
1. 1+ year battery life
1. Wearable “panic/help” button

Home Automation?

No. This is not a home automation system.

Differences between Home Alone and home automation systems:

1. Home Alone is targeted toward the Independent Elderly (IE)
1. Very focused goals
2. Home Alone is meant to be inexpensive (not for the high tech wealthy)
3. Home Alone requires no buy into proprietary hardware/software
1. Compared with other free monitoring systems, Home Alone is meant to be reliable and stable

Features

• Monitor “live alone at home” elderly family member with tiny non-intrusive sensors and get notifications of how they are doing!
• Small (book sized) base station that can be placed in kitchen
• No system interaction required by elderly family member (referred to henceforth as just ‘member’) and only log/alert reading by ‘listener’.
• Non-intrusive... don’t interfere with member’s daily life
• Mostly monitoring
• Low sense of “Big Brother”
• Don’t overlap/interfere with security systems, entertainment systems, etc.
• Reliable
• Uses same technology and techniques that is used for mission critical embedded and cloud systems
• Inexpensive
• Cheap sensors (target: ~$10-25 per sensor)
• Will need to build sensors until market catches up
• Total base equipment cost (5-10 sensors + base station) < $600
• Cloud server instance
• Sensor Ideas
• Stove heat/smoke sensor -- has stove burner been left on for unusual duration?
• Check for motion in the kitchen... is someone there attending stove?
• Log for "end of day" summary notification
• Immediate notify listener
• Bedroom monitor -- Has the IE gone to bed? Slept late? Too sick to get up?
• Log for "end of day" summary notification
• Immediate notify PC
• Room monitor -- Motion detection to tell how much the IE is moving around
• Log for "end of day" summary notification
• Door open -- How many visits? Did the IE answer the door?
• Log for "end of day" summary notification to the PC
• Water level monitor -- Basement flooded? Tub/Sink over filled?
• Beep
• Log for "end of day" summary notification
• Immediate notify PC
• Room temperature monitor -- Does the temperature match the thermostat settings?
• Log for "end of day" summary notification to PC
• Pill Minder -- Monitors usage of medicine cabinet/drawer.. did IE take pills today?
• Notify if pill box has been opened and if opened around correct time(s)
• Log for "end of day" summary notification
• Help Me! Button -- Wearable button that member can press in case of emergency
• Immediate notify listener
• Can also be programmed to (BEEP) notify wearer to take a pill, etc.
• And more to come....