Lua(JIT), Perl, Erlang (RabbitMQ) all playing nicely in Home Alone

Home Alone is a melting pot of technology, all playing together.  RabbitMQ is the transport and Linux is the foundation.

The base station (hub for all the house sensors) runs a stripped down Ubuntu 12.04 Server (to fit on a 1GB eMMC flash drive) and hosts RabbitMQ (as temporary storage and transport) along with the sensor  reading code, written in LuaJIT (calling libusb directly -- no C).  The LuaJIT talks to  RabbitMQ via STOMP (it could has just as well used AMQP, but I was itching to test a pure Lua STOMP library I've developed). The server is efficient and cranks along seamlessly.  With RabbitMQ, you can say that each base station is running Erlang!

The collected sensor data is sent via a RabbitMQ shovel to a RabbitMQ in the cloud.  There the sensor data is read by a "logger", which simply dumps the data directly into daily rolling log files.

Any house monitoring app on the cloud server is able to "subscribe" to sensor data, but currently the apps periodically read the daily log (building up a picture of the house in memory), determine the state of the house (is the stove on? are monitoring motion? is anyone home?), generate appropriate events, feed them into RabbitMQ event queues  and terminate.

This batch approach is used to keep the system manageable (performance-wise) since each arriving event doesn't trigger monitoring apps directly.  The monitoring apps are essentially cron'd to run periodically every few minutes. (Each base station is required to send data or a heartbeat every 5 minutes -- this is how we determine if the base station is running properly.)  This approach should scale well (but not fantastically well).

The first house monitors were written in Lua, but I rewrote them in Perl.  Why?  The house monitors are mostly pure logic.  They model the house and I needed a richer language than Lua (quick: how do you sort a table in Lua based on a time stamp and then walk that table in sorted order?).

Perl is old hat, but still has too much built in facility for describing data structures and algorithms to be ignored.  I've could have used Python, but I enjoy coding in Perl more. "Modern Perl" looks nice...

But, it really doesn't matter as the architecture allows using any programming language you want.  The monitors are essentially just modules in a pipeline data flow.  I may be using Haskell for some monitors.  I want to use a language that expresses what a monitor needs to do, but as succinctly as possible.

Telling a Story with Sensors

More than just reporting events, your sensors should tell a story.

Collecting temperature and motion data from around your house can tell you a lot about how you live.

My elderly monitoring system strives to tell you (the care taker) an ongoing story about what your loved one is doing during the day.  From just motion and temperature, the system can tell:

• Did she have breakfast this morning? When?
• Is she getting regular meals (spending time in the kitchen)?
• Did she get up at her usual time in the morning?
• Maybe she is isn't feeling well and is still in bed.
• Is she wandering about the home?
• Did she leave the home?
• Is she taking naps in the living room chair?
• Did she fall asleep last night in the living room chair?
• Is is too cold or too hot in the house?
• Is she constantly changing the thermostat?
• Did she leave the stove on unattended?
• Did she leave the water running in the bathroom (causing a flood)?

This is much more useful than just alert events.  At the end of the day, a care taker should be able to track the well being of the elderly, not just be notified when something is wrong.

Home Alone base station is built with Unix principles in mind.. using LuaJIT!

Home Alone leverages Unix design principles.  That is, it embraces Unix (in the current case, the flavor is Linux) rather than ignore it.

Home Alone follows the principle that if your Linux system can run for months without a hiccup, then that is a good model for reliability. Not a great model, but a good one.  For instance: There are a lot of crazy things that go on in  an Ubuntu 12.04 server (which is used for both the base station and cloud instances of Home Alone). Take a look at syslog sometime.  But it works, for months.

Home Alone uses Upstart as a process monitor daemon. There are several Home Alone processes running on the base station: X10 (CM19A) USB interface, temperature control USB interface, a process to filter messages, one to log messages to disk (for transactional delivery to the cloud), and one to push messages to the cloud.

The base station processes communicate with each other through POSIX message queues. Look them up, they're in Linux and they offer a loosely coupled way for multiple processes to talk to each other.

All Home Alone processes generate rolling log messages (courtesy of Upstart).  The ext4 filesystem is my database for message storage (why not? A good journaling filesystem can be an excellent poor man's index store -- directories and files!).

The Home Alone processes don't depend on any "third party" middleware.  Each process is written in LuaJIT (100% Lua, no C) and links directly to system libraries. Any memory leaks or crashes are *mine*. (So far: 18 days of uptime, feeding the cloud at least every 30 seconds, and with no crashes or memory leaks).

This all runs amazingly light and well coordinated. I can kill a process and Upstart restarts it and it picks up where it left off.   If I unplug the CM19a USB transceiver, it waits until it has been plugged back in.

No Home Alone processes hold state in memory. They are part of a workflow. Any message or event that cannot be dropped is placed into the ext4 filesystem until they are acknowledged to be in possession of the cloud server.  Each message is a file. This file is written "atomically" from the perspective of any consumer. (The file is created hidden, the message is written, the file is then closed and then "renamed" so that the consumer can see it).

A heartbeat is generated (at the application level) from the base station to the cloud server. This allows the ability to detect whether the base station is connected and properly communicating in a timely manner.

I've still got plenty of fault tolerance testing to do, and some recovery stategies to implement. For instance:

1. If a USB sensor has been unplugged or fails (such as the X10 CM19a), an event should be sent to the cloud.
2. The base station should be "rebootable" from the cloud server. There should be some remote control.
3. If the cloud server isn't getting (at least) "heartbeats" from the base station, there should be an email alert.

The base station, when hardened will have at least a 1 month "burn in" before I consider it done.

/todd

Thoughts on Architecture: Fault Tolerance, Reliability, etc

Let's turn our thoughts to infrastructure (or Architecture).  Your thousand dollar home monitoring considers "architecture" doesn't it?  You know, security, availability, reliability, those boring things, right?

Home Alone uses SSL to connect from the home base to the "cloud" server.  We don't want people peeking at your data (or capturing plain text passwords). I don't know if the other monitors do secure connections, but *everything* (in the internet-of-things) should.

The home base software is designed with failure in mind. There will be internet outages. They happen. There will be moments (or minutes or hours) that the you could have service interruption. The home base software is designed to log sensor data to "persistent storage" (flash or hard disk) until a good connection is made to the server. If that connection goes away, data is still retained in persistent storage until the connection comes back.

I mentioned "fault tolerance" before. While there is no real recovery from total hardware failure (if the base station hardware dies, it dies -- there is no redundancy there unless you want to supply a backup base station), I do everything I can to make sure that data isn't lost.  Above, I mentioned logging sensor data to "persistent storage" when there isn't internet connectivity. Well, the system actually logs data regardless of connectivity. As soon as sensor data is collected (and filtered) it is written to non-volatile storage. The data is only deleted when it has been "confirmed" successfully received by the server.

Bad connection? No data is lost.  Home base loses power suddenly? Any logged data is not lost (which is probably most if not all sensor data collected up to the moment of power loss).

On the internet side, the data is also persisted. It is never thrown away. It is archived. Pick a day; it can be "played back".

There are plans for server "fault tolerance" too. I plan to have mirrors (US east and US west). Data will be replicated between the servers. A server outage (Amazon East Coast I am looking at you) won't result in total failure.

Architecture matters.

Monitor your "Independent" Grandmother for under $200 (hardware cost)

I'm looking into appropriate ARM based devices for the base station, one with good wi-fi coverage and enough memory/horsepower to be the base station. The current contender is about $75.  Cheaper ones can be had for under $50, but I wanted one with a good antenna.

Pair that with a few sensors and you can get started for under $200. What would this get you?

Let's assume X10 for now (until I get my Bluetooth LE sensor project going).

Let's also assume an apartment or condo...

You'll need

• 1 Wi-fi Base Station ($75)
• 1 X10 USB RF Transceiver CM19a : (ebay *new* ~ $20 each)
• 1 Door sensor, DS-10a or DS-12a : (ebay *new* ~ $13 each)
• 4 Motion sensors MS-14a or MS-16a (ebay *new* ~ $12 each)
• Living/Dining Room
• Bed Room
• Kitchen
• Bath Room
• 1 USB temperature probe (~$15 Amazon, ebay, etc)
• 2 "I need you now" buttons KR-15a (ebay *new* ~ $8 each)
• Mounted Next to bed
• Mounted in Bathroom

So, to monitor Grandma,  you'll need to buy around $188 in hardware.

With this hardware, you can monitor stuff like:

1. Did Grandma go to bed last night?
2. Is Grandma in bed unusually late in the morning?
3. Did Grandma leave the apartment (wandering...)?
4. Did Grandma eat breakfast, lunch and dinner (activity in kitchen followed by dining room)?
5. Did Grandma leave the kitchen stove on unattended?
6. Is Grandma in the bathroom an unusually long time?
7. Does Grandma need assistance? (Pressed Big Red Button)

This isn't a scary amount of money.  What do you think?

Stove Monitoring

Right now, I have a monitor in place for stove usage. This monitor will detect if the stove is "on" but unattended. This is a non-invasive monitor (i.e. no modifications made to the stove). The monitor utilizes a motion sensor (for the whole kitchen) and a USB temperature probe (TEMPerl).

The probe is mounted on the back wall (behind the stove) up near the vent.  It is not in the direct path of heat, but it doesn't need to be.

I am taking rolling/moving averages of sampled temperature (in 30 second intervals) and any "sudden" spike in temperature (say from 73.3F average to 75F -- one or two degrees) will make the monitor believe the stove is on.   While the stove is "on", if there is no motion in the kitchen for N minutes (e.g. 30 minutes) then an alert is emailed.  If there is motion, the timer is reset. If the sampled temperature drops (a degree or two) below the now high average, then the stove is cooling down (turned off?).

There is a lot of tuning to do, but the smoothing caused by the rolling/moving average helps significant rises and declines to stand out while normal warming/cooling of the kitchen (due to the sun or thermostat) is averaged out. It may be worth looking into Bayesian classifiers to see if it can better tune out normal warming/cooling trends.

Is the Internet reliable enough?

Home Alone is based on the premise that you have 24x7 Internet connectivity. The speed of this connection doesn't matter (there isn't a lot of data sent... the system would work well on a mere kilobyte level connection), but the availability of a good connection does.

The Internet connection could be through broadband cable/fiber, ISDN, 3G or even EDGE. If you have cell phone "hot spot", this should be good enough.

An important reminder (mainly for myself): This is not a security system nor is it meant to be a "lifeline".  This system is mainly for "how are they doing" notifications. This is NOT the system for you if you are expecting it to ensure that the monitored person takes life critical medicine on time.

That being said, the system is designed to be reliable. No messages arriving at the base station should be lost and even with a flaky internet connection, the notification will be delivered to the internet server for delivery to the caretaker.

A note on reliability: No TOY technology zone

You can say what you want about X10 (toy technology, crap, unreliable, etc). It is just a placeholder right now (I'm looking to design my own Bluetooth LE -- or perhaps ANT sensors).

The real technology is in the software. There are some provisions for dealing with sensor flakiness, but the software itself is meant to be "industrial grade".  This means, among other things: fault tolerance.  The software must deal with bad sensors, bad internet connections.

What this really means: Reliability and integrity.

The system assumes that anything can fail. I will have automatic failure recovery where possible.
The weakest link will be the sensor base station. (Providing a hot swappable base station is too cost prohibitive, IMHO).

Integrity is very important to me. This system should be trusted. No spaming the caretaker with duplicate emails or losing an important notification because an internet connection flaked out.  I can't control the internet, but my software considers it unreliable. Pull a connection and messages are NOT dropped.  If someone is supposed to be notified of an event, they *will* be.

This is NOT toy technology.  I'm building this system to be rock solid.

Home Alone Status (Lua, X10, AWS, etc)

It's been over 2 years since I began thinking about this project. Now, finally, I've got something working.

Right now, I have the following installed in my house:

• 2 X10 RF door/window sensors (Kitchen Deck door and Front Door)
• 4 X10 RF Motion sensors (kitchen, living room, family room, 1 bedroom)
• 1 X10 RF Panic button
• 1 (temporary) X10 keyfob button for logging medicine taking
• (Soon to be replaced by a X10 sensor mounted in pill box)
• 1 USB temperature probe (mounted over stove)
• 1 Thinkpad T60 laptop discretely placed (closed) over the microwave
• Ubuntu 12.04
• Prototype base station software written in Lua!
• 1 USB X10 CM19 RF adapter
• 1 Amazon AWS instance (Internet host)
• RabbitMQ server for messaging
• Prototype email delivery system written in Lua

Total hardware prototype costs (not including the Thinkpad -- old laptop): < $130 (mostly X10 stuff)

I am notified on such events as

• Deck door has been left open over 5 minutes (kids forget to close door)
• No activity in the house plus Front door was recently closed = Everyone has left house
• Medicine (Pills) were taken (or not taken) 3 times a day (w/ 20 minute grace periods)
• Stove burners are on and no activity is detected in Kitchen for 15 minutes

The event periods/frequencies can be customized.

Lots of code cleanup to do and I've got new sensors coming in this weekend!

Home Alone (Home Monitoring the Independent Elderly)

Home Alone Monitor System

Goals

Home monitoring and automation has so far failed us. It is much too expensive and tend to be used exclusively by wealthy techies (for multimedia, etc) and X10 automation geeks (folks who just like messing around).

While everyone could/should afford a Home monitoring system (automation is a different beast -- I’ll discuss this later), who would benefit the most?  I think that the expanding “independent elderly” demographic is prime here. Not so much for their own purposes but to allow the children or grandchildren to keep an “eye” on their house to make sure that they are okay.  Has the stove been left on?  Is Grandma up and about?  Did she make it to her bed last night?  Has someone been ringing the door? Has the patio door been left open at night? Is the basement flooding? Is the house too warm?  Too cold? Has a window been left open?

Current solutions are too cost prohibitive and most don’t focus on the Independent Elderly.

They also use old, proprietary, non-internet saavy technology.

I propose Bluetooth LE.  It is most likely the future for low power consumption sensor nodes.

The prototype will use X10 (due to immediate availability) and wired USB solutions (temperature monitoring, etc).

For the purpose of this document/post, I refer to the independent elderly person as IE and the recipient of logs and alerts as the PC (primary caretaker).

My goals are:

2. Affordable ($10-20 per sensor, $200 base station)
3. Internet/Cloud ready (collect, log, view and distribute data)
1. Smart phone accessible
2. Notifications via email, custom smart phone app, etc
3. Web browsing of collected data
4. Zero interface once installed/callibrated
5. Non-intrusive (avoid “Big Brother” feel)
6. Open source protocol and software
1. For the tinkerer/hacker it means... FREE!
2. The value is in the backend web service subscription
1. Not open source
3. Get wider adoption and invigorate sensor market
7. Bluetooth LE based wireless sensors
1. 1+ year battery life
1. Wearable “panic/help” button

Home Automation?

No. This is not a home automation system.

Differences between Home Alone and home automation systems:

1. Home Alone is targeted toward the Independent Elderly (IE)
1. Very focused goals
2. Home Alone is meant to be inexpensive (not for the high tech wealthy)
3. Home Alone requires no buy into proprietary hardware/software
1. Compared with other free monitoring systems, Home Alone is meant to be reliable and stable

Features

• Monitor “live alone at home” elderly family member with tiny non-intrusive sensors and get notifications of how they are doing!
• Small (book sized) base station that can be placed in kitchen
• No system interaction required by elderly family member (referred to henceforth as just ‘member’) and only log/alert reading by ‘listener’.
• Non-intrusive... don’t interfere with member’s daily life
• Mostly monitoring
• Low sense of “Big Brother”
• Don’t overlap/interfere with security systems, entertainment systems, etc.
• Reliable
• Uses same technology and techniques that is used for mission critical embedded and cloud systems
• Inexpensive
• Cheap sensors (target: ~$10-25 per sensor)
• Will need to build sensors until market catches up
• Total base equipment cost (5-10 sensors + base station) < $600
• Cloud server instance
• Sensor Ideas
• Stove heat/smoke sensor -- has stove burner been left on for unusual duration?
• Check for motion in the kitchen... is someone there attending stove?
• Log for "end of day" summary notification
• Immediate notify listener
• Bedroom monitor -- Has the IE gone to bed? Slept late? Too sick to get up?
• Log for "end of day" summary notification
• Immediate notify PC
• Room monitor -- Motion detection to tell how much the IE is moving around
• Log for "end of day" summary notification
• Door open -- How many visits? Did the IE answer the door?
• Log for "end of day" summary notification to the PC
• Water level monitor -- Basement flooded? Tub/Sink over filled?
• Beep
• Log for "end of day" summary notification
• Immediate notify PC
• Room temperature monitor -- Does the temperature match the thermostat settings?
• Log for "end of day" summary notification to PC
• Pill Minder -- Monitors usage of medicine cabinet/drawer.. did IE take pills today?
• Notify if pill box has been opened and if opened around correct time(s)
• Log for "end of day" summary notification
• Help Me! Button -- Wearable button that member can press in case of emergency
• Immediate notify listener
• Can also be programmed to (BEEP) notify wearer to take a pill, etc.
• And more to come....