Driver progress and useful tools

In June this year I started with moving all the drivers of my Domotica system to Node.JS on the Raspberry Pi (Rpi). And now, 2 months later, the number of drivers running on the RPi is 15. Wooaahh, that’s fast, faster than I ever imagined! It can’t get any faster than this.. Among those drivers there are really simple ones like the one for the Roomba Robot vacuum cleaner, but also some really critical ones like the for PLCBUS driver which controls the largest number of actors in our house. And all those drivers are working great… sometimes the code is so small that you start to think ‘is this all it takes?‘Apparently it is…

There are still some drivers that need to be moved to the Raspberry though: Zigbee, Visonic Powermax, ELV Max!, Nemef Radaris, Somfy just to name a few. But with what I’ve developed so far I think I’ve got enough experience to finish those quite easily too. In the meantime I’ve also been searching for, testing and implementing some solutions that I need.

Monitoring

Monit on Raspberry PiHow do you keep an eye on a bunch of drivers running on a Raspberry Pi without a screen attached to it? Well, not by logging in, listing the running processes with ‘ps’ and see if they’re all still there. There’s a much better way for that and it’s called Monit. This really cool monitoring tool allows you to monitor a lot and alert you if something is not right. Processes, file attributes, CPU, disk space – everything you’d want to monitor. You can manually start / stop / restart processes (i.e. drivers in this case) and when a drivers fails for some reason, you can have it restarted for you automatically, without having to do anything yourself! Talking about automation… I did have to switch to upstart before I was able to start a Node process as a daemon. It took me quite some time before I figured out it wasn’t me that was doing something wrong, but in the end I found the answer to my problem in this article. After installing upstart and creating the scripts, everything went fine.

Events

Or scenarios, rules… the things that enables you to really do something with all the sensor information; a very important (indispensable!) part of any Home Automation system. Lights going on & off automatically, Roller shutters being used to keep the temperature inside at an acceptable level, all lights going on when a smoke detector is triggered, …

My current system has an event sub-system that can do all that; but it wasn’t flexible enough. if, and, or, >, =, < were the things I could do with it, and scheduling in a crontab-like way. But when it came to executing actions when something had happened in the past, or adding a lot of conditions, defining those rules became a hassle, it looked ugly. So I started searching for a rules engine that could run with Node.JS and had it’s own DSL. Well, I found it and I’m already using it! Nools subscribes to all the information (‘#’) that’s on my MQTT broker, so I can use all the information in the rules I want to have.

Here’s a very simple rule to demonstrate how an action can be triggered by a sensor:

rule garage_door_open {
  when {
    e1: Event.device == 'Garage door' && Event.value == 'open';
  }
  then {
    publish("command/speak", "Attention, garage door open", false);
    forget(m1);
  }
}

In this example, a message is being sent to my TTS  (Text To Speech) driver to speak some text when the garage door opens. I could just as easy turn on the lights in the garage…or, (handy during the winter) speak a warning message that the garage door is open for too long. Using the Nools DSL is a very convenient way of defining those rules – in a text file, in a human readable way. And Nools looks like a very powerful and flexible tool to define my rules – I love it!

Database

My current system stores the historic data in a MS SQL Server that’s running in a Hyper-V VM environment. Some information is stored forever (gas, water, power usages) and others for a shorter period like a month, so I can watch trends). The time has come to see if I want and/or really need to keep on doing that in the future – using MS SQL Server, that is. As long as my system is in a ‘dual state’ (i.e. running partly as Delphi application on Windows and partly on Raspberry Pi with Node.JS) I will need that database. And I’m not going to spend time on my Delphi application in terms of new functionality anymore – just those things that are needed to gradually destroy it to null. So until I’ve totally freed myself from the Delphi application running on Windows, I’ll also need to use my MS SQL Server.

This also means that I’ll need a way to use my MS SQL Server from Node.JS, only for retrieving information. Cause this SQL database doesn’t only contain the historic data, but also all the information about where all the devices are in the house (floor, room), what those devices are and what they can do (switch a light or a waterpump on/off, whether it’s a boiler or a media player) and so on. All this metadata is in that database too and I’ll need it in Node.JS, soon..

And again, I found a great Node module for that, named tedious. It uses the TDS protocol and allows you to connect to a MS SQL Server, execute queries and so forth – just what I need. My plan is to use tedious to store historic data in my MS SQL Server for as long as it’s needed.

The bigger picture however, the ultimate goal, is that my Domotica system runs on nothing but Raspberry Pi(s), with maybe a dedicated NAS-like device for historic data storage, web server stuff and to get rid of my 34W Hyper-V server. But there’s still a lot that needs to be done before I can do that, so ..

Onwards!

Btraced, Raspberry Pi, Node.js, MQTT to build your own GPS tracking system

Btraced-2

The Btraced app is a great tool, especially because it allows you to upload the GPS data to your own server. Once you’ve got this aspect covered, the things you can do with the GPS data is unlimited – you can do whatever you want with it!
Personally, I’m not that interested in the ‘trip’ functionality of Btraced; I’d rather use the app to allow me to do positioning: constantly letting my Domotica system know where I am. And not just my location, soon other family members will be added as well.

Although uploading the GPS data to your own server is very convenient, accomplishing this might sound scary to some people, cause “I’ve never done that”, “don’t know how”, “don’t have a server”, “too difficult”. This post will show you that it’s none of those..

During the last few days I’ve been playing with a Raspberry Pi, Node.js and a webpage on my web server. The situation before I started with this, was that I had an ASP.Net web form on my IIS web server; this page was used for uploading the Btrace GPS data to my server and storing it in my SQL Server. After a few days I started wondering why I stored all this GPS information in a database – all I did with it was SELECT-ing the newest record and display my last-known location. What a waste of disk space..

So I added MQTT publishing to the ASP.Net page and let the SQL stuff the way it was. A few days later I completely removed the SQL stuff, and realized that using an ASP.Net page on my IIS was a bit overkill for what I was doing. There must be an easier way! And there is: I’ll show you how to create your own Btraced-upload-server without a big web server and minimal cost.

Hardware

All you need is a Raspberry Pi (model B, the one with Ethernet interface) and a LAN cable to connect it to your router or LAN. I prefer to run my Raspberry’s without keyboard, video & mouse so you’ll need a way to connect to the RPi over the LAN – my favorite tool for that is Putty. Connect your RPi to the network, power it up and find out with which IP address the RPi is using.

Network

Your gateway to Internet (modem, router, firewall) should be able to do port forwarding, cause we’ll have to forward the Btraced data to the RPi. Choose a port (e.g. 8000, 8080, 8081 but preferably not 80 cause maybe you want to run a website on your RPi one day…) and forward that port to the RPi. In the Btraced App, you’ll have to change the setting for the upload server, like this: http://ww.xx.yy.zz:pppp (assuming ww.xx.yy.zz is your external IP address and pppp is the port number).

Software

You’ll need to install Node.js and an additional Node module: xml2js. Optional: mqtt. And some extra code, which will be provided below. Just follow one of those excellent guides on ‘How to install Node.js’ that can be found on the web. Also install npm, the Node Package Manager, just to make life easier. Not comfortable with using an editor on Linux? Install an Node.js and a FTP server on your PC, edit the files on your PC, test & debug them there, install an FTP client on the RPi and ‘ftp get’ the files to your RPi once the scripts are finished.

Some code

The Btraced app uploads the data as xml over http. “Oops, so I need a http server and work with XML?” Yes you do, but it only takes 75 lines of code, so don’t be scared! Keep on reading.. here’s the code that can ‘receive’ and parse the uploaded Btraced XML data:

http = require('http');
mqtt = require('mqtt');
var parseString = require('xml2js').parseString;
var ydate = '';
var position = {};

mqttClient = mqtt.createClient(process.env.DDMC_BROKER_PORT, process.env.DDMC_BROKER_HOST, {clientId: 'Btraced', keepalive: 30000});

function processPoint(p){
  if(p.date > ydate){
    ydate = p.date;
    position.date = parseFloat(p.date);
    position.latitude = parseFloat(p.lat);
    position.longitude = parseFloat(p.lon);

    position.speed = parseFloat(p.speed);
    if(position.speed > 0){
      position.speed = Math.round(position.speed * 36)/10;
    } else {
      position.speed = 0;
    }

    if(p.course.substring(0,1) == "-"){
      position.angle = 0;
    } else {
      position.angle = parseFloat(p.course);
    }
    position.batt = Math.round(parseFloat(p.bat)*100);
  }
  return parseInt(p.id);
}

http.createServer(function (req, res) {
  if(req.method == 'POST') {
    var body='';
    var travelid = '';
    var ids=[];

    req.on('data', function (data) {
      body +=data;
    });

    req.on('end',function(){
      // explicitArray false for NOT getting all nodes as arrays
      parseString(body, {explicitArray:false}, function (err, result) {
        bwiredtravel = result.bwiredtravel;
        username = bwiredtravel.username;
        travelid = parseInt(bwiredtravel.travel.id);
        ydate = '';
        position = {};
        position.username = username;
        var points = bwiredtravel.travel.point;
        if(typeof points.length === 'undefined')
        {
          ids.push(processPoint(points));
        } else {
          for(var i=0;i<points.length;i++){
            ids.push(processPoint(points[i]));
          };
        }
        console.log(JSON.stringify(position));
        mqttClient.publish('raw/'+username+'/location/gps', JSON.stringify(position));
      });
      body = '';

      response = {};
      response.id = 0;
      response.tripid = travelid;
      response.points = ids;
      response.valid = true;

      console.log(JSON.stringify(response));
      res.end(JSON.stringify(response));
    });
  }
}).listen(8000);

That’s it? Yep.. and all that needs to be done to get this code running is entering ‘node btraced’ @ the command prompt and it’s running… If you didn’t install the mqtt module mentioned earlier then just delete all the lines containing the string ‘mqtt’ and the script will keep on working, but now the information will only be displayed on the console and not published anymore – you’ll have to find yourself another way to get the information there where you want it.

With MQTT it’s very easy to create a web page that displays the information in text and also draws a map, all real-time.  I added some reverse geocoding to complement the information with an address, et voila:

Btraced-1

I haven’t tried it, but it cannot be hard to run the web page shown above on a RPi as well.

So there you have it: your own, private, fully customizable GPS tracking system for the price of a Raspberry Pi!

Raising the bar with the IRTrans driver

The last few days I’ve been busy developing a driver for my IRTrans LAN IR transmitter/receiver. I chose the IRTrans because it’s not being used that much (anymore), but still enough to detect bugs within a few hours. And another nice thing is that the IRTrans LAN Gateway accepts multiple connections, so testing a new driver is easy – I won’t have to completely shut down my previous (and still actively used) driver developed in Delphi. While working on this driver, I took the time to also improve some other things (which I should have done earlier).

Settings

I didn’t like the way some drivers have IP-addresses of hardware interfaces hard-coded, so I fixed that. Mosquitto (the MQTT broker I’m using) looks like the best place to store settings like the IP-address & port the driver has to connect to, poll intervals and other parameters that influence how a driver behaves. But first, I needed a way to find out where my broker is – for that I created 2 system-wide environment variables by adding the following to the file /etc/environment on the Raspberry Pi:

DDMC_BROKER_HOST=192.168.10.13
DDMC_BROKER_PORT=1883

If you’re using a Windows PC for development, you’ll have to add those environment variables on that machine as well, or add them to the nodevars.bat that starts the Node.js environment on Windows:

@echo off
rem Ensure this Node.js and npm are first in the PATH
set PATH=%APPDATA%npm;%~dp0;%PATH%
rem settings specific to DDMC
set DDMC_BROKER_HOST=192.168.10.13
set DDMC_BROKER_PORT=1883

A simple change in how the Node.js MQTT client connects and this issue was fixed:


var host = process.env.DDMC_BROKER_HOST;
var port = process.env.DDMC_BROKER_PORT;
mqttClient = mqtt.createClient(port, host);

When the connection has been established, the rest of the required settings can be retrieved from the MQTT broker by subscribing to a topic and wait for a message to arrive:


// get required settings
tools.mqttClient.on('connect', function(packet) {
  tools.settings.require('host');
  tools.settings.require('port');
  tools.settings.require('subdelay');
});

Of course, the script has to wait for the required settings to arrive; this is done by keeping track of all the required settings and ‘pause’ execution until all required settings are ‘set’ by the incoming messages. I picked up this idea by browsing the HomA source code which was brought to my attention in a comment recently. Now I can remove all the hard-coded stuff from the driver code and clean them up a bit.

OK, back to the IRTrans driver. During all the years I’ve been working with the IRTrans, I added some ‘features’ which I didn’t want to lose:

Channels.

I’m used to working with TV station names instead of channel numbers for my cable STB. An example: I have a page on my Philips Pronto TSU9600 with icons for all the TV station we’re able to receive. Let’s say that each icon (a button, actually) has on onClick() method which looks like this: SetDevice(‘upc’, ‘EUROSPORT’);

Why? Because I can’t remember that EuroSport is on channel 401, but I can remember the ‘short-codes’ like ‘EUROSPORT’, ‘NED1’, ‘BBC1’, ‘WDR’,’DISCOVERY’…

But the IRTrans doesn’t know what ‘EUROSPORT’ means, so I made a ‘mapping’ table on my SQL Server to translate ‘EUROSPORT’ to ‘401’. But because the Node driver has no way to connect to my SQL Server, I added an extra setting to the IRTrans driver, so I don’t have to change all the UI’s when something changes and I can keep on using these ‘short-codes’ for ever:

{"EUROSPORT":201,
"BBC1":50,
"BBC2":51,
"ANIMAL":21,
"NATGEO":18,
....
"CNN":401}

The JSON formatted data shown above is parsed and used as an associative array.

Delays

Let’s take the example above a little further. Eurosport, channel 401. All the buttons on the remote of the Cable set-top box (STB) were learned with the Philips Pronto (PEP1) and added to a IRTrans .rem file. That means that I have an IR code for the button ‘1’, another one for ‘2’, ‘3’ and so forth. But I don’t have an IR code for ‘401’. So, to mimic selecting channel 401 with the remote, I have to send the IRTrans 3 IR codes: the code for ‘4’,’0′ and ‘1’. And I have to add a delay between those 3 IR codes because if I don’t, the STB doesn’t always recognize what’s being sent! Hmm… delay, event-driven, timeouts… challenge.

Adding a delay is not such a problem, because the Node.js setTimeout() command allows you to add a minimum(!) delay before something’s being executed. But no matter what I tried, instead of sending the ‘4’,’0′,’1′ sequence to the IRTrans, my driver code sent ‘1’,’1′,’1’… Okay, those who are familiar with Javascript will probably know what is causing this, but for me this ‘scope’ problem was new!

After I read this excellent page, I knew what to do… well, that’s the down-side of immediately start coding instead of learning the language first.

Toggle codes.

A more detailed explanation of the problem with our current STB is described here as well as how I handled it, although the issue can also be fixed with newer IRTrans firmware. The new Node.js based IRTrans driver has to automatically handle this toggle problem as well of course, as if toggle codes don’t even exist. That’s because I don’t want to send up#1, up#2, up#1 from my UI’s to go from channel X to X+3, but just up, up, up – it’s ridiculous and it wouldn’t work anyway, with more than 1 UI – capiche?

So the IRTrans driver should keep track of the last-used IR code for ‘up’.  The ‘old’ Delphi driver already retrieved a lot of information from the IRTrans gateway – the identifications of all the remotes (upc, onkyo, hdmax, …) and all the commands (up, down, vol+, poweron,..) of those remotes, so I did the smae thing in my Node driver.

So the physical STB remote has 2 different IR codes for the ‘up’ button, which I named ‘up#1’ and ‘up#2’. I use the ‘#’ in the command name to detect that we’re dealing with a so-called toggle code for the ‘up’ command. So all I had to do was collect all the commands starting with ‘up#’ into a single object instance, add an index counter and all goes automatically…

During startup of the new IRTrans driver I query the IRTrans driver with ‘Agetremotes‘ and ‘Agetcommands‘ and store the information in an array. Here’s the code of the class that ‘hides’ the toggle code issue:

var InfraRedCommand = function(remotecommand) {
  this._command = remotecommand;
  this._index = 0;
  this._toggleCodes = [];
}
InfraRedCommand.prototype.addCode = function(code){
  this._toggleCodes.push(code);
}
InfraRedCommand.prototype.getCode = function(){
  if(this._toggleCodes.length > 0){
    this._index++;
    if(this._index >= this._toggleCodes.length) this._index = 0;
    return this._toggleCodes[this._index];
  } else {
  	parts = this._command.split(",");
    return parts[parts.length-1];
  }
}

module.exports = InfraRedCommand;

Now the toggle codes are hidden, in just a few lines of code 🙂

From time to time I test this new IRTrans driver by sending an ‘off’ command to the TV or STB, or switching to ‘CNN’. Judging by the amount of uproar coming from the living-room, this new driver is working very well!

 

Using Geo-fencing for triggering events

After I got my GPS location published to the MQTT broker relatively easy yesterday, I really wanted to see the geo-fencing working. How nice would it be if I (eh, my Domotica system) could determine whether I’m entering or leaving a certain area (a circle, to begin with).

For that, I needed a way to calculate distances between 2 GPS coordinates, which I found here. Not only does this site explain the math, but the Javascript is provided too, in such a way that it’s ready-to-use for what I want to do with it – brilliant, this makes life even easier than it already is 😉

After some copy&paste the code for calculating the distance between 2 GPS points was ready to be used; now all I had to do was write a small script that would subscribe to ‘my’ GPS location provider (i.e. the Btraced GPS App), define our house in GPS coordinates, calculate the distance between our house and  “me” and watch what happens when I drive from my work location back home. I also decided to draw a virtual circle around our house with a radius of 4 km and checked every incoming GPS point to see if I was in- or outside of that virtual circle. No big deal, around 25 lines of code should do the job:

var LatLon = require('./latlong.js');
var tools = require('./tools');
tools.systemId = 'geofence';

var thuis  = new LatLon(52.0071670, 6.0393751);

(function connect() {
    tools.clog('connecting...');
    tools.mqttClient.connect({clientId: tools.systemId});
})();

tools.mqttClient.on('connect', function(packet) {
  tools.clog('connected to broker, subscribing to all locations');
  tools.mqttClient.subscribe('raw/+/location');
  tools.mqttClient.on('message',function(packet){
    var txt;
    txt = packet.payload;
    lat = eval('('+packet.payload+')').latitude;
    lon = eval('('+packet.payload+')').longitude;
    txt +=', dst='+thuis.distanceTo(new LatLon(lat, lon), 4).toString()+' km';
    txt +=', in='+(thuis.distanceTo(new LatLon(lat, lon), 4)<=4?1:0).toString();
    tools.clog(txt);
  }
  );
});

And late in the afternoon, when I drove home this appeared on the screen:

16:35:04 {“latitude”:51.9812396,”longitude”:5.9806257}, dst=4.949 km, in=0
16:35:13 {“latitude”:51.9819519,”longitude”:5.9810719}, dst=4.878 km, in=0
16:35:18 {“latitude”:51.9828016,”longitude”:5.9816207}, dst=4.793 km, in=0
16:35:32 {“latitude”:51.9843738,”longitude”:5.9856472}, dst=4.467 km, in=0
16:35:41 {“latitude”:51.9861408,”longitude”:5.9893560}, dst=4.146 km, in=0
16:35:50 {“latitude”:51.9882380,”longitude”:5.9928792}, dst=3.816 km, in=1
16:35:59 {“latitude”:51.9905545,”longitude”:5.9960376}, dst=3.495 km, in=1
16:36:05 {“latitude”:51.9927831,”longitude”:5.9987393}, dst=3.209 km, in=1
16:36:14 {“latitude”:51.9949893,”longitude”:6.0015182}, dst=2.924 km, in=1

16:42:32 {“latitude”:52.0070857,”longitude”:6.0398408}, dst=0.03313 km, in=1

 

Now I have almost all the ingredients for triggering events based on geo-fencing…
Let’s start with something to improve safety – for instance, I can let the touchscreen speak a text like “Diner needs be ready in a few moments now!” or something similar – it will save me a phone call while driving home, so I can concentrate more on driving than something else! 😉

You see? Home Automation is good for a lot of things!

Btraced GPS App and geo-fencing

Btraced screenSince April this year I’ve been bèta-testing Btraced GPS Tracking App for Android; and since a week this Android version is finished too. I love it! In short, with Btraced (now available in the App Store as well as Google play) you can upload GPS coordinates, speed, course, altitude to your own web server (or to the free Btraced web server). Uploading the data can be done real-time, or afterwards (e.g. when you have a Wifi connection). Well, the feature list is too long, so go to Btraced.com for more information.

Last week I made an ASP.Net web page that enabled me to upload (post) the data to my own web server and store the data in a SQL Server table and I updated the touchscreen application that’s running in the livingroom so that all the family members can see where I am.

OK, this is nice, but I want to do more with this of course, so I added another feature to my Btraced upload web page: an MQTT client! Now I have my GPS location available on my MQTT broker:

GPS Location on MQTT brokerThat’s more like it 😉 Now I can really start doing something with the data instead of ‘just’ storing it!

Reverse geocoding was the first thing I tried. There’s a lot of code on the Internet for Node.js and I found a module (download the more recent code from GitHub for it to work) that does exactly what I needed – it uses the Google Maps API to find an (approximate) address for the GPS coordinates. Within  a quarter of an hour I got it working. Cool, now I can show an address as well, which is even more convenient then just displaying a marker on a map – although I could also combine the two of course.

Next: geo-fencing. There’s a (Dutch) TV commercial where someone turns on the central heating with his smart phone after landing at the airport – that won’t be necessary anymore with geo-fencing! When your system knows where you are and can determine whether you’re entering or leaving a geo-fence, it can do things like switching the house to Eco mode, booting the PC when coming home, starting the robot vacuum cleaner when nobody is at home – you name it and it’s possible!

All I need is a way to define my geo-fences with 1 or more GPS coordinates – either as a rectangle or as a circle with a center and radius. The rest should be easy!

Now this is CNC!

CNC millRecently I visited Robin, someone I know from the Domoticaforum. He and I share some common interests like Domotica; another one is CNC.

Robin has been sending me pictures of his home-brew CNC mill now and then for a year or so, and now it was time to visit him and have a look how it performed. Wow…

The frame of his mill is made of 10 mm steel, the portal is made of 10 mm aluminum. The X-axis has 2 and the Y-Axis has 1 x 4.8 Nm stepper-motors, the Z-Axis is driven by a 2 Nm stepper.

The 4 drivers that control the stepper-motors are connected to a Parallel breakout boards as well as the stops. The linear guide-ways are made of  hard chrome and are equipped with linear ball bearings.

The software that Robin uses for his creations, consists of 2 parts: Vectric VCarve and Mach3.

hobbyroom00

Time for a demonstration!

Robin asked me what I’d like to take home with me; “your house number?”  Hmmm.. I know something better.. So we searched for the ‘Bio Hazard‘ symbol and copy/pasted it to VCarve, added some text and within 10 minutes the design for the board on the door of my hobby-room was ready. We let VCarve generate the G-code, saved it to a file on Robins NAS and went to the CNC PC with Mach3 installed on it. We homed the milling cutter and started the milling:

Cool stuff, I want one too! What a pity that I can only spend my money once; choices, choices….

I took the board home and started painting. And after a few days, the end result is this:

2013-08-11 14.31.5400Thanks Robin!