PLCBUS as serious test case for all

Some time ago I wrote about a spare PLCBUS Interface that had been unused for quite a while; I wanted to get rid of the USB cable and see if I could do something useful with it.

Today my first thought was: why send the raw PLCBUS frames to the interface over Ethernet¬†(what I planned to do)? Why not add some more ‘intelligence‘ , closer to the hardware; why not completely remove the PLCBUS driver from my Domotica system and put it somewhere else??

I mean, MQTT has been working great right from the start and it’s here to stay, so lets see if I can ‘remove’ the PLCBUS protocol from my monolithic Domotica system and make a MQTT capable PLCBUS driver, just like I did for the Opentherm Gateway – hide the ‘ugly’ details!

There are quite a lot of PLCBUS modules in our house, so a problem with the PLCBUS driver should surface almost instantly – I can use this spare PLCBUS interface as a perfect test case and roll back to the old situation any time in case of serious problems.

USB/RS232/TTL

The first thing I did was examining the internals of the PLCBUS interface once again. USB is out of the question, RS232 will need more hardware (5V for the MAX232), so TTL looked like the right choice. I saw that the MAX232 on the PLCBUS interface PCB was connected to the rest of the hardware by 2 4N35 Optocouplers – 1 for incoming and 1 for outgoing signals. Why not solder 2 wires to these Optocouplers and directly connect this to a UART? Good question…

So I tried. I soldered a wire to each of the two Optocouplers and used a spare Arduino Duemilanove to see if the output was what I expected.

PLCBUS connected to Arduino

And some lines of code (most of them for getting each frame on a new line) for the Arduino to show what’s being received in the Serial Monitor window:

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(57600);
  Serial.println("PLCBUS I/O test");
  // set the data rate for the SoftwareSerial port
  mySerial.begin(9600);
}

long time=0;
int nl=0;
void loop() // run over and over
{
  if (mySerial.available()){
    nl=0;
    int c = mySerial.read();
    Serial.print(c,HEX);
    Serial.print(" ");
    time=millis();
  }
  if (millis() > time+100){
    if(nl==0){
      nl=1;
      Serial.print("rn");
    }
  }
}

Yep… all is well. I used a PLCBUS Mini controller to send some frames with a House code which is not in use and saw the following output being displayed in the Arduino IDE Serial Monitor window:

2 6 A5 21 22 64 0 C 3
2 6 A5 21 23 0 0 C 3
2 6 A5 11 23 0 0 C 3
2 6 A5 11 22 64 0 C 3
2 6 A5 11 23 0 0 C 3
2 6 A5 11 22 64 0 C 3
2 6 A5 A8 22 64 0 C 3
2 6 A5 A8 22 64 0 C 3

For those not that familiar with PLCBUS frames: those are all 100% valid frames ūüėČ So that is working just fine. Next test will be sending PLCBUS frames to the interface to control some appliance modules.

Driver

Next issue: where does the driver go? For this I have (had) several ideas and options. The most important thing for me is that I want all my drivers to work independent of each other – if one of them breaks down for whatever reason, the rest of ’em should not be influenced by that. Now that I have an unused interface which I can talk to @ TTL level, the options are endless – JeeNode, .NetMF boards, Arduino (Mega), Simplecortex, TTL to Ethernet adapter, anything goes, but I decided to go for the Simplecortex. Lots of processing power, Ethernet and I/O. Maybe even suitable for hosting multiple drivers (the less important ones, that is).

This PLCBUS interface will be a good excercise to translate the relatively small amount of Delphi code (about 200 lines) to C and make an application for the Simplecortex with lwIP TCP/IP Stack, DHCP, DNS and MQTT and see how this works out. DHCP to make life easier, DNS to make the application more adaptive network-wise and MQTT as transport layer on top of TCP/IP.

And if this excercise turns out the way I hope, this could be the real start of what I started thinking about in June last year; cutting my Domotica system into relatively¬†independent¬†parts – especially the essential ones like PLCBUS, Zigbee, AV equipment, IRTrans, OpenTherm and all other drivers which are heavily used and make life more comfortable…

Lets do it!

Showing off

Not that what I’m talking about in this post is something special or unique, but it’s just that the things I completed last week are a big step in the right direction – big enough to post about and show off¬†a bit in a small demonstration ūüėČ

It’s all a logical follow-up on the smart meter we have in our house since September 3rd of this year. I decided to transform this smart meter into a MQTT client so that my Domotica system wouldn’t have to parse the P1 datagrams every 10 seconds and filter what had changed since the previous datagram. Instead I used a SimpleCortex to parse the datagrams and made it publish new values to a MQTT broker (Mosquitto) and I added a MQTT client to my Domotica system to subscribe to the various smart meter topics. The last step in the smart meter project was creating a web page to display the information from the smart meter in real-time.

This all went very well; well enough to upgrade my Touchscreen application that’s running in the living room on a Asus Eee Top, from UDP broadcast to MQTT as well (to receive updated values from my Domotica system).

And I knew I had all the tools available to start working on another type of user interface – web based this time. So I bought a new Andoid tablet (we needed one for upstairs anyway), saw JQuery Mobile, liked it and just started developing and testing some basic web pages, all with the wish of being able to use this new UI on our smart phones as well.

And last week I completed the technical side of things – I (we) can now also control all our roller shutters and lights from our tablets, PCs, smart phones –¬†anything that has a browser.

And to show how it all works together, I made a small video showing (off) a light bulb being controlled in/from 3 different ways – a PC browser, a smart phone and the touchscreen application. Wow!

New User Interfaces

About 2 weeks ago I decided that something had to be done about the (number of) User Interfaces in our house. Especially for controlling the roller shutters from upstairs there was a need for a 2nd user interface. The touchscreen in the livingroom is being used a lot and we wanted the same convenience upstairs as well.

I’ve been tinkering with various touch screens on Arduino, FEZ Panda and the like, but I didn’t really like the size (~3″) and counting all the costs for a setup like that made me wonder if there could be a better choice. Why not buy a cheap 10″ Android tablet and make a web based interface? HTML5, websockets, ….

Setting Thermostat from smartphone

So I started looking around on the web for the right tools and I found jQuery Mobile.

The demos looked good, so I started creating some basic pages while keeping an eye open for a cheap tablet.

The first (test-)page I made was for changing the temperature setpoint of our central heating, just to see how hard it would be for me as a non-web-developer to create something that looked OK.

A slider, radiobuttons and an OK button were made very easy; it felt as if I’d done this before, but I haven’t – it’s very simple!

And if you look at the code (it’s great to create pages this way) I decided that jQuery Mobile will be it for the user interfaces I’m going to add.

And just look at the code; ain’t it sweet? ūüėČ

<div data-role="page" id="tstat">
	<div data-role="header">
	  <h2>Thermostaat</h2>
	</div><!-- /header -->
	<div data-role="content">
	  <br />
    <label for="slider-1">Hoe warm wil je het hebben jochie:</label>
    <input type="range" name="slider-1"
    id="tstatslider" value="20" min="15" max="25" step="0.5"
    data-highlight="true" />
    <br />
    <fieldset data-role="controlgroup" data-mini="false">
    	<input type="radio" name="radio" id="radio-1"
    	value="choice-1" checked="checked" />
    	<label for="radio-1">Tijdelijk (tot volgend
    	  programma)</label>
    	<input type="radio" name="radio" id="radio-2"
    	value="choice-2"  />
    	<label for="radio-2">Continu</label>
    </fieldset>
    <br />
    <input type="submit" value="OK" onClick='do it;'
    data-icon="check" />
	</div><!-- /content -->

	<div data-role="footer" data-position="fixed">
  <div data-role="navbar">
		<ul>
			<li><a href="#home">Home</a></li>
			<li><a href="#">Weer</a></li>
			<li><a href="#">Cams</a></li>
			<li><a href="#luiken">Rolluiken</a></li>
		</ul>
	</div><!-- /navbar -->
	</div><!-- /footer -->
</div><!-- /page -->

Yep, this is definitely what I want and what I need!
So I created some more pages (lights, the roller shutters):

And right now I’m testing this user interface, cause I want to know how well it performs compared to our VB.Net touchscreen application in the living room, before I spend more time on it. First results are very good – 10 times faster than X10 ūüėČ

There are some issues I have to work on; things like the buttons not giving an audible feedback, making it secure so that this can also be used on our smartphones, etcetera.

But for now I’m happy with how this working out so far. A tablet has been bought too yesterday, so it looks like we’ll have our user interface upstairs real soon!

Testing drivers as Windows services

Now that my OpenTherm Gateway is operational, the interface to my Remeha Calenta has become almost obsolete; I could deactivate the whole Calenta driver now, cause the information from the OpenTherm Gateway is enough for me. But just shutting down the Calenta interface is not what I’m going to do – this is a great opportunity to test a driver while it’s running decoupled from the system – as a (Windows-)¬†service.

I already did some small tests some time ago where I compiled a driver to a Console App; within minutes I had one running in my test-environment, but I didn’t go any further than that. And now that things are getting serious, ¬†I decided to jump right to building a driver as a (Windows) service, which sounds like the best thing to do – I don’t see any benefit in having lots of drivers running as a Console App – a Service sounds much better.

So this evening I started reading on how to create a Service in Delphi, cause I’ve never done that before. Well with the TService class you’ve got a service installed in just a few steps.

The first thing I needed was some indicator to make the code service-aware. With all drivers built inside the ‘kernel’, it was very easy to get in- and outgoing data from driver to the event system, device objects, etcetera. That’s all gone when the driver is running as a service. So I had to alter some code so that the driver knows whether it’s running as a service or not; cause if so, it has to use MQTT to publish outgoing data instead of pushing it into a PC queue. Likewise, it has to subscribe to some relevant topics to be able to receive information from the outside world (e.g. the other Domotica system components).

Service-awareness was not that hard; it seems that checking the value of the Application.Mainform property is enough – if it’s nil, it’s safe to assume ‘service mode’ (probably Console mode as well, but didn’t check that).

Creating a MQTT equivalent for those internal queues won’t be that hard either; a single line of code has to be replaced so that the data is being published, instead of queued:


// 20121113
//DistributeToDevice(PhysicalAddress, copy(Buffer,1, PacketSize));

if gRunningAsService
then MQTTPublish(PhysicalAddress,copy(Buffer,1,PacketSize))
else DistributeToDevice(PhysicalAddress,copy(Buffer,1,PacketSize));

I also had to define whether a driver should be started from inside my Domotica system or that it was supposed to be running stand-alone as a service; for that I created an extra field in the driver table so that I wouldn’t end up with 2 running versions of the driver – the embedded one and the service instance.

Right now I have the Remeha Calenta driver running as a service – quick and dirty, just to see what happens and what has to be added and changed to make it reliable and most important: easy to work with.

Calenta Service

All that’s left to do is MQTT-enabling the Calenta driver and I can test this old-code/new-style driver without the fear of information loss, yet test as ‘live’ as much as possible ūüôā

Realtime data with Mosquitto, Apache and mod_websocket_mosquitto

Things are getting better and better ūüėČ

This is a follow-up on my previous post about real-time data delivery to web pages (or anything else, in fact). The comments and emails I got on that post were surprising; MQTT is really hot and so is the push-mechanism that’s being used. Willem was one of them. I know Willem for some years now and we share some common interests; the first being Domotica and we’re also both tinkering with the Simplecortex, our recently installed smart meters and we both use Mosquitto as MQTT broker.

A week ago Willem developed an Apache module, which made life even more easier than it already was – with his Apache module, the only thing you need to make a real-time webpage is the mosquitto javascript that can be downloaded from¬†here – nothing more! Less components, less chance that something will collapse… ūüėČ

So, this time the components to build a real-time webpage are:

  • Mosquitto;
  • Apache;
  • mod_websocket_mosquitto from Willem;
  • Mosquitto javascript/websockets client;
  • a simple webpage.

It all started when Willem sent me an email with his¬†mod_websocket_mosquitto source code attached and some instructions on how to use it. I have Apache running on my proxy server (for hosting this weblog) but since I don’t have any development tools installed on that machine (VM, to be precise) I booted a Fedora 17 VM on my PC (with VirtualBox, cause Windows 7 Virtual PC didn’t like to boot Fedora 17), installed scons and compiled the source code. The resulting module was copied to the proxy server including a configuration file for defining some configuration details:

<IfModule mod_websocket.c>
 Loadmodule mod_websocket_mosquitto modules/mod_websocket_mosquitto.so
 <Location /mosquitto>
 MosBroker proxyserver.hekkers.lan
 MosPort 1883
 SetHandler websocket-handler
 WebSocketHandler modules/mod_websocket_mosquitto.so mosquitto_init
 </Location>
</IfModule>

But somehow it didn’t work… I went back to my Fedora17 VM, installed the Apache module on it, but the result stayed the same – no go. I couldn’t find any trace of connection attempts being made to the Mosquitto broker, I got no error messages, nothing; looked like a dead end…

But then I got the tip to disable SELinux. And that was it, phew! Why didn’t I think of that, I’ve had troubles with SELinux in the past (some 7-8 years ago) but never again since, so the thought to disable it just didn’t occur to me… grr! Hurry back to the proxy server: Bingo!

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3624-giga.

But the thought of a broker being accessible from the Internet gave me the creeps; with not much effort, anybody who knows a bit about how all this works, could easily start publishing to my broker. But that’s not going to happen, I need some way to make sure my ‘production‘ broker is not affected in any way.

So for that I went back to my Fedora17 VM, downloaded the mosquitto broker source code, built it and installed it on my proxy server. This mosquitto instance on the proxy server would be the one that the web page (“the Internet”) would connect to. And to feed this proxy-broker with information from my production broker, I defined a 1-way bridge from production to proxy, which can be done from the mosquitto.conf file, like this:

connection domotica
address domotica.hekkers.lan
topic /# in

This way the proxy server knows about all the topics on the domotica (production) server, but topics published on the proxy server won’t arrive at the domotica server. After a restart of mosquitto I saw this:

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
Connecting bridge domotica
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3728-giga.

Again, wow…I added a web page to my Domotica website, so that y’all can watch my Smart meter data in real-time as well. Enjoy!

PS.: Use Firefox or Chrome, those are tested and work.

Realtime data with MQTT, Node.js, MQTTClient.js and Socket.IO

Seeing things happen the moment they actually¬†do happen, that’s what I like the most.

And that’s why I’ve always been searching for a better solution than Ajax. I use Ajax here and there on my own website (which is actually nothing more than a showcase of what you can do with Domotica/Home Automation), but I never really liked it. It always felt like some sort of patch-up while waiting for something better to emerge.

In the past¬†I’ve tried several solutions besides Ajax; Lightstreamer, Comet-like solutions and other similar products, but they all were either too big, required too much work for a single webpage or the price was too high. But now, the time has come to get rid of Ajax and polling web pages.

Once a year or so, I browse the web to look for an Ajax alternative. And this time I read this¬†comment on a post about MQTT over Websockets. Node.js rang a bell, Websockets too and I decided to give it a try; with Flanagans Javascript bible next to me that shouldn’t be a problem.

So the ingredients for this exercise are:

  • MQTT,¬†a machine-to-machine (M2M)/”Internet of Things” connectivity protocol, especially¬†useful there where small code footprint is important;
  • Node.js,¬†a platform for easily building fast, scalable network applications. Lightweight, event-driven and non-blocking I/O; ideal for real-time applications;
  • MQTTClient.js, an MQTT client for Node.js;
  • Socket.IO, which enables you to build¬†realtime apps for virtually every browser and mobile device, using different transport mechanisms.

Hehe, it’s beginning to sound like a commercial but it ain’t – now you know what I’m talking about during the rest of this post.

I’ve already got a Smart meter and an Opentherm Gateway publishing their data to an MQTT broker, so that part was already working; I started with installing Node.js. There are several installers available and I picked the one that matches my PC: the x64 Windows installer, started it. Nnnf, done. Well, the rest was just as easy… now the code.

First thing I did was making a Javascript that would run on Node.js that would combine the Socket.IO functionality and an MQTT client to push MQTT topics to a browser; I named it the ‘pusher‘ script:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
});

That should do it.. I guess. We’ll see soon enough…

Next, a small web page:

<h1>Real Time</h1>
<script type="text/javascript" src="socket.io.min.js"></script>
<script type="text/javascript" src="jquery-1.4.2.js"></script>
<script type="text/javascript">
  var socket = io.connect('http://localhost:5000');
    socket.on('connect', function () {
      socket.on('mqtt', function (msg) {
        var elmarr=msg.topic.split("/");
        var elm=elmarr[3];
        console.log(msg.topic+' '+msg.payload);
        $('#'.concat(elm)).html(msg.payload);
     });
     socket.emit('subscribe',{topic:'/sensor/OTGW/returntemp'});
    });
</script>
<table class="tablegv" style="width: 500px;">
<tbody>
<tr class="tablegvHeader">
<td colspan="2"><center>Status</center></td>
</tr>
<tr>
<td>Return temp</td>
<td id="returntemp"></td>
</tr>
</tbody>
</table>

Here’s the result, the boiler return temperature shows up in the screen:

Pushing it!

Wow, it’s working… both the page and the console log below the page shows the incoming temperatures!

Needless to say this can easily expanded with room temperature, flow temperature, flame status, etcetera etcetera…. all updated instantly – as realtime as you can get! ūüôā

But I wanted more…

Suppose you have a web page like this displaying the Smart meter information and another one displaying the Opentherm information. Both pages will result in subscriptions to all kinds of topics, either related to the Smart meter or the Opentherm. And I saw that both pages received all the information! That’s not good – I need to a way to make sure the smart meter page only receives updates it subscribed to.

After reading some more about Socket.IO I found out there’s a feature called ‘rooms’. Sockets (clients) can join a specific room and there’s a way to emit information to just thos sockets in a specific room. That sounds nice, so I tried it. Changed the pusher script to this:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    socket.join(data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.in(topic).emit('mqtt',{'topic': String(topic),
    'payload':String(payload) });
});

As you can see, line 11 is added and line 17 (old) has been changed to line 18. That’s all, now none of the pages gets information it didn’t ask for; great, this can¬†definitely¬†be a very good Ajax replacement!

Migrating to stand-alone drivers this simple?

This evening I did a quick test to get an idea of what problems I would encounter when I would separate the drivers from the rest of the system (event handling and stuff).

So I created a new Delphi project of type Console Application, entered the code to create an instance of the PLCBUS¬†(hey, I’m mentioned there :-)) driver class (line 15) and just watched what would happen. Of course I had to create a dirty-looking loop because all the important stuff happens in a thread inside the driver, but I was surprised about how quickly I could turn this driver into a stand-alone executable – in a matter of minutes!

program DrvPLCBUS;

{$APPTYPE CONSOLE}

uses
  SysUtils,
  Utils in 'CoreUtils.pas',
  PLCBUS in 'DriversPLCBUS.pas',
  PLC1141 in 'DriversPLC1141.pas';

var
  Driver: TPLC1141;

begin
  Driver:=TPLC1141.Create('PLCBUS', True, TPLC1141CommandHandler);
  Driver.Run;
  repeat
    delay(100);
  until false;
  Driver.Shutdown;
end.

Of course, this doesn’t mean this driver is already functional in the way it is now: it will connect to the PLCBUS hardware by opening a TCP/IP connecting, but that’s it.

That’s because I have no way of sending commands to this stand-alone driver, cause in the ’embedded’ way (i.e. running inside my monolithic Domotica system) it would receive messages from internal message handlers – in this state, the internal message handler has only got 1 listener, namely the driver itself and nothing else…

But that’s where MQTT will come to the rescue!

Another great thing is that I can extend the driver code for stand-alone operation and still use the same driver in the embedded way. This is possible because Delphi’s System unit provides an¬†IsConsole variable with which you can decide what has to be done in both cases like this:

if IsConsole
then Writeln(Output, S)
else Logger.Add(S);

This way, in embedded mode everything will be logged to a file while the console version will write everything to standard output. So when I start the stand-alone driver I see this:

PLCBUS driver in console mode

This is great; just what I had in mind for a smooth transition: I can keep on extending my system as it is (build new versions, add new hardware drivers) and work on stand-alone mode for the drivers at the same time – no Big Bang!

MQTT: about dumb sensors, topics and clean code

Or, in other words: where to define the topic a device should publish/be published on. In a scalable way..

I always try to keep the number of hard-coded values to a minimum; no hard-coded device IDs, peer IP addresses and things like that.¬†And now that I’m working on MQTT clients, I was wondering how I could keep it that way.
Hard-coded MQTT topics? No way. Well, the last part in the hierarchy maybe (temperature, humidity, motion, …), but that’s it – nothing more. Driver code should be as clean as possible too; the drivers’ own IP-address is OK; peer IP address rather not; parameters that influence behavior: rather not. So the credo is: the least as possible/doable/practical.¬†That’s the only way to keep maintenance (and irritation) low and scalabilty high; lets see what can be done about that with respect to MQTT.

The diversity in all the devices I have is rather large.¬†There are devices that are really ‘dumb’, like 433 MHz temperature sensors that transmit the temperature every x seconds. ¬†And there are devices which are a bit smarter, like JeeNodes with a RF12B or Xbee transceiver.¬†And devices that can be controlled over powerline, RF, Zigbee, Infrared or TCP/IP and smart(er) devices like a EDS HA7Net, ¬†Opentherm Gateway, ELV MAX! Radiator Valves, IrTrans interface, GSM Modem etcetera.¬†And the latest addition to this list is the Smart meter, which is the first (well, the driver) that’s going to use MQTT to publish the smartmeter data.

Communication between my domotica system and all the end devices can be direct by a built-in driver (.e.g. a built-in driver that has a direct TCP/IP connection to the GSM modem), indirect (via ZigBee Coordinator, IrTrans LAN module) or device-initiated (connections are being made to an open TCP/IP port of my Domotica system, either with or without an extra hardware layer in between).

The format of the data can be a stream of bytes that need to be decoded, html output, xml, readable text, proprietary frames; you name and it’s in use here.¬†Conclusion: its one big pool of hardware and nothing works the same ūüôā

Now the challenge is to develop an easy to use, reliable and scalable solution so that I can build MQTT enabled hardware that acts as a driver which in some way ‘configures’ itself and doesn’t have to rely on hard coded topic names, IP addresses etcetera.

Lets give it a try. All sensors/devices, no matter how ‘dumb’, can be identified in some way; with a hardware address, MAC-/IP-address, Node id on a RS485 bus, whatever. The driver (the software under my control that’s the closest to the hardware) can only use this ID to publish the data; cause for the driver the device ID is the only unique thing.

For exampe, an¬†RF receiver sees an incoming RF frame and can determine that this frame its coming from a temperature sensor with ID xx.¬†So what can this driver do in terms of MQTT publishing? Publish the measured temperature of 23.1 ¬įC for sensor ID xx (topic in red, payload in blue) as:

/RF/xx/temperature 23.1

Yuck. I don’t like that at all, I don’t think I want that – those IDs don’t tell me anything!
I’d like something like ourplace/house/firstfloor/kitchen/temperature or /ourplace/garden_shed/temperature. This has some advantages… but how do I do that? Cause the thought of ‘coded’ location information gives me headaches…

With a ‘database’ of course.. From the first day I started developing my own system I’ve got a table which contains a list of all the devices with their ID, description, where it is located (Building, Floor, Room, Location) and lots of other attributes.
That is where the relation between the hardware IDentifiers and the rest of the device information is stored – and it must be, and always stay, in one place. It doesn’t necessarily need to be SQL of course as it is in my case – the most important thing is that there can only be 1 ‘source’ – cause if not, a system with more than 230 devices with nearly 1200 device values will become a big mess, very quickly.

So how do I get the information from my SQL database into that MQTT capable driver that needs it – well, why not publish the contents of that database?¬†Lets assume that for sensor ID xx the topic used for publishing the temperature should be /ourplace/garden_shed/temperature. ¬†The fun is that ‘ourplace’ and ‘garden shed’ are already in the database as attributes to a device with ID xx – so nothing more to do than just publish that information and I’m done – right? If I would publish

/config/RF/deviceinfo/xx/location/ /ourplace/garden_shed/

then the RF driver can subscribe to the topic /config/RF/deviceinfo/# and it will receive all the information it needs for the relevant devices; and it can be more than just the location information.

In this case, ‘/config‘ is just an arbitrary topic name that separates the configuration stuff from the rest of the hierarchy.¬†The next topic in the hierarchy is used to differentiate the devices per driver they are supported by (the RF driver only needs to know about the sensors it needs to ‘service’ – there’s no need for the RF driver to receive information about Zigbee devices for instance).

I could do that… that means that¬†during the startup phase the driver subscribes itself to the right topics in the ‘config’ hierarchy and the information comes in… and this can be used for other things just as well – like¬†driver specific settings as a poll interval, for instance.

Doing things this way means that, when a temperature sensor is being relocated from the kitchen to the garage, I don’t have to do anything more than I do now:¬†changing the location information in the database and the rest follows! All I have to develop is a tool that publishes the required database information in the /config hierarchy and I’m done.

But there’s another way.

Let the driver publish the ‘raw’ information (Device ID oriented) in a /raw hierarchy (/raw/RF/xx/temperature¬†23.1) and develop a tool that subscribes to the /raw hierarchy and that re-publishes everything it can (e.g. that can be found in the database)¬†to the location-oriented hierarchy, e.g.¬†/ourplace/garden_shed/

This has some advantages too… hmmm; and there are even more alternatives… but this example of the temperature sensor is maybe too simple, it doesn’t cover every aspect I have to deal with – or does it?

Reset…

The # subscriber

In a recent comment on my new ideas to make my Domotica system more flexible, the term broker was mentioned. ZeroMQ doesn’t have one, Mosquitto,¬†for instance, does. But do I really need a broker? I don’t think so; I think I can very well live without one.¬†In fact, what’s the point of a broker when there will be a part of the automation system that will need to do a¬†# subscription.

This subscribe-to-all entity is in fact the heart of the system, the brains, the glue to it all; the part of the system that does the real automation part.

Automation is actually based on events/scenarios; if all conditions for a specific event are met, the event fires and a series of actions are performed/executed. Some examples:

  1. There’s an incoming phone call which results in lowering the AV receiver volume level in the livingroom. Once the phone call has ended, the AV receiver goes back to its previous volume level. (these are actually 2 events)
  2. The garage door is being opened and a light bulb is switched on. When the door closes again, the light bulb is switched off again. (ditto)

These are 2 very simple examples that everyone can understand.
But the phone line can’t publish anything by itself, nor can the¬†AV receiver subscribe to anything all by itself. For that, a driver, interface or plugin (pick your favorite) would be needed – a piece of software that can monitor one or more phone lines, control one or more¬†AV receivers or switch one or more¬†light bulbs on/off.

First thing I want to accomplish is removing all the interfaces to the hardware (the¬†drivers) from the core application, cause those are the most error-prone; reconstructing incomplete/fragmented frames, faulty data (RF!), bad connections, misbehaving hardware and stuff like that is what’s causing the most problems. So for the sake of stability it would be best to start there.¬†Each hardware interface (driver) will eventually become a stand-alone process, taking care of communicating with the sensors and actors belonging to that specific driver. The PLCBUS will handle all of my PLCBUS modules, the IRTRANS driver enables me to communicate with all the IR controlled devices in range of the IRTRANS transmitter, and so forth. Hardware specific code is nicely tucked away inside the driver, just like it should; for example, the mandatory 400 ms. delay between 2 consecutive (RF) Roller shutter commands is taken care of inside the driver; nothing else knows this delay is needed.¬†This construction makes that it is the driver that does the publishing, not the device.¬†

Inside the System process every physical piece of hardware (temperature sensor, motion sensor, light actor, TV, AV receiver) is represented by an instance of a specific object class. To make these devices really come alive, they will need to receive the information from their physical counterparts; so all drivers will need to publish to the System process. Or better: the System process needs to subscribe to all the drivers; right?

OK, back to events (or scenarios) again.

Now it’s time for an Event that’s a bit more complicated; ‘house power save mode’. I’ve made a first attempt with this some weeks ago. Goal is to determine whether energy consuming devices can be turned off (or in standby) since there’s¬†no one¬†around anyway: PC’s, laptop, lights, garage door openers, AV equipment, you name it. I’m not going to bore you with the issues that arise once you start with this, but believe me when I say that it can be really hard to determine whether everybody’s out the door or there’s still somebody laying on the couch watching TV – you need every possible input you can get.

Likewise, the devices that will be turned off to save power are handled by various drivers – PLCBUS or X10 for lighting, TCP/IP for networked devices, Infrared or any other hardware specific driver.¬†Should the TV device subscribe to dozens of publishers so it can decide for itself when it’s time to go into power save mode? Just like the laptop(s), lights etc.? BTW, the TV also needs to locally store status information about all the sensors it uses to be able to make the ‘I can turn myself off now’-decision just like the the laptop has to do (for itself); and the light bulbs, and so on..

The effect of the last example is in fact the same as the 2 I gave at the start of this post (something happens automatically), but the numbers are bigger and suddenly direct pub/sub between devices doesn’t sound that appealing anymore – in my opinion, event handling is a typical task that performs best when a single event handler instance takes care of all the events, no matter the complexity. Devices don’t interact directly – a ‘higher power’ is needed for that- at least for how my domotica system works right now.

And this makes me believe I don’t need a broker – I will need some kind of self-made broker anyway, with events acting as some sort of advanced routing rules…

Receiving sensor data with ZeroMQ

Mobotix cameraTo see what has to be done to get ZeroMQ working in my Domotica system, I decided to do a small test. For this test I chose a sensor that’s inside one of my camera’s, the Mobotix camera you see in the picture to the left. This Mobotix has a very good illumination sensor which I use for detecting daylight. The Mobotix can be configured to send (among a lot of other things) the sensor value at a regular interval to a list of IP addresses. My camera is configured to send it’s IP address, a PIR value and the illumination value each minute,¬†separated¬†by a semicolon. This results in the following data, with a 1 minute interval between each line:

192.168.120.11;0;25;
192.168.120.11;0;30;
192.168.120.11;0;35;
192.168.120.11;0;46;
192.168.120.11;0;52;
192.168.120.11;0;63;
192.168.120.11;0;68;

Let’s see if I can get this into my system by sending it to a ZeroMQ server that’s embedded in my system; so for that I need 2 things:

  • A 0MQ client, written in whatever language I want (Delphi XE2¬†this time);
  • A 0MQ server in my Domotica system, written in Delphi 2005.

I started with the client, following the examples found in the ZeroMQ Guide:

program MobotixNotifyPlugin;
{$APPTYPE CONSOLE}
{$R *.res}

uses
  System.SysUtils,
  IdTCPServer,
  IdIOHandler,
  IdContext,
  zmq in 'zmq.pas',
  zhelper in 'zhelper.pas';

type
  // dummy class for holding the event
  TProcObject = Class(TObject)
    procedure ServerExecute(AContext: TIdContext);
  end;

var
  PO       : TProcObject;
  Context  : Pointer;
  Requestor: Pointer;
  Msg      : string;
  Server   : TIDTCPServer;

function Version:string;
var
  major, minor, patch:Integer;
begin
  zmq_version(major, minor, patch);
  Result := Format('%d.%d.%d', [major, minor, patch]);
end;

procedure TProcObject.ServerExecute(AContext: TIdContext);
var
  IO   : TIdIOHandler;
  data : string;
  reply: String;
begin
  IO := AContext.Connection.IOHandler;
  //IO.CheckForDataOnSource(10);
  if not IO.InputBufferIsEmpty then
  begin
    data := IO.InputBuffer.ExtractToString();
    Writeln(TimeToStr(Now)+' '+data);
    s_sendmore(Requestor, 'MOBOTIX');
    s_send(Requestor, data);

    reply := s_recv(Requestor);
    Writeln(TimeToStr(Now)+' '+reply);
  end;
end;

begin
  PO := TProcObject.Create;
  Writeln('ZeroMQ library version: '+Version);
  Context := zmq_init(1);
  Requestor := zmq_socket(Context, ZMQ_REQ);
  zmq_connect(Requestor, 'tcp://192.168.10.40:5563');

  Server := TidTCPServer.Create;
  Server.DefaultPort := 808;
  Server.MaxConnections := 1;
  Server.OnExecute := PO.ServerExecute;
  Server.Active := True;

  try
    while True do
    begin
      sleep(1);
    end;
  except
    on E: Exception do
    begin
      Writeln(E.ClassName, ': ', E.Message);
      zmq_close(Requestor);
      zmq_term(Context);
      FreeAndNil(PO);
    end;
  end;

end.

Receiving the sensor data from the Mobotix is handled by the Indy TIDTCPServer class and all it does is forwarding the incoming data to my system the 0MQ way, preceded by the Device ID (‘MOBOTIX’) the Mobotix camera has in my Domotica system.¬†That should do it, in a very very simplified way; on to the server part!

procedure TZMQServerThread.Execute;
var
  DeviceID, Data:string;
begin
  while not Terminated do
  begin
    // read Device ID and sensordata
    DeviceID := s_recv(Responder);
    Log('RX '+DeviceID);
    Data := s_recv(Responder);
    Log('RX '+Data);

    // transfer sensor data to Device object
    gInterfaceDataHandler.DistributeToDevice(DeviceID, Data);

    // send reply to 0MQ client
    s_send(Responder, 'OK');

  end;
end;

The code above only shows the code for the Execute() method of a class derived from a Delphi TThread. All this method does is receiving information from the 0MQ socket and ‘injecting’ it into a Interface data queue, which handles the data delivery to the device object. From there, events are triggered, data is made historic, etc. etc. Again, the method code above is very very simple and not good enough for use in a live system, but good enough for testing.

OK, now it’s time to see if this ZeroMQ Plugin (hint, hint) works. I first started up the Plugin and after it had received the first data sample from the Mobotix, I started up a development instance of my Domotica system. And this is the output of both the Plugin and my system:

Plugin:
19:29:02 192.168.120.11;0;629;;
19:29:02 OK
19:30:01 192.168.120.11;0;633;;
19:30:01 OK
19:31:01 192.168.120.11;0;640;;
19:31:01 OK
Domotica system:
19:29:02.419 ZEROMQ : RX MOBOTIX
19:29:02.419 ZEROMQ : RX 192.168.120.11;0;629;;
19:30:01.835 ZEROMQ : RX MOBOTIX
19:30:01.835 ZEROMQ : RX 192.168.120.11;0;633;;
19:31:01.561 ZEROMQ : RX MOBOTIX
19:31:01.561 ZEROMQ : RX 192.168.120.11;0;640;;

Yeeha, it works, I’m receiving data in my system and everything keeps on working!

You might wonder why I’m so thrilled about being able to do something that looks a lot like what I’ve been doing from the beginning (i.e. getting data into my system); well, it is¬†what it looks like, but it’s more!

ZeroMQ enables me to do a lot of things:

  • I can start using any programming language for which there’s a ZeroMQ binding;
  • This programming language¬†independence¬†also gives me platform independence¬†(try running a Delphi-win32 executable on a Linux based NAS..)
  • I can break up my system in lots of small parts, starting with removing all the hardware I/O related code from the ‘core’, resulting in less chance of the whole system collapsing;
  • And with all this, create a multi-language, multi-platform distributed home automation system!

Although this all sounds nice, I’m still far away from implementing all this; this new architecture comes with new questions and demands (and new trade-offs?).

For example, I’ll need to do some sort of authentication for these loosely coupled Plugins, because I don’t want just anything that can ‘do’ ZeroMQ to start talking to my system. And I will also need some sort of heartbeat messaging, so that the ‘core’ knows if all the Plugins are still online and performing like they should. And the Plugins will need to know where to find the ‘core’; and the core should be able to start & stop Plugins, and … ¬†sorry, this post ends here – I’ve got work to do!