Migrating to the future has begun

I think I’ve got it. For now… Almost a year ago I realized that something had to change; my Domotica system grew too fast, became too big to keep it all in a single executable: stability and flexibility were the two main issues that had to be addressed.

During the last 12 months I tried several things to find a better solution on how to gradually rebuild my system; ZeroMQ, SimpleCortex, MQTT, FEZ, Raspberry Pi, Node.JS, Python, Netduino, they were all tried and tested for some time. And (for me) the winners are: Raspberry Pi, MQTT and Node.JS.

The power of Node.JS enables me to very quickly develop a stable hardware driver and combining this with a very small, low power & cheap Linux box like the Raspberry Pi to run those drivers is absolutely unique in my opinion; and MQTT is the super-glue that makes all the different parts of the system talk to each other, from Arduino sketch to VB.Net application.

The last weeks have been quite busy for me, there was very little time was left for working on Domotica, but with some hours here and there I still managed to write 5 drivers for some parts of the hardware I’m using in my Domotica system. So since a week or two I have a Raspberry Pi here that has been running those 5 replacement-drivers flawlessly – for my RooWifi (Roomba), RFXCOM RF receiver, Mobotix security camera light sensor, HA7Net with 10 1-Wire sensors attached to it and for my Remeha Calenta boiler. The last one mentioned is one of the most CPU & I/O intensive drivers I have, but the Node-version of all those drivers work perfectly on a single RPi:

uptime

Still enough processing power left to add some more drivers, don’t you think?

I made some changes to my monolithic Domotica system so that it would accept ‘raw’ device information from the outside world by means of MQTT, automated starting the drivers after booting the RPi and everything has been running great so far. I still have a lot of things to do though, mostly regarding maintenance & ease of use, of which some issues have already been addressed and others need some more time to find the best solution:

  • backup procedure for the RPi;
  • remotely controlling the RPi and its drivers;
  • supervising the RPi and its drivers;
  • storing global configuration data elsewhere.

So I still have some things to do before I can concentrate on migrating all my hardware drivers to the RPi, but I think I should be able to do one driver per week (that includes developing, testing and accepting it as a reliable replacement). The advantage I have is that I already have thoroughly tested Delphi code with all the workarounds for hardware specific peculiarities in the code; for example, I know that the Remeha Calenta sometimes doesn’t respond to a query, so I already know the new driver on the RPi needs to able to handle that – knowing all those peculiarities will certainly speed up things.

Another advantage is that all my hardware interfaces are connected to my LAN, so I don’t have to worry about RS-232, -485 or USB stuff, an IP address is enough for the RPi to communicate with the hardware interface.

So if all goes well, my Domotica system will be stripped of all its built-in drivers in about 30 weeks or so (cause that’s about the number of drivers I’m using right now) and all those drivers will be migrated to the RPi. Sounds doable, and I hope this will still leave some time to do other things as well, like adding more hardware, software and features to my system.

Yeah I know, I’m always too optimistic… 😉

RooWifi, Node.js and MQTT working together

As I wrote yesterday, the RooWifi has both a Web Interface and it can also connect to a remote TCP server. The RooWifi prioritizes the TCP server connection to the Web Interface, so if there’s a remote TCP server running and accepting connections, this TCP Server is in full control. The IP address and port number can be set from the RooWifi Web Interface:

RooWifi TCP server settings

This afternoon I tried to get the RooWifi to connect to a TCP server. I decided to use Node.js this time. I’ve used Node.js before and I wanted to see if I could get it running on one my Raspberry Pi‘s, create a TCP server, accept connections, parse a JSON payload and send the results to a MQTT broker (mosquitto).

I remembered a set of commands on JeeLabs to install Node.js, so I used that:

sudo usermod -aG staff pi && sudo reboot
v=v0.10.7
cd
sudo curl http://nodejs.org/dist/$v/node-$v-linux-arm-pi.tar.gz | tar xz
cp -a node-$v-linux-arm-pi/{bin,lib} /usr/local/

The first script I made was a TCP server that kept track of the clients that connected and disconnected and wrote the incoming data to the console:

net = require('net');
var connections = [];

net.createServer(function (socket) {

socket.name = socket.remoteAddress + ":" + socket.remotePort
 connections.push(socket);

socket.on('data', function (data) {
 var newDate = new Date();
 var time = newDate.toLocaleTimeString();
 console.log(time + " | " + socket.name + " < " + data);
 var obj = eval('('+data+')');
 console.log(time + " | temp=" + obj.roomba.temp);
 console.log(time + " | dirt=" + obj.roomba.dirt);
 });

socket.on('end', function () {
 var newDate = new Date();
 var time = newDate.toLocaleTimeString();
 console.log(time + " | " + socket.name + " ended the connection");
 connections.splice(connections.indexOf(socket), 1);
 });
}).listen(8001);

console.log("Server listening on port 8001n");

That’s it?? Yep.. 26 lines of code, wow. The output looked like this:

19:57:29 | 192.168.10.201:1561 < { "roomba": { "status": "3", "cleaning": "0", "battery": ":0", "temp": "36", "dirt": "0" } }
19:57:29 | temp=36
19:57:29 | dirt=0
19:57:34 | 192.168.10.201:1561 ended the connection

I also installed the MQTT package from adamvr:

npm install mqtt

I added a mqtt client to the script so I could publish all the information inside the JSON data and made some preparations to receive commands from the outside world as well:

net = require('net');
var mqtt = require('mqtt');
var connections = [];

var mqttc = mqtt.createClient(1883, '192.168.10.17', {
 keepalive: 30000
 });

mqttc.on('connect', function() {
 mqttc.subscribe('command/roomba');
 mqttc.on('message', function(topic, message) {
 console.log('topic: ' + topic + ' payload: ' + message);
 });
});

net.createServer(function (socket) {
 socket.name = socket.remoteAddress + ":" + socket.remotePort
 connections.push(socket);

socket.on('data', function (data) {
 var newDate = new Date();
 var time = newDate.toLocaleTimeString();
 console.log(time + " | " + socket.name + " < " + data);
 var obj = eval('('+data+')');
 for(var key in obj.roomba){
 console.log(time + " | "+key+" "+obj.roomba[key]);
 mqttc.publish('/value/roomba/'+key, obj.roomba[key]);
 }
 });

socket.on('end', function () {
 var newDate = new Date();
 var time = newDate.toLocaleTimeString();
 console.log(time + " | " + socket.name + " ended the connection");
 connections.splice(connections.indexOf(socket),1);
 });
}).listen(8001);

console.log("Server listening on port 8001n");

Still a very tiny script! I started the script in the background and let it run for a while (it’s still running):

node tcpmqtt.js &

Now let’s see if I can get the RooWifi information visible on another machine, my Win7 PC for instance:

RooWifi data published

Bingo… Now it’s easy to write a webpage to display the information, make certain values historic in my SQL database, control the Roomba from any User Interface (Touchscreen, Smart phones, tablets) – whatever you can think of!

Of course, I still have to add the code to control my Roomba, but that’s just a matter of time and will probably add just a couple of lines of code to the script.

What I’ve learned today is that the Remote TCP server feature of the RooWifi is the best way to monitor & control your Roomba and that Node.js is very powerful and most important: it works like I think: event-driven.

Opentherm Gateway operational

Opentherm Gateway and SimpleCortex side by sideToday I finished the monitoring part of the OpenTherm Gateway. The code running on the Simplecortex had proven to be reliable enough to start using it, so why not do just that 🙂

The OpenTherm Gateway you see on the right is my older one, built earlier this year and will be replaced by the new one once it’s finished.

Before I could start on the OpenTherm Gateway, I had to fix some issues which were the result of what I did last week. The MQTT driver was not as stable as I hoped; it worked fine for several hours, but it never made it longer than 48 hours, so I rearranged some code and it has been running fine since then.

Another thing I had to fix was that the Touchscreen, which was transformed to an MQTT client, was disconnected from the broker by the time I came home from work. The reason: the Touchscreen goes into hibernation when nobody’s at home to save energy, which leads to the broker closing the connection! Didn’t think of that one.. before, the touchscreen was a UDP listener, in which case there is no real connection… so I added a timer to the Touchscreen application that periodically checks the connection and re-establishes it when needed.

Back to the OT Gateway. The largest part of the work that still had to be done was preparing my Domotica system. The MQTT topics were already being received, but I had to do something about the payloads. The Opentherm Gateway produces a lot of bit values like for instance CH mode (Central Heating mode); payload ‘0’ means not active, ‘1’ means active. Well, I’m not going to pester users with terms like ‘ch_mode 1’ or ‘dhw_mode 0′; Central Heating mode active’ or ‘Hot water not active’ sounds more like it, right… But again, one of my goals is not to put texts like those hard-coded in any piece of code – not on the Simplecortex which publishes the information, nor in a Touchscreen app, or webpage or whatever.

So I felt really lucky that I started thinking about those things a long time ago and introduced so-called ‘Informationtypes‘ in my system in a very early stage. In a few words, an informationtype can transform a value that represents whatever to something a human can understand. For example with a door sensor, ‘0’ will be translated to ‘closed’ and 1 becomes ‘open’; but the same ‘0’ becomes ‘off‘ in the case of an appliance module. My Domotica system already does that for me and exposes both the ‘raw’ values (0/1 in case of booleans, unformatted data in case of numeric values, etc.) as the ‘formatted’ values (on/off, open/closed, floats with the right number of decimals). Raw data is nice, for computers. But not for humans! Currently, there are 150 informationtypes in my system (in the database, of course) ranging from on/off for a switch to the selected input on my AV receiver (0=”BD/DVD”, 1=”CBL/SAT”…).

So all I had to do was publishing those formatted values as well and I could present those values on a webpage and display something really meaningful … a new MQTT root hierarchy was born: /value.

And doing things this way, enabled me to reduce the code for the OpenTherm Gateway Device class to (almost) nothing more than this (the rest is all constants and type defs):

  case DataInfo.Format of
    flag8: Pin(iTopic).AsBoolean:=(Payload='1');
    u8,s8,u16,s16: Pin(iTopic).AsInteger:=StrToInt(Payload);
    f8_8: Pin(iTopic).AsFloat:=StrToFloatX(Payload);
    dowtod:begin
      { TODO : Finish }
    end;
  end;  //case

So now, when the central heating starts working, there’s (my system being the point of view) “/sensor/OTGW/flame 1” coming in and “/value/OTGW/flame Yes” going out. Now I can make a webpage without displaying meaningless numbers or having to translate those numbers locally that don’t (have to) mean anything to a human.

And here’s the result, showing parts of the information gathered with the OpenTherm Gateway.

 

Realtime data with Mosquitto, Apache and mod_websocket_mosquitto

Things are getting better and better 😉

This is a follow-up on my previous post about real-time data delivery to web pages (or anything else, in fact). The comments and emails I got on that post were surprising; MQTT is really hot and so is the push-mechanism that’s being used. Willem was one of them. I know Willem for some years now and we share some common interests; the first being Domotica and we’re also both tinkering with the Simplecortex, our recently installed smart meters and we both use Mosquitto as MQTT broker.

A week ago Willem developed an Apache module, which made life even more easier than it already was – with his Apache module, the only thing you need to make a real-time webpage is the mosquitto javascript that can be downloaded from here – nothing more! Less components, less chance that something will collapse… 😉

So, this time the components to build a real-time webpage are:

  • Mosquitto;
  • Apache;
  • mod_websocket_mosquitto from Willem;
  • Mosquitto javascript/websockets client;
  • a simple webpage.

It all started when Willem sent me an email with his mod_websocket_mosquitto source code attached and some instructions on how to use it. I have Apache running on my proxy server (for hosting this weblog) but since I don’t have any development tools installed on that machine (VM, to be precise) I booted a Fedora 17 VM on my PC (with VirtualBox, cause Windows 7 Virtual PC didn’t like to boot Fedora 17), installed scons and compiled the source code. The resulting module was copied to the proxy server including a configuration file for defining some configuration details:

<IfModule mod_websocket.c>
 Loadmodule mod_websocket_mosquitto modules/mod_websocket_mosquitto.so
 <Location /mosquitto>
 MosBroker proxyserver.hekkers.lan
 MosPort 1883
 SetHandler websocket-handler
 WebSocketHandler modules/mod_websocket_mosquitto.so mosquitto_init
 </Location>
</IfModule>

But somehow it didn’t work… I went back to my Fedora17 VM, installed the Apache module on it, but the result stayed the same – no go. I couldn’t find any trace of connection attempts being made to the Mosquitto broker, I got no error messages, nothing; looked like a dead end…

But then I got the tip to disable SELinux. And that was it, phew! Why didn’t I think of that, I’ve had troubles with SELinux in the past (some 7-8 years ago) but never again since, so the thought to disable it just didn’t occur to me… grr! Hurry back to the proxy server: Bingo!

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3624-giga.

But the thought of a broker being accessible from the Internet gave me the creeps; with not much effort, anybody who knows a bit about how all this works, could easily start publishing to my broker. But that’s not going to happen, I need some way to make sure my ‘production‘ broker is not affected in any way.

So for that I went back to my Fedora17 VM, downloaded the mosquitto broker source code, built it and installed it on my proxy server. This mosquitto instance on the proxy server would be the one that the web page (“the Internet”) would connect to. And to feed this proxy-broker with information from my production broker, I defined a 1-way bridge from production to proxy, which can be done from the mosquitto.conf file, like this:

connection domotica
address domotica.hekkers.lan
topic /# in

This way the proxy server knows about all the topics on the domotica (production) server, but topics published on the proxy server won’t arrive at the domotica server. After a restart of mosquitto I saw this:

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
Connecting bridge domotica
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3728-giga.

Again, wow…I added a web page to my Domotica website, so that y’all can watch my Smart meter data in real-time as well. Enjoy!

PS.: Use Firefox or Chrome, those are tested and work.

Realtime data with MQTT, Node.js, MQTTClient.js and Socket.IO

Seeing things happen the moment they actually do happen, that’s what I like the most.

And that’s why I’ve always been searching for a better solution than Ajax. I use Ajax here and there on my own website (which is actually nothing more than a showcase of what you can do with Domotica/Home Automation), but I never really liked it. It always felt like some sort of patch-up while waiting for something better to emerge.

In the past I’ve tried several solutions besides Ajax; Lightstreamer, Comet-like solutions and other similar products, but they all were either too big, required too much work for a single webpage or the price was too high. But now, the time has come to get rid of Ajax and polling web pages.

Once a year or so, I browse the web to look for an Ajax alternative. And this time I read this comment on a post about MQTT over Websockets. Node.js rang a bell, Websockets too and I decided to give it a try; with Flanagans Javascript bible next to me that shouldn’t be a problem.

So the ingredients for this exercise are:

  • MQTT, a machine-to-machine (M2M)/”Internet of Things” connectivity protocol, especially useful there where small code footprint is important;
  • Node.js, a platform for easily building fast, scalable network applications. Lightweight, event-driven and non-blocking I/O; ideal for real-time applications;
  • MQTTClient.js, an MQTT client for Node.js;
  • Socket.IO, which enables you to build realtime apps for virtually every browser and mobile device, using different transport mechanisms.

Hehe, it’s beginning to sound like a commercial but it ain’t – now you know what I’m talking about during the rest of this post.

I’ve already got a Smart meter and an Opentherm Gateway publishing their data to an MQTT broker, so that part was already working; I started with installing Node.js. There are several installers available and I picked the one that matches my PC: the x64 Windows installer, started it. Nnnf, done. Well, the rest was just as easy… now the code.

First thing I did was making a Javascript that would run on Node.js that would combine the Socket.IO functionality and an MQTT client to push MQTT topics to a browser; I named it the ‘pusher‘ script:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
});

That should do it.. I guess. We’ll see soon enough…

Next, a small web page:

<h1>Real Time</h1>
<script type="text/javascript" src="socket.io.min.js"></script>
<script type="text/javascript" src="jquery-1.4.2.js"></script>
<script type="text/javascript">
  var socket = io.connect('http://localhost:5000');
    socket.on('connect', function () {
      socket.on('mqtt', function (msg) {
        var elmarr=msg.topic.split("/");
        var elm=elmarr[3];
        console.log(msg.topic+' '+msg.payload);
        $('#'.concat(elm)).html(msg.payload);
     });
     socket.emit('subscribe',{topic:'/sensor/OTGW/returntemp'});
    });
</script>
<table class="tablegv" style="width: 500px;">
<tbody>
<tr class="tablegvHeader">
<td colspan="2"><center>Status</center></td>
</tr>
<tr>
<td>Return temp</td>
<td id="returntemp"></td>
</tr>
</tbody>
</table>

Here’s the result, the boiler return temperature shows up in the screen:

Pushing it!

Wow, it’s working… both the page and the console log below the page shows the incoming temperatures!

Needless to say this can easily expanded with room temperature, flow temperature, flame status, etcetera etcetera…. all updated instantly – as realtime as you can get! 🙂

But I wanted more…

Suppose you have a web page like this displaying the Smart meter information and another one displaying the Opentherm information. Both pages will result in subscriptions to all kinds of topics, either related to the Smart meter or the Opentherm. And I saw that both pages received all the information! That’s not good – I need to a way to make sure the smart meter page only receives updates it subscribed to.

After reading some more about Socket.IO I found out there’s a feature called ‘rooms’. Sockets (clients) can join a specific room and there’s a way to emit information to just thos sockets in a specific room. That sounds nice, so I tried it. Changed the pusher script to this:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    socket.join(data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.in(topic).emit('mqtt',{'topic': String(topic),
    'payload':String(payload) });
});

As you can see, line 11 is added and line 17 (old) has been changed to line 18. That’s all, now none of the pages gets information it didn’t ask for; great, this can definitely be a very good Ajax replacement!

Smart meter online

After 3 weeks, finally the end result came dangerously close 😉

Every time I finish something new and the time comes to actually start using it, I get excited – will it work? Cause testing in the office is something completely different than seeing real live data coming in! (yep, I forgot to check if I still had a free connection on the switch in the fuse box…)

Simplecortex with Proto Shield

The Simplecortex had been running without any problems during the last 2 days so it was time for the last part: inverting the P1 signal before it arrives at the Simplecortex UART0. I had an old Arduino Proto Shield laying around and put an 7470 inverter on top.

The ‘receiving end’ i.e. my Domotica system is ready – which consisted of adding an MQTT client to it and making sure it subscribes to the necessary topics. Just 2 more wires that needed to be soldered and I could hook up the Simplecortex to my smart meter… couldn’t wait!

And this evening I decided to remove the laptop that was logging the P1 data to a file from day 1 and replaced it with the contraption you see below.

Smart meter online!

And this is what it publishes (output from a MQTT subscriber):

/P1/powerusage1 00311.543
/P1/powerusage2 00158.107
/P1/powerdeliv1 00000.000
/P1/powerdeliv2 00000.000
/P1/tariff 0001
/P1/powerusagec 0001.88
/P1/powerdelivc 0000.00
/P1/gasusage 00027.758

Amazing…

The hardest most exciting part was developing the code in C for the Simplecortex. For my work but also at home I see a lot of code, in all kinds of languages and I can follow what the code should do – but getting started with writing my own code in a language which I hadn’t really used before took me some time – I just kept on changing things because I was not satisfied. Now I am (thanks, JC!), although there are still some minor things I have to work on. But those are not necessary for now; I can flash the Simplecortex later, when everything is completely finished software-wise.

Now I can start logging gas usage again and I can stop using my DIN rail mounted KWh-meter with S0 output which I used for power usage (but not before I’ve had a chance to compare it with the smart meter, though). Source code will be available on GitHub soon (account has already been made) .

Next project that will be handled in a similar way: the Opentherm Gateway! For that one the Simplecortex needs to be able to subscribe as well!

Edit: the P1_Publisher project has been committed to Github some moments ago: https://github.com/rhekkers/P1_Publisher

 

MQTT: about dumb sensors, topics and clean code

Or, in other words: where to define the topic a device should publish/be published on. In a scalable way..

I always try to keep the number of hard-coded values to a minimum; no hard-coded device IDs, peer IP addresses and things like that. And now that I’m working on MQTT clients, I was wondering how I could keep it that way.
Hard-coded MQTT topics? No way. Well, the last part in the hierarchy maybe (temperature, humidity, motion, …), but that’s it – nothing more. Driver code should be as clean as possible too; the drivers’ own IP-address is OK; peer IP address rather not; parameters that influence behavior: rather not. So the credo is: the least as possible/doable/practical. That’s the only way to keep maintenance (and irritation) low and scalabilty high; lets see what can be done about that with respect to MQTT.

The diversity in all the devices I have is rather large. There are devices that are really ‘dumb’, like 433 MHz temperature sensors that transmit the temperature every x seconds.  And there are devices which are a bit smarter, like JeeNodes with a RF12B or Xbee transceiver. And devices that can be controlled over powerline, RF, Zigbee, Infrared or TCP/IP and smart(er) devices like a EDS HA7Net,  Opentherm Gateway, ELV MAX! Radiator Valves, IrTrans interface, GSM Modem etcetera. And the latest addition to this list is the Smart meter, which is the first (well, the driver) that’s going to use MQTT to publish the smartmeter data.

Communication between my domotica system and all the end devices can be direct by a built-in driver (.e.g. a built-in driver that has a direct TCP/IP connection to the GSM modem), indirect (via ZigBee Coordinator, IrTrans LAN module) or device-initiated (connections are being made to an open TCP/IP port of my Domotica system, either with or without an extra hardware layer in between).

The format of the data can be a stream of bytes that need to be decoded, html output, xml, readable text, proprietary frames; you name and it’s in use here. Conclusion: its one big pool of hardware and nothing works the same 🙂

Now the challenge is to develop an easy to use, reliable and scalable solution so that I can build MQTT enabled hardware that acts as a driver which in some way ‘configures’ itself and doesn’t have to rely on hard coded topic names, IP addresses etcetera.

Lets give it a try. All sensors/devices, no matter how ‘dumb’, can be identified in some way; with a hardware address, MAC-/IP-address, Node id on a RS485 bus, whatever. The driver (the software under my control that’s the closest to the hardware) can only use this ID to publish the data; cause for the driver the device ID is the only unique thing.

For exampe, an RF receiver sees an incoming RF frame and can determine that this frame its coming from a temperature sensor with ID xx. So what can this driver do in terms of MQTT publishing? Publish the measured temperature of 23.1 °C for sensor ID xx (topic in red, payload in blue) as:

/RF/xx/temperature 23.1

Yuck. I don’t like that at all, I don’t think I want that – those IDs don’t tell me anything!
I’d like something like ourplace/house/firstfloor/kitchen/temperature or /ourplace/garden_shed/temperature. This has some advantages… but how do I do that? Cause the thought of ‘coded’ location information gives me headaches…

With a ‘database’ of course.. From the first day I started developing my own system I’ve got a table which contains a list of all the devices with their ID, description, where it is located (Building, Floor, Room, Location) and lots of other attributes.
That is where the relation between the hardware IDentifiers and the rest of the device information is stored – and it must be, and always stay, in one place. It doesn’t necessarily need to be SQL of course as it is in my case – the most important thing is that there can only be 1 ‘source’ – cause if not, a system with more than 230 devices with nearly 1200 device values will become a big mess, very quickly.

So how do I get the information from my SQL database into that MQTT capable driver that needs it – well, why not publish the contents of that database? Lets assume that for sensor ID xx the topic used for publishing the temperature should be /ourplace/garden_shed/temperature.  The fun is that ‘ourplace’ and ‘garden shed’ are already in the database as attributes to a device with ID xx – so nothing more to do than just publish that information and I’m done – right? If I would publish

/config/RF/deviceinfo/xx/location/ /ourplace/garden_shed/

then the RF driver can subscribe to the topic /config/RF/deviceinfo/# and it will receive all the information it needs for the relevant devices; and it can be more than just the location information.

In this case, ‘/config‘ is just an arbitrary topic name that separates the configuration stuff from the rest of the hierarchy. The next topic in the hierarchy is used to differentiate the devices per driver they are supported by (the RF driver only needs to know about the sensors it needs to ‘service’ – there’s no need for the RF driver to receive information about Zigbee devices for instance).

I could do that… that means that during the startup phase the driver subscribes itself to the right topics in the ‘config’ hierarchy and the information comes in… and this can be used for other things just as well – like driver specific settings as a poll interval, for instance.

Doing things this way means that, when a temperature sensor is being relocated from the kitchen to the garage, I don’t have to do anything more than I do now: changing the location information in the database and the rest follows! All I have to develop is a tool that publishes the required database information in the /config hierarchy and I’m done.

But there’s another way.

Let the driver publish the ‘raw’ information (Device ID oriented) in a /raw hierarchy (/raw/RF/xx/temperature 23.1) and develop a tool that subscribes to the /raw hierarchy and that re-publishes everything it can (e.g. that can be found in the database) to the location-oriented hierarchy, e.g. /ourplace/garden_shed/

This has some advantages too… hmmm; and there are even more alternatives… but this example of the temperature sensor is maybe too simple, it doesn’t cover every aspect I have to deal with – or does it?

Reset…

Publishing smart meter P1 data

Got it!

I just had to know whether my Simplecortex code was any good; it would be nice to see some real results instead of all the test results that were created along the way. The code was in a ‘workable’ state so I was ready to test if I could turn the Simplecortex into a MQTT client and publish the relevant smart meter P1 data.

I already downloaded Mosquitto somewhere in July, tried some simple (and successful) tests with a Delphi MQTT Client, but now it was time for the real thing. The Simplecortex was already running when I started the mosquitto server from a command line and I saw the Simplecortex connecting to the mosquitto server (great, auto connect works):

Mosquitto server

No timeouts, disconnects or any other thing that looked suspicious; so lets see if the Simplecortex is actually publishing something. For that I started a mosquitto client from another command prompt to subscribe to the topic for the current power usage:

Mosquitto client

Yes! Every 10 seconds I see the current power usage being updated;  cool, it works!

The biggest hurdle I had to take was the fact that the uIP API doesn’t allow you to just send outgoing data when/where you’d like to – no, the uIP API calls the application (a C function) when data is received, data has been delivered to the other end, when a new connection has been set up, and when data has to be sent. Therefore I had to use a sort of publish queue to store the MQTT publish messages until the uIP API allowed the application (the MQTT client on the Simplecortex) to actually send the data to the MQTT broker.  But the uIP stack had some good examples of which I gladly made use of 🙂

Does this mean I’m finished? Not yet.. There are still some things I’m not happy about; the way some things are setup in code right now. Ping for example, uses a hardware timer which throws an interrupt way too often for an interval of (now) 30 seconds. And the UART is being read in main(); I want that nicely tucked away too. And there are some other things that still need to be improved, all caused by my lack of C knowledge – while developing code for part B, I realize I could better have coded part A in a different way (and better)!

But for now, I’m going to sit back, relax and watch it working 🙂