Realtime data with Mosquitto, Apache and mod_websocket_mosquitto

Things are getting better and better ūüėČ

This is a follow-up on my previous post about real-time data delivery to web pages (or anything else, in fact). The comments and emails I got on that post were surprising; MQTT is really hot and so is the push-mechanism that’s being used. Willem was one of them. I know Willem for some years now and we share some common interests; the first being Domotica and we’re also both tinkering with the Simplecortex, our recently installed smart meters and we both use Mosquitto as MQTT broker.

A week ago Willem developed an Apache module, which made life even more easier than it already was – with his Apache module, the only thing you need to make a real-time webpage is the mosquitto javascript that can be downloaded from¬†here – nothing more! Less components, less chance that something will collapse… ūüėČ

So, this time the components to build a real-time webpage are:

  • Mosquitto;
  • Apache;
  • mod_websocket_mosquitto from Willem;
  • Mosquitto javascript/websockets client;
  • a simple webpage.

It all started when Willem sent me an email with his¬†mod_websocket_mosquitto source code attached and some instructions on how to use it. I have Apache running on my proxy server (for hosting this weblog) but since I don’t have any development tools installed on that machine (VM, to be precise) I booted a Fedora 17 VM on my PC (with VirtualBox, cause Windows 7 Virtual PC didn’t like to boot Fedora 17), installed scons and compiled the source code. The resulting module was copied to the proxy server including a configuration file for defining some configuration details:

<IfModule mod_websocket.c>
 Loadmodule mod_websocket_mosquitto modules/mod_websocket_mosquitto.so
 <Location /mosquitto>
 MosBroker proxyserver.hekkers.lan
 MosPort 1883
 SetHandler websocket-handler
 WebSocketHandler modules/mod_websocket_mosquitto.so mosquitto_init
 </Location>
</IfModule>

But somehow it didn’t work… I went back to my Fedora17 VM, installed the Apache module on it, but the result stayed the same – no go. I couldn’t find any trace of connection attempts being made to the Mosquitto broker, I got no error messages, nothing; looked like a dead end…

But then I got the tip to disable SELinux. And that was it, phew! Why didn’t I think of that, I’ve had troubles with SELinux in the past (some 7-8 years ago) but never again since, so the thought to disable it just didn’t occur to me… grr! Hurry back to the proxy server: Bingo!

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3624-giga.

But the thought of a broker being accessible from the Internet gave me the creeps; with not much effort, anybody who knows a bit about how all this works, could easily start publishing to my broker. But that’s not going to happen, I need some way to make sure my ‘production‘ broker is not affected in any way.

So for that I went back to my Fedora17 VM, downloaded the mosquitto broker source code, built it and installed it on my proxy server. This mosquitto instance on the proxy server would be the one that the web page (“the Internet”) would connect to. And to feed this proxy-broker with information from my production broker, I defined a 1-way bridge from production to proxy, which can be done from the mosquitto.conf file, like this:

connection domotica
address domotica.hekkers.lan
topic /# in

This way the proxy server knows about all the topics on the domotica (production) server, but topics published on the proxy server won’t arrive at the domotica server. After a restart of mosquitto I saw this:

[fake@proxyserver sbin]# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
mosquitto version 1.0.4 (build date 2012-10-27 16:56:56+0200) starting
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
Connecting bridge domotica
New connection from 192.168.10.11.
New client connected from 192.168.10.11 as mosqpub/3728-giga.

Again, wow…I added a web page to my Domotica website, so that y’all can watch my Smart meter data in real-time as well. Enjoy!

PS.: Use Firefox or Chrome, those are tested and work.

A smarter smart meter

Well, I think my MQTT enabled smart meter is rather smart too, but this one is smart also, but in somewhat different way…

A few weeks ago my ‘good old‘ Domotica buddy Pieter Knuvers started asking questions about my new smart meter, its P1 port and if I could build something for him to get the P1 datagrams into his system. (B)Wired, preferably via Ethernet. OK, that means that Pieter needed something completely different than what I made for myself – no Simplecortex needed, no MQTT publishing and stuff like that. A simple TTL to Ethernet server would do.

And then I remembered a wiki page where I’d seen just that. I looked it up and indeed this was all Pieter needed; so I bought the EZL70A, an enclosure and started building.

Signal inverterThe EZL70A module is the biggest part and all that was needed was an additional signal inverter. For that I used a 7404 hex inverter, the same I used for my own smart meter, so I already knew that it would work. A few wires for power, signals and the hardest part was done.

I put some heat shrink around it and was ready to put all the components inside the enclosure.

Under construction

 

And this is what it looked like after the first evening. Soldering a few wires, cutting some holes in the enclosure and Pieter’s smart meter would be Ethernet enabled, exactly what he wanted to have. As you can see the EZL70A, together with the signal inverter, fits nicely into a small enclosure of 100 x 50 x 25 mm.

As power cable we decided to use a USB cable; that seemed like a good idea since it’s not that strange anymore to find a USB socket in the meter cabinet – it’s the most common place to also find a NAS, router or Internet modem. And all of those devices have USB these days ūüôā

After a brief test with my own smart meter I sent the P1 to Ethernet converter to Pieter. He had already finished the code for his own homebrew Domotica system to parse the P1 datagrams with the help of a logfile of my own smart meter.

So now all we had to do was waiting for the smart meter to be installed in his house. And last thursday it was; and Pieter connected the converter, changed the IP address and his smart meter was connected!

The¬†accessibility¬†of the P1 datagrams, the ease of building, the relatively low costs – it made us wonder why this wouldn’t work just as well for others as it did for Pieter… if we could just add some extra functionality to this (built-in web-page with charts etcetera) …

So we’ve both started doing some research… stay tuned!

Realtime data with MQTT, Node.js, MQTTClient.js and Socket.IO

Seeing things happen the moment they actually¬†do happen, that’s what I like the most.

And that’s why I’ve always been searching for a better solution than Ajax. I use Ajax here and there on my own website (which is actually nothing more than a showcase of what you can do with Domotica/Home Automation), but I never really liked it. It always felt like some sort of patch-up while waiting for something better to emerge.

In the past¬†I’ve tried several solutions besides Ajax; Lightstreamer, Comet-like solutions and other similar products, but they all were either too big, required too much work for a single webpage or the price was too high. But now, the time has come to get rid of Ajax and polling web pages.

Once a year or so, I browse the web to look for an Ajax alternative. And this time I read this¬†comment on a post about MQTT over Websockets. Node.js rang a bell, Websockets too and I decided to give it a try; with Flanagans Javascript bible next to me that shouldn’t be a problem.

So the ingredients for this exercise are:

  • MQTT,¬†a machine-to-machine (M2M)/”Internet of Things” connectivity protocol, especially¬†useful there where small code footprint is important;
  • Node.js,¬†a platform for easily building fast, scalable network applications. Lightweight, event-driven and non-blocking I/O; ideal for real-time applications;
  • MQTTClient.js, an MQTT client for Node.js;
  • Socket.IO, which enables you to build¬†realtime apps for virtually every browser and mobile device, using different transport mechanisms.

Hehe, it’s beginning to sound like a commercial but it ain’t – now you know what I’m talking about during the rest of this post.

I’ve already got a Smart meter and an Opentherm Gateway publishing their data to an MQTT broker, so that part was already working; I started with installing Node.js. There are several installers available and I picked the one that matches my PC: the x64 Windows installer, started it. Nnnf, done. Well, the rest was just as easy… now the code.

First thing I did was making a Javascript that would run on Node.js that would combine the Socket.IO functionality and an MQTT client to push MQTT topics to a browser; I named it the ‘pusher‘ script:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
});

That should do it.. I guess. We’ll see soon enough…

Next, a small web page:

<h1>Real Time</h1>
<script type="text/javascript" src="socket.io.min.js"></script>
<script type="text/javascript" src="jquery-1.4.2.js"></script>
<script type="text/javascript">
  var socket = io.connect('http://localhost:5000');
    socket.on('connect', function () {
      socket.on('mqtt', function (msg) {
        var elmarr=msg.topic.split("/");
        var elm=elmarr[3];
        console.log(msg.topic+' '+msg.payload);
        $('#'.concat(elm)).html(msg.payload);
     });
     socket.emit('subscribe',{topic:'/sensor/OTGW/returntemp'});
    });
</script>
<table class="tablegv" style="width: 500px;">
<tbody>
<tr class="tablegvHeader">
<td colspan="2"><center>Status</center></td>
</tr>
<tr>
<td>Return temp</td>
<td id="returntemp"></td>
</tr>
</tbody>
</table>

Here’s the result, the boiler return temperature shows up in the screen:

Pushing it!

Wow, it’s working… both the page and the console log below the page shows the incoming temperatures!

Needless to say this can easily expanded with room temperature, flow temperature, flame status, etcetera etcetera…. all updated instantly – as realtime as you can get! ūüôā

But I wanted more…

Suppose you have a web page like this displaying the Smart meter information and another one displaying the Opentherm information. Both pages will result in subscriptions to all kinds of topics, either related to the Smart meter or the Opentherm. And I saw that both pages received all the information! That’s not good – I need to a way to make sure the smart meter page only receives updates it subscribed to.

After reading some more about Socket.IO I found out there’s a feature called ‘rooms’. Sockets (clients) can join a specific room and there’s a way to emit information to just thos sockets in a specific room. That sounds nice, so I tried it. Changed the pusher script to this:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    socket.join(data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.in(topic).emit('mqtt',{'topic': String(topic),
    'payload':String(payload) });
});

As you can see, line 11 is added and line 17 (old) has been changed to line 18. That’s all, now none of the pages gets information it didn’t ask for; great, this can¬†definitely¬†be a very good Ajax replacement!

Same Opentherm Gateway, different approach

Yep, I’m going to build a 3rd Opentherm Gateway ūüėČ
The first one was built on perfboard and became a mess; I never finished it, because before I could the Opentherm Gateway PCB became available, which looked much better. The first PCB version was built somewhere in February 2012 and is in use since August after I replaced the Proliphix NT20e with a Honeywell modulating thermostat.

There’s only one thing I don’t really like that much about the Opentherm Gateway and that’s the serial interface. I like to have all my hardware network-attached, but I’d also like to keep the number of Serial to Ethernet servers to a minimum – right now, I have 6 Serial ports in the meter cabinet (PLCBUS, X10, GSM modem, etcetera) connected to a Serial to Ethernet server. In total, I think I have around 10 serial devices connected that way.

For the third Opentherm Gateway I’m going to try something else; I’m going to leave the MAX232 IC off the PCB and directly connect a Simplecortex¬†to the TTL level output of the PIC that’s on the OT Gateway PCB. This approach has some benefits, like

  • The Opentherm Gateway becomes Ethernet enabled;
  • Programmable TTL to Ethernet conversion;
  • Less¬†‚ā¨‚ā¨‚ā¨;
  • Big reduction on network traffic;
  • more fun!

Yesterday I started with the Simplecortex firmware for the OT Gateway. I want the firmware to be versatile, meaning that it should not only support the Data ID’s that I see going back and forth between my thermostat and boiler. Fortunately the Opentherm Protocol is documented well enough to be able to write code for Data ID’s I’ve never seen in my life. In fact, the definition of how the firmware should deal with all the Data ID’s found in the OT protocol 2.2, comes down to the following:

typedef enum {Read, Write, Both} CommandType;
typedef enum {both, highbyte, lowbyte} ByteType;
typedef enum {flag8, u8, s8, f8_8, u16, s16, dowtod} PayloadType;

typedef struct{
	uint8_t ID;
	CommandType rw;
	ByteType whichbyte;
	PayloadType format;
	uint8_t bitpos;
	char* topic;

} OTInformation;

OTInformation OTInfos[] = {
{0x00,	Read, 	highbyte, flag8, 0, "ch_enable"},
{0x00,	Read, 	highbyte, flag8, 1, "dhw_enable"},
{0x00,	Read, 	highbyte, flag8, 2, "cooling_enable"},
{0x00,	Read, 	highbyte, flag8, 3, "otc_active"},
{0x00, 	Read, 	highbyte, flag8, 4, "ch2_enable"},
...
...
{0x7f, 	Read, 	highbyte, u8,    0,  "slaveproducttype"},
{0x7f, 	Read, 	lowbyte,  u8,    0,  "slaveproductversion"}
};

That’s all there is to it… add about 150 lines of code and the complete set of Data ID’s defined in the OT Protocol 2.2 documentation is supported – no more switch (DataID) with a long list of cases and repeating code…

But I wanted more – I really don’t need to be told that CH is still enabled twice a second; just a report of a change will do. For that I’m going to add a filter that will only report changes. That filter will reduce the network traffic immensely. And of course this Opentherm Gateway will be transformed into a MQTT Publisher, just like my smart meter. And last but not least, this implies that the Opentherm Gateway will act as a MQTT subscriber too, so that I can control the behavior of the OT Gateway and override the thermostat’s temperature setpoint.

Right now, I’m watching the OT Gateway information ¬†on my screen as it is being published:

Opentherm Gateway publishing information

There’s still a lot to do, but considering the fact that I managed to get this far in only a few hours makes me confident that I can finish this project before it starts to get really cold.

Another good thing is that once this project is finished I can shut down the Remeha Calenta driver which has been running for 2 years.

The biggest disadvantage of that driver was that I had to constantly poll the Calenta and that it was based on a protocol specifically targeted at Remeha boilers.

So it all gets much, much better this way!

 

TFTP boot loader for Simplecortex is working

The problems I had with the TFTP boot loader last week are solved!

Last weekend I discovered that the InterruptTimer_CoIDE example project I used from the Simplecortex repo wasnt’t working that well as I thought. When I started with the TFTP boot loader, one of the first things I did was running that example unmodified frm the normal start address. I saw all the LEDs blinking, so there was no reason to think that something was wrong. ¬†Then, last Saturday I read this post¬†on the Simplecortex forum. Apparently timer interrupt 3 wasn’t doing what it should.. hmm, in my case, timer 3 was the only one that did work in TFTP mode – there must be some link between the 2 issues!

And I was right – after updating the¬†InterruptTimer_CoIDE example from the repo, all the LEDs were still all blinking in non-TFTP mode of course, but in TFTP mode¬†none of the 4 LEDs were blinking! That may sound bad but it isn’t; I’d rather have 4 timers not working than 3 that don’t and 1 that does ūüėČ

Cause now I had a consistent behavior and not something completely inexplicable; now the user code didn’t function ‘a bit’ – no, now it didn’t function at all! That sounds like a better starting point to find the error. Somehow I had the feeling that the ¬†interrupt vectors were wrong; but why?

The mistake I made was that I had a completely wrong perception of what was going on when the TFTP boot loader jumped to the user code:

/* Change the Vector Table to the USER_FLASH_START
in case the user application uses interrupts */
SCB->VTOR = (FLASH_START_ADDR & 0x1FFFFF80);

// Load contents of second word of user flash - the
// reset handler address in the applications
// vector table
p = (unsigned *)(FLASH_START_ADDR + 4);

// Set user_code_entry to be the address
// contained in that second word
// of user flash
user_code_entry = (void *) *p;

// Jump to user application
user_code_entry();

The vector table is set to the right address (line 3) and the user code is called.

Aarrgghh!

I don’t know why, but somehow I had the (wrong) idea that the user_code_entry() call jumped directly to main() in the user code – and this id√©e fixe withheld me from finding the solution, which was relatively simple as I found out later… Willem, someone I know from the Domoticaforum tried to help me and mentioned I should change the SystemInit() in system_LPC17xx.c. Well, the only thing I could think of was the following:

void SystemInit (void)
{

  ...
  ...

  //  Set Vector table offset value
  #if (__RAM_MODE__==1)
    SCB->VTOR  = 0x10000000 & 0x3FFFFF80;
  #else
    SCB->VTOR  = 0x00000000 & 0x3FFFFF80;
  #endif
}

This code sets the vector table back to the value needed for ‘normal’ operation, where the user code starts at 0x00000000. But… why bother, the user code doesn’t come here at all, right? After Willem mentioned SystemInit() a second time I decided to give it a try although I still didn’t think it would help…¬†WRONG!

The moment I saw all the LEDs blinking at a different rate, suddenly 2 words in the comments caught my attention: reset handler.

Of course, how could I be so blind… the user_code_entry() call mentioned earlier doesn’t call main(), but the reset handler – so this effectively runs the same code¬†as if the Simplecortex has been reset! And the default reset handler routine contains a call to SystemInit(), so the effect was that the vector table was overwritten by the user code…

Bingo!