Realtime data with MQTT, Node.js, MQTTClient.js and Socket.IO

Seeing things happen the moment they actually do happen, that’s what I like the most.

And that’s why I’ve always been searching for a better solution than Ajax. I use Ajax here and there on my own website (which is actually nothing more than a showcase of what you can do with Domotica/Home Automation), but I never really liked it. It always felt like some sort of patch-up while waiting for something better to emerge.

In the past I’ve tried several solutions besides Ajax; Lightstreamer, Comet-like solutions and other similar products, but they all were either too big, required too much work for a single webpage or the price was too high. But now, the time has come to get rid of Ajax and polling web pages.

Once a year or so, I browse the web to look for an Ajax alternative. And this time I read this comment on a post about MQTT over Websockets. Node.js rang a bell, Websockets too and I decided to give it a try; with Flanagans Javascript bible next to me that shouldn’t be a problem.

So the ingredients for this exercise are:

  • MQTT, a machine-to-machine (M2M)/”Internet of Things” connectivity protocol, especially useful there where small code footprint is important;
  • Node.js, a platform for easily building fast, scalable network applications. Lightweight, event-driven and non-blocking I/O; ideal for real-time applications;
  • MQTTClient.js, an MQTT client for Node.js;
  • Socket.IO, which enables you to build realtime apps for virtually every browser and mobile device, using different transport mechanisms.

Hehe, it’s beginning to sound like a commercial but it ain’t – now you know what I’m talking about during the rest of this post.

I’ve already got a Smart meter and an Opentherm Gateway publishing their data to an MQTT broker, so that part was already working; I started with installing Node.js. There are several installers available and I picked the one that matches my PC: the x64 Windows installer, started it. Nnnf, done. Well, the rest was just as easy… now the code.

First thing I did was making a Javascript that would run on Node.js that would combine the Socket.IO functionality and an MQTT client to push MQTT topics to a browser; I named it the ‘pusher‘ script:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
});

That should do it.. I guess. We’ll see soon enough…

Next, a small web page:

<h1>Real Time</h1>
<script type="text/javascript" src="socket.io.min.js"></script>
<script type="text/javascript" src="jquery-1.4.2.js"></script>
<script type="text/javascript">
  var socket = io.connect('http://localhost:5000');
    socket.on('connect', function () {
      socket.on('mqtt', function (msg) {
        var elmarr=msg.topic.split("/");
        var elm=elmarr[3];
        console.log(msg.topic+' '+msg.payload);
        $('#'.concat(elm)).html(msg.payload);
     });
     socket.emit('subscribe',{topic:'/sensor/OTGW/returntemp'});
    });
</script>
<table class="tablegv" style="width: 500px;">
<tbody>
<tr class="tablegvHeader">
<td colspan="2"><center>Status</center></td>
</tr>
<tr>
<td>Return temp</td>
<td id="returntemp"></td>
</tr>
</tbody>
</table>

Here’s the result, the boiler return temperature shows up in the screen:

Pushing it!

Wow, it’s working… both the page and the console log below the page shows the incoming temperatures!

Needless to say this can easily expanded with room temperature, flow temperature, flame status, etcetera etcetera…. all updated instantly – as realtime as you can get! :-)

But I wanted more…

Suppose you have a web page like this displaying the Smart meter information and another one displaying the Opentherm information. Both pages will result in subscriptions to all kinds of topics, either related to the Smart meter or the Opentherm. And I saw that both pages received all the information! That’s not good – I need to a way to make sure the smart meter page only receives updates it subscribed to.

After reading some more about Socket.IO I found out there’s a feature called ‘rooms’. Sockets (clients) can join a specific room and there’s a way to emit information to just thos sockets in a specific room. That sounds nice, so I tried it. Changed the pusher script to this:

var sys = require('sys');
var net = require('net');
var mqtt = require('mqtt');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    socket.join(data.topic);
    client.subscribe(data.topic);
  });
});

client.addListener('mqttData', function(topic, payload){
  sys.puts(topic+'='+payload);
  io.sockets.in(topic).emit('mqtt',{'topic': String(topic),
    'payload':String(payload) });
});

As you can see, line 11 is added and line 17 (old) has been changed to line 18. That’s all, now none of the pages gets information it didn’t ask for; great, this can definitely be a very good Ajax replacement!

Tagged , . Bookmark the permalink.

37 Responses to Realtime data with MQTT, Node.js, MQTTClient.js and Socket.IO

  1. airox says:

    Nice! I also use Websockets and fell in love also. It’s now powering a lot of user interfaces. Something I also do is a small flash effect with jQuery to let the user know the value has been updated. Great stuff Robert!

    See you soon on the slim wonen event :-)

    • Hi Gijs,
      Me too, I really like it. And it’s good to know that I’m not the only Domotica enthusiast working on this, as you can see on the forum. See you soon, lots of stuff to talk about!

  2. gary says:

    Hi,

    I am new to MQTT and Socket.IO and have tried for 2 days to get this working with your code.

    I have learnt a lot, so not totally wasted time, but perhaps the link you give in the requirments may not be correct?

    It looks to me as though there are items missing, as it only has 2 files, and I cannot install it with NPM ?

    Found via the bullet Point blue link = “MQTTClient.js, an MQTT client for Node.js;”

    If there are files missing would it be possible to email them to me ?

    Or am i doing something completely wrong ?

    Regards

    Gary

    • Hi Gary,

      No there are no files missing as far as I can see, although I know someone else mailed me about problems installing with npm too.
      For him, the solution was to locally copy some files, editing the ‘pusher’ script and adding the path to those required files.
      This is what he did: var mqtt = require(‘./node_mqtt_client/MQTTClient.js’);
      Another thing is that on the 1st day of the post it contained a wrong link, so please check them all once again.

      Hope this helps,

      Robert.

  3. Gary says:

    Hi Robert,

    Thanks for that, I forgot to mention I am using Windows XP

    I had to do the following:

    var mqtt = require(‘MQTTClient/MQTTClient.js’);

    MQTTClient is the named of the folder (this folder can be called whatever you like) and this contains the file MQTTClient.js

    Regards

    Gary

  4. Gary says:

    Hi Again,

    For clarity I have started a new comment from my previous one.

    I now have the console displaying the following

    info – socket.io started
    Connected as :Pusher
    Session opend

    ‘and a flashing underscore here’

    Is this correct so far ?

    Where should I be looking for data:
    127.0.0.1 : not sure of the port number to use ?

    Or should I be sending data to the HTML file that you created ?

    I am using Really Small Message Broker from IBM and IA92 (to simulate a sensor) and send data to Topic mqttData.

    As you can see I still have not got my head around all of this yet and how it all interacts, but slowly its making sense, just need a little help !

    Thanks

    regards

    Gary

    • >Is this correct so far ?
      Correct, that’s node.js running the pusher script.

      You should change 127.0.0.1:1883 to where your RSMB is running.

      Ports 5000 can be changed, but must remain the same in pusher.js and the html page. This is for the socket between html page and the pusher script.

      And one last thing, the “localhost” in io.connect(‘http://localhost:5000′) in the html page
      may require a change too; “localhost” assumes that the pusher script is running on the same machine as the html page (e.g. your webserver). If you’re running everything on a single machine (as I did), this should be fine.

      Oh and beware the “/” splitting of the topic, another pitfall which can result in nothing being displayed… ;-)

      HTH,
      Robert.

      (BTW, I used mosquitto, but I don’t see any problem in that)

    • BTW this is what I get in node when the html and script connect (yes I shortened the html page on the weblog and left just a single subscription in it to keep only the important stuff in):

      debug - client authorized
      info - handshake authorized d25Jg3n47K2z5LZBKycm
      debug - setting request GET /socket.io/1/websocket/d25Jg3n47K2z5LZBKycm
      debug - set heartbeat interval for client d25Jg3n47K2z5LZBKycm
      debug - client authorized for
      debug - websocket writing 1::
      subscribe to /sensor/OTGW/ch_mode
      Subcribe to:/sensor/OTGW/ch_mode
      subscribe to /sensor/OTGW/dhw_mode
      Subcribe to:/sensor/OTGW/dhw_mode
      subscribe to /sensor/OTGW/flame
      Subcribe to:/sensor/OTGW/flame
      subscribe to /sensor/OTGW/roomtemp
      Subcribe to:/sensor/OTGW/roomtemp
      subscribe to /sensor/OTGW/flowtemp
      Subcribe to:/sensor/OTGW/flowtemp
      subscribe to /sensor/OTGW/returntemp
      Subcribe to:/sensor/OTGW/returntemp
      subscribe to /sensor/OTGW/controlsetpoint
      Subcribe to:/sensor/OTGW/controlsetpoint

  5. Gary says:

    It works !!!

    Hi Robert,

    Thanks very much for your help, firstly I want to let you know that it is both yourself and Bwired.nl that inspired me to get into home automation. You both have brilliant websites and very informative and interesting!

    And secondly yes I have got it working and for anyone else reading this in the future I would just like to point out a few things that weren’t obvious to me being new to JavaScript and Web pages.

    The ‘small web page’ mentioned in your article (incidentally I named this Default.html) has to be run on a server, I used a local server on my PC = ‘Abyys Web server’ a free version can be found on the Internet.

    Also in the same folder as the Default.html, you have to include the files ‘jquery-1.4.2.js’ and also ‘socket.io.min.js’ as the webpage requires these to work. Both can be found with a search on the Internet.

    Also you require a MQTT server to be running in the background and I used ‘really small message broker’ by IBM, again this can be found by searching on the Internet.

    http://mqtt.org/wiki/doku.php/ia92

    I used ‘IA92 utility’ by IBM, and ‘subscribed to’ /sensor/OTGW/returntemp
    You can then publish a number for example: 10 and if everything is working correctly this will be shown on your webpage.

    Hope this helps someone !

    Regards

    Gary

    • Great! And thanks for the additional information, I’m sure it will be helpful for others. And I learned something too – I need to be more elaborate in what I write ;-)

      But the story ain’t over yet – there’s an Apache module in the make that enables you to do things even simpler, with less components – more on that later! :-)

  6. Pingback: Realtime data with Mosquitto, Apache and mod_websocket_mosquitto - Digits Domotica Blog

  7. Gary says:

    Hi Robert,

    Have been gradually working my way through how this all works and think I understand most of it now.

    My next task is have a button on the HTML page send back to MQTT server, the number 21

    I have the button on my web page but am not sure how I send the number out ?

    I would be ‘subscribed’ on the RSMB to ‘Data_Store’ awaitng that number 21.

    Hopefully you can help me out ?

    Regards

    Gary

    • Hi Gary,
      I guess you didn’t add any code to the pusher script? Because the way it is in my post, it can only handle subscriptions!
      I moved on to another setup as you can read here. With a bit of luck, you should be able to find some examples elsewhere.
      The only thing I can help you with is that you’ll have to use the client.publish() method in the pusher script, probably in a socket.on(‘publish’,….).
      Regards,
      Robert.

  8. Gary says:

    Hi Robert,

    Thanks for that info, I will try and find out how to modify the pusher script, at least you have given me a clue that its in the pusher script.

    I like the Smart meter demo !

    May go over to the new way you have achieved this using apache, but for now I will pursue the original version.

    Thanks for your help

    Regards

    Gary

  9. Thomas Jin says:

    Hi Robert and Gary,
    I would like first to thank you guys for this amazing post and comments!
    This is the most useful resource I have found so far that directs the support of js client over mqtt.

    Next is the summary of my steps,
    1) I already have mosquitto installed, and it works with my java mqtt clients. I want to add js client to my project, such that different clients can coorporate.
    2) run mosquito -p 8080 in my case.
    3) install node.js, install and socket IO, and download MQTTClient.js, add pusher.js. put pusher.js and MQTTClient.js under one directory, change the line 3 of pusher.js to
    var mqtt = require(‘./MQTTClient.js’);
    4) run node pusher.js, I can see its connection at mqtt terminal. If I publish sth from java client, it can be displayed on the pusher.js terminal as well. Works good so far.
    5) put the small html file, and js files (jquery and socketio) under an apache www directory on the same server.
    6) from my client, when accessing the html file on the web server, I only see the static text, no real time info. However, I can see that there is the handshake info and the output under the pusher.js terminal.

    Could you please give me some hint regarding this? I guess sth is wrong betweent the html and js, but have not detailed clue (I am new to node.js and socket io. less than 1 week knowledge of mqtt).

    My understanding about the solution is that, pusher.js works as a bridge between the mosquitto and browsers. In th case here we are forwarding the msg received by pusher to the browser, Am I right?

    Also, I did see there is an apache module in the later post but cannot find the src/installation. Will Willem be willing to share it?

    Thanks!

  10. Gary says:

    Hi Robert and Thomas,

    This helped me a lot to see what was happening in the HTML side of things and made debugging much easier.

    Use firefox as your browser, get an add on called Firebug.

    When installed, click on its icon top right on my browser, the firebug will then show a panel at the botton of the screen.

    From there select the ‘Console’ tab.

    Now when you send data to the web page it will be displayed in the console window.

    If you see nothing then search for the fault in your code.

    Hope this helps!

    Regards

    Gary

  11. Javier Gonzalez Ferreyra says:

    I am also buiding small devices for lighthning homes and watering stuff, I think this is the guide I was looking for. Integration of MQTT , thanks I hope I can start easier , thanks again for sharing

  12. Marnix says:

    very interesting stuff, but definitely hardcore. God knows what I’ve been trying but I feel i’m quite close.
    However, publishing a message using
    mosquitto_pub -t /sensor/OTGW/returntemp -m 22
    does not give me anyting in firefox (no action).
    I got the js-stuff in the correct directories (thanks to Gary), but just don’t see anything in the node-terminal let alone the webpage.
    Help…

  13. Marnix says:

    after a night sleep (and a lot of trying the day before) in the the html file I changed io.connect from 127.0.0.1 to the actual ip-adress of the raspberry.
    time to experiment further

  14. Pingback: Arduino mqtt and web socket home automation | Bogdi's web site

  15. Arun says:

    Hi Robert,
    i have tried those code to get the message from topic and show in web. but i got error ..So please u can share the source to my mail id.

    Thanks,
    Arun S

  16. Patrick says:

    Hi Robert,

    Maybe it’s something simple but I can’t get your example to work.
    I’ve read all the comments and tried them all but the debug window in Chrome keeps displaying:
    GET http://localhost:5000/socket.io/1/?t=1401554152306 net::ERR_CONNECTION_REFUSED

    In the console where pusher.js runs I only get:
    pi@domoserver ~/domoweb $ node pusher.js
    Connected as :pusher

    Session opend

    There is no line saying: ‘info – socket.io started’
    Could it be that socket.io isn’t running?

    Thanks, Patrick

    • I think it has to do with this being an old post; line 6 should now be something like
      var client = mqtt.createClient(1883, ‘127.0.0.1’);
      See the documentation for the mqtt module: https://github.com/adamvr/MQTT.js
      BTW, no matter what line 6 looks like, I always get this on my screen:
      info – socket.io started
      Did you install socket.io? (“npm install socket.io”)?
      And of course a MQTT broker running.

      • Patrick says:

        Hi Robert,

        I have done all the steps on a clean install of raspbian, including the installation of socket.io.
        Still no ‘info – socket.io started’ in the console.
        I even tried Pieted Knuver’s smartmeter example on a fresh raspbian install, also no message.
        What can i try else? I can’t find any similar problems reported on the internet.
        If you’re willing you could ssh into my raspberry for a look.

        Thanks anyway, Patrick

        • Patrick,
          In that case the default logging level must have changed?
          Is it running? Use netstat to see if the port is listed as listening, then you know it’s OK.

  17. Gary says:

    Hi Robert and Patrick,

    Patrick, not sure which version socket.io you are using but V 1.0 seems not to output anything anymore, i think I am using V0.7.2 which does output ‘info – socket.io started’

    Try these links as they expalin more and may help you ?

    http://codehenge.net/blog/2011/12/getting-started-with-node-js-and-socket-io-v0-7-part-2/

    http://socket.io/docs/logging-and-debugging/

    Regards

    Gary

    • Hi Gary,
      Thanks for your comment, just what I thought, I couldn’t think of anything else anymore.
      I think Patrick is using a very recent version of socket.io, mine is still 0.9.16.

      • Patrick says:

        Hi Gary and Robert,

        Finally! I found it…
        I did a rollback to socket.io version 0.9.16: the ‘info – socket.io started’ message appeared.
        But the ‘ERR_CONNECTION REFUSED’ message keep showing in Chrome’s console, and no mqtt values were displayed.
        Then I found a post regarding accessing socket.io on a RaspPi from another computer in your LAN, seems that ‘127.0.0.1’ in the pusher.js script needs to be the actual IP address of the RaspPI.
        I have been testing from my laptop and not from the midori browser.
        After I changed every reference of ‘localhost’ and ‘127.0.0.1’ to the address of the RaspPi it all worked like a charm!
        Thank you for your patience!

        Greetz, Patrick

  18. Aruns says:

    Pls can u share the full source code of this project to this mail id arun4san@gmail.com. it will help me.

    Thanks,
    Arun S

    • Hi Arun,
      All I can do is give you the same answer as the 1st time, about 3 months ago:
      All the code that I still have is in the post. But maybe, if you tell me what the error message was, I might be able to help.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>