MQTT: about dumb sensors, topics and clean code

Or, in other words: where to define the topic a device should publish/be published on. In a scalable way..

I always try to keep the number of hard-coded values to a minimum; no hard-coded device IDs, peer IP addresses and things like that. And now that I’m working on MQTT clients, I was wondering how I could keep it that way.
Hard-coded MQTT topics? No way. Well, the last part in the hierarchy maybe (temperature, humidity, motion, …), but that’s it – nothing more. Driver code should be as clean as possible too; the drivers’ own IP-address is OK; peer IP address rather not; parameters that influence behavior: rather not. So the credo is: the least as possible/doable/practical. That’s the only way to keep maintenance (and irritation) low and scalabilty high; lets see what can be done about that with respect to MQTT.

The diversity in all the devices I have is rather large. There are devices that are really ‘dumb’, like 433 MHz temperature sensors that transmit the temperature every x seconds.  And there are devices which are a bit smarter, like JeeNodes with a RF12B or Xbee transceiver. And devices that can be controlled over powerline, RF, Zigbee, Infrared or TCP/IP and smart(er) devices like a EDS HA7Net,  Opentherm Gateway, ELV MAX! Radiator Valves, IrTrans interface, GSM Modem etcetera. And the latest addition to this list is the Smart meter, which is the first (well, the driver) that’s going to use MQTT to publish the smartmeter data.

Communication between my domotica system and all the end devices can be direct by a built-in driver (.e.g. a built-in driver that has a direct TCP/IP connection to the GSM modem), indirect (via ZigBee Coordinator, IrTrans LAN module) or device-initiated (connections are being made to an open TCP/IP port of my Domotica system, either with or without an extra hardware layer in between).

The format of the data can be a stream of bytes that need to be decoded, html output, xml, readable text, proprietary frames; you name and it’s in use here. Conclusion: its one big pool of hardware and nothing works the same :-)

Now the challenge is to develop an easy to use, reliable and scalable solution so that I can build MQTT enabled hardware that acts as a driver which in some way ‘configures’ itself and doesn’t have to rely on hard coded topic names, IP addresses etcetera.

Lets give it a try. All sensors/devices, no matter how ‘dumb’, can be identified in some way; with a hardware address, MAC-/IP-address, Node id on a RS485 bus, whatever. The driver (the software under my control that’s the closest to the hardware) can only use this ID to publish the data; cause for the driver the device ID is the only unique thing.

For exampe, an RF receiver sees an incoming RF frame and can determine that this frame its coming from a temperature sensor with ID xx. So what can this driver do in terms of MQTT publishing? Publish the measured temperature of 23.1 °C for sensor ID xx (topic in red, payload in blue) as:

/RF/xx/temperature 23.1

Yuck. I don’t like that at all, I don’t think I want that – those IDs don’t tell me anything!
I’d like something like ourplace/house/firstfloor/kitchen/temperature or /ourplace/garden_shed/temperature. This has some advantages… but how do I do that? Cause the thought of ‘coded’ location information gives me headaches…

With a ‘database’ of course.. From the first day I started developing my own system I’ve got a table which contains a list of all the devices with their ID, description, where it is located (Building, Floor, Room, Location) and lots of other attributes.
That is where the relation between the hardware IDentifiers and the rest of the device information is stored – and it must be, and always stay, in one place. It doesn’t necessarily need to be SQL of course as it is in my case – the most important thing is that there can only be 1 ‘source’ – cause if not, a system with more than 230 devices with nearly 1200 device values will become a big mess, very quickly.

So how do I get the information from my SQL database into that MQTT capable driver that needs it – well, why not publish the contents of that database? Lets assume that for sensor ID xx the topic used for publishing the temperature should be /ourplace/garden_shed/temperature.  The fun is that ‘ourplace’ and ‘garden shed’ are already in the database as attributes to a device with ID xx – so nothing more to do than just publish that information and I’m done – right? If I would publish

/config/RF/deviceinfo/xx/location/ /ourplace/garden_shed/

then the RF driver can subscribe to the topic /config/RF/deviceinfo/# and it will receive all the information it needs for the relevant devices; and it can be more than just the location information.

In this case, ‘/config‘ is just an arbitrary topic name that separates the configuration stuff from the rest of the hierarchy. The next topic in the hierarchy is used to differentiate the devices per driver they are supported by (the RF driver only needs to know about the sensors it needs to ‘service’ – there’s no need for the RF driver to receive information about Zigbee devices for instance).

I could do that… that means that during the startup phase the driver subscribes itself to the right topics in the ‘config’ hierarchy and the information comes in… and this can be used for other things just as well – like driver specific settings as a poll interval, for instance.

Doing things this way means that, when a temperature sensor is being relocated from the kitchen to the garage, I don’t have to do anything more than I do now: changing the location information in the database and the rest follows! All I have to develop is a tool that publishes the required database information in the /config hierarchy and I’m done.

But there’s another way.

Let the driver publish the ‘raw’ information (Device ID oriented) in a /raw hierarchy (/raw/RF/xx/temperature 23.1) and develop a tool that subscribes to the /raw hierarchy and that re-publishes everything it can (e.g. that can be found in the database) to the location-oriented hierarchy, e.g. /ourplace/garden_shed/

This has some advantages too… hmmm; and there are even more alternatives… but this example of the temperature sensor is maybe too simple, it doesn’t cover every aspect I have to deal with – or does it?


Tagged , . Bookmark the permalink.

7 Responses to MQTT: about dumb sensors, topics and clean code

  1. Willem says:

    I partly agree, but what do we want to get out of it. Should it easy to be understand for a human or a machine …. like our famous or should i say notorious soccer play Johan Kruif says; for each pro there is a con ……

    • You’re right, it will mean extra work. But for me, it’s much easier to remember an ID like ‘boiler.waterpressure’ than some other meaningless number; and the machine won’t mind. Just like I prefer to work with hostnames instead of IP addresses – there was a time when I knew all the IP addresses by heart (60 of them in use right now) and what device it was used for, but I’ve given up on that 2-3 years ago . Scalability….for us, humans ;-)

  2. Kyle Gordon says:

    I’ve been thinking the same recently… with the huge variety of hardware and information out there, what’s the common naming behind them all?

    It’s almost impossible to make, so I started pushing data into a MySQL database. I don’t like it, but at the time there was no other way. I like your idea of republishing the MQTT data though… that’s nice. hmmmm, I spy a project :-)

    If you have a github repo for this, please let me know!

    • Hi Kyle,
      Naming can be very hard – like my boiler, it doesn’t even have a hardware address or ID! So I chose what seemed to be its address in the Modbus-like communication I saw.

      I don’t have a github repo yet, but that’s going to change. In the meantime, feel free to contact me and I’ll help you any way I can. I think you’ll know my first name by now, so it won’t be that hard for you to send an email ;-)

  3. Pingback: The Lodge | MQTT Republishing Itch

  4. Rodan says:

    Hi, I’ve read this article many times, I really liked it!
    I’m proposing a “I am a temperature sensor” (someting like plug and play) in a way more like: “functions that a sensor has”, so the MQTT broker can get (from some server) an “interface” that the sensor can “fill up” (through broker), in a way that a broker could know what a sensor can offer, can point me in a direction (I’m actually reading a lot!) Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>