Ready for the future?

My Domotica systemAlthough my Domotica system is very stable, fast and everything, I’ve become somewhat unhappy with how it has evolved through the years.

It all started in 2006 with monitoring gas usage with a CNY70 and 1-Wire counter (DS2423).

Next came RFXCOM, X10, PLCBUS, IR, and so on. Right now, my system has about 30 interfaces to the outside world, all coded in Delphi; Borland Delphi 2005 to be precise. And all that code results in a monolithic executable of 2.9 MB, uses 1500 KB of memory at run-time with an average of <1% CPU usage on a Intel i3 in a Hyper-V VM.

Nothing wrong with these numbers, right? I could go on for years like this with no problem at all. However, I started having second thoughts about that statement a year ago.

  • With the ever growing number of stuff built into a single executable, the system becomes more and more vulnerable to fatal runtime errors that could make the whole thing crash.
  • I am bound to using Delphi as single programming language for the system. Yes, I’m also using other programming languages like VB.Net, ASP.Net, Arduino, (Pronto-/Java-)script, C# but those are only used for User Interfacing and hardware.
  • Delphi restricts me to Windows as platform for my Domotica system. For example, my NAS could be a good candidate to host some pieces of my Domotica system; I can’t do that right now.
  • It’s hard to make use of other non-Delphi code that can help me to ‘enrich’ my system.

6 years ago I could never have imagined that my Domotica system would grow as fast as it did, can do what it does right now and how important it would become in our current daily lives. And the time will come where the current state of my system becomes a burden with respect to adding new functionality.

Conclusion: I have to become more flexible – break the system up in small pieces and find a way to get rid of the programming language and platform dependency. Cause if I don’t, I foresee problems – either by not being able to keep up with the pace in software development, getting stuck on old operating systems or facing huge investments in writing my own Delphi code for something that’s freely available in some other language.

The big question is: how to do that? I don’t like the idea of porting more than 150000 lines of Delphi code to another programming language, cause that would get me in the same situation as it is now. Nor do I want to stop using the code I wrote during the last 6 years; it works fine, so that would be a complete waste! No, I want to keep using the code I have as much as possible, I should be able to gradually transform my system piece by piece and not be forced to some ‘Big Bang’.

So the first thing I have to do is break the current system up in the building blocks it’s being made of; right from the start I have been working in terms of ‘hardware interfaces’, ‘devices’, ‘sensors’, ‘events’, ‘actors’ in an object-oriented way, so breaking things up into smaller pieces should not be that hard; transforming a interface unit with some classes into an executable should be doable but the biggest problem is that a hardware interface object has to be able to get data from itself to the device object and vice versa: a device object (let’s say an IR controlled TV) also has to be able to send to the IR interface; some internal queues take care of that now, but I can’t use those anymore; those building blocks I mentioned need some way to get data across from one process/executable to another. And I think I’ve got the right solution for that – ZeroMQ (0MQ).

ZeroMQ was first brought to my attention during a meeting last year with some friends who are just as addicted to Domotica as I am and it has been on my mind ever since; but I never took the time to have a really good look at it; since 3 weeks I’ve started reading about it, looking at code samples and got more and more excited and realized that this is just what I need. And I’m not the only one that feels that way, as can be seen here and here.

Last Friday I had a day off and spent most part of the day to get 0MQ up and running in Delphi. I found some Delphi bindings for 0MQ (some more finished than others), tried to let 2 Delphi programs talk to each other and it worked. Another test with .Net <–> Delphi was succesful too; promising.

Now, but not after I’ve tested some basic needs very thoroughly, I can start dissecting my monolithic system into smaller pieces and gradually create an even more stable, flexible and adaptive Domotica system. One of the upcoming projects will be integration of a Smart Meter for gas and electricity; I think I’ll do that one the 0MQ way – exciting!

 

Tagged . Bookmark the permalink.

14 Responses to Ready for the future?

  1. airox says:

    I should also checkout ZeroMQ someday. Did you consider beanstalkd? This is the component which I’m using for over a year in my system. If you have, I would love to hear the ZeroMQ benefits over it 😉

  2. No, I didn’t… but I had a quick look just now and to me beanstalk looks like it was developed with web servers in mind? I think the main difference is that beanstalk needs a daemon running and focuses at getting jobs done, where ZeroMQ doesn’t need a daemon and it’s main goal is getting data (no matter what it represents) from A to B as fast as possible. (but correct me if I’m wrong, I might have missed a lot w.r.t. beanstalkd) 😉
    BTW this doesn’t mean that beanstalkd is unable to do what I’m going to do with ZeroMQ 🙂

  3. Frans says:

    You might consider Mosquitto an mqtt implementation. Some inspiration: on slideshare

  4. Pingback: Digits Domotica Blog | Smart meter is on its way

  5. Soitjes says:

    Did you consider xPL ? I went that way and it offers a very nice abstraction layer, keeping the software part very simple. The RFXLAN can talk xPL as well.

    • Hi Soitjes,
      I know about the existence of xPL for some years, but I think that it’s a bit too much in this case; xPL seems to also impose the way data has to be transported (but correct me if I’m wrong) – all I was looking for was a way to transport data, with as less restrictions or ‘formats’ to obey as possible. My perception is that if I want to communicate by means of xPL, I would write some xPL specific code for it, just like I would if I would want Homeseer to export its data in XML format.
      Maybe it’s a good idea to share some thoughts and ideas on developing our own Home Automation system, especially since we’re both heavily involved in Delphi 😉

      • soitjes says:

        Dag Robert,

        Ik wil heel graag ideeën uitwisselen, en je uitleggen waarom ik voor XPL gekozen heb. Maar ik gebruik nog andere communicatieprotocollen om te communiceren tussen componenten.

        Ik heb ook wat Delphi code liggen die je kan interesseren. Mijn applicatie is geen open source, maar code die ik zelf bijeengesprokkeld heb, of die ik gewoon “XE compatibel ” gemaakt heb beschouw ik niet als eigen code en geef ik vrij. Je hebt misschien op domoticaforum.eu mijn thread over de Digital Home Server gezien, die kan je ook eens doorlopen om een idee te krijgen wat waar ik zoal mee bezig ben. DHS is de basis applicatie, en ik ben bezig met een home automation prototype waar ik mijn ideeën in uittest. Nadien gaat dat dan ook in DHS.

        Laat maar iets weten,

        Walter (soitjes is een alias).

        • Hi Walter,
          Yep, I’ve been following your adventures on the Domoticaforum from the start! Always nice to see and read about other Domoticans developing their own system 🙂
          I’ll get in touch soon, but I’m one the few thousands of people that are caught by Pertussis (kinkhoest) and it’s really starting to make me very tired after 2 weeks of coughing… I have to slow down a bit, but I’ll contact you as soon as I’m back to normal!

  6. Roger says:

    As Frans suggests, this is an ideal situation for mqtt or other pub/sub style messaging system. There is a central broker, but I would assert that this is a benefit rather than something to avoid.

    The idea is that data producers publish data to relevant topic, perhaps rooms/lounge/temperature. Likewise, anything interested in that data can subscribe to the same topic or groups of topics. This decouples the producer and consumer and makes it trivial to add in other producers and subscribers. For example, you could have a light bulb controller by the switch on the wall. The switch publishes to the topic that the bulb controller listens to. You could then write an Android/iPhone client that publishes to the same topic to control the lights with your phone, and have a subscriber that tweets when the lights are turned on and off – all without disrupting the operation of the original switch.

    Some disclaimers: I write Mosquitto. I’m not sure about the current state of the Delphi mqtt library. On the flip side, mqtt is straightforward to code (there’s even an arduino implementation).

    • Hi Roger,
      Thanks for your comments; Franks comment already triggered me to read more about MQTT/Mosquito than I already did in the past, and the final decision hasn’t been made yet cause this is not something to decide overnight – I’m still searching for the best solution! But things are not always that simple as your example of the light bulb, especially when events are involved (the stuff that does the real automation part in home automation). I’m still struggling with how to implement that the (only) right way; enough food for another post!

      • Roger says:

        Good to hear you’re still considering how to do this. I know from experience that the first new thing you come across that is better than what you’ve already got isn’t necessarily the best solution.

        I realise things aren’t always as simple as I’ve described, but I was of course just making a simple example… 🙂 I’d argue that events are just a type of message and how you deal with them is no different than you would now.

        Everything you’ve described in your next post on 0mq is true of mqtt, except that there is already authentication and access control built in and there is the possibility in the future to move to a more hierarchical system so the device descriptor (ip address) becomes part of the topic structure rather than part of the message data. This makes it very easy to say “give me data from all of the illumination sensors at once, but nothing else”, or a single sensor or everything on the system.

        A particularly nice feature of the protocol with regards sensor systems is the “last will and testament”. This is a message that a client can register with the server. If the client disconnects from the server unexpectedly, the server will publish this message on its behalf. This makes it very useful for presence detection: when your sensor connects to the server, it registers a will message “0” to be published on a topic. Your sensor also publishes a message “1” to the same topic when it connects. You can then monitor the presence of that sensor by checking for a 1 or 0 – and it will work even if the sensor runs out of batteries or breaks or any other unexpected event.

        I’d also like to mention some mqtt http json bridge demos that give an example of bridging a sensor to the web (again, this wouldn’t disrupt whatever else you have listening to the same data):

        (btw, I don’t think your comment subscription emails are working)

        • Hi Roger,
          Thanks again for your input, I really appreciate it. I tried mosquitto (downloaded the XP binary and used the Delphi MQTT library to write a publisher and subscriber) and got it working within an hour or so. I think I’ll need some more time with both to make a good choice; one of the issues I need to get clear for myself is broker vs. broker-less messaging system; I guess I can do without a broker, but I’m sure there will also be arguments for having a broker 😉 I’ll start a post soon in which I will discuss this – your comments are welcome!

          (regarding the subscriptions: you’re right, I don’t see your email in the subscriptions; strange. Maybe you used the mobile interface? That’s the only thing I can think of to cause this)

          • Roger says:

            Hi Robert,

            Subscriptions seem to be fixed… I don’t think I did anything different.

            On the broker vs. brokerless subject – it sounds a lot like the core you describe in “Receiving sensor data with ZeroMQ” is not that dissimilar to a broker.

  7. Pingback: Digits Domotica Blog | The # subscriber

Leave a Reply

Your email address will not be published. Required fields are marked *