Making a scene

Being able to define scenes is one of the most important aspects in a Home Automation (HA) system. Not being able to define your own scenes, things you want to happen automatically, degrades a HA system to nothing more than a big expensive monitoring system or just a remote control. That’s not Home Automation, that’s Home Control!

ReteScenes let you automate certain things. Lowering the roller shutters so that the temperature inside doesn’t become unpleasantly high, switching lights on or off based on motion, sunset, sunrise, sending notifications to your smart phone, etcetera. Anything is possible, as long as your HA system is equipped with the right sensors (information) to trigger that specific scene you’d like it to execute for you.

When I started (re)developing my system in NodeJS, one of the first items on my to-do list was finding some sort of starting point to create my own scenes. Preferably developed for Node.JS, with its own DSL (Domain Specific Language), the ability to embed simple if/else statements and calculations inside the scene definitions and crontab-ish triggered scenes were my most important conditions to look for. When I looked at HomA (a NodeJS based framework for building smart homes) when I started exploring whether NodeJS was a good choice for rewriting my HA system, I noticed that it uses nools as a rules engine; Nools is a rete based rules engine and when I read the documentation it looked like a good starting point. HomA also provided a good starting point in terms of combining MQTT & nools.

In fact, I shamelessly copied this file from the HomA repository and added the things I missed and changed what I wanted to do different. Here’s a couple of things worth mentioning about how I integrated nools in my HA system and how I got scenes to actually automate things.

Lets have a quick look at how easy it is to use nools as the scene engine or rules engine. I’ll take some of my own scenes as example.

rule testrule {
  when {
    m1: Message m1.t == 'testsensor/status' && m1.changedTo('open');
  }
  then {
    unchange(m1);
    log('Execute rule Office testlight on');
    execute('command','{"address":"testactor", "command":"ON"}');
 }
}

Easy right?… when the status of the testsensor (in this case a Visonic door/window sensor) on my desk changes to ‘open’ , the testactor (a PLCBUS on/off module) which is also on my desk, will switch on a light.

But with this scene, when I close the sensor again, nothing happens. I would need a second rule to switch off the light when the sensor changes back to ‘closed’, in a similar way as with the rule above.

But that won’t be necessary, cause I can add some code to the rule, like this:

rule testrule {
  when {
    m1: Message m1.t == 'testsensor/status' && m1.changed;
  } then {
    unchange(m1);
    log('Execute rule Office test open');
    if (m1.p === 'open') {
      execute('command','{"address":"testactor", "command":"ON"}');
    } else {
      execute('command','{"address":"testactor", "command":"OFF"}');
    }
  }
}

With this scene, the light ‘follows’ the door/window sensor. And it’s totally immune to a particular brand or technology – cause all those are virtually interconnected by my HA system.

Here’s another one; this one switches off the pump (with a Plugwise Circle) of the floor heating when the temperature in the living room exceeds the thermostat setpoint on my Honeywell Chronotherm :

rule floorpump_off {
  when {
    c: Clock isTrue(c.inMinutes([0,30]));
    m1: Message m1.t == 'otgw/roomtemp';
    m2: Message m2.t == 'otgw/roomsetpoint';
  } then {
    unchange(c);
    log('Execute rule Floor pump off?');
    if(m1.p > m2.p){
      execute('command','{"address":"236D7E", "command":"off"}');
    }
  }
}

Especially the ‘code’ part is very nice to have – you can even call functions defined elsewhere in the rules file; nice!

As I said earlier, there were some things I had to do to make nools work; here are some of the things I had to take care of.

The amount of topics

I’ve got a MQTT root topic called value/. This topic contains all the so-called device values available in my system. They’re all retained, so by letting the rules engine subscribe to the value/# topic, the rules engine will always have access to all the last-known device values. With device values I also mean a great deal of virtual device values, values not produced by physical devices, but mostly calculated ones. Examples of those virtual device values are: GPS locations of our house and ourselves, todays usages of power and gas, position of sun, moon, house mode (eco, deepsleep, awake) and so forth.

All this adds up to a total of more than 1000 device values – so in practice there’s always something changing, almost each second – a temperature, usage value, motion, location, whatever. But nools was just too busy with all those ever-changing values it had subscribed to. So the first task for me was to reduce the number of topics it would subscribe to: no longer subscribing to the value/# MQTT topic, but only those that nools really needs, based on the contents of my rules (scenes) file. So I wrote a small routine that parses the rules file and extracts those topics being actively used in the rules file. This reduced the number of topics nools had to subscribe to 15. This was a big relief for the Raspberry Pi on which nools initially ran, cause it drastically reduced the amount of CPU cost for nools 😉

Time related stuff

In Homa, the matching of the rules with the facts was only performed when a new MQTT message was received. This made it hard to implement rules that had a time trigger.
Taking into account that I’m doing my best to keep the workload for nools as small as possible, I added some additional code so that the minimal interval of matching is set to 1 minute.
Now I could do things like shown below, which is executing a script every 4th minute of the hour:

rule bwiredxml {
  when {
    c: Clock isTrue(c.isMinute(04));
  } then {
    forget(c);
    log('Execute rule Bwired XML');
    execute('command','bwiredxml');
  }
}

Triggers & conditions

A scene is triggered by something – a door opening or closing, motion, temperature, position of the sun, your smart phone, whatever. But that’s just a small part of the story – 90% of the scenes will also need one or more conditions that need to be met for the scene to be allowed to execute. Think of cases like lowering a roller shutter while the window is open – a temperature triggers the roller shutter going down, but when the (outward opening) window is open, you don’t want that to happen of course.

Or take the sunset scene for example; it will probably turn on some lights for you (front- and back door, garden), make a couple of roller shutters go down and do some more stuff you’d otherwise do manually. Using the calculated sunset time as a trigger is not good enough, cause that won’t take into account whether it’s cloudy or not (and therefore the brightness outside & inside). OTOH, what if there’s a heavy storm with lots of dark clouds  in the middle of the day? You don’t really want the sunset scene to execute either; so just a simple light sensor won’t suffice either.

Only when the measured light outside dropping below a certain minimal value and the current time being within a certain margin of the calculated sunset must allow a scene to be executed – for that, conditions come to the rescue. This is what  a sunset rule could look like:

rule sunset {
    when {
        m1: Message m1.t == 'mbtxls1/light' && m1.droppedBelow(170);
        c : Clock;
    } then {
      if(c.hoursIsBetween(15,22)) {
       ...
       ...
       ...
      }
    }

That’s better; now the scene will only be executed when it’s getting dark and time is between 15:00 and 22:59.

Even better would be this:

rule sunset {
    when {
        m1: Message m1.t == 'mbtxls1/light' && m1.p < 170;
        c : Clock;
    } then {
      if ((abs(sunset - c) < 1800) | (c - sunset > 1800)){
       ...
       ...
       ...
      }
    }

With this rule, the sunset scene would only be executed when dusk set in and:

  • the current time is within half an hour (1800 seconds) of the calculated sunset,
  • or
  • the calculated sunset is more than half an hour ago.

That should do the trick 😉

Creating scenes that will always do the right thing is probably the hardest thing to do, cause every time I think of a new scene which should automate things for us, the first thing I realize is that I should have more sensors to make it really work – always when it should, but never when it shouldn’t; that last part is the hardest and most important, cause there’s nothing more irritating then scenes getting in your way, for example switching off lights when you don’t want that to happen…

Sticking to the facts

Homa retracts a fact (if you’ve come this far reading and don’t know what this means, start reading the nools documentation now) after it has triggered a rule; but that’s not that handy actually, cause you might wanna use that same fact (e.g. that door sensor) as a condition as well – in that case, retracting will break things. So instead of doing

  when {
    m1: Message m1.t == 'testsensor/status' && m1.p == 'open';
  } then {
    forget(m1);
    ...
  }

it’s better to do it this way:

  when {
    m1: Message m1.t == 'testsensor/status' && m1.changedTo('open');
  } then {
    unchange(m1);
    ...
  }

This will no longer retract (forget) the fact, but just disable it to trigger more than once (unchange) and keep the fact available to be used in other scenes as a condition.

One small disadvantage (the way it’s working now) is that it’s still not possible to define multiple rules with the same trigger(s) – after triggering the first fact, the fact is being modified to not trigger again, but that’s being done while more rules are waiting to be evaluated; I’ll have to come up with some sort of solution for that.

If you want to see how powerful nools really is, have a look at these (non-Home Automation related) examples – they show how great nools really is; have fun!

 Phew, it was really hard to finish a post with all that soccer on TV 😉

Adding a smart meter to the Fibaro HC2

This is (sort of) a follow-up on my previous post about the cheap Serial to Ethernet converter. It has kept me busy for a couple of evenings, for several reasons:

  • I wanted to use the Arduino Nano as a web server;
  • Make the Nano return a JSON object containing the smart meter data;
  • Learn more about the Fibaro HC2 on the job;
  • The new sketch was unstable.

It was mostly that last item that kept me busy for some time – well, waiting describes it better actually, cause the sketch I initially wrote was doing what it should, but wasn’t stable – it stopped working after 2 hours, the next time after 20 hours.. totally unpredictable. I also saw the Nano was sometimes missing incoming serial P1 data. And of course, these sort of things never happen while you’re there to witness it: I was always either at work or asleep (or both 😉 ).

So for a few days I tried to improve things and had to wait until the next day to see if the changes fixed the issue. It wasn’t lack of free RAM, it wasn’t the USB hub misbehaving, it wasn’t RF interference, nor was it a grounding issue. After I found a solution, the cause was very obvious actually: the Nano doesn’t do multitasking. Somehow taking care of receiving and processing serial data and handling an Ethernet shield interfered with each other – I still don’t know why, but after I denied the web server part of the sketch of servicing HTTP requests while the smart meter data was being received, this problem was solved. Detecting the beginning and end of a P1 packet is easy: it starts with ‘/’ and ends with ‘!’. So now the loop() looks like this:

void loop() {
  c = mySerial.read();
  switch (c) {
    case '/':
      receiving = true;
      ...
      break;
    case '!':
      receiving = false;
      ...
      break;
  }
  ...
  if(!receiving) {
    if (ether.packetLoop(ether.packetReceive())) {
      ...
      ether.httpServerReply(readingsPage());
    }
  }
}

With a P1 packet being about 500 bytes in size and 9600 baud this would ‘stall’ the response for about half a second (max.), which should not be a problem.

After I finally had a sketch that kept working for >48 hours it was time to have a look a the Fibaro HC2 and try to get the smart meter information (visibly) available in there.

Because I’ve spent very little time with the Fibaro HC2, please don’t put too much weight in what I have to say about the HC2, but: there’s one word that’s now associated with the HC2 in my head: impressive. The HC2 case feels solid, looks good, gets warm, the UI looks good, the power consumption is about 13.5 W doing nothing (I had no Z-Wave hardware to add), and the HC2 has something called Virtual Devices. For me, that’s the thing I’m interested in the most – using the HC2 is not my thing, but exploring what I can do to add my own things to it is..

So after I hooked up the HC2 to my LAN, performed some upgrades and changed some settings, I immediately focused on those Virtual Devices. P1 Smart meter in Fibaro HC2I know I’ve just scratched the surface so far, but being able to create a Virtual Device within minutes and being able to get the smart meter data from the Nano web-server into Virtual Device properties and being displayed nicely as well with just a lua script of ~20 lines of code – that’s impressive! Not bad for a beginner 😉 And for a price <20 Euro! Add a database solution, charting and I think the HC2 has a great future ahead.

 

Although I liked what I’ve seen so far, I do have some things that might be not so handy.

The lua script I wrote, was only 20 lines or so. Editing the code from within the browser, in a small window is not really as comfortable as editing code can/should be. And more important, everything you need in a script has to be in there (AFAIK) – so suppose you have to calculate a CRC, the CRC code has to be inside the script that needs it – no way of including source code files which will probably be used in more ‘main loops’ than just 1. I’d really like to see some sort of ‘Plugin’-ish way to add support for exotic hardware; a bunch of files with code, UI pages that can be embedded in the HC2 UI (e.g. for configuration stuff), creating Plugin specific triggers etcetera. In other words: really embed your own creations into HC2.

If Fibaro can accomplish that, then the HC2 can become the killer solution for Home Automation for a lot of people; with Z-Wave as the base technology, yet still expandable with support for other popular hardware.

Oh my, almost forgot the lua script, here it is:

--[[
%% properties
%% globals
--]]
fibaro:debug("start")
if (nano == nil) then
  nano = Net.FHttp("192.168.10.191", 80)
end
response, status, errorcode = nano:GET("/")
fibaro:debug(response)
local data
data = json.decode(response)
-- not 0 based!
jso = data[1]
v181  = jso.v181
v182  = jso.v182
v170  = jso.v170 * 1000
v2421 = jso.v2421
-- deviceid 7
fibaro:call(7, "setProperty", "ui.lbl181.value", v181.." kWh")
fibaro:call(7, "setProperty", "ui.lbl182.value", v182.." kWh")
fibaro:call(7, "setProperty", "ui.lbl170.value", v170.." W")
fibaro:call(7, "setProperty", "ui.lbl2421.value", v2421.." m3")
fibaro:sleep(10*1000)

Have fun!

Flexible Serial to Ethernet for less than 20 Euro

Nano v3 with Ethernet Shield

Wired connections have always been my favorite for connecting hardware to my Home Automation System. This week I found a new way of doing that – costing only 17 Euros, purchased at AliExpress.

What you see in the image above is a ENC28J60 based Ethernet shield with a Arduino Nano 3 on top of it. Measuring about 70 x 20 x 37 mm (with the lower pins cut to half their length). Very small and programmable 😉

Last Friday 2 sets (of shield & Nano) arrived and I just couldn’t resist giving them a try and since I’ve still got a Raspberry Pi near the smart meter for the sole purpose of collecting the data it produces, it looked like a good idea to see if I could replace the RPi with this Ethernet-Nano.

Finding a library for the ENC28J60 based shield wasn’t hard – I had already worked with Jean-Claude Wipplers Ethercard before and which is still in use as our doorbell controller, so that wouldn’t be any problem. But first I had to solve the problem I had when I connected the Nano to my PC- ‘Device enumeration failed’ is what USBView told me. It took a while before I got the idea to place a powered USB hub in between the two…. problem solved.

This Ethernet shield uses pin 10 for CS (Clock Select) but this was easy to change in the code; instead of

if (ether.begin(sizeof Ethernet::buffer, mymac) == 0)

all I had to do was supply an extra parameter and it all worked instantly:

if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0) 

There’s one small problem I still have to fix, which is doing a DNS query on the Nano. It times out and I have no clue why; for now I added a ‘fallback’ IP address in the code until it’s fixed.

EthernetNano test setupI used a 2nd Arduino to play the smart meter ‘role’ by sending a P1 datagram to the Nano over a serial connection with an interval of 10 seconds – that would be as close to reality as it could get. I connected the TX pin of the Arduino Ethernet (the bottom one)  to the RX pin on the EthernetNano and all that was left to do was writing a sketch.

Sometimes it looks like everything has been developed before already, the only thing you have to do is find it or remembering where you saw it – well, that was the case here also. With the Ethercard library come a couple of examples which I could use for the Ethernet part of the sketch. And I knew that Jean-Claude Wippler had blogged about P1 data a couple of times – strip the RF12 code and replace it with Ethernet code and I would be ready to go… and that’s exactly what I did 😉

Here’s the code for my first EthernetNano handling the smart meter data and uploading it (with HTTP) to my HA system! Bye bye RPi…

/// @dir p1scanner
/// Parse P1 data from smart meter and send as compressed packet over RF12.
/// @see http://jeelabs.org/2013/01/02/encoding-p1-data/
// 2012-12-31 <jc@wippler.nl> http://opensource.org/licenses/mit-license.php

// Changed to work with Ethernet shield, 2014-06 Robert Hekkers

#include <SoftwareSerial.h>
#include <EtherCard.h>

#define DEBUG 1   // set to 1 to use fake data instead of SoftwareSerial
#define LED   0   // set to 0 to disable LED blinking

SoftwareSerial mySerial (7,17 /*, true*/); // rx, tx, inverted logic
#define NTYPES (sizeof typeMap / sizeof *typeMap)
// list of codes to be sent out (only compares lower byte!)
const byte typeMap [] = {181, 182, 281, 282, 96140, 170, 270, 2410, 2420, 2440};

byte Ethernet::buffer[700];
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x00,0x01 };
const char website[] PROGMEM = "devbox.hekkers.lan";
uint8_t hisip[] = { 192,168,10,179 };
Stash stash;

byte type;
uint32_t value;
uint32_t readings[NTYPES+1];

static bool p1_scanner (char c) {
  switch (c) {
    case ':':
      type = 0;
      value = 0;
      break;
    case '(':
      if (type == 0)
        type = value; // truncates to lower byte
      value = 0;
    case '.':
      break;
    case ')':
      if (type)
        return true;
      break;
    default:
      if ('0' <= c && c <= '9')
        value = 10 * value + (c - '0');
  }
  return false;
}

static void collectData (bool empty =false) {
  if (!empty) {
    for (byte i = 0; i < NTYPES; ++i) {
      Serial.print("@ ");
      Serial.print(typeMap[i]);
      Serial.print('=');
      Serial.println(readings[i]);
    }
    byte sd = stash.create();
    stash.print("0,");
    stash.println(millis() / 1000);
    for (byte i = 0; i < NTYPES; ++i) {
      stash.print(typeMap[i]);
      stash.print(",");
      stash.println(readings[i]);
    }
    stash.save();
    // generate the header with payload - note that the stash size is used,
    // and that a "stash descriptor" is passed in as argument using "$H"
    Stash::prepare(PSTR("GET http://$F/p1/ HTTP/1.0" "rn"
                        "Host: $F" "rn"
                        "Content-Length: $D" "rn"
                        "rn"
                        "$H"),
            website, website, stash.size(), sd);
    // send the packet - this also releases all stash buffers once done
    ether.tcpSend();
  }
}

void setup () {
  if (DEBUG) {
    Serial.begin(115200);
    Serial.println("n[p1poster]");
  }
  mySerial.begin(9600);
  // digitalWrite(7, 1); // enable pull-up
  collectData(true); // empty packet on power-up

  delay(2000);
  if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
    Serial.println( "Failed to access Ethernet controller");
  if (!ether.dhcpSetup())
    Serial.println("DHCP failed");

  ether.printIp("IP:  ", ether.myip);
  ether.printIp("GW:  ", ether.gwip);
  ether.printIp("DNS: ", ether.dnsip);

  if (!ether.dnsLookup(website)) {
    Serial.println("DNS failed");
    ether.copyIp(ether.hisip, hisip);
  }
  ether.hisport = 80;
  ether.printIp("SRV: ", ether.hisip);
}

void serialLoop() {
  byte c;
  while(mySerial.available()){

    c = mySerial.read();
    if (c > 0) {
      //c &= 0x7F;
    }
    switch (c) {
      case '/':
        break;
      case '!':
        collectData();
        memset(readings, 0, sizeof readings);
        break;
      default:
        if (p1_scanner(c)) {
          for (byte i = 0; i < NTYPES; ++i)
            if (type == typeMap[i]) {
              readings[i] = value;
              break;
            }
        }
    }
  }
}

void loop () {
  ether.packetLoop(ether.packetReceive());
  serialLoop();
}

I’m sure I’m going to use this EthernetNano more in the future – the very small size, price, built-in flexibility make this a great solution for a lot of things!

 

 

 

 

 

Meet the RFXCOM RFXtrx433E

 

RFXCOM RFXtrx433EHere’s the newest addition to my ever-growing list of hardware that make my HA system to what it is: the RFXCOM RFXtrx433E USB 433.92MHz Transceiver. The grey enclosure measures 82 x 58 x 22 mm which is really small, compared to the other RFXCOM transmitters and receivers I bought back in 2007. But size doesn’t matter at all in this case – the list of supported actors and sensors seams to have exploded in the last 7 years! The same goes for the number of Home Automation Systems that support the RFXCOM products. And if the Home Automation system you use is not listed, there’s an Open Source SDK to make your own plugin/driver for RFXCOM.

RFXCOM RFXtrx433EThe RFXCOM has a USB-B connector and 2 LEDs: the left (red) one is lit while the RFXtrx is booting while the other (yellow) one will light up when the RFXtrx has received an RF transmission it could decode.

Along with the hardware comes a range of documentation and software to test, configure and use the RFXCOM products; RFXMngr is probably the most important one for most – look here for a complete list of all the available downloads.

So let’s connect this small yet very powerful transceiver to one of my SBCs and see how it works.

Since I’m into Node.JS since a year or so, the first thing I did after unpacking the transceiver was searching for a NodeJS module for it and I found this one, made by Kevin McDermott. Well, life just can’t get easier than this; install the module and a script of <100 lines of code will suffice to receive all your sensors!

RFXCOM TH sampleHere you see the information from a received RF packet, transmitted by one of my Oregon Scientific Temperature/Humidity sensors – all stored in a convenient JSON object and ready for further processing. What more can you wish for? It just doesn’t get better than this 🙂

 

Well… there is a special reason why this new RFXtrx433E is very interesting in my case; that’s because this RFXtrx433E supports the Somfy RTS protocol. That would mean I can combine an old RFXLAN receiver, an old RFXLAN transmitter and my expensive Somfy RTS485 transmitter in a single product! That would be really nice.

But for that, the reliability of the Somfy RTS protocol is crucial, so I focused on that during my first hours working with the RFXtrx433E. Today I ‘paired’ our 12 Somfy roller shutters with the RFXtrx433E so that I could use it to control those roller shutters with it. I sent a ‘PROG’ command to the Somfy RTS485 transmitter, followed by a RFXCOM RFY ‘program’ command after the roller shutter responded to the ‘PROG’ command by 2 small movements.

    switch(command.toLowerCase()){
      case 'up':
        rfy.up(deviceID);
        break;
      case 'down':
        rfy.down(deviceID);
        break;
      case 'stop':
        rfy.stop(deviceID);
        break;
      case 'program':
        rfy.program(deviceID);
        break;
      default:
        console.log('Command unknown: '+command);
        break;
    }

After that I tested whether the roller shutters obeyed the ‘up’ and ‘down’ commands sent by the RFXtrx433E and 2 hours later I was done programming all 12 roller shutters. Now I can use my new RFXtrx433E as a remote for all of ’em – what’s left to do is ’embed’ the RFXtrx433E in my HA system and see what happens during the next couple of weeks.

Exciting, although I think I already know what the result will be – cause RFXCOM products have never let me down before!