RaZberry: upgrading to 2.0.0 and modules

RaZberry on a PiI’m beginning to like the RaZberry, despite the problems I had with it the last few days. Why? Cause it’s open, you can add your own stuff to it- it’s different from all those other closed systems you see so often. Should have done more with it much sooner … I bought the RaZberry @ CeBIT 2013, ordered a Wall Plug when I got home and tinkered with both for a while when that Wall Plug arrived. But since I had no plans on buying more Z-wave hardware (it wouldn’t add anything I didn’t already have – well, problemz maybe), the RaZberry board soon ended up on a shelf cause I needed the Raspberry for something else.

Until I got my hands on a Fibaro motion sensor 2 weeks ago … this sensor was almost ‘irresistible’, I just had to use it. So that’s when I picked up the RaZberry project again to see if I could get the RazBerry integrated into my own HA system.

After I made some progress, I mailed Pieter (who also bought a RaZberry at the CeBIT and ordered the same motion sensor) and he installed my code. But he encountered strange things; weird behavior and messages like “Cannot set port to 8083: Cannot bind to port”, “Object.keys called on non-object”, needing to reboot in order to get things going again etcetera. Strange …

Browsing the web I found out that there was a new 2.0.0 firmware for the RaZberry so I decided to upgrade. Suddenly I had exactly the same problems as Pieter; so I reverted back to 1.7.2 and the problems disappeared again.. So Pieter must be running on firmware v2.0.0 – we checked this and the source of the problems was found. Something must have changed that resulted in my code crippling the whole RaZberry environment…? So I kept my RaZberry running on v1.7.2 until I had the time to find a solution for this, cause staying on v1.7.2 just wasn’t a solution.

This afternoon I had some time to resolve the issue. Although being a complete newbie on Z-Wave (and RaZberry in particular), I did realize that editing the main.js file to get my own code running was not such a good idea; an upgrade made the changes to main.js disappear so there would probably be a better way to extend the RaZberry system with my own add-ons. User Modules seemed like the way to go, as if it was made for what I was doing ;-)

So I put my code, almost unchanged, into a User module. It still didn’t work… same problems. Binding worked and the callback was still being executed. Although not completely, so something in my code was still blocking something … I figured out that it must be the blocking /ZWaveAPI/Data/… HTTP call I did, so I rewrote that one to an async version and my problems were gone. Why? I still don’t know; for that I don’t know enough (about how the RaZberry API’s work internally – but I’m glad I found a way to get my upload to my HA system working again :-).

After a while I found out that my Usermodule didn’t always start – it felt like some dependency wasn’t met after a restart (sudo service z-way-server restart) so I changed the module.json and added a autoloadPriority and set it to 9999, like this:

{
  "autoload": true,
  "singleton": false,
  "autoloadPriority":9999,
  "defaults": {
    "title": "HA system binding",
    "description": "Report sensor values to my HA system by HTTP",
    "device": ""
  }
}

Now my module will be loaded later then all the rest; found it on the Z-Wave-Me/home-automation Wiki on Github. The code is still very fresh, it’ll need some more time to prove itself, but so far everything is looking good.

BTW, how nice would it be if MQTT would become part of the RaZberry system …

The Fibaro FGMS-001 on RaZberry and Z-Wave, The Interoperable Standard ?

Well, maybe the fact that the Z-Wave logo can only be found at the bottom of the package of the FGMS-001 was an omen for another story about bad luck. Here it goes.

Fibaro FGMS-001After successfully binding to events in the Z-Way API and getting the required information into my home-brew Home Automation system, I focused on the FGMS-001.

It’s a real beauty; small, flexible, you could almost say that Fibaro reinvented the motion sensor and even managed to fit some additional sensors into the ‘ball’ which are useful @ lots of places in the house.

So I was very excited to see this sensor working. Inclusion went fine, but then the trouble started. My binding worked fine, but no sign of Temp & Lux values? I also saw an error in the z-way-server.log:

Packet CC::MULTI_CHANNEL_CMD_ENCAP_V2 is too short: required at least 5 bytes, got 3
Error returned from _zway_cc_call_handler(zway, command, controller->id, 0, data[4], &data[5]): Wrong packet from Z-Wave network or Discovery got bad data (stick communication failed) (-9)

That doesn’t look good, so I searched for the error message; according to this seems I need a better/newer XML file, so I did. Motion was working, but Temp & Lux weren’t being updated. After a while I found out that this doesn’t happen by design – you have to change some parameters in the configuration. OK, did that. Wrong – you can’t use the RaZberry Expert UI to change those values, cause it messes up the entered values – enter “10” but this is changed to “32769” once you tab your way to the next textbox. Imagine what happens after applying such a value – temperature measurement every 9 hours.

So to finally get things going, I had to do it myself from a command prompt (setting Temperature interval to 600 seconds):

wget http://192.168.10.xxx:8083/ZWaveAPI/Run/devices[7].instances[0].commandClasses[112].Set(64,600,2)

Yep, by the time I got this far the device counter had already reached 7, with just 2 sensors. And the battery is at 88% now, after almost 3 days – what was it, 2 years battery life? I’ll believe it when I see it. The FGMS-001 has lost a lot of its ‘glamour’ in just 2 days.

So what is this, a sensor from a bad batch? Still in beta stage? Z-wave.me forum blames Fibaro for not following the standards of the holy grail of Home Automation called Z-Wave. Is this true and if so, why do this? Can’t they read?  Or is there some other purpose for this kind of deviation? I don’t know and frankly it doesn’t even really matter – what it results in does matter.

Cause all these little deviations make Z-Wave unreliable, immature – you never know whether it’ll work out of the box or not. Lots of techies will be able to solve this sort of issues, but the majority won’t. Killing. And imagine the IoT, with a predicted >25 billion devices around 2020, being driven by Z-Wave … it gives me the shivers!

Razberry, binding to events and communicating device state changes

It all started about a week ago, when I became the owner of a very good looking Fibaro Motion sensor:

Fibaro FGMS-001

Compared to a Visonic motion sensor (to the right), this Fibaro is small. And very nice to see as well if I may say so. And that’s not it, there’s more inside than just motion:

  • A temperature sensor;
  • Light intensity;
  • an accelerometer to detect tampering.

And now that I’ve got one, it was time to give the Razberry, which has been unused for the last 18 months, another try.

So I took a spare Raspberry Pi, downloaded a complete image from here and started fiddling with the web-interface which is at http://<RPi-IP-address>:8083 . I have more Zwave devices than just this Fibaro FGMS-001 Motion sensor, so I chose to start with including a Fibaro Door/Window sensor, the FGK-101 – because it’s easier to test with a sensor on your desk that doesn’t require you to wave at it all the time ;-) All went well, I saw the sensor switching from on to off and vice versa.

Now how do I get this infomation into my own Home Automation system without too much hassle? I’m used to subscribing to MQTT topics lately, so I started searching for a way to catch events or hook up to ‘something’ that could help me to export device data in some way. The Developers Documentation PDF gave a clue: binding. Ah, so that’s how it’s called in Z-Way world.

The PDF says: “A very powerful function of the JS API is the ability to bind functions to certain values of the device tree. They get then executed when the value changes.” Yeah, that sounds like what I want. Lets give it a try.

To run the “binding to *everything*” code I had to make sure an additional (Java-)script was executed when the z-way-server service was started; for that I added an additional line to the end of the file /opt/z-way-server/automation/main.js file (showing just the last 3 lines here):

  });
  executeFile('bind_all.js')
}

And this bind_all.js file, which has to be placed in the same directory as the main.js we just changed, goes like this:

var callback = function(type,arg) {
  console.log('### here I am ###');
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

I restarted the z-way-server service (sudo service z-way-server restart) and moved the magnet of the Door/Window sensor away; bingo! The ‘### here I am ###’ showed up in the z-way-server log (tail -f /var/log/z-way-server.log). OK, nice.

Now what? I had a look at what others did; the only thing  I could find was executing a system() call to notify the outside world that something had changed but I didn’t like that approach; sounds like too much overhead? Getting a MQTT client running would be nice, but way out of reach with what the JS API has to offer ;-)

I preferred to keep it all wrapped up in a single script without any other dependencies, so I went back to the Developers Documentation PDF I mentioned earlier. It says: “The JavaScript implementation of Z-Way allows to directly accessing HTTP objects”. So I can do a HTTP request from inside my callback? Let’s try it. With a few lines of code I managed to POST the result of a JSON.stringify(this) to a NodeJS HTTP server that was running on my PC. Nice, and the information I was looking for was in there, great. But this POST resulted in 15 KB of data (including headers and other usual stuff), just for an opening/closing binary sensor? Come on, still not good enough!

Now the JSON API came to the rescue; again from the Developer Documentation PDF: “/ZWaveAPI/Data/<timestamp> returns an associative array of changes in Z-Way data tree since <timestamp>”

Let’s use that and POST that response to my little HTTP server. Yes! Same result (as in: it also contains the information I need) and the size has decreased to 1/9th of what it was.

About 1700 bytes to ‘communicate’ a state change of a binary sensor is still too much, but I couldn’t find any other way to do this. Maybe I’ll parse the result further inside the callback to minimize the I/O to a more ‘sane’ amount. So here’s the complete bind_all.js script that’s now communicating state changes of my Fibaro Door/Window sensor:

var callback = function(type,arg) {
 try {
   var options = {};
   // timestamp
   var ts = (new Date()).getTime();
   ts = Math.floor(ts/1000).toString();
   // query local ZWave API
   options.url = "http://127.0.0.1:8083/ZWaveAPI/Data/"+ts;
   options.method = "GET";
   var res = http.request(options);
   // send response to HA system
   options.url = "http://192.168.10.xxx:8181";
   options.method = "POST";
   options.data = JSON.stringify(res);
   res = http.request(options);
 } catch(err) {
   debugPrint("Failed to upload device state changes to HA system: " + err);
 }
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

Now back to the sensor which started all of this – the Fibaro FGMS-001.

In a 2nd post, because it’s quite a story … soon!

Solar Panels and the Omnik Wifi Kit

As announced, last Thursday we went solar!

Men at workIt took me some time to take that final step and order the solar system, cause I’ve been interested in solar energy for a long time. But now that we finally have 12 solar panels on our roof I’m very satisfied with the end result.

I decided not to do this myself cause I knew that it would take me much too long – much, much longer than the 2 professionals did. They were done in about 6 hours and left me with a fully functional solar system:

Almost finished

4 rows of 3 x JA Solar JAM6-60/280-PR panels and an Omniksol-3k-TL inverter in the garage:

Omink inverterI’m very satisfied with how it all went – from the first contact with Sungevity to the point where the solar system was working.

sungevity

 

 

 

No loose ends, everything is taken care of for you, from notifying 3rd parties that I’m gonna start producing energy soon and even the VAT refund (the EU considers you as a energy producer) is handled  for a small fee.

All I had to do was sign some papers and provide the password of my Wifi Access Point to connect the Omnik inverter to Internet. Great job guys (and gals)!

Now on to the tech part of this post: the Omnik Internal Data Collector and how to use it.

Now that we have 12 solar panels on our roof I’d really like to be able to monitor them – myself, local and not on some portal. Fortunately Sungevity made it a standard procedure to install the Omnik Wifi kit. This kit uploads the performance data to the cloud and can be viewed from the Omnik portal.

Wifi Kit inside the inverterBut it would be even better if I’d be able to receive this information directly from the Omnik Wifi kit instead of heaving to log in on some portal and probably do some web scraping to get the data.

You can see the Wifi antenna of the kit on the picture to the left, right next to the grey cable. Now the really great part of this Wifi kit is that you can add your own ‘remote servers’ from the web interface – for instance, you can add your PC (bad choice) or a Raspberry Pi (better), or whatever – just enter the right IP address and port number and you’ll receive the same information that you’ll see on the Omnik portal. (you’ll have to do the charting thing yourself though ;-)

The information looks like this (this one is invalid for obvious reasons):

68 81 41 b0 77 6f 7e 5f 77 6f 7e 5f 81 02 01 4e  h?A°wo~_wo~_?..N
4c 44 4e 33 30 32 30 31 34 35 53 32 33 35 35 00  LDN3020145S2355.
45 08 42 08 3d ff ff 00 00 00 00 ff ff 00 00 ff  E.B.=ÿÿ....ÿÿ..ÿ
ff ff ff 08 89 ff ff ff ff 13 8b 00 00 ff ff ff  ÿÿÿ.?ÿÿÿÿ.?..ÿÿÿ
ff ff ff ff ff 00 00 00 00 00 31 00 00 00 17 00  ÿÿÿÿÿ.....1.....
00 00 00 00 00 ff ff 00 00 00 00 00 00 00 00 00  .....ÿÿ.........
00 00 00 00 01 4e 4c 31 2d 56 31 2e 30 2d 30 30  .....NL1-V1.0-00
38 33 2d 34 00 00 00 00 00 56 32 2e 30 2d 30 30  83-4.....V2.0-00
32 38 00 00 00 00 00 00 00 00 00 00 00 6d 16     28...........}.

Having the luxury of knowing what the Omnik Wifi kit uploads to the cloud and being able to see what it results in on the Omnikportal should not make it too hard to figure out the ‘where is what’.

But hey, wait, I’m not the first one that can think of this – let’s do some serious searching – maybe someone already figured it all out? And yes, there are various sources, even with code. With that information I had my NodeJS script running in no-time and saw the information being received. Nice, cause now I have the same information as can be seen on the Omnik portal. But I’m not done yet, cause I wanted some additional checks on the data received from the Omnik Wifi kit.

From the source code made by others I knew that the ‘checksum’ used was just a sum of all the individual bytes in the packet. So it wasn’t so hard to find out that the packets being received used the same method; in the example packet above, the 0x6d at the end is the low byte of the sum of all preceding bytes except the first one (the start-byte, 0x68).

Another thing is that it looks like the packet has a fixed header and always ends with the checksum + stop-byte (0x16). The size of this all is 14 bytes. So with a packet of 143 bytes this leaves 143-14 = 129 (0x81) bytes for the payload – aha, the 2nd byte in the packet is the payload size! Some more validity checks ;-)

Right now my NodeJS script is running on a Cubieboard, collecting data and checking if the additional checks really work.

Have fun!

FTTH, finally

Today there’s been a mechanic to connect us to XS4All FTTH. Wow..

FTTH speed

Yep, it really is 50/50, as promised by the ISP ;-) 50/50 is the ‘lowest’ FTTH subscription in terms of bandwidth, actually it’s the same as we had with our ADSL subscription, but ADSL only gave us about 10/1 Mbps. Last year has been an almost constant ‘battle’ for bandwidth here (especially between my son and me), so it was time for an upgrade.

A good illustration of the ‘hunger’ some of us here have is the downstream chart the new FitzBox 7490 produces:

FritzDownstream

“Just some game updates…” Ah well, maybe soon it will be images for his Raspberry ;-)

Installation went fine; after succesfully testing the new Internet connection on a laptop I changed the gateway in the DHCP options of my DHCP server. And because I had all devices on a short leash, within a matter of a few hours everything switched to the new FTTH connection. As always DNS is a bit slower in understanding what’s going on, but this evening almost everything worked as it should.

On to the next adventure: next Thursday 12 solar panels will be installed on the (flat) roof of our house!

My Web-app (and more)

Since I lack the time to write a lot of text but did want to share some thing going on here, I thought it would be nice to share some images of my Web-app which I’ve developed. I started with it in 2012 but most of the work has been done in some spare time since the last half year.

Below the first and last page I made: the Home screen and a handy page with just the five most important buttons while relaxing on the couch, watching TV after another day of work.

The icons on the Home screen take you to pages for controlling lights, roller shutters, thermostat, garage door, front door lock, robot vacuum cleaner, alarm, what’s left, info pages, TV and house modes & settings.

Home screen   Watching TV

The first remark after I deployed the web-app to my wife and kids was: “it’s nice that I can control everything, but I can’t see what’s happening at home while I’m away” – that’s what some of the pages below have been added as well:

Doors and windows   Motion

Temperatuees, humidity, energy consumption; the latter one helped me to conclude that the thermostat of an old refrigerator was indeed broken.

Temperatures   Energy panel

Roller shutters, lights and some various things…

Roller shutters   Misc. devices

As you can see: just a bunch of simple icons, no flashy colors, animations or other distracting things – hey, I’m no graphic designer and I don’t wanna be – much too hard for me to do… but it’s working great and that’s the only thing that really matters to me ;-)

More

The more part in the title is about a couple of things that will all happen very soon and which are all just as exciting and worthy enough to get their own dedicated posts:

  • finally, in about 3 weeks we’ll have 12 solar panels on our roof, the Node script for monitoring the inverter is already on stand-by :-)
  • fiber Internet connection of 50/50 Mbit should be functional around the same time, working on a real router to take over the function the Fritz!Box 7490;
  • this WordPress blog will probably be moved to an Odroid or Banana Pi;
  • I’m also switching from Windows-based hMailserver to Postfix/Dovecot (+ add-ons)

Somewhere in December all of this should be fully functional – you’ll read all about it in the near future!

Again… and at the wrong time!

This is the 3rd time in ~1.5 year that a HDD in my primary NAS (a Synology DS209) crashes:

Volume 2 crashed

I do have a backup of the most important stuff that’s in those 903.5 GBytes , but preserving it all is even better, so I’m trying to save as much as I can right now.

And the timing of this couldn’t be worse – I’m quite busy the last few months (which is also the reason why I haven’t posted for more than 3 months) and I just can’t have this kind of unplanned, time-consuming, non-productive jobs right now.. ah well, I’ll just have to live with the fact that I can’t control everything ;-)

Making a scene

Being able to define scenes is one of the most important aspects in a Home Automation (HA) system. Not being able to define your own scenes, things you want to happen automatically, degrades a HA system to nothing more than a big expensive monitoring system or just a remote control. That’s not Home Automation, that’s Home Control!

ReteScenes let you automate certain things. Lowering the roller shutters so that the temperature inside doesn’t become unpleasantly high, switching lights on or off based on motion, sunset, sunrise, sending notifications to your smart phone, etcetera. Anything is possible, as long as your HA system is equipped with the right sensors (information) to trigger that specific scene you’d like it to execute for you.

When I started (re)developing my system in NodeJS, one of the first items on my to-do list was finding some sort of starting point to create my own scenes. Preferably developed for Node.JS, with its own DSL (Domain Specific Language), the ability to embed simple if/else statements and calculations inside the scene definitions and crontab-ish triggered scenes were my most important conditions to look for. When I looked at HomA (a NodeJS based framework for building smart homes) when I started exploring whether NodeJS was a good choice for rewriting my HA system, I noticed that it uses nools as a rules engine; Nools is a rete based rules engine and when I read the documentation it looked like a good starting point. HomA also provided a good starting point in terms of combining MQTT & nools.

In fact, I shamelessly copied this file from the HomA repository and added the things I missed and changed what I wanted to do different. Here’s a couple of things worth mentioning about how I integrated nools in my HA system and how I got scenes to actually automate things.

Lets have a quick look at how easy it is to use nools as the scene engine or rules engine. I’ll take some of my own scenes as example.

rule testrule {
  when {
    m1: Message m1.t == 'testsensor/status' && m1.changedTo('open');
  }
  then {
    unchange(m1);
    log('Execute rule Office testlight on');
    execute('command','{"address":"testactor", "command":"ON"}');
 }
}

Easy right?… when the status of the testsensor (in this case a Visonic door/window sensor) on my desk changes to ‘open’ , the testactor (a PLCBUS on/off module) which is also on my desk, will switch on a light.

But with this scene, when I close the sensor again, nothing happens. I would need a second rule to switch off the light when the sensor changes back to ‘closed’, in a similar way as with the rule above.

But that won’t be necessary, cause I can add some code to the rule, like this:

rule testrule {
  when {
    m1: Message m1.t == 'testsensor/status' && m1.changed;
  } then {
    unchange(m1);
    log('Execute rule Office test open');
    if (m1.p === 'open') {
      execute('command','{"address":"testactor", "command":"ON"}');
    } else {
      execute('command','{"address":"testactor", "command":"OFF"}');
    }
  }
}

With this scene, the light ‘follows’ the door/window sensor. And it’s totally immune to a particular brand or technology – cause all those are virtually interconnected by my HA system.

Here’s another one; this one switches off the pump (with a Plugwise Circle) of the floor heating when the temperature in the living room exceeds the thermostat setpoint on my Honeywell Chronotherm :

rule floorpump_off {
  when {
    c: Clock isTrue(c.inMinutes([0,30]));
    m1: Message m1.t == 'otgw/roomtemp';
    m2: Message m2.t == 'otgw/roomsetpoint';
  } then {
    unchange(c);
    log('Execute rule Floor pump off?');
    if(m1.p > m2.p){
      execute('command','{"address":"236D7E", "command":"off"}');
    }
  }
}

Especially the ‘code’ part is very nice to have – you can even call functions defined elsewhere in the rules file; nice!

As I said earlier, there were some things I had to do to make nools work; here are some of the things I had to take care of.

The amount of topics

I’ve got a MQTT root topic called value/. This topic contains all the so-called device values available in my system. They’re all retained, so by letting the rules engine subscribe to the value/# topic, the rules engine will always have access to all the last-known device values. With device values I also mean a great deal of virtual device values, values not produced by physical devices, but mostly calculated ones. Examples of those virtual device values are: GPS locations of our house and ourselves, todays usages of power and gas, position of sun, moon, house mode (eco, deepsleep, awake) and so forth.

All this adds up to a total of more than 1000 device values – so in practice there’s always something changing, almost each second – a temperature, usage value, motion, location, whatever. But nools was just too busy with all those ever-changing values it had subscribed to. So the first task for me was to reduce the number of topics it would subscribe to: no longer subscribing to the value/# MQTT topic, but only those that nools really needs, based on the contents of my rules (scenes) file. So I wrote a small routine that parses the rules file and extracts those topics being actively used in the rules file. This reduced the number of topics nools had to subscribe to 15. This was a big relief for the Raspberry Pi on which nools initially ran, cause it drastically reduced the amount of CPU cost for nools ;-)

Time related stuff

In Homa, the matching of the rules with the facts was only performed when a new MQTT message was received. This made it hard to implement rules that had a time trigger.
Taking into account that I’m doing my best to keep the workload for nools as small as possible, I added some additional code so that the minimal interval of matching is set to 1 minute.
Now I could do things like shown below, which is executing a script every 4th minute of the hour:

rule bwiredxml {
  when {
    c: Clock isTrue(c.isMinute(04));
  } then {
    forget(c);
    log('Execute rule Bwired XML');
    execute('command','bwiredxml');
  }
}

Triggers & conditions

A scene is triggered by something – a door opening or closing, motion, temperature, position of the sun, your smart phone, whatever. But that’s just a small part of the story – 90% of the scenes will also need one or more conditions that need to be met for the scene to be allowed to execute. Think of cases like lowering a roller shutter while the window is open – a temperature triggers the roller shutter going down, but when the (outward opening) window is open, you don’t want that to happen of course.

Or take the sunset scene for example; it will probably turn on some lights for you (front- and back door, garden), make a couple of roller shutters go down and do some more stuff you’d otherwise do manually. Using the calculated sunset time as a trigger is not good enough, cause that won’t take into account whether it’s cloudy or not (and therefore the brightness outside & inside). OTOH, what if there’s a heavy storm with lots of dark clouds  in the middle of the day? You don’t really want the sunset scene to execute either; so just a simple light sensor won’t suffice either.

Only when the measured light outside dropping below a certain minimal value and the current time being within a certain margin of the calculated sunset must allow a scene to be executed – for that, conditions come to the rescue. This is what  a sunset rule could look like:

rule sunset {
    when {
        m1: Message m1.t == 'mbtxls1/light' && m1.droppedBelow(170);
        c : Clock;
    } then {
      if(c.hoursIsBetween(15,22)) {
       ...
       ...
       ...
      }
    }

That’s better; now the scene will only be executed when it’s getting dark and time is between 15:00 and 22:59.

Even better would be this:

rule sunset {
    when {
        m1: Message m1.t == 'mbtxls1/light' && m1.p < 170;
        c : Clock;
    } then {
      if ((abs(sunset - c) < 1800) | (c - sunset > 1800)){
       ...
       ...
       ...
      }
    }

With this rule, the sunset scene would only be executed when dusk set in and:

  • the current time is within half an hour (1800 seconds) of the calculated sunset,
  • or
  • the calculated sunset is more than half an hour ago.

That should do the trick ;-)

Creating scenes that will always do the right thing is probably the hardest thing to do, cause every time I think of a new scene which should automate things for us, the first thing I realize is that I should have more sensors to make it really work – always when it should, but never when it shouldn’t; that last part is the hardest and most important, cause there’s nothing more irritating then scenes getting in your way, for example switching off lights when you don’t want that to happen…

Sticking to the facts

Homa retracts a fact (if you’ve come this far reading and don’t know what this means, start reading the nools documentation now) after it has triggered a rule; but that’s not that handy actually, cause you might wanna use that same fact (e.g. that door sensor) as a condition as well – in that case, retracting will break things. So instead of doing

  when {
    m1: Message m1.t == 'testsensor/status' && m1.p == 'open';
  } then {
    forget(m1);
    ...
  }

it’s better to do it this way:

  when {
    m1: Message m1.t == 'testsensor/status' && m1.changedTo('open');
  } then {
    unchange(m1);
    ...
  }

This will no longer retract (forget) the fact, but just disable it to trigger more than once (unchange) and keep the fact available to be used in other scenes as a condition.

One small disadvantage (the way it’s working now) is that it’s still not possible to define multiple rules with the same trigger(s) – after triggering the first fact, the fact is being modified to not trigger again, but that’s being done while more rules are waiting to be evaluated; I’ll have to come up with some sort of solution for that.

If you want to see how powerful nools really is, have a look at these (non-Home Automation related) examples – they show how great nools really is; have fun!

 Phew, it was really hard to finish a post with all that soccer on TV ;-)

Adding a smart meter to the Fibaro HC2

This is (sort of) a follow-up on my previous post about the cheap Serial to Ethernet converter. It has kept me busy for a couple of evenings, for several reasons:

  • I wanted to use the Arduino Nano as a web server;
  • Make the Nano return a JSON object containing the smart meter data;
  • Learn more about the Fibaro HC2 on the job;
  • The new sketch was unstable.

It was mostly that last item that kept me busy for some time – well, waiting describes it better actually, cause the sketch I initially wrote was doing what it should, but wasn’t stable – it stopped working after 2 hours, the next time after 20 hours.. totally unpredictable. I also saw the Nano was sometimes missing incoming serial P1 data. And of course, these sort of things never happen while you’re there to witness it: I was always either at work or asleep (or both ;-) ).

So for a few days I tried to improve things and had to wait until the next day to see if the changes fixed the issue. It wasn’t lack of free RAM, it wasn’t the USB hub misbehaving, it wasn’t RF interference, nor was it a grounding issue. After I found a solution, the cause was very obvious actually: the Nano doesn’t do multitasking. Somehow taking care of receiving and processing serial data and handling an Ethernet shield interfered with each other – I still don’t know why, but after I denied the web server part of the sketch of servicing HTTP requests while the smart meter data was being received, this problem was solved. Detecting the beginning and end of a P1 packet is easy: it starts with ‘/’ and ends with ‘!’. So now the loop() looks like this:

void loop() {
  c = mySerial.read();
  switch (c) {
    case '/':
      receiving = true;
      ...
      break;
    case '!':
      receiving = false;
      ...
      break;
  }
  ...
  if(!receiving) {
    if (ether.packetLoop(ether.packetReceive())) {
      ...
      ether.httpServerReply(readingsPage());
    }
  }
}

With a P1 packet being about 500 bytes in size and 9600 baud this would ‘stall’ the response for about half a second (max.), which should not be a problem.

After I finally had a sketch that kept working for >48 hours it was time to have a look a the Fibaro HC2 and try to get the smart meter information (visibly) available in there.

Because I’ve spent very little time with the Fibaro HC2, please don’t put too much weight in what I have to say about the HC2, but: there’s one word that’s now associated with the HC2 in my head: impressive. The HC2 case feels solid, looks good, gets warm, the UI looks good, the power consumption is about 13.5 W doing nothing (I had no Z-Wave hardware to add), and the HC2 has something called Virtual Devices. For me, that’s the thing I’m interested in the most – using the HC2 is not my thing, but exploring what I can do to add my own things to it is..

So after I hooked up the HC2 to my LAN, performed some upgrades and changed some settings, I immediately focused on those Virtual Devices. P1 Smart meter in Fibaro HC2I know I’ve just scratched the surface so far, but being able to create a Virtual Device within minutes and being able to get the smart meter data from the Nano web-server into Virtual Device properties and being displayed nicely as well with just a lua script of ~20 lines of code – that’s impressive! Not bad for a beginner ;-) And for a price <20 Euro! Add a database solution, charting and I think the HC2 has a great future ahead.

 

Although I liked what I’ve seen so far, I do have some things that might be not so handy.

The lua script I wrote, was only 20 lines or so. Editing the code from within the browser, in a small window is not really as comfortable as editing code can/should be. And more important, everything you need in a script has to be in there (AFAIK) – so suppose you have to calculate a CRC, the CRC code has to be inside the script that needs it – no way of including source code files which will probably be used in more ‘main loops’ than just 1. I’d really like to see some sort of ‘Plugin’-ish way to add support for exotic hardware; a bunch of files with code, UI pages that can be embedded in the HC2 UI (e.g. for configuration stuff), creating Plugin specific triggers etcetera. In other words: really embed your own creations into HC2.

If Fibaro can accomplish that, then the HC2 can become the killer solution for Home Automation for a lot of people; with Z-Wave as the base technology, yet still expandable with support for other popular hardware.

Oh my, almost forgot the lua script, here it is:

--[[
%% properties
%% globals
--]]
fibaro:debug("start")
if (nano == nil) then
  nano = Net.FHttp("192.168.10.191", 80)
end
response, status, errorcode = nano:GET("/")
fibaro:debug(response)
local data
data = json.decode(response)
-- not 0 based!
jso = data[1]
v181  = jso.v181
v182  = jso.v182
v170  = jso.v170 * 1000
v2421 = jso.v2421
-- deviceid 7
fibaro:call(7, "setProperty", "ui.lbl181.value", v181.." kWh")
fibaro:call(7, "setProperty", "ui.lbl182.value", v182.." kWh")
fibaro:call(7, "setProperty", "ui.lbl170.value", v170.." W")
fibaro:call(7, "setProperty", "ui.lbl2421.value", v2421.." m3")
fibaro:sleep(10*1000)

Have fun!

Flexible Serial to Ethernet for less than 20 Euro

Nano v3 with Ethernet Shield

Wired connections have always been my favorite for connecting hardware to my Home Automation System. This week I found a new way of doing that – costing only 17 Euros, purchased at AliExpress.

What you see in the image above is a ENC28J60 based Ethernet shield with a Arduino Nano 3 on top of it. Measuring about 70 x 20 x 37 mm (with the lower pins cut to half their length). Very small and programmable ;-)

Last Friday 2 sets (of shield & Nano) arrived and I just couldn’t resist giving them a try and since I’ve still got a Raspberry Pi near the smart meter for the sole purpose of collecting the data it produces, it looked like a good idea to see if I could replace the RPi with this Ethernet-Nano.

Finding a library for the ENC28J60 based shield wasn’t hard – I had already worked with Jean-Claude Wipplers Ethercard before and which is still in use as our doorbell controller, so that wouldn’t be any problem. But first I had to solve the problem I had when I connected the Nano to my PC- ‘Device enumeration failed’ is what USBView told me. It took a while before I got the idea to place a powered USB hub in between the two…. problem solved.

This Ethernet shield uses pin 10 for CS (Clock Select) but this was easy to change in the code; instead of

if (ether.begin(sizeof Ethernet::buffer, mymac) == 0)

all I had to do was supply an extra parameter and it all worked instantly:

if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0) 

There’s one small problem I still have to fix, which is doing a DNS query on the Nano. It times out and I have no clue why; for now I added a ‘fallback’ IP address in the code until it’s fixed.

EthernetNano test setupI used a 2nd Arduino to play the smart meter ‘role’ by sending a P1 datagram to the Nano over a serial connection with an interval of 10 seconds – that would be as close to reality as it could get. I connected the TX pin of the Arduino Ethernet (the bottom one)  to the RX pin on the EthernetNano and all that was left to do was writing a sketch.

Sometimes it looks like everything has been developed before already, the only thing you have to do is find it or remembering where you saw it – well, that was the case here also. With the Ethercard library come a couple of examples which I could use for the Ethernet part of the sketch. And I knew that Jean-Claude Wippler had blogged about P1 data a couple of times – strip the RF12 code and replace it with Ethernet code and I would be ready to go… and that’s exactly what I did ;-)

Here’s the code for my first EthernetNano handling the smart meter data and uploading it (with HTTP) to my HA system! Bye bye RPi…

/// @dir p1scanner
/// Parse P1 data from smart meter and send as compressed packet over RF12.
/// @see http://jeelabs.org/2013/01/02/encoding-p1-data/
// 2012-12-31 <jc@wippler.nl> http://opensource.org/licenses/mit-license.php

// Changed to work with Ethernet shield, 2014-06 Robert Hekkers

#include <SoftwareSerial.h>
#include <EtherCard.h>

#define DEBUG 1   // set to 1 to use fake data instead of SoftwareSerial
#define LED   0   // set to 0 to disable LED blinking

SoftwareSerial mySerial (7,17 /*, true*/); // rx, tx, inverted logic
#define NTYPES (sizeof typeMap / sizeof *typeMap)
// list of codes to be sent out (only compares lower byte!)
const byte typeMap [] = {181, 182, 281, 282, 96140, 170, 270, 2410, 2420, 2440};

byte Ethernet::buffer[700];
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x00,0x01 };
const char website[] PROGMEM = "devbox.hekkers.lan";
uint8_t hisip[] = { 192,168,10,179 };
Stash stash;

byte type;
uint32_t value;
uint32_t readings[NTYPES+1];

static bool p1_scanner (char c) {
  switch (c) {
    case ':':
      type = 0;
      value = 0;
      break;
    case '(':
      if (type == 0)
        type = value; // truncates to lower byte
      value = 0;
    case '.':
      break;
    case ')':
      if (type)
        return true;
      break;
    default:
      if ('0' <= c && c <= '9')
        value = 10 * value + (c - '0');
  }
  return false;
}

static void collectData (bool empty =false) {
  if (!empty) {
    for (byte i = 0; i < NTYPES; ++i) {
      Serial.print("@ ");
      Serial.print(typeMap[i]);
      Serial.print('=');
      Serial.println(readings[i]);
    }
    byte sd = stash.create();
    stash.print("0,");
    stash.println(millis() / 1000);
    for (byte i = 0; i < NTYPES; ++i) {
      stash.print(typeMap[i]);
      stash.print(",");
      stash.println(readings[i]);
    }
    stash.save();
    // generate the header with payload - note that the stash size is used,
    // and that a "stash descriptor" is passed in as argument using "$H"
    Stash::prepare(PSTR("GET http://$F/p1/ HTTP/1.0" "\r\n"
                        "Host: $F" "\r\n"
                        "Content-Length: $D" "\r\n"
                        "\r\n"
                        "$H"),
            website, website, stash.size(), sd);
    // send the packet - this also releases all stash buffers once done
    ether.tcpSend();
  }
}

void setup () {
  if (DEBUG) {
    Serial.begin(115200);
    Serial.println("\n[p1poster]");
  }
  mySerial.begin(9600);
  // digitalWrite(7, 1); // enable pull-up
  collectData(true); // empty packet on power-up

  delay(2000);
  if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
    Serial.println( "Failed to access Ethernet controller");
  if (!ether.dhcpSetup())
    Serial.println("DHCP failed");

  ether.printIp("IP:  ", ether.myip);
  ether.printIp("GW:  ", ether.gwip);
  ether.printIp("DNS: ", ether.dnsip);  

  if (!ether.dnsLookup(website)) {
    Serial.println("DNS failed");
    ether.copyIp(ether.hisip, hisip);
  }
  ether.hisport = 80;
  ether.printIp("SRV: ", ether.hisip);
}

void serialLoop() {
  byte c;
  while(mySerial.available()){

    c = mySerial.read();
    if (c > 0) {
      //c &= 0x7F;
    }
    switch (c) {
      case '/':
        break;
      case '!':
        collectData();
        memset(readings, 0, sizeof readings);
        break;
      default:
        if (p1_scanner(c)) {
          for (byte i = 0; i < NTYPES; ++i)
            if (type == typeMap[i]) {
              readings[i] = value;
              break;
            }
        }
    }
  }
}

void loop () {
  ether.packetLoop(ether.packetReceive());
  serialLoop();
}

I’m sure I’m going to use this EthernetNano more in the future – the very small size, price, built-in flexibility make this a great solution for a lot of things!