RaZberry: upgrading to 2.0.0 and modules part 3

Now that I finally had a working setup with RazBerry on 2.0.0 I could start doing some serious logging. Not like seeing the results on the screen for a few minutes and concluding it’s all OK, but logging everything to a text file and analyzing the results during a period that could easily take 2 or 3 weeks. I always do that before adding a new type/kind of hardware to my system – it has to be 100% reliable, because I don’t like searching for an answer to questions like “Why didn’t that Visonic sensor trigger a rule in my rules engine to set the brightness of that Insteon controlled light bulb to 20%?”.

But this time it only took 24 hours to see something was wrong here. And it’s not the result of some bug in the module, but coming directly from the RaZberry – and that’s what worries me the most, actually. Here’s what went wrong during the last 48 hours, all very wrong sensor values:

Temp Sensor: 7.0 = -2971.1
Lux Sensor: 7.0 = -9463
Lux Sensor: 7.0 = 0.0601529
Lux Sensor: 7.0 = -28353
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 16.7
Temp Sensor: 7.0 = -1742.8
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 23.5
Lux Sensor: 7.0 = 30569
Temp Sensor: 7.0 = 21.9
Temp Sensor: 7.0 = -309.6
Temp Sensor: 7.0 = 15220400
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 3
Lux Sensor: 7.0 = 0.0006696
Lux Sensor: 7.0 = 26
Temp Sensor: 7.0 = 5.8
Temp Sensor: 7.0 = 2198.9
Temp Sensor: 7.0 = 15028.7
Temp Sensor: 7.0 = 23
Lux Sensor: 7.0 = 1758910
Lux Sensor: 7.0 = 17664
Temp Sensor: 7.0 = -2664.5
Lux Sensor: 7.0 = 29475
Lux Sensor: 7.0 = 20798
Lux Sensor: 7.0 = 0.16683
Lux Sensor: 7.0 = 0.017478
Lux Sensor: 7.0 = 4.47507
Lux Sensor: 7.0 = -31466
Power Sensor: 8.0 = 6302.02
Temp Sensor: 7.0 = 60.757
Lux Sensor: 7.0 = 4893
Lux Sensor: 7.0 = 1950140
Temp Sensor: 7.0 = -565.1
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 0.413177
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 113
Lux Sensor: 7.0 = 12608
Lux Sensor: 7.0 = 1657
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = -16060
Lux Sensor: 7.0 = 0.017486
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Temp Sensor: 7.0 = 14950600

Line 1: no comment. Line 2: lights will be switched on, roller shutters will go down. Line 13: fire alarm! Line 23 during daytime: roller shutters will go down because the room will soon be warmed up too much by sunlight. And what about lines 52 and 53: where the hell do they come from?? Need I go on? With this poor result, I don’t even trust the binary (true/false) sensors anymore…

Mister Bwired experienced the same with identical hardware & module – also strange values. And something similar was also mentioned  the z-wave.me forum. Same issue? Hmm. I was ready to stop with all this ZWave stuff; back to hardware that does work.

In the meantime René Klootwijk, who’s also creating his own Home Automation system, also developed a module to interface with the RaZberry. His module can be used to forward events from zway to your own server using a REST API. René was my last hope actually; his module worked different from mine so maybe the results were different also (as in: better)? I tried his module during the last 24 hours and guess what: perfect! I can’t explain why and I’m not even gonna spend time on it – it’s working, so be happy! All you have to do is create a little webserver to receive the information, add some MQTT and  you’re done:

RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":3,"vDevId":"ZWayVDev_zway_7-0-49-3","value":32,"timestamp":1422388041}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388068}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"on","timestamp":1422388104}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-49-1","value":23.7,"timestamp":1422388154}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388170}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"on","timestamp":1422388631}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"off","timestamp":1422388634}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.8,"timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"off","timestamp":1422388683}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"on","timestamp":1422388685}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"off","timestamp":1422388687}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"on","timestamp":1422388689}
RECEIVED: {"status":"online","nodeId":9,"instanceId":2,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-2-37","value":"off","timestamp":1422388691}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.6,"timestamp":1422388692}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.7,"timestamp":1422388693}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388694}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388695}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388696}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388697}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388701}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388701}

Now the only thing that still bothers me is the battery which was empty after 2 weeks…

RaZberry: upgrading to 2.0.0 and modules

RaZberry on a PiI’m beginning to like the RaZberry, despite the problems I had with it the last few days. Why? Cause it’s open, you can add your own stuff to it- it’s different from all those other closed systems you see so often. Should have done more with it much sooner … I bought the RaZberry @ CeBIT 2013, ordered a Wall Plug when I got home and tinkered with both for a while when that Wall Plug arrived. But since I had no plans on buying more Z-wave hardware (it wouldn’t add anything I didn’t already have – well, problemz maybe), the RaZberry board soon ended up on a shelf cause I needed the Raspberry for something else.

Until I got my hands on a Fibaro motion sensor 2 weeks ago … this sensor was almost ‘irresistible’, I just had to use it. So that’s when I picked up the RaZberry project again to see if I could get the RazBerry integrated into my own HA system.

After I made some progress, I mailed Pieter (who also bought a RaZberry at the CeBIT and ordered the same motion sensor) and he installed my code. But he encountered strange things; weird behavior and messages like “Cannot set port to 8083: Cannot bind to port”, “Object.keys called on non-object”, needing to reboot in order to get things going again etcetera. Strange …

Browsing the web I found out that there was a new 2.0.0 firmware for the RaZberry so I decided to upgrade. Suddenly I had exactly the same problems as Pieter; so I reverted back to 1.7.2 and the problems disappeared again.. So Pieter must be running on firmware v2.0.0 – we checked this and the source of the problems was found. Something must have changed that resulted in my code crippling the whole RaZberry environment…? So I kept my RaZberry running on v1.7.2 until I had the time to find a solution for this, cause staying on v1.7.2 just wasn’t a solution.

This afternoon I had some time to resolve the issue. Although being a complete newbie on Z-Wave (and RaZberry in particular), I did realize that editing the main.js file to get my own code running was not such a good idea; an upgrade made the changes to main.js disappear so there would probably be a better way to extend the RaZberry system with my own add-ons. User Modules seemed like the way to go, as if it was made for what I was doing 😉

So I put my code, almost unchanged, into a User module. It still didn’t work… same problems. Binding worked and the callback was still being executed. Although not completely, so something in my code was still blocking something … I figured out that it must be the blocking /ZWaveAPI/Data/… HTTP call I did, so I rewrote that one to an async version and my problems were gone. Why? I still don’t know; for that I don’t know enough (about how the RaZberry API’s work internally – but I’m glad I found a way to get my upload to my HA system working again :-).

After a while I found out that my Usermodule didn’t always start – it felt like some dependency wasn’t met after a restart (sudo service z-way-server restart) so I changed the module.json and added a autoloadPriority and set it to 9999, like this:

{
  "autoload": true,
  "singleton": false,
  "autoloadPriority":9999,
  "defaults": {
    "title": "HA system binding",
    "description": "Report sensor values to my HA system by HTTP",
    "device": ""
  }
}

Now my module will be loaded later then all the rest; found it on the Z-Wave-Me/home-automation Wiki on Github. The code is still very fresh, it’ll need some more time to prove itself, but so far everything is looking good.

BTW, how nice would it be if MQTT would become part of the RaZberry system …

The Fibaro FGMS-001 on RaZberry and Z-Wave, The Interoperable Standard ?

Well, maybe the fact that the Z-Wave logo can only be found at the bottom of the package of the FGMS-001 was an omen for another story about bad luck. Here it goes.

Fibaro FGMS-001After successfully binding to events in the Z-Way API and getting the required information into my home-brew Home Automation system, I focused on the FGMS-001.

It’s a real beauty; small, flexible, you could almost say that Fibaro reinvented the motion sensor and even managed to fit some additional sensors into the ‘ball’ which are useful @ lots of places in the house.

So I was very excited to see this sensor working. Inclusion went fine, but then the trouble started. My binding worked fine, but no sign of Temp & Lux values? I also saw an error in the z-way-server.log:

Packet CC::MULTI_CHANNEL_CMD_ENCAP_V2 is too short: required at least 5 bytes, got 3
Error returned from _zway_cc_call_handler(zway, command, controller->id, 0, data[4], &data[5]): Wrong packet from Z-Wave network or Discovery got bad data (stick communication failed) (-9)

That doesn’t look good, so I searched for the error message; according to this seems I need a better/newer XML file, so I did. Motion was working, but Temp & Lux weren’t being updated. After a while I found out that this doesn’t happen by design – you have to change some parameters in the configuration. OK, did that. Wrong – you can’t use the RaZberry Expert UI to change those values, cause it messes up the entered values – enter “10” but this is changed to “32769” once you tab your way to the next textbox. Imagine what happens after applying such a value – temperature measurement every 9 hours.

So to finally get things going, I had to do it myself from a command prompt (setting Temperature interval to 600 seconds):

wget http://192.168.10.xxx:8083/ZWaveAPI/Run/devices[7].instances[0].commandClasses[112].Set(64,600,2)

Yep, by the time I got this far the device counter had already reached 7, with just 2 sensors. And the battery is at 88% now, after almost 3 days – what was it, 2 years battery life? I’ll believe it when I see it. The FGMS-001 has lost a lot of its ‘glamour’ in just 2 days.

So what is this, a sensor from a bad batch? Still in beta stage? Z-wave.me forum blames Fibaro for not following the standards of the holy grail of Home Automation called Z-Wave. Is this true and if so, why do this? Can’t they read?  Or is there some other purpose for this kind of deviation? I don’t know and frankly it doesn’t even really matter – what it results in does matter.

Cause all these little deviations make Z-Wave unreliable, immature – you never know whether it’ll work out of the box or not. Lots of techies will be able to solve this sort of issues, but the majority won’t. Killing. And imagine the IoT, with a predicted >25 billion devices around 2020, being driven by Z-Wave … it gives me the shivers!

Razberry, binding to events and communicating device state changes

It all started about a week ago, when I became the owner of a very good looking Fibaro Motion sensor:

Fibaro FGMS-001

Compared to a Visonic motion sensor (to the right), this Fibaro is small. And very nice to see as well if I may say so. And that’s not it, there’s more inside than just motion:

  • A temperature sensor;
  • Light intensity;
  • an accelerometer to detect tampering.

And now that I’ve got one, it was time to give the Razberry, which has been unused for the last 18 months, another try.

So I took a spare Raspberry Pi, downloaded a complete image from here and started fiddling with the web-interface which is at http://<RPi-IP-address>:8083 . I have more Zwave devices than just this Fibaro FGMS-001 Motion sensor, so I chose to start with including a Fibaro Door/Window sensor, the FGK-101 – because it’s easier to test with a sensor on your desk that doesn’t require you to wave at it all the time 😉 All went well, I saw the sensor switching from on to off and vice versa.

Now how do I get this infomation into my own Home Automation system without too much hassle? I’m used to subscribing to MQTT topics lately, so I started searching for a way to catch events or hook up to ‘something’ that could help me to export device data in some way. The Developers Documentation PDF gave a clue: binding. Ah, so that’s how it’s called in Z-Way world.

The PDF says: “A very powerful function of the JS API is the ability to bind functions to certain values of the device tree. They get then executed when the value changes.” Yeah, that sounds like what I want. Lets give it a try.

To run the “binding to *everything*” code I had to make sure an additional (Java-)script was executed when the z-way-server service was started; for that I added an additional line to the end of the file /opt/z-way-server/automation/main.js file (showing just the last 3 lines here):

  });
  executeFile('bind_all.js')
}

And this bind_all.js file, which has to be placed in the same directory as the main.js we just changed, goes like this:

var callback = function(type,arg) {
  console.log('### here I am ###');
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

I restarted the z-way-server service (sudo service z-way-server restart) and moved the magnet of the Door/Window sensor away; bingo! The ‘### here I am ###’ showed up in the z-way-server log (tail -f /var/log/z-way-server.log). OK, nice.

Now what? I had a look at what others did; the only thing  I could find was executing a system() call to notify the outside world that something had changed but I didn’t like that approach; sounds like too much overhead? Getting a MQTT client running would be nice, but way out of reach with what the JS API has to offer 😉

I preferred to keep it all wrapped up in a single script without any other dependencies, so I went back to the Developers Documentation PDF I mentioned earlier. It says: “The JavaScript implementation of Z-Way allows to directly accessing HTTP objects”. So I can do a HTTP request from inside my callback? Let’s try it. With a few lines of code I managed to POST the result of a JSON.stringify(this) to a NodeJS HTTP server that was running on my PC. Nice, and the information I was looking for was in there, great. But this POST resulted in 15 KB of data (including headers and other usual stuff), just for an opening/closing binary sensor? Come on, still not good enough!

Now the JSON API came to the rescue; again from the Developer Documentation PDF: “/ZWaveAPI/Data/<timestamp> returns an associative array of changes in Z-Way data tree since <timestamp>”

Let’s use that and POST that response to my little HTTP server. Yes! Same result (as in: it also contains the information I need) and the size has decreased to 1/9th of what it was.

About 1700 bytes to ‘communicate’ a state change of a binary sensor is still too much, but I couldn’t find any other way to do this. Maybe I’ll parse the result further inside the callback to minimize the I/O to a more ‘sane’ amount. So here’s the complete bind_all.js script that’s now communicating state changes of my Fibaro Door/Window sensor:

var callback = function(type,arg) {
 try {
   var options = {};
   // timestamp
   var ts = (new Date()).getTime();
   ts = Math.floor(ts/1000).toString();
   // query local ZWave API
   options.url = "http://127.0.0.1:8083/ZWaveAPI/Data/"+ts;
   options.method = "GET";
   var res = http.request(options);
   // send response to HA system
   options.url = "http://192.168.10.xxx:8181";
   options.method = "POST";
   options.data = JSON.stringify(res);
   res = http.request(options);
 } catch(err) {
   debugPrint("Failed to upload device state changes to HA system: " + err);
 }
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

Now back to the sensor which started all of this – the Fibaro FGMS-001.

In a 2nd post, because it’s quite a story … soon!