RaZberry: upgrading to 2.0.0 and modules part 3

Now that I finally had a working setup with RazBerry on 2.0.0 I could start doing some serious logging. Not like seeing the results on the screen for a few minutes and concluding it’s all OK, but logging everything to a text file and analyzing the results during a period that could easily take 2 or 3 weeks. I always do that before adding a new type/kind of hardware to my system – it has to be 100% reliable, because I don’t like searching for an answer to questions like “Why didn’t that Visonic sensor trigger a rule in my rules engine to set the brightness of that Insteon controlled light bulb to 20%?”.

But this time it only took 24 hours to see something was wrong here. And it’s not the result of some bug in the module, but coming directly from the RaZberry – and that’s what worries me the most, actually. Here’s what went wrong during the last 48 hours, all very wrong sensor values:

Temp Sensor: 7.0 = -2971.1
Lux Sensor: 7.0 = -9463
Lux Sensor: 7.0 = 0.0601529
Lux Sensor: 7.0 = -28353
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 16.7
Temp Sensor: 7.0 = -1742.8
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 23.5
Lux Sensor: 7.0 = 30569
Temp Sensor: 7.0 = 21.9
Temp Sensor: 7.0 = -309.6
Temp Sensor: 7.0 = 15220400
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 3
Lux Sensor: 7.0 = 0.0006696
Lux Sensor: 7.0 = 26
Temp Sensor: 7.0 = 5.8
Temp Sensor: 7.0 = 2198.9
Temp Sensor: 7.0 = 15028.7
Temp Sensor: 7.0 = 23
Lux Sensor: 7.0 = 1758910
Lux Sensor: 7.0 = 17664
Temp Sensor: 7.0 = -2664.5
Lux Sensor: 7.0 = 29475
Lux Sensor: 7.0 = 20798
Lux Sensor: 7.0 = 0.16683
Lux Sensor: 7.0 = 0.017478
Lux Sensor: 7.0 = 4.47507
Lux Sensor: 7.0 = -31466
Power Sensor: 8.0 = 6302.02
Temp Sensor: 7.0 = 60.757
Lux Sensor: 7.0 = 4893
Lux Sensor: 7.0 = 1950140
Temp Sensor: 7.0 = -565.1
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 0.413177
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 113
Lux Sensor: 7.0 = 12608
Lux Sensor: 7.0 = 1657
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = -16060
Lux Sensor: 7.0 = 0.017486
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Temp Sensor: 7.0 = 14950600

Line 1: no comment. Line 2: lights will be switched on, roller shutters will go down. Line 13: fire alarm! Line 23 during daytime: roller shutters will go down because the room will soon be warmed up too much by sunlight. And what about lines 52 and 53: where the hell do they come from?? Need I go on? With this poor result, I don’t even trust the binary (true/false) sensors anymore…

Mister Bwired experienced the same with identical hardware & module – also strange values. And something similar was also mentioned  the z-wave.me forum. Same issue? Hmm. I was ready to stop with all this ZWave stuff; back to hardware that does work.

In the meantime René Klootwijk, who’s also creating his own Home Automation system, also developed a module to interface with the RaZberry. His module can be used to forward events from zway to your own server using a REST API. René was my last hope actually; his module worked different from mine so maybe the results were different also (as in: better)? I tried his module during the last 24 hours and guess what: perfect! I can’t explain why and I’m not even gonna spend time on it – it’s working, so be happy! All you have to do is create a little webserver to receive the information, add some MQTT and  you’re done:

RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":3,"vDevId":"ZWayVDev_zway_7-0-49-3","value":32,"timestamp":1422388041}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388068}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"on","timestamp":1422388104}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-49-1","value":23.7,"timestamp":1422388154}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388170}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"on","timestamp":1422388631}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"off","timestamp":1422388634}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.8,"timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"off","timestamp":1422388683}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"on","timestamp":1422388685}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"off","timestamp":1422388687}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"on","timestamp":1422388689}
RECEIVED: {"status":"online","nodeId":9,"instanceId":2,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-2-37","value":"off","timestamp":1422388691}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.6,"timestamp":1422388692}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.7,"timestamp":1422388693}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388694}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388695}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388696}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388697}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388701}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388701}

Now the only thing that still bothers me is the battery which was empty after 2 weeks…

RaZberry: upgrading to 2.0.0 and modules part 2

RaZberryWell, I messed up things very badly right after finishing my previous post about the RaZberry, cause I somehow managed to create a complete mess. Going from firmware 1.7.2 to 2.0.0, back to 1.7.2, 2.0.0 again, several re-installs of the Z-Way for RaZberry software, messing with modules, stopping in the middle of something because it was getting too late etc… Not good; it wasn’t stable anymore.

And I just didn’t have enough time to figure out what I’d done wrong… so last Saturday I decided to start all over again -flashed a fresh Raspbian image on the SD card and did a clean install of v2.0.0. After re-including all the Z-Wave devices I saw them all returning in the Z-Way Home Automation UI; so far so good. I copied my ‘BindHA’ module to the ../automation/modules directory, restarted the z-way-server and went to the Modules section in the Preferences.

And there it was (back again):

Zway modules

After selecting it and NNF (and I think restarting the z-way-server once more, can’t recall anymore) the NodeJS script on my PC was starting to get updates:

Sun Jan 18 2015 21:15:10 GMT+0100
On/Off 8.0 = false
Sun Jan 18 2015 21:15:12 GMT+0100
On/Off 9.1 = true
Sun Jan 18 2015 21:14:21 GMT+0100
On/Off 9.1 = false
Sun Jan 18 2015 21:14:22 GMT+0100
Power: 8.0 = 0.5
Sun Jan 18 2015 21:14:57 GMT+0100
On/Off 9.1 = true
Sun Jan 18 2015 21:14:58 GMT+0100
Power: 8.0 = 0.6
On/Off 9.1 = true
Sun Jan 18 2015 21:14:59 GMT+0100
Power: 8.0 = 0.7
On/Off 9.0 = false
Sun Jan 18 2015 21:14:59 GMT+0100
On/Off 9.0 = true
Sun Jan 18 2015 21:15:28 GMT+0100
Motion, Open/Close 3.0 = true
Sun Jan 18 2015 21:15:32 GMT+0100
Motion, Open/Close 3.0 = false
Sun Jan 18 2015 21:15:33 GMT+0100
Motion, Open/Close 3.0 = true
etc...

Back in business! :-) So what does a line like “Motion, Open/Close 3.0 = false” mean? That a binary sensor value of device devices[3].instances[0] became false (0, zero). Likewise “Power: 8.0 = 0.6″ means that the power usage for devices[8].instances[0] became 0.6 W and “On/Off 9.1 = true” indicates that devices[9].instances[1] switched to on.

That’s it, I can use this to feed my HA system with information from the RaZberry by turning this information into a MQTT topic & payload – done!

Some things still need to be done though; like support for newly added Z-Wave devices without the need to restart the module, some code cleanup etc., but I can start using Z-Wave devices now if I want to.

MQTT publishing with the ESP8266 + Arduino

This is more or less a follow-up on my previous post about using the ESP8266 to MQTT-enable an Arduino (or a JeeNode in my case). The first time I installed the espduino library there was no publish available yet; one day later it was, so this afternoon I installed the latest version in the Arduino IDE and gave publishing a try.

All I had to do was adding some additional code to the example:

int reportInterval = 10 * 1000;
unsigned long now = 0;
unsigned long nextPub = reportInterval;

void loop() {
  esp.process();

  // will work for 49 days; that's OK.
  now = millis();
  if (now >= nextPub) {
    nextPub += reportInterval;
    String payload = "Payload ";
    payload += now;
    char char_array[payload.length()+1];
    payload.toCharArray(char_array, payload.length()+1);
    esp.publish("topic", char_array, payload.length()+1, 0, 0);
  }
}

The result:

Espduino libaray publishing

I noticed that the keepalive packets are still being sent even though the publishing is done every 10 seconds; I can’t recall seeing that behavior with other MQTT clients – as far as I know keepalive packets are only needed to ping the broker when there hasn’t been any other communication within the keepalive interval, but I might be wrong about that. I’ll have to check that some day.

For now this is not a real issue but it probably will be when the library will be used in a setup that’s battery powered. Another question I have is: what would be an acceptable keepalive interval for a battery operated MQTT client? For some sensors even a value of 3600 seconds would suffice I guess..

Last thing on my mind is a good auto-reconnect feature for this; a closed connection (e.g. because your Wifi AP has been down for a short period) should not be a reason to power cycle all those ESP connected devices in your house..!

MQTT client for Arduino with the ESP8266

A few days ago I saw a new (first commit on GitHub just 5 days ago) library that made it possible to use MQTT on an Arduino-ish board and use the ESP8266 Wifi Serial Transceiver to connect to a MQTT broker. I just had to try it out :-)

ESP8266 MQTT JeeNode

I used the same setup as before cause it was still on my desk, installed the library in the Arduino IDE, checked the SERIAL_BUFFER_SIZE in [your program files]\Arduino\hardware\arduino\cores\arduino\HardwareSerial.cpp (OK in Arduino 1.0.6, didn’t check older versions), changed the example regarding MQTT broker address, Wifi SSID & password, client id (JeeNode_on_your_desk), set the debug serial port to pins 6 & 7 and uploaded the sketch to the JeeNode.

And it’s working, as the Mosquitto MQTT broker messages show below:

D:\Program Files (x86)\mosquitto>mosquitto -c mosquitto.conf
mosquitto version 1.1.2 (build date 30/01/2013 20:46:29.67) starting
Config loaded from mosquitto.conf.
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.155.
New client connected from 192.168.10.155 as JeeNode_on_your_desk.
New connection from ::1.
New client connected from ::1 as mosqpub/3228-devbox.

Nice! Let’s try to send an MQTT message to the JeeNode:

mosquitto_pub -h localhost -t /topic -m "Hi there, it's me!"

And the debug port showed this:

[espduino_mqttclient]
RESET ESP done
WIFI: Connected
TCP: Connected
AT+CIPSEND=36
MQTT: Connected
MQTT:Connected
MQTT: subscribe, topic
AT+CIPSEND=13
MQTT: Subscribe successful

MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
Received, topic:/topic, data:Hi there, it's me!
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
...

And here’s what the communication between JeeNode & ESP8266 looks like:

AT+RST
ATE0
AT+CWMODE=1
AT+CWJAP="SSID","password"
AT+CIPMUX=0
AT+CIPCLOSE
AT+CIPSTART="TCP","192.168.10.179",1883
AT+CIPSEND=36
 "  MQIsdp     JeeNode_on_your_deskAT+CIPSEND=13
‚,    /topic AT+CIPSEND=2
AT+CIPSEND=2
AT+CIPSEND=2
...

Nice; not that I have any immediate use case for it (my Zigbee network based on XBee modules is still working fine), but it’s fun to see this working and good to know it exists.

Next ‘experiment’ will be an MQTT client on the ESP8266 itself (yep, without MCU). I’ve already installed the ESP8266 Windows Eclipse IDE in a VM so that I can flash the ESP8266 with a firmware that includes this MQTT client; that will be even more exciting; stay tuned!

Meet the ESP 8266 WiFi Serial Transceiver

esp8266Look what I found under the Christmas tree about a week ago – ESP8266 modules ;-)

A tiny module with a size comparable to a Fibaro Universal Sensor. Basically, the ESP8266 enables you to add wireless connectivity to your hardware – to anything that does serial; for example your Arduino or Raspberry Pi. But there’s more to this module – you can build your own firmware, program it in Lua and do stuff that might make that additional MCU obsolete.

But for now, all I have to tell is about my experiences during the first few hours with the ESP8266.

After unwrapping the modules the first thing I had to find out the purpose of the 8 pins of the ESP8266 board. VCC, GND, RX, TX and found out about those really quick; I also found a small Arduino sketch that would allow me to test some basic functions of the ESP8266 so I set things up on a breadboard and uploaded a sketch to a spare Arduino.

No go.. I had been a bit too hasty, thinking that a single webpage would provide all the information I needed to get things going. After a while I found out that the CH_PD pin had to be pulled up to VCC to set the ESP8266 to ‘normal operation’ mode. Cause when pulled to GND, the ESP8266 will just sit and wait for a new firmware to arrive … OK, got that. Still no go; that’s when I read about the different baud rates – some models work with 9600 bps, others with 57600 bps so maybe the ESP8266 and the Arduino weren’t using the same bps. So I decided to just hook up the 2nd ESP8266 to my PC with a PL2303 serial to USB cable and see if I could get the module to respond to some AT commands.

ESP8266 with PuTTYPuTTY is one of my favorite tools for telnet, ssh and it can also do serial communication, so I used PuTTY to send some AT commands to the ESP8266, but after some time I gave up… I knew I was using the right speed (9600 bps in my case) but the module just didn’t respond after- no OK; nothing.

ESP8266 with TermiteChanged some settings, still no answer. Made PutTTY add an additional LF after a CR but still no luck? Eventually I switched to Termite, cause I found out that I wasn’t the first having problems with the combination PuTTY + ESP8266.

 

Now that I knew it was 9600 instead of 57600 bps I could proceed with the sketch mentioned earlier. Another ‘problem’ was that the Arduino operates @5V and the ESP8266 @ 3.3V, so I had to do something about that too; I saw all kinds of solutions with level shifters, resistors, diodes and decided to go the easy way – a JeeNode which operates @ 3.3V:

ESP8266 on a JeeNode

The result in the PuTTY window that was connected to the ‘debug’ Serial port (pins 6 & 7):

ESP8266 PuTTY screen

Neat. Needless to say that this is just a small example of what this ESP8266 can do; it can do much more than just fetching a small web page – I haven’t even scratched the surface yet! A Wifi Door sensor for the price of an ESP8266, a reed switch + magnet and power supply.. the latter is the only downside of the ESP8266: it consumes too much power to run on batteries. Well, you can’t have ‘em all … (yet?) Besides that, I love this little powerful Wifi module.

Happy new year!

 

I’m ready for it as you can see, are you? :-)

Ready for New Year's Eve

 

Let’s hope 2015 will bring just as much Home Automation fun as 2014 or even more! Some exciting Kickstarter projects will probably arrive and I’ve got some plans of my own, and they’re all fighting for my attention ;-) As usual, you’ll read all about it over here. Happy New Year!

RaZberry: upgrading to 2.0.0 and modules

RaZberry on a PiI’m beginning to like the RaZberry, despite the problems I had with it the last few days. Why? Cause it’s open, you can add your own stuff to it- it’s different from all those other closed systems you see so often. Should have done more with it much sooner … I bought the RaZberry @ CeBIT 2013, ordered a Wall Plug when I got home and tinkered with both for a while when that Wall Plug arrived. But since I had no plans on buying more Z-wave hardware (it wouldn’t add anything I didn’t already have – well, problemz maybe), the RaZberry board soon ended up on a shelf cause I needed the Raspberry for something else.

Until I got my hands on a Fibaro motion sensor 2 weeks ago … this sensor was almost ‘irresistible’, I just had to use it. So that’s when I picked up the RaZberry project again to see if I could get the RazBerry integrated into my own HA system.

After I made some progress, I mailed Pieter (who also bought a RaZberry at the CeBIT and ordered the same motion sensor) and he installed my code. But he encountered strange things; weird behavior and messages like “Cannot set port to 8083: Cannot bind to port”, “Object.keys called on non-object”, needing to reboot in order to get things going again etcetera. Strange …

Browsing the web I found out that there was a new 2.0.0 firmware for the RaZberry so I decided to upgrade. Suddenly I had exactly the same problems as Pieter; so I reverted back to 1.7.2 and the problems disappeared again.. So Pieter must be running on firmware v2.0.0 – we checked this and the source of the problems was found. Something must have changed that resulted in my code crippling the whole RaZberry environment…? So I kept my RaZberry running on v1.7.2 until I had the time to find a solution for this, cause staying on v1.7.2 just wasn’t a solution.

This afternoon I had some time to resolve the issue. Although being a complete newbie on Z-Wave (and RaZberry in particular), I did realize that editing the main.js file to get my own code running was not such a good idea; an upgrade made the changes to main.js disappear so there would probably be a better way to extend the RaZberry system with my own add-ons. User Modules seemed like the way to go, as if it was made for what I was doing ;-)

So I put my code, almost unchanged, into a User module. It still didn’t work… same problems. Binding worked and the callback was still being executed. Although not completely, so something in my code was still blocking something … I figured out that it must be the blocking /ZWaveAPI/Data/… HTTP call I did, so I rewrote that one to an async version and my problems were gone. Why? I still don’t know; for that I don’t know enough (about how the RaZberry API’s work internally – but I’m glad I found a way to get my upload to my HA system working again :-).

After a while I found out that my Usermodule didn’t always start – it felt like some dependency wasn’t met after a restart (sudo service z-way-server restart) so I changed the module.json and added a autoloadPriority and set it to 9999, like this:

{
  "autoload": true,
  "singleton": false,
  "autoloadPriority":9999,
  "defaults": {
    "title": "HA system binding",
    "description": "Report sensor values to my HA system by HTTP",
    "device": ""
  }
}

Now my module will be loaded later then all the rest; found it on the Z-Wave-Me/home-automation Wiki on Github. The code is still very fresh, it’ll need some more time to prove itself, but so far everything is looking good.

BTW, how nice would it be if MQTT would become part of the RaZberry system …

The Fibaro FGMS-001 on RaZberry and Z-Wave, The Interoperable Standard ?

Well, maybe the fact that the Z-Wave logo can only be found at the bottom of the package of the FGMS-001 was an omen for another story about bad luck. Here it goes.

Fibaro FGMS-001After successfully binding to events in the Z-Way API and getting the required information into my home-brew Home Automation system, I focused on the FGMS-001.

It’s a real beauty; small, flexible, you could almost say that Fibaro reinvented the motion sensor and even managed to fit some additional sensors into the ‘ball’ which are useful @ lots of places in the house.

So I was very excited to see this sensor working. Inclusion went fine, but then the trouble started. My binding worked fine, but no sign of Temp & Lux values? I also saw an error in the z-way-server.log:

Packet CC::MULTI_CHANNEL_CMD_ENCAP_V2 is too short: required at least 5 bytes, got 3
Error returned from _zway_cc_call_handler(zway, command, controller->id, 0, data[4], &data[5]): Wrong packet from Z-Wave network or Discovery got bad data (stick communication failed) (-9)

That doesn’t look good, so I searched for the error message; according to this seems I need a better/newer XML file, so I did. Motion was working, but Temp & Lux weren’t being updated. After a while I found out that this doesn’t happen by design – you have to change some parameters in the configuration. OK, did that. Wrong – you can’t use the RaZberry Expert UI to change those values, cause it messes up the entered values – enter “10” but this is changed to “32769” once you tab your way to the next textbox. Imagine what happens after applying such a value – temperature measurement every 9 hours.

So to finally get things going, I had to do it myself from a command prompt (setting Temperature interval to 600 seconds):

wget http://192.168.10.xxx:8083/ZWaveAPI/Run/devices[7].instances[0].commandClasses[112].Set(64,600,2)

Yep, by the time I got this far the device counter had already reached 7, with just 2 sensors. And the battery is at 88% now, after almost 3 days – what was it, 2 years battery life? I’ll believe it when I see it. The FGMS-001 has lost a lot of its ‘glamour’ in just 2 days.

So what is this, a sensor from a bad batch? Still in beta stage? Z-wave.me forum blames Fibaro for not following the standards of the holy grail of Home Automation called Z-Wave. Is this true and if so, why do this? Can’t they read?  Or is there some other purpose for this kind of deviation? I don’t know and frankly it doesn’t even really matter – what it results in does matter.

Cause all these little deviations make Z-Wave unreliable, immature – you never know whether it’ll work out of the box or not. Lots of techies will be able to solve this sort of issues, but the majority won’t. Killing. And imagine the IoT, with a predicted >25 billion devices around 2020, being driven by Z-Wave … it gives me the shivers!

Razberry, binding to events and communicating device state changes

It all started about a week ago, when I became the owner of a very good looking Fibaro Motion sensor:

Fibaro FGMS-001

Compared to a Visonic motion sensor (to the right), this Fibaro is small. And very nice to see as well if I may say so. And that’s not it, there’s more inside than just motion:

  • A temperature sensor;
  • Light intensity;
  • an accelerometer to detect tampering.

And now that I’ve got one, it was time to give the Razberry, which has been unused for the last 18 months, another try.

So I took a spare Raspberry Pi, downloaded a complete image from here and started fiddling with the web-interface which is at http://<RPi-IP-address>:8083 . I have more Zwave devices than just this Fibaro FGMS-001 Motion sensor, so I chose to start with including a Fibaro Door/Window sensor, the FGK-101 – because it’s easier to test with a sensor on your desk that doesn’t require you to wave at it all the time ;-) All went well, I saw the sensor switching from on to off and vice versa.

Now how do I get this infomation into my own Home Automation system without too much hassle? I’m used to subscribing to MQTT topics lately, so I started searching for a way to catch events or hook up to ‘something’ that could help me to export device data in some way. The Developers Documentation PDF gave a clue: binding. Ah, so that’s how it’s called in Z-Way world.

The PDF says: “A very powerful function of the JS API is the ability to bind functions to certain values of the device tree. They get then executed when the value changes.” Yeah, that sounds like what I want. Lets give it a try.

To run the “binding to *everything*” code I had to make sure an additional (Java-)script was executed when the z-way-server service was started; for that I added an additional line to the end of the file /opt/z-way-server/automation/main.js file (showing just the last 3 lines here):

  });
  executeFile('bind_all.js')
}

And this bind_all.js file, which has to be placed in the same directory as the main.js we just changed, goes like this:

var callback = function(type,arg) {
  console.log('### here I am ###');
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

I restarted the z-way-server service (sudo service z-way-server restart) and moved the magnet of the Door/Window sensor away; bingo! The ‘### here I am ###’ showed up in the z-way-server log (tail -f /var/log/z-way-server.log). OK, nice.

Now what? I had a look at what others did; the only thing  I could find was executing a system() call to notify the outside world that something had changed but I didn’t like that approach; sounds like too much overhead? Getting a MQTT client running would be nice, but way out of reach with what the JS API has to offer ;-)

I preferred to keep it all wrapped up in a single script without any other dependencies, so I went back to the Developers Documentation PDF I mentioned earlier. It says: “The JavaScript implementation of Z-Way allows to directly accessing HTTP objects”. So I can do a HTTP request from inside my callback? Let’s try it. With a few lines of code I managed to POST the result of a JSON.stringify(this) to a NodeJS HTTP server that was running on my PC. Nice, and the information I was looking for was in there, great. But this POST resulted in 15 KB of data (including headers and other usual stuff), just for an opening/closing binary sensor? Come on, still not good enough!

Now the JSON API came to the rescue; again from the Developer Documentation PDF: “/ZWaveAPI/Data/<timestamp> returns an associative array of changes in Z-Way data tree since <timestamp>”

Let’s use that and POST that response to my little HTTP server. Yes! Same result (as in: it also contains the information I need) and the size has decreased to 1/9th of what it was.

About 1700 bytes to ‘communicate’ a state change of a binary sensor is still too much, but I couldn’t find any other way to do this. Maybe I’ll parse the result further inside the callback to minimize the I/O to a more ‘sane’ amount. So here’s the complete bind_all.js script that’s now communicating state changes of my Fibaro Door/Window sensor:

var callback = function(type,arg) {
 try {
   var options = {};
   // timestamp
   var ts = (new Date()).getTime();
   ts = Math.floor(ts/1000).toString();
   // query local ZWave API
   options.url = "http://127.0.0.1:8083/ZWaveAPI/Data/"+ts;
   options.method = "GET";
   var res = http.request(options);
   // send response to HA system
   options.url = "http://192.168.10.xxx:8181";
   options.method = "POST";
   options.data = JSON.stringify(res);
   res = http.request(options);
 } catch(err) {
   debugPrint("Failed to upload device state changes to HA system: " + err);
 }
};

for (var dv in zway.devices) {
 var dv = zway.devices[dv];
 dv.data.bind(callback, null, true);
}

Now back to the sensor which started all of this – the Fibaro FGMS-001.

In a 2nd post, because it’s quite a story … soon!

Solar Panels and the Omnik Wifi Kit

As announced, last Thursday we went solar!

Men at workIt took me some time to take that final step and order the solar system, cause I’ve been interested in solar energy for a long time. But now that we finally have 12 solar panels on our roof I’m very satisfied with the end result.

I decided not to do this myself cause I knew that it would take me much too long – much, much longer than the 2 professionals did. They were done in about 6 hours and left me with a fully functional solar system:

Almost finished

4 rows of 3 x JA Solar JAM6-60/280-PR panels and an Omniksol-3k-TL inverter in the garage:

Omink inverterI’m very satisfied with how it all went – from the first contact with Sungevity to the point where the solar system was working.

sungevity

 

 

 

No loose ends, everything is taken care of for you, from notifying 3rd parties that I’m gonna start producing energy soon and even the VAT refund (the EU considers you as a energy producer) is handled  for a small fee.

All I had to do was sign some papers and provide the password of my Wifi Access Point to connect the Omnik inverter to Internet. Great job guys (and gals)!

Now on to the tech part of this post: the Omnik Internal Data Collector and how to use it.

Now that we have 12 solar panels on our roof I’d really like to be able to monitor them – myself, local and not on some portal. Fortunately Sungevity made it a standard procedure to install the Omnik Wifi kit. This kit uploads the performance data to the cloud and can be viewed from the Omnik portal.

Wifi Kit inside the inverterBut it would be even better if I’d be able to receive this information directly from the Omnik Wifi kit instead of heaving to log in on some portal and probably do some web scraping to get the data.

You can see the Wifi antenna of the kit on the picture to the left, right next to the grey cable. Now the really great part of this Wifi kit is that you can add your own ‘remote servers’ from the web interface – for instance, you can add your PC (bad choice) or a Raspberry Pi (better), or whatever – just enter the right IP address and port number and you’ll receive the same information that you’ll see on the Omnik portal. (you’ll have to do the charting thing yourself though ;-)

The information looks like this (this one is invalid for obvious reasons):

68 81 41 b0 77 6f 7e 5f 77 6f 7e 5f 81 02 01 4e  h?A°wo~_wo~_?..N
4c 44 4e 33 30 32 30 31 34 35 53 32 33 35 35 00  LDN3020145S2355.
45 08 42 08 3d ff ff 00 00 00 00 ff ff 00 00 ff  E.B.=ÿÿ....ÿÿ..ÿ
ff ff ff 08 89 ff ff ff ff 13 8b 00 00 ff ff ff  ÿÿÿ.?ÿÿÿÿ.?..ÿÿÿ
ff ff ff ff ff 00 00 00 00 00 31 00 00 00 17 00  ÿÿÿÿÿ.....1.....
00 00 00 00 00 ff ff 00 00 00 00 00 00 00 00 00  .....ÿÿ.........
00 00 00 00 01 4e 4c 31 2d 56 31 2e 30 2d 30 30  .....NL1-V1.0-00
38 33 2d 34 00 00 00 00 00 56 32 2e 30 2d 30 30  83-4.....V2.0-00
32 38 00 00 00 00 00 00 00 00 00 00 00 6d 16     28...........}.

Having the luxury of knowing what the Omnik Wifi kit uploads to the cloud and being able to see what it results in on the Omnikportal should not make it too hard to figure out the ‘where is what’.

But hey, wait, I’m not the first one that can think of this – let’s do some serious searching – maybe someone already figured it all out? And yes, there are various sources, even with code. With that information I had my NodeJS script running in no-time and saw the information being received. Nice, cause now I have the same information as can be seen on the Omnik portal. But I’m not done yet, cause I wanted some additional checks on the data received from the Omnik Wifi kit.

From the source code made by others I knew that the ‘checksum’ used was just a sum of all the individual bytes in the packet. So it wasn’t so hard to find out that the packets being received used the same method; in the example packet above, the 0x6d at the end is the low byte of the sum of all preceding bytes except the first one (the start-byte, 0x68).

Another thing is that it looks like the packet has a fixed header and always ends with the checksum + stop-byte (0x16). The size of this all is 14 bytes. So with a packet of 143 bytes this leaves 143-14 = 129 (0x81) bytes for the payload – aha, the 2nd byte in the packet is the payload size! Some more validity checks ;-)

Right now my NodeJS script is running on a Cubieboard, collecting data and checking if the additional checks really work.

Have fun!