RaZberry: upgrading to 2.0.0 and modules part 3

Now that I finally had a working setup with RazBerry on 2.0.0 I could start doing some serious logging. Not like seeing the results on the screen for a few minutes and concluding it’s all OK, but logging everything to a text file and analyzing the results during a period that could easily take 2 or 3 weeks. I always do that before adding a new type/kind of hardware to my system – it has to be 100% reliable, because I don’t like searching for an answer to questions like “Why didn’t that Visonic sensor trigger a rule in my rules engine to set the brightness of that Insteon controlled light bulb to 20%?”.

But this time it only took 24 hours to see something was wrong here. And it’s not the result of some bug in the module, but coming directly from the RaZberry – and that’s what worries me the most, actually. Here’s what went wrong during the last 48 hours, all very wrong sensor values:

Temp Sensor: 7.0 = -2971.1
Lux Sensor: 7.0 = -9463
Lux Sensor: 7.0 = 0.0601529
Lux Sensor: 7.0 = -28353
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 16.7
Temp Sensor: 7.0 = -1742.8
Temp Sensor: 7.0 = 23.5
Temp Sensor: 7.0 = 23.5
Lux Sensor: 7.0 = 30569
Temp Sensor: 7.0 = 21.9
Temp Sensor: 7.0 = -309.6
Temp Sensor: 7.0 = 15220400
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 27
Lux Sensor: 7.0 = 3
Lux Sensor: 7.0 = 0.0006696
Lux Sensor: 7.0 = 26
Temp Sensor: 7.0 = 5.8
Temp Sensor: 7.0 = 2198.9
Temp Sensor: 7.0 = 15028.7
Temp Sensor: 7.0 = 23
Lux Sensor: 7.0 = 1758910
Lux Sensor: 7.0 = 17664
Temp Sensor: 7.0 = -2664.5
Lux Sensor: 7.0 = 29475
Lux Sensor: 7.0 = 20798
Lux Sensor: 7.0 = 0.16683
Lux Sensor: 7.0 = 0.017478
Lux Sensor: 7.0 = 4.47507
Lux Sensor: 7.0 = -31466
Power Sensor: 8.0 = 6302.02
Temp Sensor: 7.0 = 60.757
Lux Sensor: 7.0 = 4893
Lux Sensor: 7.0 = 1950140
Temp Sensor: 7.0 = -565.1
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 0.413177
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 63
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 113
Lux Sensor: 7.0 = 12608
Lux Sensor: 7.0 = 1657
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 64
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = -16060
Lux Sensor: 7.0 = 0.017486
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Lux Sensor: 7.0 = 68
Temp Sensor: 7.0 = 14950600

Line 1: no comment. Line 2: lights will be switched on, roller shutters will go down. Line 13: fire alarm! Line 23 during daytime: roller shutters will go down because the room will soon be warmed up too much by sunlight. And what about lines 52 and 53: where the hell do they come from?? Need I go on? With this poor result, I don’t even trust the binary (true/false) sensors anymore…

Mister Bwired experienced the same with identical hardware & module – also strange values. And something similar was also mentioned  the z-wave.me forum. Same issue? Hmm. I was ready to stop with all this ZWave stuff; back to hardware that does work.

In the meantime René Klootwijk, who’s also creating his own Home Automation system, also developed a module to interface with the RaZberry. His module can be used to forward events from zway to your own server using a REST API. René was my last hope actually; his module worked different from mine so maybe the results were different also (as in: better)? I tried his module during the last 24 hours and guess what: perfect! I can’t explain why and I’m not even gonna spend time on it – it’s working, so be happy! All you have to do is create a little webserver to receive the information, add some MQTT and  you’re done:

RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":3,"vDevId":"ZWayVDev_zway_7-0-49-3","value":32,"timestamp":1422388041}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388068}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"on","timestamp":1422388104}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":49,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-49-1","value":23.7,"timestamp":1422388154}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388170}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"on","timestamp":1422388631}
RECEIVED: {"status":"online","nodeId":3,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_3-0-48-1","value":"off","timestamp":1422388634}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388641}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.8,"timestamp":1422388653}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"off","timestamp":1422388683}
RECEIVED: {"status":"online","nodeId":9,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-0-37","value":"on","timestamp":1422388685}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"off","timestamp":1422388687}
RECEIVED: {"status":"online","nodeId":9,"instanceId":1,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-1-37","value":"on","timestamp":1422388689}
RECEIVED: {"status":"online","nodeId":9,"instanceId":2,"cmdClass":37,"vDevId":"ZWayVDev_zway_9-2-37","value":"off","timestamp":1422388691}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.6,"timestamp":1422388692}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.7,"timestamp":1422388693}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388694}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"off","timestamp":1422388695}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0,"timestamp":1422388696}
RECEIVED: {"status":"online","nodeId":7,"instanceId":0,"cmdClass":48,"sensorType":1,"vDevId":"ZWayVDev_zway_7-0-48-1","value":"off","timestamp":1422388697}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":37,"vDevId":"ZWayVDev_zway_8-0-37","value":"on","timestamp":1422388701}
RECEIVED: {"status":"online","nodeId":8,"instanceId":0,"cmdClass":49,"sensorType":4,"vDevId":"ZWayVDev_zway_8-0-49-4","value":0.9,"timestamp":1422388701}

Now the only thing that still bothers me is the battery which was empty after 2 weeks…

RaZberry: upgrading to 2.0.0 and modules part 2

RaZberryWell, I messed up things very badly right after finishing my previous post about the RaZberry, cause I somehow managed to create a complete mess. Going from firmware 1.7.2 to 2.0.0, back to 1.7.2, 2.0.0 again, several re-installs of the Z-Way for RaZberry software, messing with modules, stopping in the middle of something because it was getting too late etc… Not good; it wasn’t stable anymore.

And I just didn’t have enough time to figure out what I’d done wrong… so last Saturday I decided to start all over again -flashed a fresh Raspbian image on the SD card and did a clean install of v2.0.0. After re-including all the Z-Wave devices I saw them all returning in the Z-Way Home Automation UI; so far so good. I copied my ‘BindHA’ module to the ../automation/modules directory, restarted the z-way-server and went to the Modules section in the Preferences.

And there it was (back again):

Zway modules

After selecting it and NNF (and I think restarting the z-way-server once more, can’t recall anymore) the NodeJS script on my PC was starting to get updates:

Sun Jan 18 2015 21:15:10 GMT+0100
On/Off 8.0 = false
Sun Jan 18 2015 21:15:12 GMT+0100
On/Off 9.1 = true
Sun Jan 18 2015 21:14:21 GMT+0100
On/Off 9.1 = false
Sun Jan 18 2015 21:14:22 GMT+0100
Power: 8.0 = 0.5
Sun Jan 18 2015 21:14:57 GMT+0100
On/Off 9.1 = true
Sun Jan 18 2015 21:14:58 GMT+0100
Power: 8.0 = 0.6
On/Off 9.1 = true
Sun Jan 18 2015 21:14:59 GMT+0100
Power: 8.0 = 0.7
On/Off 9.0 = false
Sun Jan 18 2015 21:14:59 GMT+0100
On/Off 9.0 = true
Sun Jan 18 2015 21:15:28 GMT+0100
Motion, Open/Close 3.0 = true
Sun Jan 18 2015 21:15:32 GMT+0100
Motion, Open/Close 3.0 = false
Sun Jan 18 2015 21:15:33 GMT+0100
Motion, Open/Close 3.0 = true
etc...

Back in business! 🙂 So what does a line like “Motion, Open/Close 3.0 = false” mean? That a binary sensor value of device devices[3].instances[0] became false (0, zero). Likewise “Power: 8.0 = 0.6” means that the power usage for devices[8].instances[0] became 0.6 W and “On/Off 9.1 = true” indicates that devices[9].instances[1] switched to on.

That’s it, I can use this to feed my HA system with information from the RaZberry by turning this information into a MQTT topic & payload – done!

Some things still need to be done though; like support for newly added Z-Wave devices without the need to restart the module, some code cleanup etc., but I can start using Z-Wave devices now if I want to.

MQTT publishing with the ESP8266 + Arduino

This is more or less a follow-up on my previous post about using the ESP8266 to MQTT-enable an Arduino (or a JeeNode in my case). The first time I installed the espduino library there was no publish available yet; one day later it was, so this afternoon I installed the latest version in the Arduino IDE and gave publishing a try.

All I had to do was adding some additional code to the example:

int reportInterval = 10 * 1000;
unsigned long now = 0;
unsigned long nextPub = reportInterval;

void loop() {
  esp.process();

  // will work for 49 days; that's OK.
  now = millis();
  if (now >= nextPub) {
    nextPub += reportInterval;
    String payload = "Payload ";
    payload += now;
    char char_array[payload.length()+1];
    payload.toCharArray(char_array, payload.length()+1);
    esp.publish("topic", char_array, payload.length()+1, 0, 0);
  }
}

The result:

Espduino libaray publishing

I noticed that the keepalive packets are still being sent even though the publishing is done every 10 seconds; I can’t recall seeing that behavior with other MQTT clients – as far as I know keepalive packets are only needed to ping the broker when there hasn’t been any other communication within the keepalive interval, but I might be wrong about that. I’ll have to check that some day.

For now this is not a real issue but it probably will be when the library will be used in a setup that’s battery powered. Another question I have is: what would be an acceptable keepalive interval for a battery operated MQTT client? For some sensors even a value of 3600 seconds would suffice I guess..

Last thing on my mind is a good auto-reconnect feature for this; a closed connection (e.g. because your Wifi AP has been down for a short period) should not be a reason to power cycle all those ESP connected devices in your house..!

MQTT client for Arduino with the ESP8266

A few days ago I saw a new (first commit on GitHub just 5 days ago) library that made it possible to use MQTT on an Arduino-ish board and use the ESP8266 Wifi Serial Transceiver to connect to a MQTT broker. I just had to try it out 🙂

ESP8266 MQTT JeeNode

I used the same setup as before cause it was still on my desk, installed the library in the Arduino IDE, checked the SERIAL_BUFFER_SIZE in [your program files]ArduinohardwarearduinocoresarduinoHardwareSerial.cpp (OK in Arduino 1.0.6, didn’t check older versions), changed the example regarding MQTT broker address, Wifi SSID & password, client id (JeeNode_on_your_desk), set the debug serial port to pins 6 & 7 and uploaded the sketch to the JeeNode.

And it’s working, as the Mosquitto MQTT broker messages show below:

D:Program Files (x86)mosquitto>mosquitto -c mosquitto.conf
mosquitto version 1.1.2 (build date 30/01/2013 20:46:29.67) starting
Config loaded from mosquitto.conf.
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.155.
New client connected from 192.168.10.155 as JeeNode_on_your_desk.
New connection from ::1.
New client connected from ::1 as mosqpub/3228-devbox.

Nice! Let’s try to send an MQTT message to the JeeNode:

mosquitto_pub -h localhost -t /topic -m "Hi there, it's me!"

And the debug port showed this:

[espduino_mqttclient]
RESET ESP done
WIFI: Connected
TCP: Connected
AT+CIPSEND=36
MQTT: Connected
MQTT:Connected
MQTT: subscribe, topic
AT+CIPSEND=13
MQTT: Subscribe successful

MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
Received, topic:/topic, data:Hi there, it's me!
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
...

And here’s what the communication between JeeNode & ESP8266 looks like:

AT+RST
ATE0
AT+CWMODE=1
AT+CWJAP="SSID","password"
AT+CIPMUX=0
AT+CIPCLOSE
AT+CIPSTART="TCP","192.168.10.179",1883
AT+CIPSEND=36
 "  MQIsdp     JeeNode_on_your_deskAT+CIPSEND=13
‚,    /topic AT+CIPSEND=2
AT+CIPSEND=2
AT+CIPSEND=2
...

Nice; not that I have any immediate use case for it (my Zigbee network based on XBee modules is still working fine), but it’s fun to see this working and good to know it exists.

Next ‘experiment’ will be an MQTT client on the ESP8266 itself (yep, without MCU). I’ve already installed the ESP8266 Windows Eclipse IDE in a VM so that I can flash the ESP8266 with a firmware that includes this MQTT client; that will be even more exciting; stay tuned!

Meet the ESP 8266 WiFi Serial Transceiver

esp8266Look what I found under the Christmas tree about a week ago – ESP8266 modules 😉

A tiny module with a size comparable to a Fibaro Universal Sensor. Basically, the ESP8266 enables you to add wireless connectivity to your hardware – to anything that does serial; for example your Arduino or Raspberry Pi. But there’s more to this module – you can build your own firmware, program it in Lua and do stuff that might make that additional MCU obsolete.

But for now, all I have to tell is about my experiences during the first few hours with the ESP8266.

After unwrapping the modules the first thing I had to find out the purpose of the 8 pins of the ESP8266 board. VCC, GND, RX, TX and found out about those really quick; I also found a small Arduino sketch that would allow me to test some basic functions of the ESP8266 so I set things up on a breadboard and uploaded a sketch to a spare Arduino.

No go.. I had been a bit too hasty, thinking that a single webpage would provide all the information I needed to get things going. After a while I found out that the CH_PD pin had to be pulled up to VCC to set the ESP8266 to ‘normal operation’ mode. Cause when pulled to GND, the ESP8266 will just sit and wait for a new firmware to arrive … OK, got that. Still no go; that’s when I read about the different baud rates – some models work with 9600 bps, others with 57600 bps so maybe the ESP8266 and the Arduino weren’t using the same bps. So I decided to just hook up the 2nd ESP8266 to my PC with a PL2303 serial to USB cable and see if I could get the module to respond to some AT commands.

ESP8266 with PuTTYPuTTY is one of my favorite tools for telnet, ssh and it can also do serial communication, so I used PuTTY to send some AT commands to the ESP8266, but after some time I gave up… I knew I was using the right speed (9600 bps in my case) but the module just didn’t respond after- no OK; nothing.

ESP8266 with TermiteChanged some settings, still no answer. Made PutTTY add an additional LF after a CR but still no luck? Eventually I switched to Termite, cause I found out that I wasn’t the first having problems with the combination PuTTY + ESP8266.

 

Now that I knew it was 9600 instead of 57600 bps I could proceed with the sketch mentioned earlier. Another ‘problem’ was that the Arduino operates @5V and the ESP8266 @ 3.3V, so I had to do something about that too; I saw all kinds of solutions with level shifters, resistors, diodes and decided to go the easy way – a JeeNode which operates @ 3.3V:

ESP8266 on a JeeNode

The result in the PuTTY window that was connected to the ‘debug’ Serial port (pins 6 & 7):

ESP8266 PuTTY screen

Neat. Needless to say that this is just a small example of what this ESP8266 can do; it can do much more than just fetching a small web page – I haven’t even scratched the surface yet! A Wifi Door sensor for the price of an ESP8266, a reed switch + magnet and power supply.. the latter is the only downside of the ESP8266: it consumes too much power to run on batteries. Well, you can’t have ’em all … (yet?) Besides that, I love this little powerful Wifi module.