MQTT publishing with the ESP8266 + Arduino

This is more or less a follow-up on my previous post about using the ESP8266 to MQTT-enable an Arduino (or a JeeNode in my case). The first time I installed the espduino library there was no publish available yet; one day later it was, so this afternoon I installed the latest version in the Arduino IDE and gave publishing a try.

All I had to do was adding some additional code to the example:

int reportInterval = 10 * 1000;
unsigned long now = 0;
unsigned long nextPub = reportInterval;

void loop() {
  esp.process();

  // will work for 49 days; that's OK.
  now = millis();
  if (now >= nextPub) {
    nextPub += reportInterval;
    String payload = "Payload ";
    payload += now;
    char char_array[payload.length()+1];
    payload.toCharArray(char_array, payload.length()+1);
    esp.publish("topic", char_array, payload.length()+1, 0, 0);
  }
}

The result:

Espduino libaray publishing

I noticed that the keepalive packets are still being sent even though the publishing is done every 10 seconds; I can’t recall seeing that behavior with other MQTT clients – as far as I know keepalive packets are only needed to ping the broker when there hasn’t been any other communication within the keepalive interval, but I might be wrong about that. I’ll have to check that some day.

For now this is not a real issue but it probably will be when the library will be used in a setup that’s battery powered. Another question I have is: what would be an acceptable keepalive interval for a battery operated MQTT client? For some sensors even a value of 3600 seconds would suffice I guess..

Last thing on my mind is a good auto-reconnect feature for this; a closed connection (e.g. because your Wifi AP has been down for a short period) should not be a reason to power cycle all those ESP connected devices in your house..!

MQTT client for Arduino with the ESP8266

A few days ago I saw a new (first commit on GitHub just 5 days ago) library that made it possible to use MQTT on an Arduino-ish board and use the ESP8266 Wifi Serial Transceiver to connect to a MQTT broker. I just had to try it out 🙂

ESP8266 MQTT JeeNode

I used the same setup as before cause it was still on my desk, installed the library in the Arduino IDE, checked the SERIAL_BUFFER_SIZE in [your program files]ArduinohardwarearduinocoresarduinoHardwareSerial.cpp (OK in Arduino 1.0.6, didn’t check older versions), changed the example regarding MQTT broker address, Wifi SSID & password, client id (JeeNode_on_your_desk), set the debug serial port to pins 6 & 7 and uploaded the sketch to the JeeNode.

And it’s working, as the Mosquitto MQTT broker messages show below:

D:Program Files (x86)mosquitto>mosquitto -c mosquitto.conf
mosquitto version 1.1.2 (build date 30/01/2013 20:46:29.67) starting
Config loaded from mosquitto.conf.
Opening ipv6 listen socket on port 1883.
Opening ipv4 listen socket on port 1883.
New connection from 192.168.10.155.
New client connected from 192.168.10.155 as JeeNode_on_your_desk.
New connection from ::1.
New client connected from ::1 as mosqpub/3228-devbox.

Nice! Let’s try to send an MQTT message to the JeeNode:

mosquitto_pub -h localhost -t /topic -m "Hi there, it's me!"

And the debug port showed this:

[espduino_mqttclient]
RESET ESP done
WIFI: Connected
TCP: Connected
AT+CIPSEND=36
MQTT: Connected
MQTT:Connected
MQTT: subscribe, topic
AT+CIPSEND=13
MQTT: Subscribe successful

MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
Received, topic:/topic, data:Hi there, it's me!
MQTT: Send keepalive packet
AT+CIPSEND=2
MQTT: Send keepalive packet
AT+CIPSEND=2
...

And here’s what the communication between JeeNode & ESP8266 looks like:

AT+RST
ATE0
AT+CWMODE=1
AT+CWJAP="SSID","password"
AT+CIPMUX=0
AT+CIPCLOSE
AT+CIPSTART="TCP","192.168.10.179",1883
AT+CIPSEND=36
 "  MQIsdp     JeeNode_on_your_deskAT+CIPSEND=13
‚,    /topic AT+CIPSEND=2
AT+CIPSEND=2
AT+CIPSEND=2
...

Nice; not that I have any immediate use case for it (my Zigbee network based on XBee modules is still working fine), but it’s fun to see this working and good to know it exists.

Next ‘experiment’ will be an MQTT client on the ESP8266 itself (yep, without MCU). I’ve already installed the ESP8266 Windows Eclipse IDE in a VM so that I can flash the ESP8266 with a firmware that includes this MQTT client; that will be even more exciting; stay tuned!

Meet the ESP 8266 WiFi Serial Transceiver

esp8266Look what I found under the Christmas tree about a week ago – ESP8266 modules 😉

A tiny module with a size comparable to a Fibaro Universal Sensor. Basically, the ESP8266 enables you to add wireless connectivity to your hardware – to anything that does serial; for example your Arduino or Raspberry Pi. But there’s more to this module – you can build your own firmware, program it in Lua and do stuff that might make that additional MCU obsolete.

But for now, all I have to tell is about my experiences during the first few hours with the ESP8266.

After unwrapping the modules the first thing I had to find out the purpose of the 8 pins of the ESP8266 board. VCC, GND, RX, TX and found out about those really quick; I also found a small Arduino sketch that would allow me to test some basic functions of the ESP8266 so I set things up on a breadboard and uploaded a sketch to a spare Arduino.

No go.. I had been a bit too hasty, thinking that a single webpage would provide all the information I needed to get things going. After a while I found out that the CH_PD pin had to be pulled up to VCC to set the ESP8266 to ‘normal operation’ mode. Cause when pulled to GND, the ESP8266 will just sit and wait for a new firmware to arrive … OK, got that. Still no go; that’s when I read about the different baud rates – some models work with 9600 bps, others with 57600 bps so maybe the ESP8266 and the Arduino weren’t using the same bps. So I decided to just hook up the 2nd ESP8266 to my PC with a PL2303 serial to USB cable and see if I could get the module to respond to some AT commands.

ESP8266 with PuTTYPuTTY is one of my favorite tools for telnet, ssh and it can also do serial communication, so I used PuTTY to send some AT commands to the ESP8266, but after some time I gave up… I knew I was using the right speed (9600 bps in my case) but the module just didn’t respond after- no OK; nothing.

ESP8266 with TermiteChanged some settings, still no answer. Made PutTTY add an additional LF after a CR but still no luck? Eventually I switched to Termite, cause I found out that I wasn’t the first having problems with the combination PuTTY + ESP8266.

 

Now that I knew it was 9600 instead of 57600 bps I could proceed with the sketch mentioned earlier. Another ‘problem’ was that the Arduino operates @5V and the ESP8266 @ 3.3V, so I had to do something about that too; I saw all kinds of solutions with level shifters, resistors, diodes and decided to go the easy way – a JeeNode which operates @ 3.3V:

ESP8266 on a JeeNode

The result in the PuTTY window that was connected to the ‘debug’ Serial port (pins 6 & 7):

ESP8266 PuTTY screen

Neat. Needless to say that this is just a small example of what this ESP8266 can do; it can do much more than just fetching a small web page – I haven’t even scratched the surface yet! A Wifi Door sensor for the price of an ESP8266, a reed switch + magnet and power supply.. the latter is the only downside of the ESP8266: it consumes too much power to run on batteries. Well, you can’t have ’em all … (yet?) Besides that, I love this little powerful Wifi module.

Opentherm Monitor finished

This post will be the last one about the Opentherm Monitor. OTOH, when is something really completely finished…

Arduino Serial Monitor

I could spend some more hours on the Opentherm (OT) Monitor and in particular the sketch, but for now it’s good enough. I should add some extra code to validate the OT frame but that would also mean I won’t be able to analyze ‘strange’ frames with unknown Data ID etcetera on my PC. So I’ll leave it as-is for  now. The Opentherm Gateway is waiting 😉

From what I’ve seen during the last 24 hours, the ‘quality’ of the frames I receive is quite good; somehow there seems to be an invalid frame on the wires every minute or so, and I can’t find out what it is. This same thing happens with the Opentherm Gateway Monitor, so I think both are having the same problem. The Data ID tells me it’s probably an OEM frame…?

OpenTherm Decoder

The Opentherm Decoder running on my PC receives the 4 OT bytes from a serial port and decodes those bytes to something human readable: whether the frame came from the Thermostat or the Boiler, Message type and the meaning of the Data ID. The 16-bit data value (there where you can find the temperatures, pressure and status bits) is not decoded yet; well, it’s all in the Opentherm Protocol documentation, so that should be no problem.

Now I can use this Opentherm Monitor as an additional display near the boiler! The Remeha Calenta already has a rather large display showing stuff like status, water pressure, whether the pump is running, but it doesn’t display flow- and return temperature, control setpoint and I’m sure I can think of some more interesting stuff I wanna see – that’s what the Opentherm Monitor is going to do for me. I already have a 16×4 LCD, so all I have left to do is finding a suitable enclosure, build everything in there and I’m done!

I really liked getting this Opentherm Monitor to work without errors; in fact, getting it to work was more exciting than building it. Learning on the job about ATMega timers, Manchester decoding and programming the whole thing in C from scratch was one big adventure.

The most important references I used were:

And here‘s the sketch- no additional libraries needed, free to use and no guarantees that it will work for you just as well as it does for me. Have fun!

 

Got it!

Yes, yes, YES! I love it 🙂

The rethinking of the OpenTherm bit capturing strategy I did a few days ago really did improve things quite a lot, as can be seen on a screendump of my OT Decoder:

"My" OpenTherm decoder

Below is what the Opentherm Monitor, a tool that belongs to the OpenTherm Gateway (yep, I’ve got one of those too, since a week or so 😉 ) is showing in the log:

Gateway Opentherm Monitor

This is great, wonderful result! And the time it took to come this far was well spent, cause I’ve learned a lot in the past days. I’m starting to know the Opentherm Data ID’s by heart, I know some more about the ATMega timers and Manchester decoding has no secrets for me anymore.

You may notice that in my case only the Master (lines with a ‘T’ in them, in both screendumps) frames are decoded and not the Slave response frames (Slave=Boiler, starting with a ‘B’ in the lower screendump), but that’s because I added a delay that causes the sketch to skip the Slave responses.

So the sketch is going to be the next thing I’m going to work on in the next couple of days. But for today, I’m going to relax and and stop thinking about microseconds, timers, prescalers and ISRs;  it’s time for something completely different: a Somfy RS-485 RTS Transmitter for our rolling shutters which will (hopefully) arrive in  a few weeks! That’s why I like Domotica so much – the diversity of things to do and learn.

 

RF to Zigbee gateway

The last piece of missing hardware is finished. The picture below shows the 2nd RF to Zigbee gateway I had to make to be able to receive all the Hydronic balancing sensors I made earlier this week. One of those sensors just couldn’t make it through 3 walls all day long, so I created  a temporary solution on a breadboard to solve this.

A very simple yet effective way (for me) to get the sensor data where I want it (in my Domotica system) with minimal effort.

The JeeNode acts as a RF receiver and just echoes everything with a valid CRC to the Digi XBee; from there it eventually arrives at my Zigbee Coordinator with which I can communicate over TCP/IP.

The JeeNode runs a slightly modified version of the RF12Demo sketch made by Jean-Claude Wippler. I used the NewSoftSerial library to create an additional Serial port, and added a few print statements for the XBee port, right there where the RF12Demo Serial.println()’s the received RF data to the Serial port. Compile, Upload, setting the RF band,  group- and node ID and I’m done!

This JeeNode is powered by a 5V USB adapter and the XBee gets its power from the 3.3V JeeNode ports. The XBee uses a Zigbee End Device AT firmware (2864) with the Sleep Mode set to Pin Hibernate. But because pin 9 is wired to GND, this means that the XBee is permanently on.  Only 3 wires are needed to connect the XBee to the JeeNode: 3.3V, GND and a JeeNode digital pin to the XBee DOUT.

That’s it – moving on with where this all started with: understanding the flow of  heating energy in our house!

Hydronic balancing

This weekend I came to the conclusion that whatever I do, I’ll never get a well performing central heating without hydronic balancing. I’ve been watching how the temperatures in all the rooms of our house react and where the energy goes to – it’s a mess! No matter how well I’ll be able to control the kettle, temperature control in all the rooms will still be a mess without a hydronicly balanced system.

So this weekend I decided to stop what I was doing (building the Opentherm Gateway) and first try to do something about this balancing issue.

Hydronic balancing is not something I’m familiar with, and I certainly don’t have the tools for it ; but what i can do is provide enough information with these sensors; I don’t know if I’ll succeed, but it’s worth to give it a try.

Sensor for Hydronic balancing The first thing I need to know is how much energy flows through the radiators. Well, I can do that, I guess… A JeeNode with RF transmitter and 2 1-Wire DS18B20 sensors can provide me information about how much energy each radiator produces by measuring the flow- and return temperatures of each radiator.

I’ve got a bunch of JeeNode kits still waiting to be used, enough 1-Wire sensors, batteries and all other components needed, so what am I waiting for??

So this weekend I built a first sensor and a RF-to-Zigbee ‘gateway’ so I can receive all the sensors without the need of USB, RS232 or an additional Ethernet port.

The first sensor is operational now; more will follow!

Red = flow temperature, Blue = temperature drop

Something old and something new

While I’m logging data and waiting for it to become substantial enough to test another project with, I thought it would be nice to kill the time and start building some new sensors: motion, temperature and light.

My goal is to monitor the complete second floor. For the 3 bedrooms I want to monitor motion, temperature and light. The passage will only require motion and light, while measuring the humidity in the bathroom would be nice too.

So, for that I need the following:

  • 5 * motion;
  • 4 * temperature;
  • 5 * light;
  • 1 * humidity.

I already have the motion sensors (PIR), the temperature sensors (1-Wire DS18B20), light sensors (LDR), enough JeeNodes, so all I need is a new humidity sensor.

OK; the amount of sensors I want to use is a bit too much for a single JeeNode I guess. Maybe 2 JeeNodes will suffice? I didn’t do any math on this yet, but I don’t think a single JeeNode can do the job all on its own. 2 JeeNodes maybe, cause the temperature sensors can be on a single 1-Wire bus, so those sensors won’t use up much ports on the JeeNode; and maybe I can use an Analog Plug to measure the LDRs? We’ll see. But for now, I assume I’ll need more than 1 JeeNode. However, I don’t like to spend an XBee module on each of those JeeNodes, cause I want to keep it relatively low cost. Yeah I know, there are cheaper alternatives for XBee..

But wait, I’ve still got an Arduino Fio; I’ve bought it sometime in January, not for a specific purpose but just to have a look at it, and I never found a good use for it yet. This Fio is also based on the ATmega328P and runs at 3.3V and 8 MHz. And it also has an XBee socket and USB connector. And it’s getting old, gathering dust on the shelf…

Now why don’t I use this Fio as a ‘hub’, or ‘central node’? It shouldn’t be too hard to make the (1,2,..) JeeNodes talk to this Arduino Fio over a few wires and use the Fio to transmit all the sensor values to the Zigbee coordinator; now that would be nice! I could power this Fio with a small USB power adapter and run a few wires (for power and serial connection) to the JeeNodes. Maybe I could just as easily use 1 JeeNode per room; this leaves enough ports available for future expansion… well, let’s think this one over for some time!

Ethernet Doorbell

Yep, today the old Doorbell has been removed and my new Ethernet Doorbell is in use! As you can see the white LED was on when I took the picture on the left, which means that it was dark outside at that time. By polling my Domotica system, the doorbell ‘knows’ whether it’s dark outside or not. Of course it’s not the doorbell itself, but the hardware behind it that does the ‘knowing’: a JeeNode with Ethercard.

All I had to do today was some soldering, finishing the doorbell sketch and pulling the UTP cable through the hole in the wooden door frame. Those last 2 items made it all take a total of 9 hours before I could actually say that the job was completely finished…

Using XMLRPC to communicate to my Domotica system was a bit too much for the 2 kilobytes of RAM on the ATMega. Strings like

<?xml version="1.0"?><methodCall><methodName>GetDevice</methodName><params>
<param><value>DUSKDAWN1.DAYLIGHT</value></param></params></methodCall>

just to get the current value of a dusk/dawn sensor made the amount of free RAM disappear very quickly, so I decided to use another approach I already used some years ago. It hasn’t been used for some time, but it was still available and ready to use. Now all I have to send to my Domotica system is a

GET DUSKDAWN1.DAYLIGHT

and a Indy IdCmdTCPServer takes care of supplying the response. This saves a lot of RAM! Free RAM went up to 900 bytes again, where the XMLRPC method made RAM drop below 200 bytes. And an additional bonus: no need for XML parsing of the response 😉

 

The picture below shows the hardware that’s making it all possible: a JeeNode, an EtherCard and a modified Utility Plug. The cable at the bottom goes to the doorbell, the yellow cable on the right goes to an Ethernet switch and the top right cable is the power supply.

 

And below the sketch that’s currently running on the JeeNode:

 

//
// Doorbell
//
#include <EtherCard.h>
#include <Ports.h>
#include <RF12.h> // needed to avoid a linker error :(
// ethernet interface mac address, must be unique on the LAN
//char website[] PROGMEM = "virtualxp.hekkers.lan";
char website[] PROGMEM = "domoticaserver.hekkers.lan";
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x30,0x31 };
static byte myip[] = { 192,168,10,75 };
static byte gwip[] = {192,168,10,60 };
static byte dnsip[] = {192,168,10,10 };
static byte hisip[] = {192,168,10,40 };
Stash stash;
MilliTimer DuskDawn;
MilliTimer HeartBeat;
byte Ethernet::buffer[600];
byte HP3=0;
byte VP3=0;
byte night = 0;
byte vnight = 0;
#define DDINTERVAL 60000  // 10 minutes = 600 seconds
#define HBINTERVAL 9000   // 1.5 minutes = 90 seconds
//
// LED & button related stuff
//
Port BlueLed (1);
Port Button (3);
Port WhiteLed (4);
int freeRam () {
extern int __heap_start, *__brkval;
int v;
return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}
void(* resetFunc) (void) = 0; //declare reset function @ address 0
void setup () {
Serial.begin(57600);
Serial.println("[Doorbell_01]");
Serial.println(freeRam());
if (ether.begin(sizeof Ethernet::buffer, mymac) == 0)
Serial.println("Failed to access Ethernet controller");
if (!ether.staticSetup(myip, gwip, dnsip))
Serial.print("IP address setup failed");
ether.printIp("IP:  ", ether.myip);
ether.printIp("GW:  ", ether.gwip);
ether.printIp("DNS: ", ether.dnsip);
if (!ether.dnsLookup(website))
Serial.print("DNS failed");
ether.printIp("SRV: ", ether.hisip);
ether.hisport = 8000;
// port 3 = doorbell button
Button.mode(INPUT);
Button.digiWrite(HIGH);   // pull-up DIO
// BlueLed = push LED
BlueLed.mode(OUTPUT);
BlueLed.digiWrite(0);
// WhiteLed = night LED
WhiteLed.mode(OUTPUT);
WhiteLed.digiWrite(0);
DuskDawn.set(1);
}
static void send_callback (byte status, word off, word len) {
Serial.println("<P<rn");
Ethernet::buffer[off+600] = 0;
Serial.print((const char*) Ethernet::buffer + off);
Serial.println("...");
}
static void poll_callback (byte status, word off, word len) {
const char* res = (const char*) Ethernet::buffer + off;
Serial.println("<G<rn");
Ethernet::buffer[off+600] = 0;
Serial.println(res);
if((res[0]=='R')&&((res[1]=='='))) {
night = (res[2]=='1')?0:1;
Serial.print("n=");
Serial.println(night, DEC);
}
if(res[0]=='E') {
resetFunc();
}
Serial.println("...");
if (night!=vnight)
{
WhiteLed.digiWrite(night);
vnight = night;
}
}
void PollDuskDawn(){
//return;
Serial.println("?");
byte sd = stash.create();
stash.println("DUSKDAWN1.DAYLIGHT");
stash.save();
// generate the header with payload - note that the stash size is used,
// and that a "stash descriptor" is passed in as argument using "$H"
Stash::prepare(PSTR("GET $H"), sd);
// send the packet - this also releases all stash buffers once done
ether.tcpSend(poll_callback);
Serial.println(".");
}
void SendStatus() {
Serial.print("P");
byte sd = stash.create();
stash.print("DB2=");
stash.println(HP3, DEC);
stash.save();
// generate the header with payload - note that the stash size is used,
// and that a "stash descriptor" is passed in as argument using "$H"
Stash::prepare(PSTR("PUT $H"), sd);
// send the packet - this also releases all stash buffers once done
ether.tcpSend(send_callback);
Serial.println(".");
}
void loop(){
ether.packetLoop(ether.packetReceive());
if (ether.clientWaitingGw()) return;
if (DuskDawn.poll(DDINTERVAL)) {
PollDuskDawn();
}
if (HeartBeat.poll(HBINTERVAL)) {
SendStatus();
}
HP3 = !Button.digiRead();
if (HP3 != VP3)
{
if (HP3) {
Serial.println("Button Pressed!");
BlueLed.digiWrite(HIGH);
WhiteLed.digiWrite(LOW);
SendStatus();
HeartBeat.set(HBINTERVAL);
}
else
{
Serial.print("Button Releasedrn");
BlueLed.digiWrite(LOW);
WhiteLed.digiWrite(night);
}
}
VP3=HP3;
}

That’s it! On to the next adventure 😉

 

Doorbell software

Today I continued working on the Doorbell controller hardware & software. First I had to solder a Utility Plug, but I did it somewhat different than usual, I guess:

Utility Plug

(sorry JC, I seem to have some sort of “destructive” way of using your creations 😉

The way I’m going to use this Utility Plug is not standard; the only reason I want to use this Utility Plug is to have a good-fitting RJ12 plug sticking out of the enclosure in a way that it keeps the whole construction as strong as possible. I could just as easily have glued a RJ12 Plug into the enclosure, but with the headers plugged into the carrier board I think I have a stronger construction than without the use of this Plug.

Since I will need to use more than 1 JeeNode Port (a button and 2 LEDs), I made sure that the Utility Plug headers were isolated from the Plug but were still usable construction-wise, so I cut all connections between the headers and the PCB, as can be seen in the picture above. Instead of using the headers, I soldered an Extension cable to the Utility Plug; this way I can do whatever I want with those 6 wires! After checking the header isolation with a multimeter and connecting the Extension cable wires to the JeeNode ports, I was ready to start writing some code. But first, let’s specify how this Ethernet enabled doorbell should behave. It should:

  • Periodically query my system and ask whether it’s dark outside or not and turn on the “night LED” inside the doorbell based on that. If the query fails, it should always turn on the night LED;
  • Switch on the blue “signalling LED” when the button is pressed; if the white night LED is on at that moment, it should switch that one off;
  • Switch off the blue “signalling LED” when the button is released; if it’s dark outside, it should switch the white night LED on;
  • Periodically send a heartbeat to my system;
  • Only send a “Doorbell pressed” message to my system once in a specific time-frame (1 second or so, to eliminate jitter).

I’m almost there! Most of these items are already working, but there are still some small issues to be resolved.  As always, the details take most of the time..