Adding a smart meter to the Fibaro HC2

This is (sort of) a follow-up on my previous post about the cheap Serial to Ethernet converter. It has kept me busy for a couple of evenings, for several reasons:

  • I wanted to use the Arduino Nano as a web server;
  • Make the Nano return a JSON object containing the smart meter data;
  • Learn more about the Fibaro HC2 on the job;
  • The new sketch was unstable.

It was mostly that last item that kept me busy for some time – well, waiting describes it better actually, cause the sketch I initially wrote was doing what it should, but wasn’t stable – it stopped working after 2 hours, the next time after 20 hours.. totally unpredictable. I also saw the Nano was sometimes missing incoming serial P1 data. And of course, these sort of things never happen while you’re there to witness it: I was always either at work or asleep (or both ūüėČ ).

So for a few days I tried to improve things and had to wait until the next day to see if the changes fixed the issue. It wasn’t lack of free RAM, it wasn’t the USB hub misbehaving, it wasn’t RF interference, nor was it a grounding issue. After I found a solution, the cause was very obvious actually: the Nano doesn’t do multitasking. Somehow taking care of receiving and processing serial data and handling an Ethernet shield interfered with each other – I still don’t know why, but after I denied the web server part of the sketch of servicing HTTP requests while the smart meter data was being received, this problem was solved. Detecting the beginning and end of a P1 packet is easy: it starts with ‘/’ and ends with ‘!’. So now the loop() looks like this:

void loop() {
  c = mySerial.read();
  switch (c) {
    case '/':
      receiving = true;
      ...
      break;
    case '!':
      receiving = false;
      ...
      break;
  }
  ...
  if(!receiving) {
    if (ether.packetLoop(ether.packetReceive())) {
      ...
      ether.httpServerReply(readingsPage());
    }
  }
}

With a P1 packet being about 500 bytes in size and 9600 baud this would ‘stall’ the response for about half a second (max.), which should not be a problem.

After I finally had a sketch that kept working for >48 hours it was time to have a look a the Fibaro HC2 and try to get the smart meter information (visibly) available in there.

Because I’ve spent very little time with the Fibaro HC2, please don’t put too much weight in what I have to say about the HC2, but: there’s one word that’s now associated with the HC2 in my head: impressive.¬†The HC2 case feels solid, looks good, gets warm, the UI looks good, the power consumption is about 13.5 W doing nothing¬†(I had no Z-Wave hardware to add), and the HC2 has something called¬†Virtual Devices. For me, that’s the thing I’m interested in the most – using the HC2 is not my thing, but exploring what I can do to add my own things to it is..

So after I hooked up the HC2 to my LAN, performed some upgrades and changed some settings, I immediately focused on those Virtual Devices. P1 Smart meter in Fibaro HC2I know I’ve just scratched the surface so far, but being able to create a Virtual Device within minutes and being able to get the smart meter data from the Nano web-server into Virtual Device properties and being displayed nicely as well with just a lua script of ~20 lines of code – that’s impressive! Not bad for a beginner ūüėČ And for a price <20 Euro! Add a database solution, charting and I think the HC2 has a great future ahead.

 

Although I liked what I’ve seen so far, I do have some things that might be not so handy.

The lua script I wrote, was only 20 lines or so. Editing the code from within the browser, in a small window is not really as comfortable as editing code can/should be. And more important, everything you need in a script has to be in there (AFAIK) – so suppose you have to calculate a CRC, the CRC code has to be inside the script that needs it – no way of including source code files which will probably be used in more ‘main loops’ than just 1. I’d really like to see some sort of ‘Plugin’-ish way to add support for exotic hardware; a bunch of files with code, UI pages that can be embedded in the HC2 UI (e.g. for configuration stuff), creating Plugin specific triggers etcetera. In other words: really embed your own creations into HC2.

If Fibaro can accomplish that, then the HC2 can become the killer solution for Home Automation for a lot of people; with Z-Wave as the base technology, yet still expandable with support for other popular hardware.

Oh my, almost forgot the lua script, here it is:

--[[
%% properties
%% globals
--]]
fibaro:debug("start")
if (nano == nil) then
  nano = Net.FHttp("192.168.10.191", 80)
end
response, status, errorcode = nano:GET("/")
fibaro:debug(response)
local data
data = json.decode(response)
-- not 0 based!
jso = data[1]
v181  = jso.v181
v182  = jso.v182
v170  = jso.v170 * 1000
v2421 = jso.v2421
-- deviceid 7
fibaro:call(7, "setProperty", "ui.lbl181.value", v181.." kWh")
fibaro:call(7, "setProperty", "ui.lbl182.value", v182.." kWh")
fibaro:call(7, "setProperty", "ui.lbl170.value", v170.." W")
fibaro:call(7, "setProperty", "ui.lbl2421.value", v2421.." m3")
fibaro:sleep(10*1000)

Have fun!

Flexible Serial to Ethernet for less than 20 Euro

Nano v3 with Ethernet Shield

Wired connections have always been my favorite for connecting hardware to my Home Automation System. This week I found a new way of doing that – costing only 17 Euros, purchased at AliExpress.

What you see in the image above is a¬†ENC28J60 based Ethernet shield with a Arduino Nano 3 on top of it. Measuring about¬†70 x 20 x 37 mm (with the lower pins cut to half their length). Very small and programmable ūüėČ

Last Friday 2 sets (of shield & Nano) arrived and I just couldn’t resist giving them a try and since I’ve still got a Raspberry Pi near the smart meter for the sole purpose of collecting the data it produces, it looked like a good idea to see if I could replace the RPi with this Ethernet-Nano.

Finding a library for the¬†ENC28J60 based shield wasn’t hard – I had already worked with Jean-Claude Wipplers Ethercard before and which is still in use as our doorbell controller, so that wouldn’t be any problem. But first I had to solve the problem I had when I connected the Nano to my PC- ‘Device enumeration failed’ is what USBView told me. It took a while before I got the idea to place a powered USB hub in between the two…. problem solved.

This Ethernet shield uses pin 10 for CS (Clock Select) but this was easy to change in the code; instead of

if (ether.begin(sizeof Ethernet::buffer, mymac) == 0)

all I had to do was supply an extra parameter and it all worked instantly:

if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0) 

There’s one small problem I still have to fix, which is doing a DNS query on the Nano. It times out and I have no clue why; for now I added a ‘fallback’ IP address in the code until it’s fixed.

EthernetNano test setupI used a 2nd Arduino to play the smart meter ‘role’ by sending a P1 datagram to the Nano over a serial connection with an interval of 10 seconds – that would be as close to reality as it could get. I connected the TX pin of the Arduino Ethernet (the bottom one) ¬†to the RX pin on the EthernetNano and all that was left to do was writing a sketch.

Sometimes it looks like everything has been developed before already, the only thing you have to do is find it or remembering where you saw it – well, that was the case here also. With the Ethercard library come a couple of examples which I could use for the Ethernet part of the sketch. And I knew that Jean-Claude Wippler had blogged about P1 data a couple of times – strip the RF12 code and replace it with Ethernet code and I would be ready to go… and that’s exactly what I did ūüėČ

Here’s the code for my first EthernetNano handling the smart meter data and uploading it (with HTTP) to my HA system! Bye bye RPi…

/// @dir p1scanner
/// Parse P1 data from smart meter and send as compressed packet over RF12.
/// @see http://jeelabs.org/2013/01/02/encoding-p1-data/
// 2012-12-31 <jc@wippler.nl> http://opensource.org/licenses/mit-license.php

// Changed to work with Ethernet shield, 2014-06 Robert Hekkers

#include <SoftwareSerial.h>
#include <EtherCard.h>

#define DEBUG 1   // set to 1 to use fake data instead of SoftwareSerial
#define LED   0   // set to 0 to disable LED blinking

SoftwareSerial mySerial (7,17 /*, true*/); // rx, tx, inverted logic
#define NTYPES (sizeof typeMap / sizeof *typeMap)
// list of codes to be sent out (only compares lower byte!)
const byte typeMap [] = {181, 182, 281, 282, 96140, 170, 270, 2410, 2420, 2440};

byte Ethernet::buffer[700];
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x00,0x01 };
const char website[] PROGMEM = "devbox.hekkers.lan";
uint8_t hisip[] = { 192,168,10,179 };
Stash stash;

byte type;
uint32_t value;
uint32_t readings[NTYPES+1];

static bool p1_scanner (char c) {
  switch (c) {
    case ':':
      type = 0;
      value = 0;
      break;
    case '(':
      if (type == 0)
        type = value; // truncates to lower byte
      value = 0;
    case '.':
      break;
    case ')':
      if (type)
        return true;
      break;
    default:
      if ('0' <= c && c <= '9')
        value = 10 * value + (c - '0');
  }
  return false;
}

static void collectData (bool empty =false) {
  if (!empty) {
    for (byte i = 0; i < NTYPES; ++i) {
      Serial.print("@ ");
      Serial.print(typeMap[i]);
      Serial.print('=');
      Serial.println(readings[i]);
    }
    byte sd = stash.create();
    stash.print("0,");
    stash.println(millis() / 1000);
    for (byte i = 0; i < NTYPES; ++i) {
      stash.print(typeMap[i]);
      stash.print(",");
      stash.println(readings[i]);
    }
    stash.save();
    // generate the header with payload - note that the stash size is used,
    // and that a "stash descriptor" is passed in as argument using "$H"
    Stash::prepare(PSTR("GET http://$F/p1/ HTTP/1.0" "rn"
                        "Host: $F" "rn"
                        "Content-Length: $D" "rn"
                        "rn"
                        "$H"),
            website, website, stash.size(), sd);
    // send the packet - this also releases all stash buffers once done
    ether.tcpSend();
  }
}

void setup () {
  if (DEBUG) {
    Serial.begin(115200);
    Serial.println("n[p1poster]");
  }
  mySerial.begin(9600);
  // digitalWrite(7, 1); // enable pull-up
  collectData(true); // empty packet on power-up

  delay(2000);
  if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
    Serial.println( "Failed to access Ethernet controller");
  if (!ether.dhcpSetup())
    Serial.println("DHCP failed");

  ether.printIp("IP:  ", ether.myip);
  ether.printIp("GW:  ", ether.gwip);
  ether.printIp("DNS: ", ether.dnsip);

  if (!ether.dnsLookup(website)) {
    Serial.println("DNS failed");
    ether.copyIp(ether.hisip, hisip);
  }
  ether.hisport = 80;
  ether.printIp("SRV: ", ether.hisip);
}

void serialLoop() {
  byte c;
  while(mySerial.available()){

    c = mySerial.read();
    if (c > 0) {
      //c &= 0x7F;
    }
    switch (c) {
      case '/':
        break;
      case '!':
        collectData();
        memset(readings, 0, sizeof readings);
        break;
      default:
        if (p1_scanner(c)) {
          for (byte i = 0; i < NTYPES; ++i)
            if (type == typeMap[i]) {
              readings[i] = value;
              break;
            }
        }
    }
  }
}

void loop () {
  ether.packetLoop(ether.packetReceive());
  serialLoop();
}

I’m sure I’m going to use this EthernetNano more in the future – the very small size, price, built-in flexibility make this a great solution for a lot of things!

 

 

 

 

 

Raspberry Pi, P1 smart meter packets and Python

Yesterday evening I was looking for some fun thing to do; getting more familiar with the Raspberry Pi sounded nice. But what could I do? I needed something to work on, not just some fooling around with no end result.. and it should include working with Python, cause I’ve wanted to learn more about Python for quite some time but never got to it..

But when I landed on the website of G√© Janssen, I knew what I was going to do the next couple of hours: let’s see how easy it is to get the data of my smart meter into a MySQL database on the RPi!

RPi with USB Flash drive connected to an Arduino

RPi with USB Flash drive connected to an Arduino

Since I didn’t want to disrupt my own smart meter readings, I used a logfile that contains the ‘raw’ smart meter data of almost a week:

/ISk52ME382-1003
0-0:96.1.1(4B414C37303035313039)
1-0:1.8.1(00180.724*kWh)
1-0:1.8.2(00001.416*kWh)
1-0:2.8.1(00000.000*kWh)
1-0:2.8.2(00000.000*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(0000.42*kW)
1-0:2.7.0(0000.00*kW)
0-0:17.0.0(0999.00*kW)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:24.1.0(3)
0-1:96.1.0(32383031313030313231)
0-1:24.3.0(120911010000)(00)(60)(1)(0-1:24.2.1)(m3)
(00007.646)
0-1:24.4.0(1)
!

Although it’s still not the real thing, with over a 50000 packets like the one above, I should be able to see if it’s working ūüėČ

Now how do I get that data to the Raspberry Pi, in a way that’s as close to reality as possible? The log file that contains the P1 packets is on my NAS somewhere, but I wanted to feed the data through one of the USB ports of the RPi. Well, I can use an Arduino and a USB BUB for that. A VB.Net console app, Arduino sketch, USB BUB and some wires should get me going. First the VB.Net app, which feeds the Arduino:

Imports System.IO
Imports System.IO.Ports

Imports Microsoft.VisualBasic

Module Module1
  Private COMPort As SerialPort

  Sub InitPort()
    COMPort = New SerialPort
    COMPort.PortName = "COM16"
    COMPort.BaudRate = 9600
    COMPort.DataBits = 8
    COMPort.Parity = Parity.None
    COMPort.StopBits = 1
    COMPort.Open()
    Console.WriteLine("COMPort open.")
  End Sub

  Sub Main()
    InitPort()
    Dim reader As StreamReader
    Dim LineInput As String = ""
    Dim Counter As Integer = 0

    reader = My.Computer.FileSystem.OpenTextFileReader("U:P1data.txt")
    Console.WriteLine("File opened.")

    While Not reader.EndOfStream
      LineInput = reader.ReadLine()
      Console.WriteLine(LineInput)
      COMPort.Write(LineInput & Chr(13) & Chr(10))
      If LineInput.Equals("!") Then
        System.Threading.Thread.Sleep(5000)
      End If
    End While
  End Sub
End Module

And an Arduino sketch:

#include <SoftwareSerial.h>
// rx, tx
SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(9600);
  Serial.println("SM Feeder");
  // set the data rate for the SoftwareSerial port
  mySerial.begin(9600);
}

void loop() // run over and over
{
  if (Serial.available()){
    char c = Serial.read();
    mySerial.print(c);
  }
}

Ok, now lets see if the smart meter data is arriving @ the RPi USB port:

slim@raspberrypi:/root $ cu -l /dev/ttyUSB0 -s 9600 --parity=none 
Connected.
/ISk52ME382-1003
0-0:96.1.1(4B414C37303035313039)
1-0:1.8.1(00180.912*kWh)
1-0:1.8.2(00001.416*kWh)
1-0:2.8.1(00000.000*kWh)
1-0:2.8.2(00000.000*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(0000.48*kW)
1-0:2.7.0(0000.00*kW)
0-0:17.0.0(0999.00*kW)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:24.1.0(3)
0-1:96.1.0(32383031313030313231)
0-1:24.3.0(120911020000)(00)(60)(1)(0-1:24.2.1)(m3)
(00007.654)
0-1:24.4.0(1)
!

Cool. But we’re not there yet… now I need a Python script to read the smart meter data, ¬†filter out the important stuff and put it in a MySQL table.

First I had to install some additional packages: python-dev, python-pip, python-mysqldb, pyserial and mysql-server. I installed apache2 and php5 as well, cause I had a feeling this wasn’t going to end with a Python script or a MySQL table being filled…

Although this Python script is ‘just’ ¬†about 80 lines, this was the part that consumed most of the time (and one of the reasons I started with all this):

#!/usr/bin/python
# -*- coding: utf-8 -*-

import MySQLdb as mdb
import sys
import serial
con = None
try:
    con = mdb.connect('localhost', 'p1writer', 'thisissecret', 'p1data')
except mdb.Error, e:

    print "Error %d: %s" % (e.args[0], e.args[1])
    sys.exit(1)

finally:
    if con:
         print("MySQL OK, proceeding...")

ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.EIGHTBITS
ser.parity=serial.PARITY_NONE
ser.port="/dev/ttyUSB0"
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=0
ser.rtscts=0

try:
    ser.open()
except:
    sys.exit ("Error opening port")

# parse function

type=''
value=''
dot=0

def p1parse(c):
  global type
  global value
  global dot

  if c == ":":
    type=''
    value=''
  elif c == "(":
    if type == '':
      type=value
    value=''
    dot=0
  elif c == ".":
    if type <> '':
      value = value + c
    dot=1
  elif c == ")":
    if dot and type <> '':
      return 1
  else:
    if c in ['0', '1','2','3','4','5','6','7','8','9']:
      value = value + c
  return 0

cur = con.cursor()
while 1:
    try:
        c = ser.read(1)
    except:
        sys.exit ("Error reading port")
    if not c:
        break
    if p1parse(c):
        with con:
            cur.execute("INSERT INTO p1values(InfoType,Value) VALUES(%s, %s)", (type, value))

The p1parse() routine was borrowed from Jean-Claude Wippler as posted on his daily weblog here.

After some trial and error, debugging, tweaking mySQL to use the USB Flash drive (to keep the database separated from the OS) as database storage, I got it working this evening:

Smart meter data in MySQL database

Not a bad result after spending only a couple of hours… this really makes me curious whether the rest of what I (we) have in mind will go just as smooth…

A smarter smart meter

Well, I think my MQTT enabled smart meter is rather smart too, but this one is smart also, but in somewhat different way…

A few weeks ago my ‘good old‘ Domotica buddy Pieter Knuvers started asking questions about my new smart meter, its P1 port and if I could build something for him to get the P1 datagrams into his system. (B)Wired, preferably via Ethernet. OK, that means that Pieter needed something completely different than what I made for myself – no Simplecortex needed, no MQTT publishing and stuff like that. A simple TTL to Ethernet server would do.

And then I remembered a wiki page where I’d seen just that. I looked it up and indeed this was all Pieter needed; so I bought the EZL70A, an enclosure and started building.

Signal inverterThe EZL70A module is the biggest part and all that was needed was an additional signal inverter. For that I used a 7404 hex inverter, the same I used for my own smart meter, so I already knew that it would work. A few wires for power, signals and the hardest part was done.

I put some heat shrink around it and was ready to put all the components inside the enclosure.

Under construction

 

And this is what it looked like after the first evening. Soldering a few wires, cutting some holes in the enclosure and Pieter’s smart meter would be Ethernet enabled, exactly what he wanted to have. As you can see the EZL70A, together with the signal inverter, fits nicely into a small enclosure of 100 x 50 x 25 mm.

As power cable we decided to use a USB cable; that seemed like a good idea since it’s not that strange anymore to find a USB socket in the meter cabinet – it’s the most common place to also find a NAS, router or Internet modem. And all of those devices have USB these days ūüôā

After a brief test with my own smart meter I sent the P1 to Ethernet converter to Pieter. He had already finished the code for his own homebrew Domotica system to parse the P1 datagrams with the help of a logfile of my own smart meter.

So now all we had to do was waiting for the smart meter to be installed in his house. And last thursday it was; and Pieter connected the converter, changed the IP address and his smart meter was connected!

The¬†accessibility¬†of the P1 datagrams, the ease of building, the relatively low costs – it made us wonder why this wouldn’t work just as well for others as it did for Pieter… if we could just add some extra functionality to this (built-in web-page with charts etcetera) …

So we’ve both started doing some research… stay tuned!

Smart meter online

After 3 weeks, finally the end result came dangerously close ūüėČ

Every time I finish something new and the time comes to actually start using it, I get excited – will it work? Cause testing in the office is something completely different than seeing real live data coming in! (yep, I forgot to check if I still had a free connection on the switch in the fuse box…)

Simplecortex with Proto Shield

The Simplecortex had been running without any problems during the last 2 days so it was time for the last part: inverting the P1 signal before it arrives at the Simplecortex UART0. I had an old Arduino Proto Shield laying around and put an 7470 inverter on top.

The ‘receiving end’ i.e. my Domotica system is ready – which consisted of adding an MQTT client to it and making sure it subscribes to the necessary topics.¬†Just 2 more wires that needed to be soldered and I could hook up the Simplecortex to my smart meter… couldn’t wait!

And this evening I decided to remove the laptop that was logging the P1 data to a file from day 1 and replaced it with the contraption you see below.

Smart meter online!

And this is what it publishes (output from a MQTT subscriber):

/P1/powerusage1 00311.543
/P1/powerusage2 00158.107
/P1/powerdeliv1 00000.000
/P1/powerdeliv2 00000.000
/P1/tariff 0001
/P1/powerusagec 0001.88
/P1/powerdelivc 0000.00
/P1/gasusage 00027.758

Amazing…

The hardest¬†most exciting part was developing the code in C for the Simplecortex. For my work but also at home I see a lot of code, in all kinds of languages and I can follow what the code should do – but getting started with writing my own code in a language which I hadn’t really used before took me some time – I just kept on changing things because I was not satisfied. Now I am (thanks, JC!), although there are still some minor things I have to work on. But those are not necessary for now; I can flash the Simplecortex later, when everything is completely finished software-wise.

Now I can start logging gas usage again and I can stop using my DIN rail mounted KWh-meter with S0 output which I used for power usage (but not before I’ve had a chance to compare it with the smart meter, though). Source code will be available on GitHub soon (account has already been made) .

Next project that will be handled in a similar way: the Opentherm Gateway! For that one the Simplecortex needs to be able to subscribe as well!

Edit: the P1_Publisher project has been committed to Github some moments ago: https://github.com/rhekkers/P1_Publisher

 

Publishing smart meter P1 data

Got it!

I just had to know whether my Simplecortex code was any good; it would be nice to see some real results instead of all the test results that were created along the way. The code was in a ‘workable’ state so I was ready to test if I could turn the Simplecortex into a MQTT client and publish the relevant smart meter P1 data.

I already downloaded Mosquitto somewhere in July, tried some simple (and successful) tests with a Delphi MQTT Client, but now it was time for the real thing. The Simplecortex was already running when I started the mosquitto server from a command line and I saw the Simplecortex connecting to the mosquitto server (great, auto connect works):

Mosquitto server

No timeouts, disconnects or any other thing that looked suspicious; so lets see if the Simplecortex is actually publishing something. For that I started a mosquitto client from another command prompt to subscribe to the topic for the current power usage:

Mosquitto client

Yes! Every 10 seconds I see the current power usage being updated;  cool, it works!

The biggest hurdle I had to take was the fact that the uIP API doesn’t allow you to just send outgoing data when/where you’d like to – no, the uIP API¬†calls the application (a C function) when data is received, data has been¬†delivered to the other end, when a new connection has been set up, and when data has to be sent. Therefore I had to use a sort of publish¬†queue¬†to store the MQTT publish messages until the uIP API allowed the application (the MQTT client on the Simplecortex) to actually send the data to the MQTT broker. ¬†But the uIP stack had some good examples of which I gladly made use of ūüôā

Does this mean I’m finished? Not yet.. There are still some things I’m not happy about; the way some things are setup in code right now. Ping for example, uses a hardware timer which throws an interrupt way too often for an interval of (now) 30 seconds. And the UART is being read in main(); I want that nicely tucked away too. And there are some other things that still need to be improved, all caused by my lack of C knowledge – while developing code for part B, I realize I could better have coded part A in a different way (and better)!

But for now, I’m going to sit back, relax and watch it working ūüôā

Smart meter P1 project is doing fine

Surprise! Today I saw the following:

/ISk52ME382-1003

0-0:96.1.1(4B414C37303033334359373537343132)
1-0:1.8.1(00192.614*kWh)
1-0:1.8.2(00053.476*kWh)
1-0:2.8.1(00000.000*kWh)
1-0:2.8.2(00000.000*kWh)
0-0:96.14.0(0002)
1-0:1.7.0(0001.12*kW)
1-0:2.7.0(0000.00*kW)
0-0:17.0.0(0999.00*kW)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:24.1.0(3)
0-1:96.1.0(3238303131303031322356723034343132)
0-1:24.3.0(120913210000)(00)(60)(1)(0-1:24.2.1)(m3)
(00010.741)
0-1:24.4.0(1)
!

Yep, this means the gas usage is working too now! I called Liander last Monday to tell them my gas usage still wasn’t working and that I still got all zeroes. I had a bit of a problem explaining what I was doing (dunno why…), but at the end of the conversation I got the feeling my problem was taken seriously. That same day (late in the evening, as I found out today) the gas usage suddenly popped up; good, very good!

This evening I also managed to get the Simplecortex to automatically recover from lost connections. This is very very important because not being able to do that automatically would be a real show-stopper for me. Just think of it, that you would have to press the reset button after network problems or any other reason a connection has to be re-established; sounds horrible!

This weekend I’m going to have a look at 2 options to invert the P1 signal: a 7404 inverter IC or a BS170 transistor. These 2 options enable me to connect the cable from the P1 port directly to the UART of the Simplecortex; no additional components needed. I read about that last option (BS170)¬†here¬†and I just couldn’t resist ordering the parts for both options to see how they both work. Just out of curiosity…

And I’ll have to modify the UART library a bit cause as this library is now, it only supports 8 bits, no parity and 1 stopbit.

Next week will probably be dominated by adding MQTT Publishing support to the Simplecortex project.

This means that if all goes well,  at the end of next week I can install the Simplecortex near the smart meter and start subscribing to the smart meter topics!

Processing the smart meter data (3)

Well, 2 days later and I’m getting somewhere ūüôā

First of all, I must have been drunk when I said the Simplecortex was sending its data with an interval of 200 ms – no, that’s the timer resolution of my logger! The simplecortex sends a packet each time a newline (or line feed, LF) is received from the P1 port. So that’s solved.

First thing on the todo list was parsing the P1 data and extracting the numbers I want to receive in my system:

#define P1PU1  "1-0:1.8.1"      // total power usage normal tariff
#define P1PU2  "1-0:1.8.2"      // total power usage low tariff
#define P1PD1  "1-0:2.8.1"	// total delivered power normal tariff
#define P1PD2  "1-0:2.8.2"	// total delivered power low tariff
#define P1TIP  "0-0:96.14.0"    // tariff indicator power
#define P1PUC  "1-0:1.7.0"	// current power usage
#define P1PDC  "1-0:2.7.0"	// current power delivery
#define P1GTS  "0-1:24.3.0"	// timestamp gas usage sample

A routine was made to check each received line for the occurrence of the above-mentioned OBIS references (identification codes for data in DLMS/COSEM compliant metering equipment), get the value out of the received line and store the result in a buffer:

...
else if(strncmp(P1PU1, line, 9) == 0)
{
  strncpy(value, line+10, 9);
  value[9]=0;
  sendbuf("powertotalN;", value);
}
...

… and another routine that’s being called whenever the application is allowed (by the uIP TCP/IP stack) to send the data over to the other side of the connection. And this is another intermediate result, received by my domotica system:

22:41:54.406  TCP < powertotalN;00003.725
22:41:54.406  TCP < powertotalL;00000.000
22:41:54.406  TCP < powerdelivN;00000.000
22:41:54.406  TCP < powerdelivL;00000.000
22:41:54.406  TCP < powertariff;0001
22:41:54.506  TCP < powercurrent;0000.96
22:41:54.506  TCP < delivcurrent;0000.00
22:41:54.707  TCP < gastimestamp;120808010000
22:41:54.707  TCP < gasusage;00000000
22:42:04.722  TCP < powertotalN;00003.728
22:42:04.722  TCP < powertotalL;00000.000
22:42:04.842  TCP < powerdelivN;00000.000
22:42:04.842  TCP < powerdelivL;00000.000
22:42:04.842  TCP < powertariff;0001
22:42:04.842  TCP < powercurrent;0000.97
22:42:04.842  TCP < delivcurrent;0000.00
22:42:05.043  TCP < gastimestamp;120808010000
22:42:05.243  TCP < gasusage;00000000

That’s much better than what I had 2 days ago; well 4 hours ago actually, cause time is limited and I’m also learning more and more about C while working on this.

Next step: sending this data as payload in a MQTT Publish message! The protocol is not so hard in its simplest form and I’ve got working Delphi MQTT code, so this should not be a problem.

In the meantime some parts have arrived: an enclosure for the simplecortex, 7404 hex inverters and transistors to invert the P1 signal.

Onwards!

Processing the smart meter data (2)

Today I made some progress on processing the smart meter data with a SimpleCortex.

I started collecting the P1 data from our smart meter from the first day it was installed. A laptop, an RX-inverted USB cable and HyperTerminal take care of that for almost a week now.

Logging P1 smart meter data with HyperTerminal Now it was time to do something with the data I logged (currently 25 MB in size).

For that, I needed to create a test setup, with which I could simulate that the SimpleCortex was directly connected to the Smart meters P1 port.

Well, that was not so hard; I wrote a quick & dirty VB.Net tool that mimics the P1 ports behaviour, by reading the logged data and writing it to a Serial port.

This Serial Port is a USB-BUB connected to  UART0 of the SimpleCortex:

Feeding the SimpleCortex with smart meter data

Now I only needed a receiver for what the Simplecortex wants to send; for that I used a small Delphi app with a Indy IdTCPServer component – this logs everything it receives to a file, so I can easily monitor what’s being received.

With the VB.Net “P1 feeder”, the USB-BUB and the Delphi TCP server app I can now code, flash, test & debug with the Simplecortex right beside me in my office; I think I’ll need it, cause I’ve still got a long way to go.

Now that I had setup the right¬†test-environment¬†I could start with writing the code for the Simplecortex. If only I could get some sort of ‘Hello world’ example for the uIP TCP/IP stack to get me going – the Simplecortex webserver example looked like overkill; too much to learn at once, I needed something simpler to start with. Mark, who commented on an earlier post about the Simplecortex, was so kind to help me out and provided me with an example of a project that echoed the TCP data it received; and that was all I needed – suddenly it was clear to me how I should start.

And here it is, the output that’s being received from the SimpleCortex:

15:34:11.078  TCP < !
15:34:20.952  TCP < /ISk52ME382-1003
15:34:20.952  TCP < 137
15:34:21.112  TCP < 0-0:96.1.1(4B414C37444035313039373537343132)
15:34:21.112  TCP < 1-0:1.8.1(00002.368*kWh)
15:34:21.112  TCP < 1-0:1.8.2(00000.000*kWh)
15:34:21.112  TCP < 1-0:2.8.1(00000.000*kWh)
15:34:21.112  TCP < 1-0:2.8.2(00000.000*kWh)
15:34:21.312  TCP < 0-0:96.14.0(0001)
15:34:21.312  TCP < 1-0:1.7.0(0000.88*kW)
15:34:21.312  TCP < 1-0:2.7.0(0000.00*kW)
15:34:21.312  TCP < 0-0:17.0.0(0999.00*kW)
15:34:21.312  TCP < 0-0:96.3.10(1)
15:34:21.312  TCP < 0-0:96.13.1()
15:34:21.312  TCP < 0-0:96.13.0()
15:34:21.312  TCP < 0-1:24.1.0(3)
15:34:21.312  TCP < 0-1:96.1.0(3238303131303031344439323034343132)
15:34:21.513  TCP < 0-1:24.3.0(120807230000)(00)(60)(1)(0-1:24.2.1)()
15:34:21.513  TCP < (00000000)
15:34:21.513  TCP < 0-1:24.4.0(1)
15:34:21.513  TCP < !

For now, the only thing the Simplecortex does is waiting for data on UART0 and transmitting the data it received to the IP address of my PC. Sounds like a simple task, and it was; I was actually surprised with how fast I finished this first step.

But as I said: I still have a long way to go, cause this is still on my todo-list for this project:

  • Parse the incoming P1 data and collect the data I need (power usages, gas usage)
  • Act as a MQTT client and publish the collected data;
  • Being able to handle 7-E-1 with the UART instead of 8-N-1
  • Automatically reconnect when the connection fails for whatever reason;

And of course there are some technical issues I don’t understand fully yet; one of them being the interval with which the Simplecortex seems to send the data – every 200 ms? Strange.

So there’s still a lot to do, but the preliminary results¬†make me happy and give me confidence for a good end result ūüôā

Processing the smart meter data

Gas usage still doesn’t work, but I consider that a minor issue – getting the P1 data processed and into my system is what keeps my busy right now.

And I think I know how I’m going to do it; with a Simplecortex. I could take the (relatively) easy road and use an Arduino(-Ethernet) or a JeeNode (with Ether Card), but using a Simplecortex sounds much better actually ūüėČ

Because it’s something new.. I’ll get the chance to learn more about the uIP TCP/IP stack, I’ll have to enhance the Simplecortex UART library so that it will also support 7-E-1 instead of only the standard 8-N-1, I’ll have to browse through the¬†LPC1769 manual and I’m sure those things won’t be the only challenges I’m going to face in the next couple of weeks.

I know that I will need more time doing things this way, but I don’t care, actually.Cause for me creating is more fun than having/using it.

So today I ordered my 2nd Simplecortex, selected an enclosure and some ‘extras’ to take care of inverting the P1 signal. And this evening I’ve started with the software for the Simplecortex which should parse the P1 data, reduce it to the stuff I need from it and make sure it arrives in my domotica system.

And this is why Domotica is the perfect hobby for me; yesterday I had responsive web design on my mind, today it was about smart meters, and I seriously don’t know what tomorrow will bring – but what I do know is that it will be another exciting subject I want to know more about!