TFTP Bootloader for Simplecortex

About a week ago a¬†topic¬†on the LPCXpresso forum was brought to my attention. A TFTP boot loader, now that’s something I’d really like to have!

Whenever the MCU is (re-)started the boot loader will try to download a specific file from a specific TFTP server. If the boot loader can successfully download a valid file, it will flash the MCU and jump to the ‘user code’ at a predefined address.

So I mentioned this TFTP boot loader on the Simplecortex forum to see if anyone had already tried it. Nope, I was the 1st one, but one of the forum admins was so kind to have a look and created a project for it.

Lets see if I can get this working…

Setting up the environment was easy. I created a small project that uses interrupts to blink LEDs 6 to 9 at different intervals. In the project configuration I had to set the start address to a different value and decrease the size as well:

User code project settings

This is necessary so that the ‘user code’ won’t be flashed from address 0x00000000 which would result in the boot loader being overwritten – the start address had to be set to 0x0010000 (64 kb). This way the TFTP boot loader (which does start at 0x00000000) will stay intact and the ‘user code’ will be in the flash memory after the boot loader.

I copied the resulting .bin file to my TFTP server, flashed a Simplecortex with the TFTP boot loader and watched …

TFTP boot loader downloading file

OK, the first part works fine; I saw the file being downloaded by the Simplecortex. But that was all… nothing happened!

The boot loader has a built-in diagnostic option of a LED that blinks a number of times, depending on the kind of problem it encounters. LEDs see what this LED can reveal…

I changed the LED port to 0.26 (the easiest/closest pin to 3.3V) and I got 1 blink, meaning that the download was successful but the user code was invalid. Reading the code that validated the user code, I saw some sort of checksum being calculated. Ah, yes, I saw a checksum being mentioned somewhere. But I don’t have a ‘checksum’ program? I searched the web and I found a ‘lpcrc’ tool that was able to calculate the checksum of a .bin file and write that checksum into the .bin file: 4 bytes, starting at position 0x1c. Time for a post-build script:

Post build script

lpcrc Debugbin%1.bin
copy Debugbin%1.bin c:TFTP-Roottest.bin

Now the user code .bin file contained the correct checksum; retry!

New tests confirmed that the boot loader does¬†jump to the user code at address 0x00010000 now, but the user code runs ‘crippled’ – where it should toggle LEDs 6..9 each with its own interval, only LED9 is working…. and why? Dunno yet… something that needs to be reinitialized, reset, ??? A big mystery…

Interrupts maybe? The user code makes use of 4 TIMER interrupts; only TIMER3 seems to be working? I¬†hope I can find out what is going wrong here…

#include "LPC17xx.h"
#include "lpc_types.h"
#include "GPIO.h"
#include "timer.h"

uint32_t toggle = 1;
uint32_t toggle1 = 1;
uint32_t toggle2 = 1;
uint32_t toggle3 = 1;

int main(void)
  init_timer(0, 15000000);
  init_timer(1, 30000000);
  init_timer(2, 60000000);
  init_timer(3, 120000000);
  GPIOSetDir(1, 18, 1);GPIOSetValue(1, 18, toggle);   // LED6
  GPIOSetDir(1, 19, 1);GPIOSetValue(1, 19, toggle1);  // LED7
  GPIOSetDir(1, 24, 1);GPIOSetValue(1, 24, toggle2);  // LED8
  GPIOSetDir(1, 25, 1);GPIOSetValue(1, 25, toggle3);  // LED9
void TIMER0_IRQHandler (void){
  toggle = ~toggle;
  GPIOSetValue(1, 18, toggle);
void TIMER1_IRQHandler (void){
  toggle1 = ~toggle1;
  GPIOSetValue(1, 19, toggle1);
void TIMER2_IRQHandler (void){
  toggle2 = ~toggle2;
  GPIOSetValue(1, 24, toggle2);
void TIMER3_IRQHandler (void){
  toggle3 = ~toggle3;
  GPIOSetValue(1, 25, toggle3);

Migrating to stand-alone drivers this simple?

This evening I did a quick test to get an idea of what problems I would encounter when I would separate the drivers from the rest of the system (event handling and stuff).

So I created a new Delphi project of type Console Application, entered the code to create an instance of the PLCBUS¬†(hey, I’m mentioned there :-)) driver class (line 15) and just watched what would happen. Of course I had to create a dirty-looking loop because all the important stuff happens in a thread inside the driver, but I was surprised about how quickly I could turn this driver into a stand-alone executable – in a matter of minutes!

program DrvPLCBUS;


  Utils in 'CoreUtils.pas',
  PLCBUS in 'DriversPLCBUS.pas',
  PLC1141 in 'DriversPLC1141.pas';

  Driver: TPLC1141;

  Driver:=TPLC1141.Create('PLCBUS', True, TPLC1141CommandHandler);
  until false;

Of course, this doesn’t mean this driver is already functional in the way it is now: it will connect to the PLCBUS hardware by opening a TCP/IP connecting, but that’s it.

That’s because I have no way of sending commands to this stand-alone driver, cause in the ’embedded’ way (i.e. running inside my monolithic Domotica system) it would receive messages from internal message handlers – in this state, the internal message handler has only got 1 listener, namely the driver itself and nothing else…

But that’s where MQTT will come to the rescue!

Another great thing is that I can extend the driver code for stand-alone operation and still use the same driver in the embedded way. This is possible because Delphi’s System unit provides an¬†IsConsole variable with which you can decide what has to be done in both cases like this:

if IsConsole
then Writeln(Output, S)
else Logger.Add(S);

This way, in embedded mode everything will be logged to a file while the console version will write everything to standard output. So when I start the stand-alone driver I see this:

PLCBUS driver in console mode

This is great; just what I had in mind for a smooth transition: I can keep on extending my system as it is (build new versions, add new hardware drivers) and work on stand-alone mode for the drivers at the same time – no Big Bang!

Smart meter online

After 3 weeks, finally the end result came dangerously close ūüėČ

Every time I finish something new and the time comes to actually start using it, I get excited – will it work? Cause testing in the office is something completely different than seeing real live data coming in! (yep, I forgot to check if I still had a free connection on the switch in the fuse box…)

Simplecortex with Proto Shield

The Simplecortex had been running without any problems during the last 2 days so it was time for the last part: inverting the P1 signal before it arrives at the Simplecortex UART0. I had an old Arduino Proto Shield laying around and put an 7470 inverter on top.

The ‘receiving end’ i.e. my Domotica system is ready – which consisted of adding an MQTT client to it and making sure it subscribes to the necessary topics.¬†Just 2 more wires that needed to be soldered and I could hook up the Simplecortex to my smart meter… couldn’t wait!

And this evening I decided to remove the laptop that was logging the P1 data to a file from day 1 and replaced it with the contraption you see below.

Smart meter online!

And this is what it publishes (output from a MQTT subscriber):

/P1/powerusage1 00311.543
/P1/powerusage2 00158.107
/P1/powerdeliv1 00000.000
/P1/powerdeliv2 00000.000
/P1/tariff 0001
/P1/powerusagec 0001.88
/P1/powerdelivc 0000.00
/P1/gasusage 00027.758


The hardest¬†most exciting part was developing the code in C for the Simplecortex. For my work but also at home I see a lot of code, in all kinds of languages and I can follow what the code should do – but getting started with writing my own code in a language which I hadn’t really used before took me some time – I just kept on changing things because I was not satisfied. Now I am (thanks, JC!), although there are still some minor things I have to work on. But those are not necessary for now; I can flash the Simplecortex later, when everything is completely finished software-wise.

Now I can start logging gas usage again and I can stop using my DIN rail mounted KWh-meter with S0 output which I used for power usage (but not before I’ve had a chance to compare it with the smart meter, though). Source code will be available on GitHub soon (account has already been made) .

Next project that will be handled in a similar way: the Opentherm Gateway! For that one the Simplecortex needs to be able to subscribe as well!

Edit: the P1_Publisher project has been committed to Github some moments ago:


MQTT: about dumb sensors, topics and clean code

Or, in other words: where to define the topic a device should publish/be published on. In a scalable way..

I always try to keep the number of hard-coded values to a minimum; no hard-coded device IDs, peer IP addresses and things like that.¬†And now that I’m working on MQTT clients, I was wondering how I could keep it that way.
Hard-coded MQTT topics? No way. Well, the last part in the hierarchy maybe (temperature, humidity, motion, …), but that’s it – nothing more. Driver code should be as clean as possible too; the drivers’ own IP-address is OK; peer IP address rather not; parameters that influence behavior: rather not. So the credo is: the least as possible/doable/practical.¬†That’s the only way to keep maintenance (and irritation) low and scalabilty high; lets see what can be done about that with respect to MQTT.

The diversity in all the devices I have is rather large.¬†There are devices that are really ‘dumb’, like 433 MHz temperature sensors that transmit the temperature every x seconds. ¬†And there are devices which are a bit smarter, like JeeNodes with a RF12B or Xbee transceiver.¬†And devices that can be controlled over powerline, RF, Zigbee, Infrared or TCP/IP and smart(er) devices like a EDS HA7Net, ¬†Opentherm Gateway, ELV MAX! Radiator Valves, IrTrans interface, GSM Modem etcetera.¬†And the latest addition to this list is the Smart meter, which is the first (well, the driver) that’s going to use MQTT to publish the smartmeter data.

Communication between my domotica system and all the end devices can be direct by a built-in driver (.e.g. a built-in driver that has a direct TCP/IP connection to the GSM modem), indirect (via ZigBee Coordinator, IrTrans LAN module) or device-initiated (connections are being made to an open TCP/IP port of my Domotica system, either with or without an extra hardware layer in between).

The format of the data can be a stream of bytes that need to be decoded, html output, xml, readable text, proprietary frames; you name and it’s in use here.¬†Conclusion: its one big pool of hardware and nothing works the same ūüôā

Now the challenge is to develop an easy to use, reliable and scalable solution so that I can build MQTT enabled hardware that acts as a driver which in some way ‘configures’ itself and doesn’t have to rely on hard coded topic names, IP addresses etcetera.

Lets give it a try. All sensors/devices, no matter how ‘dumb’, can be identified in some way; with a hardware address, MAC-/IP-address, Node id on a RS485 bus, whatever. The driver (the software under my control that’s the closest to the hardware) can only use this ID to publish the data; cause for the driver the device ID is the only unique thing.

For exampe, an¬†RF receiver sees an incoming RF frame and can determine that this frame its coming from a temperature sensor with ID xx.¬†So what can this driver do in terms of MQTT publishing? Publish the measured temperature of 23.1 ¬įC for sensor ID xx (topic in red, payload in blue) as:

/RF/xx/temperature 23.1

Yuck. I don’t like that at all, I don’t think I want that – those IDs don’t tell me anything!
I’d like something like ourplace/house/firstfloor/kitchen/temperature or /ourplace/garden_shed/temperature. This has some advantages… but how do I do that? Cause the thought of ‘coded’ location information gives me headaches…

With a ‘database’ of course.. From the first day I started developing my own system I’ve got a table which contains a list of all the devices with their ID, description, where it is located (Building, Floor, Room, Location) and lots of other attributes.
That is where the relation between the hardware IDentifiers and the rest of the device information is stored – and it must be, and always stay, in one place. It doesn’t necessarily need to be SQL of course as it is in my case – the most important thing is that there can only be 1 ‘source’ – cause if not, a system with more than 230 devices with nearly 1200 device values will become a big mess, very quickly.

So how do I get the information from my SQL database into that MQTT capable driver that needs it – well, why not publish the contents of that database?¬†Lets assume that for sensor ID xx the topic used for publishing the temperature should be /ourplace/garden_shed/temperature. ¬†The fun is that ‘ourplace’ and ‘garden shed’ are already in the database as attributes to a device with ID xx – so nothing more to do than just publish that information and I’m done – right? If I would publish

/config/RF/deviceinfo/xx/location/ /ourplace/garden_shed/

then the RF driver can subscribe to the topic /config/RF/deviceinfo/# and it will receive all the information it needs for the relevant devices; and it can be more than just the location information.

In this case, ‘/config‘ is just an arbitrary topic name that separates the configuration stuff from the rest of the hierarchy.¬†The next topic in the hierarchy is used to differentiate the devices per driver they are supported by (the RF driver only needs to know about the sensors it needs to ‘service’ – there’s no need for the RF driver to receive information about Zigbee devices for instance).

I could do that… that means that¬†during the startup phase the driver subscribes itself to the right topics in the ‘config’ hierarchy and the information comes in… and this can be used for other things just as well – like¬†driver specific settings as a poll interval, for instance.

Doing things this way means that, when a temperature sensor is being relocated from the kitchen to the garage, I don’t have to do anything more than I do now:¬†changing the location information in the database and the rest follows! All I have to develop is a tool that publishes the required database information in the /config hierarchy and I’m done.

But there’s another way.

Let the driver publish the ‘raw’ information (Device ID oriented) in a /raw hierarchy (/raw/RF/xx/temperature¬†23.1) and develop a tool that subscribes to the /raw hierarchy and that re-publishes everything it can (e.g. that can be found in the database)¬†to the location-oriented hierarchy, e.g.¬†/ourplace/garden_shed/

This has some advantages too… hmmm; and there are even more alternatives… but this example of the temperature sensor is maybe too simple, it doesn’t cover every aspect I have to deal with – or does it?


Publishing smart meter P1 data

Got it!

I just had to know whether my Simplecortex code was any good; it would be nice to see some real results instead of all the test results that were created along the way. The code was in a ‘workable’ state so I was ready to test if I could turn the Simplecortex into a MQTT client and publish the relevant smart meter P1 data.

I already downloaded Mosquitto somewhere in July, tried some simple (and successful) tests with a Delphi MQTT Client, but now it was time for the real thing. The Simplecortex was already running when I started the mosquitto server from a command line and I saw the Simplecortex connecting to the mosquitto server (great, auto connect works):

Mosquitto server

No timeouts, disconnects or any other thing that looked suspicious; so lets see if the Simplecortex is actually publishing something. For that I started a mosquitto client from another command prompt to subscribe to the topic for the current power usage:

Mosquitto client

Yes! Every 10 seconds I see the current power usage being updated;  cool, it works!

The biggest hurdle I had to take was the fact that the uIP API doesn’t allow you to just send outgoing data when/where you’d like to – no, the uIP API¬†calls the application (a C function) when data is received, data has been¬†delivered to the other end, when a new connection has been set up, and when data has to be sent. Therefore I had to use a sort of publish¬†queue¬†to store the MQTT publish messages until the uIP API allowed the application (the MQTT client on the Simplecortex) to actually send the data to the MQTT broker. ¬†But the uIP stack had some good examples of which I gladly made use of ūüôā

Does this mean I’m finished? Not yet.. There are still some things I’m not happy about; the way some things are setup in code right now. Ping for example, uses a hardware timer which throws an interrupt way too often for an interval of (now) 30 seconds. And the UART is being read in main(); I want that nicely tucked away too. And there are some other things that still need to be improved, all caused by my lack of C knowledge – while developing code for part B, I realize I could better have coded part A in a different way (and better)!

But for now, I’m going to sit back, relax and watch it working ūüôā

Smart meter P1 project is doing fine

Surprise! Today I saw the following:



Yep, this means the gas usage is working too now! I called Liander last Monday to tell them my gas usage still wasn’t working and that I still got all zeroes. I had a bit of a problem explaining what I was doing (dunno why…), but at the end of the conversation I got the feeling my problem was taken seriously. That same day (late in the evening, as I found out today) the gas usage suddenly popped up; good, very good!

This evening I also managed to get the Simplecortex to automatically recover from lost connections. This is very very important because not being able to do that automatically would be a real show-stopper for me. Just think of it, that you would have to press the reset button after network problems or any other reason a connection has to be re-established; sounds horrible!

This weekend I’m going to have a look at 2 options to invert the P1 signal: a 7404 inverter IC or a BS170 transistor. These 2 options enable me to connect the cable from the P1 port directly to the UART of the Simplecortex; no additional components needed. I read about that last option (BS170)¬†here¬†and I just couldn’t resist ordering the parts for both options to see how they both work. Just out of curiosity…

And I’ll have to modify the UART library a bit cause as this library is now, it only supports 8 bits, no parity and 1 stopbit.

Next week will probably be dominated by adding MQTT Publishing support to the Simplecortex project.

This means that if all goes well,  at the end of next week I can install the Simplecortex near the smart meter and start subscribing to the smart meter topics!

Processing the smart meter data (3)

Well, 2 days later and I’m getting somewhere ūüôā

First of all, I must have been drunk when I said the Simplecortex was sending its data with an interval of 200 ms – no, that’s the timer resolution of my logger! The simplecortex sends a packet each time a newline (or line feed, LF) is received from the P1 port. So that’s solved.

First thing on the todo list was parsing the P1 data and extracting the numbers I want to receive in my system:

#define P1PU1  "1-0:1.8.1"      // total power usage normal tariff
#define P1PU2  "1-0:1.8.2"      // total power usage low tariff
#define P1PD1  "1-0:2.8.1"	// total delivered power normal tariff
#define P1PD2  "1-0:2.8.2"	// total delivered power low tariff
#define P1TIP  "0-0:96.14.0"    // tariff indicator power
#define P1PUC  "1-0:1.7.0"	// current power usage
#define P1PDC  "1-0:2.7.0"	// current power delivery
#define P1GTS  "0-1:24.3.0"	// timestamp gas usage sample

A routine was made to check each received line for the occurrence of the above-mentioned OBIS references (identification codes for data in DLMS/COSEM compliant metering equipment), get the value out of the received line and store the result in a buffer:

else if(strncmp(P1PU1, line, 9) == 0)
  strncpy(value, line+10, 9);
  sendbuf("powertotalN;", value);

… and another routine that’s being called whenever the application is allowed (by the uIP TCP/IP stack) to send the data over to the other side of the connection. And this is another intermediate result, received by my domotica system:

22:41:54.406  TCP < powertotalN;00003.725
22:41:54.406  TCP < powertotalL;00000.000
22:41:54.406  TCP < powerdelivN;00000.000
22:41:54.406  TCP < powerdelivL;00000.000
22:41:54.406  TCP < powertariff;0001
22:41:54.506  TCP < powercurrent;0000.96
22:41:54.506  TCP < delivcurrent;0000.00
22:41:54.707  TCP < gastimestamp;120808010000
22:41:54.707  TCP < gasusage;00000000
22:42:04.722  TCP < powertotalN;00003.728
22:42:04.722  TCP < powertotalL;00000.000
22:42:04.842  TCP < powerdelivN;00000.000
22:42:04.842  TCP < powerdelivL;00000.000
22:42:04.842  TCP < powertariff;0001
22:42:04.842  TCP < powercurrent;0000.97
22:42:04.842  TCP < delivcurrent;0000.00
22:42:05.043  TCP < gastimestamp;120808010000
22:42:05.243  TCP < gasusage;00000000

That’s much better than what I had 2 days ago; well 4 hours ago actually, cause time is limited and I’m also learning more and more about C while working on this.

Next step: sending this data as payload in a MQTT Publish message! The protocol is not so hard in its simplest form and I’ve got working Delphi MQTT code, so this should not be a problem.

In the meantime some parts have arrived: an enclosure for the simplecortex, 7404 hex inverters and transistors to invert the P1 signal.


Processing the smart meter data (2)

Today I made some progress on processing the smart meter data with a SimpleCortex.

I started collecting the P1 data from our smart meter from the first day it was installed. A laptop, an RX-inverted USB cable and HyperTerminal take care of that for almost a week now.

Logging P1 smart meter data with HyperTerminal Now it was time to do something with the data I logged (currently 25 MB in size).

For that, I needed to create a test setup, with which I could simulate that the SimpleCortex was directly connected to the Smart meters P1 port.

Well, that was not so hard; I wrote a quick & dirty VB.Net tool that mimics the P1 ports behaviour, by reading the logged data and writing it to a Serial port.

This Serial Port is a USB-BUB connected to  UART0 of the SimpleCortex:

Feeding the SimpleCortex with smart meter data

Now I only needed a receiver for what the Simplecortex wants to send; for that I used a small Delphi app with a Indy IdTCPServer component – this logs everything it receives to a file, so I can easily monitor what’s being received.

With the VB.Net “P1 feeder”, the USB-BUB and the Delphi TCP server app I can now code, flash, test & debug with the Simplecortex right beside me in my office; I think I’ll need it, cause I’ve still got a long way to go.

Now that I had setup the right¬†test-environment¬†I could start with writing the code for the Simplecortex. If only I could get some sort of ‘Hello world’ example for the uIP TCP/IP stack to get me going – the Simplecortex webserver example looked like overkill; too much to learn at once, I needed something simpler to start with. Mark, who commented on an earlier post about the Simplecortex, was so kind to help me out and provided me with an example of a project that echoed the TCP data it received; and that was all I needed – suddenly it was clear to me how I should start.

And here it is, the output that’s being received from the SimpleCortex:

15:34:11.078  TCP < !
15:34:20.952  TCP < /ISk52ME382-1003
15:34:20.952  TCP < 137
15:34:21.112  TCP < 0-0:96.1.1(4B414C37444035313039373537343132)
15:34:21.112  TCP < 1-0:1.8.1(00002.368*kWh)
15:34:21.112  TCP < 1-0:1.8.2(00000.000*kWh)
15:34:21.112  TCP < 1-0:2.8.1(00000.000*kWh)
15:34:21.112  TCP < 1-0:2.8.2(00000.000*kWh)
15:34:21.312  TCP < 0-0:96.14.0(0001)
15:34:21.312  TCP < 1-0:1.7.0(0000.88*kW)
15:34:21.312  TCP < 1-0:2.7.0(0000.00*kW)
15:34:21.312  TCP < 0-0:17.0.0(0999.00*kW)
15:34:21.312  TCP < 0-0:96.3.10(1)
15:34:21.312  TCP < 0-0:96.13.1()
15:34:21.312  TCP < 0-0:96.13.0()
15:34:21.312  TCP < 0-1:24.1.0(3)
15:34:21.312  TCP < 0-1:96.1.0(3238303131303031344439323034343132)
15:34:21.513  TCP < 0-1:24.3.0(120807230000)(00)(60)(1)(0-1:24.2.1)()
15:34:21.513  TCP < (00000000)
15:34:21.513  TCP < 0-1:24.4.0(1)
15:34:21.513  TCP < !

For now, the only thing the Simplecortex does is waiting for data on UART0 and transmitting the data it received to the IP address of my PC. Sounds like a simple task, and it was; I was actually surprised with how fast I finished this first step.

But as I said: I still have a long way to go, cause this is still on my todo-list for this project:

  • Parse the incoming P1 data and collect the data I need (power usages, gas usage)
  • Act as a MQTT client and publish the collected data;
  • Being able to handle 7-E-1 with the UART instead of 8-N-1
  • Automatically reconnect when the connection fails for whatever reason;

And of course there are some technical issues I don’t understand fully yet; one of them being the interval with which the Simplecortex seems to send the data – every 200 ms? Strange.

So there’s still a lot to do, but the preliminary results¬†make me happy and give me confidence for a good end result ūüôā

Processing the smart meter data

Gas usage still doesn’t work, but I consider that a minor issue – getting the P1 data processed and into my system is what keeps my busy right now.

And I think I know how I’m going to do it; with a Simplecortex. I could take the (relatively) easy road and use an Arduino(-Ethernet) or a JeeNode (with Ether Card), but using a Simplecortex sounds much better actually ūüėČ

Because it’s something new.. I’ll get the chance to learn more about the uIP TCP/IP stack, I’ll have to enhance the Simplecortex UART library so that it will also support 7-E-1 instead of only the standard 8-N-1, I’ll have to browse through the¬†LPC1769 manual and I’m sure those things won’t be the only challenges I’m going to face in the next couple of weeks.

I know that I will need more time doing things this way, but I don’t care, actually.Cause for me creating is more fun than having/using it.

So today I ordered my 2nd Simplecortex, selected an enclosure and some ‘extras’ to take care of inverting the P1 signal. And this evening I’ve started with the software for the Simplecortex which should parse the P1 data, reduce it to the stuff I need from it and make sure it arrives in my domotica system.

And this is why Domotica is the perfect hobby for me; yesterday I had responsive web design on my mind, today it was about smart meters, and I seriously don’t know what tomorrow will bring – but what I do know is that it will be another exciting subject I want to know more about!