The All-in-1 solution for Btraced Track & Trace

Somehow the Btraced post I wrote some time ago was somewhat half-finished. That post explained how to use a Raspberry Pi as your private Btraced upload server and briefly shows what you can do with your GPS data. What I did not cover was how to get the Btraced information in a web page. Btraced on Google MapsFor me personally, getting the Btraced information to a MQTT broker was the only thing I needed to do to get it all working, because the rest was already covered – and I figured that creating a webpage that could receive the Btraced information and display it was a bit too trivial to mention. Nevertheless, last week I did a small ‘remake’ of the Btraced code I wrote, cause I thought it would be nice to also create an All-in-1 solution for Btraced for those that don’t use MQTT or don’t have a web-server (or both ;-)).

Combining the Btraced upload server, reverse geo-coding and a web-server in one ‘package’ running on a single Raspberry Pi, fully self-supporting, that was the idea. No need for another web-server anymore, nothing – just a Raspberry Pi and the Btraced app. And of course everything’s working real-time, cause nowadays you don’t have to settle for less anymore.

Btraced codeMost of the code was already finished, all that needed to be added was a web-server and a way to get the uploaded, parsed & extended Btraced information to the web clients (the browser(s)).

And of course some static content like a html page, style sheet and some Javascript.

And the result is great – a light-weight web page, served by the Raspberry Pi that gives you real-time access to the Btraced GPS information on any device with a web browser on board – and all with the power usage of a single Raspberry Pi. Saving the uploaded data in a database is not supported yet but should be very easy to accomplish. And there are more things that can be easily added once you have a Raspberry running..not just Btraced related; Pieter and I are currently discussing some options.

Interested? Let me know. In the mean time, happy tracking!

Staircase project, part 5: the end is in sight

Today some important milestones were achieved – painting is finished now as well as soldering the 16 channel LED driver. Birthdays, parties, the flu and other physical discomfort delayed the whole thing more than we anticipated but hey, who cares – we sure don’t 😉

16 Ch. LED driver in its enclosureEspecially soldering all the components for the 16 channel LED driver on a piece of perfboard was quite a job – but now that it’s finished and all 16 channels are tested and working, I’m very pleased with it.

The sketch for the Arduino Ethernet is also progressing nicely – MQTT is operational, i2c is working fine too  and I haven’t seen any glitches which I did see when the Raspberry Pi was controlling the LED driver. So all that’s left to do is connecting the light barriers as inputs to the Arduino so that the sketch knows when someone’s walking up or down the stairs.

And of course, I still have to saw 12 aluminum profiles to the right length, put in the LED strips, solder wires to the strips and mount those profiles under the steps. So I think I’ll need another weekend to finish the hardware side before I can focus on the software – but the end is in sight. Now that I think of it – how do I change the behavior of the sketch without having to flash new firmware every time I want to change something in how the LED strips are being controlled… maybe an alternative bootloader and the SD card can help here?

Staircase project, part 4: progress and changes

Almost finished A small update. As you can see the stairs are almost done. The banisters still need to be painted and some minor ‘errors‘ need to be resolved, but the major part of the things I’m not good at are done!

Now I can finally spend some more time to spend on the fun part: the LED strips, the light barriers and creating some nice light effects with it all. Can’t wait to see it in action 😉

The light barriers have been on a desk in my lab for quite some time and are working very well. The only thing left to be done is conditioning the signal so I can feed it to a I/O pin.

I’ve also had a piece of LED strip connected to a RPi-controlled Dimmer Plug on that same desk for more than a week or so; not because I like having a LED strip next to me which is constantly changing in brightness, but because I wanted to be sure that the non real-time way of controlling the Dimmer Plug wouldn’t result in visible effects like pausing or flickering LED strips. It’s about lighting, so even the smallest hick-up will be seen.

And guess what, in a period of say 20 hours I noticed that the LED strips were indeed flickering, about 3 times – they were not doing what I instructed! I could only think of 1 reason for this: the Operating System  of the RPi decided that something else had a higher priority than my LED strip program. OK, I know this is not much of a scientific approach, there are other possible causes (Node.JS and its garbage collection?), but I didn’t like what I saw so I decided to buy a Arduino Ethernet, just for the sake of convenience. The Arduino should be able to control the DimmerPlug & LED strips much more real-time and hence more smoothly all the time.

Last week the 4 x 5m. LED strip arrived, including 12 x 1m. aluminum profile with opal cover. And about 16 x IRLZ34N, terminal blocks, perfboard, etcetera. With what I have laying around, this should be enough to create a box with 16 screw terminal for 16 LED strips. I don’t need all 16 of ’em for this staircase project, but I’ve already got some things in mind for the remaining 4.

Testing (again)The last thing I did (last evening) was that I successfully soldered a screw terminal, MOSFET and a resistor on a piece of perfboard occupying a surface small enough so that I could solder another 15 copies of it on a piece of perfboard that would still fit in the enclosure that I want to use. And of course, I also did a live test with the Dimmer Plug, a 12V adapter and 5m. LED strip: beautiful!

Sharing metadata between 2 one-legged systems

The time has come to implement some drivers that need more information than just some low-level information like IP address/port number of the hardware they have to connect to.

For instance, I need an XMLRPC server on the Raspberry Pi, cause XMLRPC  is the method that’s being used by 2 internal User Interfaces (my Philips Pronto TSU9600 and a VB.Net application on a Asus EEE TOP). Those 2 UI’s use XMLRPC to send commands (that eventually end up at the hardware driver) like switching lights, controlling roller shutters, setting temperatures on the radiator valves etc. I don’t want to change that yet, especially on the Pronto, cause it’s a battery powered device where it’s not that smart to keep a network connection open all the time. So I’ll stick with XMLRPC on the Pronto, until I have a better alternative.

The way in which the Pronto sends it commands is like this (a Prontoscript that’s being executed when someone touches the ‘Media’ icon on the Pronto touchscreen):

// UPC box, set channel 1 (to make sure power state is on)
// followed by off // (toggle key!)
scheduleAfter(5000, function() {
//Dune hd max power on
// onkyo power on
// onkyo, select HDMI input
// set default volume level
//TV livingroom on

The Pronto doesn’t ‘talk’ directly to hardware drivers, but to a device object, using XMLRPC as the transport mechanism – this UI only communicates what has to be done – it doesn’t care about the how..

And that’s good, because sometimes multiple drivers are involved in controlling a piece of hardware. For instance, my media player has a built-in http interface to which I can send commands like play, pause,  stop and so on. But there’s one thing I can not do: turn the device on via this http interface, because it the http interface doesn’t exist while the player is off.. so when the media player is off, I have to send the Infrared code for ‘on’ to the media player with the IRTrans driver. So more than 1 driver can be needed to control a device, even though this is rare.

This kind of knowledge/logic doesn’t belong in a UI, hidden somewhere in a small script, running on the Pronto; the fact that I need to control my media player this way is something that’s hardware specific – it belongs to the devices’ metadata so to speak.

So when the XMLRPC server receives a command like ‘hdmax, on‘, it needs to know more about the hdmax device (i.e. the media player) than just that it exists – it needs to be able to route the command to the device object and the device object will know what to do to have a command actually being executed. In this case: if the command is ‘on‘, forward it to the IRTrans driver, otherwise forward it to the hdmax http driver. That’s roughly how my system works internally. With the XMLRPC server embedded in my Delphi system this wasn’t a problem, cause all the device objects lived in the same memory space as the XMLRPC server – a simple internal message system was all that was needed to get information from A to B.

But with the way my Raspberry Pi based system works, things are more separated. I have an XMLRPC server running as a single process, an IRTrans driver as another single process and the hdmax driver is a separate process as well. And sometimes those processes need information about the devices –  the metadata. And not all those processes need the same metadata…

During the limited time I’ve had during the last few weeks, I began with retrieving the metadata from my MS SQL Server in a way so that it could be used for various purposes. For example, the XMLRPC server has to know some things about devices, their device values and so on. Another process (for example for storing historic data) will need information about what device values to store, the storage interval (number of minutes/hours, minimal, maximum), how long the historic data should be kept in the database, etcetera.

Retrieving information from a MS SQL Server with Node.js was already covered (tedious), so now I needed a way to transform ‘flat’ rows and columns from a SQL database to something that could be handled better in Node.js – I got the idea of creating an Array of device metadata objects, searchable on multiple properties and the ability to define a customized subset of the required metadata per process (for those that need it, cause only a few will) – all nicely tucked away in a module.

First I created an SQL query that would retrieve as much metadata as possible in a single query:

select devices.*, values.*, floors.*, rooms.* from devices
left join values on values.deviceid = devices.deviceid
left join rooms on rooms.roomid = devices.roomid
left join floors on floors.floorid = rooms.floorid
order by address, id

This query results in 1668 rows of 68 columns, containing all the information (metadata) of 285 devices. Most of the time this is way too much, so I added a ‘filter’ so that I could reduce the amount of information, like this:

var coldefs = [];
coldefs[‘DEVICEID’] = {‘name’:’devid’};
coldefs[‘ADDRESS’] = {‘name’:’address’};
coldefs[‘DRIVER’] = {‘name’:’driver’};
coldefs[‘VALUEID’] = {‘name’:’devicevalue’};

Now, only the 4 query result columns specified on the left hand side will be used, the rest of the other 64 columns will be discarded. The remaining ‘metadata result set’ is then transformed into an array of 285 device metadata objects, of which one looks like this:

 { devid: 'dressoir',
   address: 'b02',
   driver: 'plcbus',
   devicevalues: [
    { devicevalue: 'dressoir.signal' },
    { devicevalue: 'dressoir.noise' },
    { devicevalue: 'dressoir.on' }

Cause that’s all the XMLRPC driver needs to know – the name (id) used throughout the system, its physical address (in this case a PLCBUS address, but it could just as well be an IP address, MAC address, Visonic sensor address, whatever), the driver used for controlling this device and information about all the values this device has (signal strenth, noise level, on).

With this array of device metadata I can do lots of handy stuff, like:

// this will display "b02":

// this will display "plcbus":

// this will display "dressoir":

Much better than a Delphi TStringList! 🙂

Although this may look like some next trivial addition to my Node.JS based Home Automation system, it certainly is not. Retrieving information from a MS SQL Server is covered, so I don’t have to worry about ‘databases’ in general anymore for some time, until I’m ready for migrating to (probably even) another type of database. This also means I can continue walking down the migration path as I’ve been doing during the last 2-3 months: gradually rebuilding (and sometimes rethinking) my system.

And now, it’s back to the staircase again!