Got it!

Yes, yes, YES! I love it 🙂

The rethinking of the OpenTherm bit capturing strategy I did a few days ago really did improve things quite a lot, as can be seen on a screendump of my OT Decoder:

"My" OpenTherm decoder

Below is what the Opentherm Monitor, a tool that belongs to the OpenTherm Gateway (yep, I’ve got one of those too, since a week or so 😉 ) is showing in the log:

Gateway Opentherm Monitor

This is great, wonderful result! And the time it took to come this far was well spent, cause I’ve learned a lot in the past days. I’m starting to know the Opentherm Data ID’s by heart, I know some more about the ATMega timers and Manchester decoding has no secrets for me anymore.

You may notice that in my case only the Master (lines with a ‘T’ in them, in both screendumps) frames are decoded and not the Slave response frames (Slave=Boiler, starting with a ‘B’ in the lower screendump), but that’s because I added a delay that causes the sketch to skip the Slave responses.

So the sketch is going to be the next thing I’m going to work on in the next couple of days. But for today, I’m going to relax and and stop thinking about microseconds, timers, prescalers and ISRs;  it’s time for something completely different: a Somfy RS-485 RTS Transmitter for our rolling shutters which will (hopefully) arrive in  a few weeks! That’s why I like Domotica so much – the diversity of things to do and learn.

 

Oops and yeah!

I don’t know what happened, but yesterday’s post is completely wrong. Maybe it was too late in the evening, maybe because I was trying to do 2 things simultaneously,  I don’t know – but the fact is, that yesterday’s post is totally messed up – it doesn’t reflect what I had in my mind at all… so that post was a major f***-up, if I may say so. Well, I’ll fix that some day. Soon, I hope. Maybe I can explain things better, once my thoughts are documented in code 😉

Fortunately, I didn’t use my own post for what I did today. It’s all in my head, right?  The status so far is that the JeeNode is correctly detecting the short and long periods and it’s reporting those on the Serial port (during the time where there’s no OT communication going on), in a format like this:

FT 1
0S
1S
0S
1L
0S
1S
0L
1L

The FT stands for ‘First Transition’, which tells me that a new transition from 0 to 1 has been spotted.  After that (line 2), the JeeNode input went LOW (0) for a short period (S), HIGH (1) for a short period, LOW for a short period and HIGH for a Long (L) period, …

With this information  I can ‘rebuild’ the signal and the bit stream. For now, I’ve  done this in my favorite programming language Delphi, just to speed things up a bit. The sketch has been running for just half an hour or so and the fact that the reconstructed bit streams have had a length of 68 bits all the time, is encouraging. Cause 68 bits lead to 34 Manchester-decoded bits, which is exactly the size of an OpenTherm frame of 32 bits +1 start- and stop bit. Yeah!!

When I’m 100% sure of how everything should work, I’ll embed all the bit calculations & manipulations  into the sketch, so that all I’ll only receive the resulting 4 bytes per OpenTherm frame from the JeeNode.

But there’s still a lot of work to do before I get there!

Time based sampling vs signal duration

Not being ‘hindered’ by any knowledge about Manchester decoding and signal processing, I came up with the following idea. Rereading the OpenTherm Protocol 2.2 documentation, the following is said about the bitrate and timing:

Bit rate                           : 1000 bits/sec
Period between mid-bit transitions : 900 .. 1150 μs (nominal 1 ms)

Furthermore, there’s a built-in time margin where a transition can take place; When TØ is the start of a bit period of ≈1000 μs, the transition must take place between TØ+400 μs and TØ+650 μs. If not, the transition wouldn’t be conform OT protocol specs. This in fact means that the time window in which the transition can take place is rather large, namely 250 (100+150) μs. That’s 25% of the total bit period, which sounds like a lot to me, actually.

I want to try to code something that’s just as flexible – maybe even more flexible, considering the fact that it’s not unusual for things to be out of spec. Wouldn’t it be better to search for other things that can be measured just as well but which are less dependent on time? Determining long vs. short periods should be enough, maybe? So lets forget about the timing and have a look at the highs and lows for a change. One thing that could be useful is the fact that the signal must always be stable for a period of at least 250 μs; cause if not, it would be out of OT protocol specs.

You could also say that a short period should be between 400 and 650 μs and a long period between 750 and 1250 μs. That means there’s a gap of 100 μs between the longest ‘short period’ and the shortest ‘long period’, but all still within specs. I should be able to determine whether a signal has been stable during a short period or a long one… right?

Update:

The rest of this post has been deleted, because it was totally rubbish and incorrect – what was I thinking?? Too much beer perhaps… 😉

This will soon be fixed…

Transition timing

Somehow the OpenTherm Monitor I built  a week ago has some problems to correctly decode the signal to Manchester code and produce the correct 32 bits. And the strange thing is, that my logic analyzer which I attached to the input pin on he JeeNode, doesn’t seem to have these problems (click):

BugLogic

The latest version of the Saleae Logic software has Manchester decoding included and the software can make sense of all this, so… Maybe the analyzer is more tolerant? I don’t know.

Time to figure out how ‘good’ the signal really is, and whether it’s worth the time to try to improve the OT Monitor – I don’t just want to give up!

The first thing I did, was having a look at the distribution of the periods between all the transitions (high <–> low). For that I wrote a sketch that used TIMER2 to check the value of the input pin every 80 µs. The time between each transition was measured and stored into an array during sampling. And after a certain number of transitions, it would write the array to the Serial port.

The Serial Output looked something like this:

2 0
3 1
4 1
5 217
6 2795
7 1246
8 23
9 0
10 0
11 16
12 325
13 281
14 17
15 0
16 1
17 0

Copy & paste to Excel and creating a chart revealed the following:

 

Here you see the distribution of all the measured periods between a transition (for both high to low and vice versa) .The numbers on the X-Axis should be multiplied with 80 µs to get the real duration of all the periods. Well, it’s clear that there’s a peak at ≈500 µs and a second one at ≈1000 µs (1 ms); that’s good and what can be expected for a signal with 1 ms bit rate and where the difference in duration between the short and long periods is a factor of 2. So far so good.

Another test I did was to see how the ‘low’ periods and ‘high’ periods related to each other; did they both perform equally well?

Don’t mind the X-axis numbers, I’m not that good with Excel 😉 – it’s the red (high periods) and blue (low periods) lines that count. Again, the time of the second peak (63)  is twice as larga as the first (31..32). That’s good.

But the chart is not that ‘clean’ anymore… some yet unexplainable things popped up. For example,  the leftmost peak of the high periods (e.g. the short high periods) shows a strange curve to the right. In words, this means that there are quite a lot of short high periods that take longer than you’d want – cause the ideal situation would be a chart with only 2 narrow, high peaks for both the highs and lows, right? Could this mess up things? It could very well be that 1 single transition messes up the timing and hence the decoding of a complete OT frame…

I don’t know if that last thought (1 single period messing up things)  is right  – I have to dig some more, understand the OT Monitor sketch, try to find out where in the frame the timing goes wrong and see if I can find a pattern somewhere. If there’s a pattern, maybe I can create a work-around for it to improve the decoding..

Ow, and why’s there a huge dip in the blue (low) chart?

To be continued…

 

Arduino OpenTherm Monitor Shield

Yesterday Freddy Martens contacted me. His name rang a bell, because I had seen his name before.. yep,  he is/has also been working on the Pale Blue Dot Opentherm Monitor as can be seen in the comments at the bottom of the page. He sent me an email and told me that  he had some Eagle designs for an Arduino Shield for the OpenTherm Monitor. His design is based on the schematic he found on the Pale Blue Dot site which I mentioned before – here and in earlier posts. And he asked me if I could host his Eagle files. Sure!

Arduino OpenTherm Monitor Shield

 

The archive contains both a single-layer and a 2-layer design. He also told me that the designs are not perfect – for example, some of the pads are too small. But this could be a good starting point for others. So if you’re interested, just download the OpenTherm4Arduino archive.

Freddy also provided some images he made of his Arduino OpenTherm Monitor shield, which can be seen here.

Have fun!

Current status on my own OpenTherm Monitor project is that I’m getting lots of decoding errors- but I’m not giving up yet!

 

Getting closer

Yesterday the last parts for the OpenTherm Monitor arrived, so I could finish that part last evening:

– Adding the last part to the PCB (a 4.7 Ω resistor);
– Removing a diode that was there to power the PCB from the RS232 port that I won’t use;
– Replace the 2nd “RS232 power” diode with a wire;
– Cut away the unused headers.

And this is the end result:

OpenTherm Monitor PCB finished So the OT Monitor part is finished now. As you can see the DTR connection is now unused and the DSR connection doesn’t have a diode anymore.

Next step was to have a close look at the additional parts that are needed to connect it all to a JeeNode:

Additional parts

On this part of the schematic (here’s the complete original), 5 new parts are being used:

  • 2 resistors of 1 kΩ;
  • 2 * 2N7000 MOSFET (available here)
7N2000 MOSFET

7N2000 MOSFETThese 2 pictures and the the datasheet saying that

pin 1 = Source
pin 2 = Gate
pin 3 = Drain

is enough to make no mistake with the wiring, even if  you have no clue what the 2N7000 does 😉

 

  • A SN7432N which is a Quadruple 2 input positive OR Gate and can be purchased here

SN7432N

As can be seen in the schematic, a Single 2 input positive OR Gate would suffice just as well; maybe they exist too, but I didn’t bother to search

 

SN7432N powerBesides the 2 inputs and the output signal, the SN7432N als needs to be powered (pin 7 and 14), so the total number of connected pins results in 2+1+2=5.

 

 

 

With a breadboard, a JeeNode, a 4×20 LCD with a JeeLabs LCD Plug on its back, some wires and the SN7432N I mentioned above, this is the end result for now:

 Almost finished

Yep, I’m waiting for the MOSFETs to arrive. As soon as they arrive and I have some time to finish the breadboard, I’m going to put this OT Monitor between the Honeywell Evotouch Wireless OpenTherm RF module and our Remeha Calenta.

Assuming everything keeps working and that I’m not confronted with errors on the display of the boiler, I can try to have a look at the signal on the data pin – and if that looks OK, the next step will be connecting the data pin to the JeeNode and watch the LCD… exciting!

Preparing for OpenTherm Monitoring

More and more my weblog is becoming an ‘archive’ of what I did and, more important, how I did it. I know that I can find that specific picture or other piece of information I’m looking for on my NAS or a local HDD somewhere, but a search on my own weblog is much quicker, and it works too, more and more. So I decided to write down every step I take with this OpenTherm project on my blog and in more detail than I’ve done before – I think I’ll thank myself for that later 😉

Before heating up the soldering iron, I spent some time reading the OT Protocol Specification v2-2. I won’t go into all the details, but here are some important things to know.

OpenTherm is a point-to-point communication system between boilers and room units (the thermostat). The room unit is supposed to calculate a heat demand which is sent to the boiler; the boiler should act on the information provided (more or less heating) and the boiler can also send back various types of information like diagnostics and other useful stuff (to display on the thermostat LCD, for example).

The physical way in which the boiler and thermostat exchange information over the 2 wires is that the boiler transmits a signal by changing the current (high=17 .. 23 mA, low=5 .. 9mA), while a thermostat does this by changing the voltage (high= 15 .. 18V, low = 7V).  The boiler is also supposed to provide the room unit with power, so no batteries should be needed.

The encoding method that’s being used is called Manchester, where a transition from low to high represents a ‘0’ bit value and a transition from high to low is a ‘1’; more about Manchester code here. OpenTherm uses a bitrate of 1000 bits/second.

An OpenTherm frame consists of 32 bits e.g. 4 bytes and those 32 bits are transmitted with a leading start- and a trailing stop-bit. These 32 bits contain information like parity, message type (read, write, read-acknowledge, invalid data), data identifier and 1 or 2 (used) data bytes. In a OpenTherm conversation it’s always the thermostat (room unit) that acts as master; hence, the boiler is the slave. Communication is therefore always initiated by the thermostat, not by the boiler and the latter is supposed to reply within 800 ms. Furthermore, the protocol documentation states that a master must communicate at least every second. (question for myself: how does this relate to a wireless connection between thermostat and boiler?)

That’s enough info for now; lets have a look at the first thing I want to do: monitoring OpenTherm traffic with the Elektor OpenTherm Monitor. Lets have a look at the schematic of the Elektor Opentherm Monitor (OT Monitor) and what has to be done to connect this OT Monitor to an Arduino (or a JeeNode, in my case) – cause that’s the goal of this first step.

 

(the image can be a bit blurry due to the resizing; clicking it will reveal a better image)

The right side is where the builder of the OT Monitor is supposed to hook up his/her RS232 port. But that’s not going to happen – at least not if I can manage to modify the Elektor OT Monitor to this schematic, which I found on a site I mentioned before: http://www.palebluedot.nl/jml/projects/arduino/24-openthermmon.html

I can clearly see the modifications on the right side of the schematic (duh), but to make sure I don’t forget or oversee any of the modifications, I opened the original Elektor schematic in MSPaint and started modifying the original to get to the end result that I need. All modifications were done in red, so that it would become much clearer to me what had to be changed on the original PCB and what additional components had to be added (on a separate piece of perfboard):

 

It might look ugly or silly (or both), but it works for me and that’s all that counts, right? It reduces the labyrinth..

So, 1 diode needs to be replaced by a wire and the other connection with a diode will (must) not be used anymore – that takes care of the PCB-side of what has to be done. That should not be too hard.

Low Temperatures & OpenTherm

I don’t know why, but the cold weather of the last few weeks makes the word “OpenTherm” go through my mind all the time… temperatures are in control of what keeps me busy, how about that 😉

While the Nemef Radaris Plugin is being tested, I have some spare time to try and make that first syllable in OpenTherm a bit more true. At least for me, cause I don’t know that much about the OpenTherm protocol yet – time to do something about it!

First I spent quite a lot of time searching for sources regarding OpenTherm and what I could do to make OpenTherm work for me. Sure, I’m building the OpenTherm Gateway that is described in detail here, but that’s not enough; I want to know more – everything there is to know! The biggest problem I’m facing is my lack of knowledge of electronics. So when it comes down to building stuff to interface with OpenTherm hardware, I have to fall back to schematics made by others. Sad but true.

So I started off with a document I already know about for a long long time (and that anyone who has ever searched the net for information about OpenTherm must have seen before): the Elektuur/Elektor article about the OpenTherm Monitor that was published in 2001. I’ve always wanted to build this Opentherm Monitor, but there’s a tiny problem with this OT Monitor: it needs to be connected to a serial port; not my favorite way of doing this. However, searching some more resulted in this site. Here you can find detailed information on how to connect a slightly modified OT Monitor and some extra components directly to an Arduino; that’s much better, for me.

Another good source of information is Alex Vieira’s blog, which is also mentioned on the JeeLabs Café. Alex’s approach is (as far as I can remember) different from others, cause Alex is using a JeeNode to act as a Thermostat instead of the ‘man in the middle’ approach that is being used by the Opentherm Gateway (injecting/modifying OT frames  between boiler and thermostat) that I’m building.

This made me wonder what would be the best thing to do; yes, I do want to control my central heating and yes, I do want to be able to change the temperature setpoint. But do I want to take over the role of the thermostat all the way and in fact make the current thermostat obsolete? No, cause in my opinion, our central heating system as a ‘basic facility’ that should always stay operational, no matter what. Computers can break down, the additional (DIY) hardware can suddenly stop working by some unforseen reason, or Murpy’s Law kicks in somewhere unexpected – all these things should never affect the service level of the central heating! OK, comfort level will decrease (cause if not, I’m doing something terribly wrong 😉 ), but we won’t have to be afraid of frozen water pipes, cause the ‘real’ thermostat will still keep the house warm enough. And just as important, problems can be fixed very easily by removing the additional hardware and things are back to how it was – as in a ‘normal house’.

You might think I don’t trust a DIY thermostat enough to replace the real one? No, that’s not the issue here – it’s more about creating something that can keep on working forever. Without me, my domotica system or any other piece of hardware I installed. Nothing I do should ever create an unusable house; neither should I be needed to fix things – that says it all.

Back to the original plan again, which is getting to know OpenTherm better.

The first step I made was ordering the OT Monitor @ Eurocircuits. Cost: 24 Euro for the PCB and with some components, the total cost is around 30 Euro.

 

OpenTherm Monitor Some components are still missing, but that won’t take long anymore. Next step will be to add the ‘extras’ I mentioned before so I can connect the OT Monitor to a JeeNode. These ‘extras’ have been ordered a few days ago and a JeeNode is already waiting to display some results on the 16×2 display. Total cost: not much.

Another great thing is that I managed to obtain a PCB out of a very small series that was produced for the OpenTherm Gateway I mentioned above; although I already started building this Gateway on perfboard, I’m happy to start all over again now I have this PCB 😉

OpenTherm Gateway PCB


To be continued!

Bedroom lighting almost finished

A small update with some results on the bedroom floor lighting project.

Bedroom floor lighting

The small red dot in the center of the image above is where the hardware is (the boxed Arduino) with the PIR to the left. It’s hard to make a picture in almost complete darkness – keeping the picture sharp but also getting a result with colors that match with what my own eyes see is not that easy.. But the picture above resembles what I see when I get out of bed during the night well enough – and it’s exactly what I wanted! The LED strips (1.5 meters on both sides of the bed) light up the floor well enough to spot any obstacles, but it doesn’t light up the whole room. And the 15 seconds turn out to be the right amount of time to light up the floor; and the LDR  takes care of unnecessary lighting during daytime; so in one word, the only conclusion can be: perfect! 

One small glitch though, which points out the difference between a clean testing environment and usage in real life: some days ago, when I had to change the alarm clock, I switched on the bedside lamp so I could see the buttons on the alarm clock. This triggered the LED strips to go on!

This means that the switching of the 230V lamp resulted in a ‘spike’ on the PIR sensor input on the Arduino… hmm. I think I’ll have to redo some wiring and use shielded cables instead of the unshielded & untwisted ones I’m using right now. My bad…

Perhaps another solution would be to watch the incoming PIR signal on the Arduino more closely and detect whether it’s a short ‘spike’ or not and decide not to turn on the LEDs on a short ‘high’; but that just doesn’t feel right – cables first!

Jenkins, unsigned integers and VB.NET

Fotunately I had my hair cut short recently, cause otherwise I’d have pulled them all out yesterday! 🙁

For a project I’m working on, I need to use the so-called “Jenkins’s one-at-a-time” hash. The algorithm looks like this, in C code:

uint32_t jenkins_one_at_a_time_hash(char *key, size_t len)
{
  uint32_t hash, i;
  for(hash = i = 0; i < len; ++i)
  {
    hash += key[i];
    hash += (hash << 10);
    hash ^= (hash >> 6);
  }
  hash += (hash << 3);
  hash ^= (hash >> 11);
  hash += (hash << 15);
  return hash;
}

No big deal. Right? Until you have to rewrite this to VB.Net. Here's what I started with, and when things got weird:

Public Function Jenkins_one_at_a_time_hash(ByVal data() As Byte) As UInt32

  Dim hash As UInt32 = 0

  For Each b As Byte In data
   hash = hash + b
   hash = hash + (hash << 10)
   hash = hash Xor (hash >> 6)
  Next
  hash = hash + (hash << 3)
  hash = hash Xor (hash >> 11)
  hash = hash + (hash << 15)

  Return hash

End Function

While looking at the original code, I knew that with all the bit-shifting going on, there would definitely be overflows; this algorithm has overflow written all over it. So I anticipated some small bumps, but not real big ones...

OK. I don't care about bits falling out of the bucket, as long as this doesn't result in exceptions. How wrong could I be...nearly every line gave me exceptions. But, surprisingly, those exceptions weren't thrown by the bit-shifting code, but by intermediate results that exceeded the UInt32's maximum value, for example:

3281982255 + 2085403648 = 5367385903; yep, that's 33 bits.

OK. Nevermind; I'll use a UInt64 and only keep the 32 LSBs. That should do the trick. No! The VB.Net OverlowException told me 'Arithmetic operation resulted in an overflow.' Yeah I know, with 32 bits it does, but not with 64!
What's going on here ?. Lets isolate the problem a bit...

Dim a As UInt32 = 3281982255 ' 11000011100111110001001100101111
Dim b As UInt32 = 2085403648 ' 01111100010011001011110000000000
Dim result As UInt64 = a + b

Line 3 also throws the exception I mentioned above. But why?? The result of a+b is 5367385903, which should perfectly fit in 64 bits. This is becoming really weird ...
Changing the result type to Decimal, with a largest possible value of +/-79228162514264337593543950335 didn't help either. Same Exception.

This code works though:

Dim a As UInt64 = 3281982255
Dim b As UInt64 = 2085403648
Dim result As UInt64 = a + b

But why should I have to this; it's stupid - using 64 bits to store 32 bit values; huh? Since I don't work with VB.Net that much, I started looking for compiler directives to ignore overflows in this hash routine. The only thing I could find was this:

Yeah, right, remove integer overflow checks... at project level. Who invented that? Seems I can't use compiler directives in the code at all, and I sure don't want to remove all the overflow checks from the project of course. I did a little more testing with the Intxx type variables and saw some strange things happening:

Dim a As UInt32 = 1073741824
Dim b As UInt32 = 1073741824
Dim result1 As UInt64 = (2 * a) + (2 * b)
Dim result2 As UInt64 = a + a + b + b
Dim result3 As UInt64 = (a + a) + (b + b)
Dim result4 As UInt64 = 5 * a

Line 3 does not result in an Overflow Exception, but line 4 does. Line 5 does too, but line 6 doesn't. It looks like only additions can trigger those overflows? What is this??

Finally I found out that the UInt32 and UInt64 types are not CLS-compliant and that CLS-compliant alternative for UInt32 is Int64. Again, I don't want to use 64 bits. I only need 32 of them!

I can't be the only one having this trouble, so I started searching what this CLS compliance meant. From what I've read about this, I know now that .NET doesn't treat UInt32 values as such internally, which results in the MSB getting back its role as sign bit, which in turn results in a sign bit being shifted out during the arithmetic which eventually results in an Exception...
So .NET doesn't natively support unsigned data types! OK..

After multiple hours of trying all kinds of stupid things to get the right results without exceptions, I ended up with this code:

Public Function Jenkins_one_at_a_time_hash(ByVal data() As Byte) As UInt32
  Dim hash As UInt32 = 0

  For Each b As Byte In data
    hash = ((hash + b) And &HFFFFFFFFL)
    hash = (hash + ((hash << 10) And &HFFFFFFFFL)) And &HFFFFFFFFL
    hash = hash Xor ((hash >> 6) And &HFFFFFFFFL)

  Next

  hash = (hash + ((hash << 3) And &HFFFFFFFFL)) And &HFFFFFFFFL
  hash = hash Xor ((hash >> 11) And &HFFFFFFFFL)
  hash = (hash + ((hash << 15) And &HFFFFFFFFL)) And &HFFFFFFFFL

  Return hash

End Function

So actually, it comes down to 'manually' preventing overflows wherever they can occur; even in intermediate results; then why did it take me so long to figure this out???

Hurrying back to Delphi again, yuck!