More Kinect: skeletons and thoughts

This is fun! Today I created a C# application to view skeletons on my PC monitor. Below is the result (available in HD res):

I’m just doing some silly moves in front of the camera like lifting my arms etcetera, so nothing spectacular there; it’s the skeleton being displayed on the monitor what’s spectacular, and fun!

Every time I did some tests, the kids came running from the livingroom and started jumping, waving their arms and doing all kinds of crazy stuff 🙂

At the end of the video, where I walk back towards my keyboard and the distance to the Kinect reduces to 50 cm or so, you can see that the Kinect is getting a hard time tracking all the joints; it becomes a big mess. But that’s okay.

After tinkering with the Kinect and the Kinect SDK for 8 hours or so, my first impression is better than I expected, acually. The way speech is being recognized, how well a skeleton is being tracked is better then I had expected. Ready for use! Is it, or…?

My goal is to use the Kinect for speech and gesture recognition; for that I want to develop a small, lean and mean app that does just that and that can interface with my Domotica system in some way. Either by adding a TCP server to the app, or by uploading information with XMLRPC, or whatever – haven’t really thought about that yet. But I also need an application to ‘record’ gestures and an easy way to define new speech commands this app should recognize; and not just “light on”, but also commands with variables in them, like “we will be back at 16:00”. My system can handle this kind of ‘dynamic events’, so the Kinect should too, right?

But some other things give me second thoughts about the usability of the Kinect.

First, while I was capturing this video, the CPU usage on my Intel E8200 Core 2 Duo (2.66 GHz) Windows 7 Ultimate PC with 8GB RAM was around 50-60%; that’s rather high. Well, I guess most of the CPU cycles were consumed by displaying the skeleton on the screen (which is something I won’t need in the end), but I really need to know what the remainder of the current 50-70% will be when I strip this app to what it is supposed to do: simply recognizing gestures and speech without UI. Cause I’m not that fond of spending that much processing power (and the power consumption that comes with it) on this.

Second, the SDK is limited to Windows 7 (and above, I guess). That’s too bad; now that I think of it; wouldn’t I be better off with something more versatile, platform independent? Wouldn’t it be ideal to connect a Kinect to some local, low-power processor unit (Raspberry Pi, xyz Plug, etc) and let that device take care of the processing and only communicate “the stuff that matters” to the central system? Hmm, maybe I should have a look at some alternatives as well.

Third, the Kinect uses USB to connect to the PC. What if I want to use more than 1 Kinect, say 2 or 3, throughout the house? How do I fix that, USB and max 5 meters of cable length?

So although I’m impressed by the Kinect, there’s still a huge pile of things to do, discover, explore and.. decide!

The Kinect and Home Automation

Microsoft Kinect

A new, exciting piece of hardware arrived today – a Microsoft Kinect. The Microsoft Kinect (formerly known as ‘Project Natal’) was primarily sold as an accessory for the XBOX game console, but that’s going to change. Microsoft released a Kinect for Windows SDK earlier this year, which enables developers to unlock the possibilities of the Kinect to the Windows platform.

Now this Kinect is really becoming interesting for Domoticans 🙂 Interacting with your Home Automation system by using gestures, speech, tracking people – that really sounds like the next step to me. So when Robin, who’s also a member of Domoticaforum Europe asked me if I would like to have a look at the Kinect, I didn’t hesitate one second – yes, of course!

Before I could start I had to download & install Visual Studio 2010 Express and the Kinect for Windows SDK. Most of the SDK samples are written in C#, so I installed VS C# 2010 Express and the SDK on my Windows 7 machine. After the Kinect arrived, I plugged it in and all the drivers that were needed were installed. I started Visual Studio, started the Speech sample and started saying some colors, cause this sample was setup to recognize the colors red, green and blue. And it did, very well.

One strange thing though – I had to press the ENTER key to stop that sample application? Okay.. there’s a better way to do that for an application that can recognize speech, so the first thing I changed was adding the word “shutdown” to the list of words that should be recognized and added some code that would do just that: shutting down the sample application. And although I’ve never ever done C# before, I did it in a matter of minutes… cool! I think I can work with this.

Although this was a very brief encounter with the Kinect, as far as I can see now, there are great opportunities for this Kinect in the world of Home Automation/Domotica. Now let’s see if I can also recognize gestures; which sounds much more complex and maybe it is – well, I can tell more about that in a few days, or weeks maybe…

Happy Easter!