Home Automation and Voice Control

HAL-9000 (Space Odyssey), Mother (Alien), The Matrix, Jarvis (Iron Man), KITT – who doesn’t know them? And since a few days there’s Jasper, voice control for the Raspberry Pi.
An RPi, microphone, speaker and network connection is all you need (and the Jasper software package of course).

Interacting with computers by voice has always been a very appealing feature to have in my Home Automation System. There’s a button on the touchscreen in the living room which controls a light bulb – when you press that button, you hear Darth Vader saying “Yes, Master“. My son and I liked it; it was funny. But there had to be more…

So when 2 Princeton students released ‘Jasper’ a few days ago, I was triggered to revisit the subject ‘voice control’ once again.

My first thought was to give Jasper a try as soon as I had the time – but after I read some parts of the API documentation I became a bit hesitant – stuff like defining the words that the user is allowed to speak (or better: which will be recognized and processed further by Jasper) in the code is not how I’d like to do things. Another thing I didn’t like is that it would become a more or less isolated ‘sub-system’ to my HA system – answering questions, controlling Spotify and such. Create a module for every type of hardware here in our house? Neh. No chance.

Maybe it’s better to revisit Voicecommand, a tool developed by Steven Hickson and his PiAUISuite which I read about a year ago or so. Voicecommand (at least, the demo-videos are) seems to be made primarily to initiate actions (playing video, music, start the browser) on the local computer(/Raspberry). But why not try to extend it, remove some of the (local) action initiation parts of the code and replace that with a MQTT client?

That would make it a perfect fit for my HA system – this way I can let my rules engine receive the voice commands and let the rules engine be the definition of what has to be accepted as valid command and what actions should be executed.

So I ‘freed’ a Raspberry Pi and downloaded the PiAUISuite. The first problem was that I didn’t have a USB microphone – ahh, but our kids do,Β for things like Skype, online gaming and other things I never do. I found an old speaker set in the garage and I was good to go.

After some tinkering with the Voicecommand tool as-is, it’s configuration, trying different keywords and stuff like that, it was time to change some things.

First thing I wanted to change was the language. Voicecommand uses the Google Speech API, so using Dutch as language should not be a problem; all I had to do was change lang = “en” to lang = “nl”. Done! It improved the voice recognition quite a bit too! πŸ˜‰

I also wanted to change the response (“Yes, Sir?”) in to a simple short beep. This would significantly shorten the duration of the whole conversation which was a bit too long for my taste. I searched for a ‘beep’ MP3 on the internet that was short and loud enough to be noticed, searched the Voicecommand code for Speak(response) and replaced that call with Play(beep), a new function that I added to the code.

Another thing I changed was the matching of spoken command with a list of predefined commands (and their associated actions) in the ~/.commands.conf. Right now, I just send every wordΒ to my HA system and let my system decide if the spoken command contains something useful.

The last thing I did to do get communication between Voicecommand and my HA system going was building the Mosquitto MQTT client on the Raspberry Pi and call that client (mosquitto_pub) with the right parameters from Voicecommand with a system() call. It’s a bit of a quick & dirty trick to get things going though; it would be much better to incorporate the MQTT protocol in the Voicecommand code, but that’s too much work for now – first I want to see how this works out in practice with a better microphone and some useful commands & rules…

The only rule I have right now is this one, for controlling a small night lamp in the office:

rule office_test_light {
  when {
    m1: Message m1.t == 'voice' && m1.contains('licht');
  }
  then {
    if (m1.contains('aan')) {
      publish("command",'{"address":"B02", "command":"ON"}');
    } else
    if (m1.contains('uit')) {
      publish("command",'{"address":"B02", "command":"OFF"}');
    } else {
      log('Snap het niet');
    }
  }
}

Voicecommand has, for as far as I can see now, one drawback: no Internet connection means no Voice Control. The (very!) big plus is that the TTS voice is superior to what I’ve heard with Jasper.

Future plans:

  • sending textual (MQTT) messages to Voicecommand and let it speak them;
  • returning an error message when the rules engine was not able to process the command;
  • adding the RPi hostname to the message that goes to my HA system, which can be useful when having multiple Voicecommand Rpi’s throughout the house – cause a “light off” command in the garage implies a different action than “light off” in the kitchen.. πŸ˜‰

Right now, after a few hours of tinkering, I think I’ve got something that’s worth spending more time on. We’ll see! Here’s a video of what I’ve accomplished so far:

 

 

Tagged , , . Bookmark the permalink.

28 Responses to Home Automation and Voice Control

  1. Lukas says:

    Mag ik vragen hoe uw config file (command.conf) er uitziet?

    • Natuurlijk Lukas! Al zal het nut bijzonder gering zijn, aangezien ik in de voicecommand code diverse aanpassingen heb gedaan om hetgeen je ziet gebeuren voor elkaar te krijgen. Inmiddels heb ik voicecommand ‘verlaten’ en ben nu met node-speakable bezig; deze ‘luistert continu’, is eenvoudiger intergreerbaar in mijn systeem en heb ik omgebouwd naar Google Speech API v2. Deze bevalt me beter eigenlijk. Hier is mijn ~/.command.conf:

      licht== ...
      !continuous==1
      !verify==1
      !com_dur==3
      !duration==2
      !improper==begrijp ik niet
      !thresh==0.5
      !keyword==James
      !filler==0
      !ignore==1
      !response==Tot uw dienst

  2. Eric says:

    Robert,
    That’s awesome! I’d really like to know how you linked up Voicecommand to send MQTT messages. Right now, I’m using OpenHAB and Mosquitto, so if I can get Voicecommand to publish some things depending on the command, it would fit in really well.

    If you develop this more, I’d love to hear more details.

    • Quick, dirty & simple…but OK for testing purposes πŸ˜‰
      Here you see how I changed the ProcessMessage():
      inline void VoiceCommand::ProcessMessage(const char* message) {
      printf("ProcessMessage() %sn",message);
      //Speak(message);
      unsigned int i = 0, loc = 0;
      string tmp = message;
      string sTmp = message;
      to_upper(sTmp);
      broker = "192.168.1.2";
      string pubcommand = "";
      pubcommand += "mosquitto_pub -h ";
      pubcommand += broker;
      pubcommand += " -t voice";
      pubcommand += " -m "";
      pubcommand += message;
      pubcommand += """;
      printf("publishing: %sn",pubcommand.c_str());
      system(pubcommand.c_str());
      return;

      And of course you’ll need mosquitto installed on the same box (for the mosquitto_pub command).
      That’s it!

  3. Marcus Martini says:

    “I also wanted to change the response (β€œYes, Sir?”) in to a simple short beep. This would significantly shorten the duration of the whole conversation which was a bit too long for my taste. I searched for a β€˜beep’ MP3 on the internet that was short and loud enough to be noticed, searched the Voicecommand code for Speak(response) and replaced that call with Play(beep), a new function that I added to the code.”

    How did you implement the beep, can you please post the Code.

    • I had 2 beeps actually, for ACK and NAK.
      The ACK:
      int VoiceCommand::BeepAck() {
      printf("Beepn");
      if (quiet) {
      return 0;
      }
      string command = "vcbeep";
      system(command.c_str());

      return 0;
      }

      And a script called vcbeep:

      #!/bin/bash
      #for the Raspberry Pi, we need to insert some sort of FILLER here since it cuts off the first bit of audio
      mpg123 /home/pi/beep-23.mp3 1>>/dev/shm/beep.log 2>>/dev/shm/beep.log

      • Marcus Martini says:

        Where exactly i need to put the code an the script file. and how i need to configure the confog file

        • The code should be added to the voicecommand.cpp, but will of course only work if you call it πŸ˜‰
          The script can be saved in /usr/bin where some other voicecommand scripts can be found as well.
          The contents of my config file can be found in a previous reply.

  4. Marcus Martini says:

    can you make me a example to call it i am not so god with c++

    • Not an example, but here’s my voicecommand code.
      There will be some things in the code you’ll probably not want, but that should be easy to change (back to the original code).

      • Martin says:

        Thanks Robert! The stuff you create is so inspirational. Voice control is still the missing piece in my home automation system and your code gave me the right base to start. It’s easier to adjust and tweak existing stuff than reinventing the wheel πŸ™‚

        Keep up the good work!

  5. Xtrusion says:

    Hi Robert

    how’s your node-speakable project going. i already have voice control via home-made android app, and yesterday started with node-speakable on an raspberry (also ordered an condensor microphone to get better audio from the room)

    some problems i have till now
    – sometimes sox is keep listening, even if you are not speaking anymore, only solution then is make some more noise, then it will send to google. also played with the mic volume in aslamixer seems to help a little but it’s not yet stable

    did you changed the sox arguments in the library ?

    other issue is in some cases it’s missing the first words i say, for example “televisie aanzetten”, then it’s only recognising the word “aanzetten”

    • It’s on hold, just like almost everything over here – some other ‘forces’ consume a lot of time right now.
      I had the same issues as you; experimented with the sox arguments but I never got it 100% right all the time.
      I hope I’ll get some more time in the near future …

  6. Pingback: Voice control revisited: the Web Speech API - Digits Domotica Blog

  7. Geert-Jan says:

    Hoi Robbert,

    Ik ben al 3 dagen bezig in steven hickson’s voice control op mij Rpi aan de gang te krijgen. Maar geen bananen…..
    heb jij of weet jij een image online van een werkende nl setup?
    Het gaat bij mij al mis als ik de een webcam wil instellen als microfoon. Een heeft een Thresh van 0.001582 en de andere wilt helemaal niet:
    arecord: set_params:1145: Unable to install hw params:
    ACCESS: RW_INTERLEAVED

    Met Jasper werken ze wel.
    Nu ben ik niet erg thuis in linux en heb ik al de site al wel 5x bezocht voor antwoorden.

    Ik heb geen idee meer hoe ik het nog aan de gang kan krijgen, het lijkt allemaal een hele simpele installatie. Maar het ik weet het voor mezelf toch heel lastig te maken.

    • Robert Hekkers says:

      Die foutmelding zegt me niet veel, nooit gehad. En de SD card waarmee ik destijds heb gewerkt is al een tijdje naar de eeuwige jachtvelden. Binnenkort komt Homey, ik wacht daar even op tot ik weer met Voice Control verder ga. Ik heb nu een eigen brouwsel werkend op de smartphones, maar momenteel staat dit project even in de koelkast.

  8. Johan says:

    Hello Robert, really nice guide. I was trying to change the answer to a “beep” i have copied your code and overwritten the original .cpp and .h files. Created the vcbeep script under home/scripts and made it +x (runable) however voicecommand seems to ignore the changes i’ve made to the .cpp and .h files. It still run the “show” as usual.

    Do i need to recompile the code or something? Or maybe specificy the full path to my vcbeep script in the .cpp file? I would deeply appriciate your help, big thanks in advance.

    //Johan

    • Robert Hekkers says:

      Thanks, yes you’ll need to recompile. IIRC it’ll take a while.

      • Johan says:

        Okey thank you. I am a real newb at compiling but i tried the g++ /voicecommand.cpp didnt work so well im afraid. Is there any other file i need to run the compilation on? Like the voicecommand.h? Or how do i run this proceedure when the code is in place?

        And secondly, do the filepath to the vcbeep script need to be specified in the function? (For example)

        }
        string command = “/home/vcbeep”;
        system(command.c_str());

        return 0;
        }

        //Johan

  9. Csnorlax says:

    I tried to change the response to a beep. but it does not work. Is it also possible to change it to a mp4 ? So it will play a short movie instead of only a sound response

  10. Csnorlax says:

    But first I have to get that beep working. And I can not even get that to work

  11. Joshua Falken says:

    Hi Robert,
    at first, sorry for my english. I play now a little while with the VC and find you’r Blog and i compile the source and generate the vcbeep-script as well. 1 little error coming up, but work! My question now, why is my VC-File more then 5 times bigger then the original and the beep comes 3 times, when i call.
    Any idea? BTW -> Great work!

    • Robert Hekkers says:

      Hi Joshua, nothing wrong with your english, mein deutsch ist eigentlich auch nicht so gut…lesen geht schon aber fehlerlos schreiben ist viel schwieriger πŸ˜‰ (DE is a 10 minute drive from here, so.. πŸ˜‰ Suddenly I get a lot of questions about this post, I don’t really understand why *now* but it’s nice to hear. In fact, I’ll redo my search for the complete source code I once had – it was on a SD card without backup so I know it’s not on my backup NAS, but maybe on my local HDD’s somewhere… I’ll do a 2nd search and see if I can help you..
      Robert

      • Joshua Falken says:

        Hi Robert, thanks for your fast answer!
        I do a mistake, the beep-sound works, the orignal sound was a 3 times beep, sorry!

        “You need not looking for your source!!”
        Everthing is fine and works πŸ™‚ Now i’m trying to talk to my Arduino. The goal is that the Arduino give me a status-call back. I have a project with Processing on PC for visualization and i’ll a audio-answer of my actions. So when i say “Lampe an”, i like to have a “Lampe ist an!” if Arduino switch the light on. I think it is not so heavy, the serial communication works with Processing, so i have only to check the output. Thank for your help, happy greetings from germany.
        Joshua

  12. Jos Reijnders says:

    Hallo allemaal, ik ben ook begonnen mijn home automation uit te breiden met voice control.
    Ik heb de aanwijzingen gevolgd van de software van Steven Hickson maar mij Pi reageert zonder dat ik het keyword heb gezegd.
    Even te aanvulling: Mijn verlichting ed. kan ik al bedienen met een afstandbediening, een (oude van hetzelfde type) heb ik (middels opto-couplers) aan een bedieningspaneel gehangen. Als de voice recognition goed werkt kan ik uitgangen van de Pi besturen en daarmee m’n verlichting of radio of versterker of tuinsproeiers ed.
    Maar eerst moet de VR nog werken. Heeft iemand hetzelfde probleem gehad of kan ik beter Jasper VR gaan gebruiken?

Leave a Reply

Your email address will not be published. Required fields are marked *