I’ve been doing some excursions during the last 12 months to figure out what would be the best way to replace my Windows-based Home Automation system. It has become rather large through the years and I want to get rid of that single point of failure, being that Windows executable that does it all. A few weeks ago I tried some things with a Raspberry Pi (RPi) and Python and was actually very much surprised by the speed with which I could build something from scratch in a matter of hours.
Yesterday afternoon was the first time (after a long, long week full of all kinds of problems) that I could sit down behind my PC again and spend some time on things I like to do. That afternoon I tried to get a Python script to run as a daemon. Running a Python script from the command-line is not very useful for me (only for testing & debugging) cause what I really want is that the RPi starts all the tasks I want it to run on it automatically at boot time – just power-up the RPi and the rest goes automagically, no user interaction should be needed. So I decided to combine 3 subjects into this small experiment: a Python MQTT client, daemonizing Python scripts and web scraping; lets see how far I can get.
I started with a brand new RPi, SD card and new image written to it. Did the usual things like setting the hostname, time zone etcetera. After that I installed some Python related tools (e.g. installing modules) to make life easier. An MQTT client written in Python wasn’t very hard to find – Mosquitto (the OS MQTT server that I’m using since September last year) has one, so I installed that one too. I searched for info on how to create a daemon and found the python-daemon library. The last things I needed were some tools to make the process of web scraping somewhat more comfortable: regular expressions and requests. I found some examples on how to use all these libraries/tools mentioned above and started coding.
Well, is this something you can really call coding? Reading and understanding the examples I found, copying code snippets, adding some extra lines, deleting others – it feels more like blending actually 😉
I use web scraping to show various kinds of information on the User interfaces in our house – things like the amount and total length of traffic jams, a 2-hour rain forecast (BBQ!) for our specific location and stuff like that, so it looked like a good idea to have a look if I could make one of those scrapers run in a Python-based daemon on the RPi.
3 hours later I was finished, with this code as a result:
import logging import time import mosquitto import requests import re from daemon import runner class App(): def __init__(self): self.stdin_path = '/dev/null' self.stdout_path = '/dev/tty' self.stderr_path = '/dev/tty' self.pidfile_path = '/var/run/webfetcherd.pid' self.pidfile_timeout = 5 def on_connect(self, mosq, obj, rc): logger.info("on_connect:"+str(rc)) def on_message(self, mosq, obj, msg): logger.info(msg.topic+" "+msg.payload) def de_html(self, html): pattern = re.compile("<.*?>| |&",re.DOTALL|re.M) return pattern.sub("",html) def run(self): while True: self.mqttc = mosquitto.Mosquitto("webfetcher") self.mqttc.on_message = self.on_message self.mqttc.on_connect = self.on_connect self.mqttc.connect("192.168.10.40", 1883, 60) rc = 0 prvtime = 0.0 while rc == 0: rc = self.mqttc.loop() if (time.time()-prvtime) > 60: prvtime = time.time() rq = requests.get('http://m.fileindex.nl/files.js') rex = re.compile('"(.*)"') m = rex.search(rq.text) if m: res = self.de_html(m.group()) logger.info('Match: %s', res) self.mqttc.publish("test/trafficjams",str(res)) else: logger.info('NO match: %s', rq.text) logger = logging.getLogger("Webfetcher") logger.setLevel(logging.INFO) formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(messa handler = logging.FileHandler("./webfetcher.log") handler.setFormatter(formatter) logger.addHandler(handler) app = App() daemon_runner = runner.DaemonRunner(app) daemon_runner.daemon_context.files_preserve=[handler.stream] daemon_runner.do_action()
Done! Ok, not completely – I didn’t make an init script yet and I don’t really like the way the MQTT client is called (line 36) and there are some other things that can probably be done better, especially when there’s I/O involved with hardware connected to the RPi; but for this goal and as result for a first alpha version this is OK I guess..
On to alpha2!