Something I found out very recently, is the fact that Websockets don’t always work as expected. And neither the client side nor the server side can do anything about it- in my case, it’s the telco that messes up things. Either intentionally or not, I don’t know. And to be honest, I don’t really care – it doesn’t work, that’s the only thing that matters.
There are several sources that warn about proxy servers in combination with Websockets. According to one of those sources, the chance that Websockets will work when the traffic has to pass a proxy server is about 60%.
The day after I had my first Websocket-enabled web page running, I tested it from my PC at my work location. But unfortunately, it didn’t work.. my first thought was that it must be the transparent proxy server we use over there – I installed that one myself, so I know the state it is in 😉 But hey, never mind, I still got my smart phone, an Android with Firefox Mobile which supports Websockets, right?
Wrong. It didn’t work either. First thought was that I must have been dreaming. But that same day, back at home, the page worked again. Strange, cause I don’t use the company network with my smart phone.
So what’s the difference? The distance? Well, in fact it is.. 😉 The distance from my home Wifi to be precise! That’s why I couldn’t get websockets working 50 km away from home – I needed Wifi to get things running..
Time for some serious WSI (Websocket Scene Investigation).
While at home, my websocket page worked fine. But after I disabled Wifi on the smart phone, it stopped working. But after turning Wifi back on, it all worked perfect again, as before. What is this; what’s the big difference? It’s not like I’m addressing local IP addresses while I’m at home, so firewalls and all other things that can block or otherwise mess up things can be ruled out…I got the feeling that there was some unknown entity in between that was bugging me and I just had to find out what it was.
The first thing I did was setting the log level of my Apache reverse proxy server to debug. Maybe it was my own proxy server that was misbehaving? Wifi-mode told me it couldn’t be that, but it’s always good to check and double-check the things under your own control before blaming something or someone else, so I did. I performed 2 tests with my smart phone – 1 with Wifi enabled and the other one with Wifi disabled and watched the Apache logs.
The good, with Wifi enabled:
18.104.22.168 - - [28/Nov/2012:20:44:05 +0100] "GET /mosquitto/ HTTP/1.1" 101 20 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"
Perfect. Response 101.
Now the bad (Apache error log, with Wifi disabled):
[debug] proxy_util.c(1415): [client 22.214.171.124] proxy: ws: found worker ws://localhost:80/ for ws://localhost:80// [debug] mod_proxy.c(849): Running scheme ws handler (attempt 0) [debug] mod_proxy_http.c(1812): proxy: HTTP: declining URL ws://localhost:80// [debug] mod_proxy_ajp.c(524): proxy: AJP: declining URL ws://localhost:80// [debug] mod_proxy_ftp.c(807): proxy: FTP: declining URL ws://localhost:80// - not ftp: [debug] mod_proxy_connect.c(100): proxy: CONNECT: declining URL ws://localhost:80// [warn] proxy: No protocol handler was valid for the URL /mosquitto/. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule
And the access log says:
62.111.137.xx - - [28/Nov/2012:20:41:15 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0" 62.111.137.xx - - [28/Nov/2012:20:41:21 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0" 62.111.137.xx - - [28/Nov/2012:20:41:24 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"
Aha, response 500, which means Internal server error, the error code you get when there’s something going wrong. But what and why? I still didn’t get it. Same phone, same browser, same web page, .. the only differences were connection and time of day.
Digging a bit deeper was all I could think of to find the cause. tcpdump, a network traffic analysis tool I haven’t used for quite some time, was the first thing that popped up in my mind, so after re-reading the man pages I set up tcpdump to capture the port 80 traffic to and from my Apache server – one capture from my smart phone with Wifi enabled and the other one without Wifi. Here are the results:
GET./mosquitto/.HTTP/1.1.. Host:.ws.hekkers.net.. User-Agent:.Mozilla/5.0.(Android;.Mobile;.rv:17.0).Gecko/17.0.Firefox/17.0.. Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.. Accept-Language:.nl,en-us;q=0.7,en;q=0.3.. Accept-Encoding:.gzip,.deflate.. Connection:.keep-alive,.Upgrade.. Sec-WebSocket-Version:.13.. Origin:.http://www.hekkers.net.. Sec-WebSocket-Protocol:.mqtt.. Sec-WebSocket-Key:.7ribPC7W8+/Jd6d58JElFA==.. Cookie:.__qca=P0-1711558157-1343631612547.. Pragma:.no-cache.. Cache-Control:.no-cache.. Upgrade:.websocket..
And with Wifi disabled, the headers are:
GET./mosquitto/.HTTP/1.1.. User-Agent:.Mozilla/5.0.(Android;.Mobile;.rv:17.0).Gecko/17.0.Firefox/17.0.. Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.. Accept-Language:.nl,en-us;q=0.7,en;q=0.3.. Accept-Encoding:.gzip,.deflate.. Sec-WebSocket-Version:.13.. Origin:.http://www.hekkers.net.. Sec-WebSocket-Protocol:.mqtt.. Sec-WebSocket-Key:.5OeSLgzyhItbIYt7PcZMJw==.. Cookie:.__qca=P0-1711558157-1343631612547.. Pragma:.no-cache.. X-VF-ACR:.204004DYNMVFNLACRXXYYYYYYYYYYXXXVd97865Ex9p M2tFXP2+lEtG2MEYYX0XKm9e9LXijY/GrxAd/k9UpEIfGAbmwfFqj GvB68WhBsSlB3s2K5F9XAGS9nSoIURdX6hNviIdtqq6uVGZsQBmMO xVqeNiA9tmuf/pdv7Ag+dkQBzBs9XpKHB3zUJMojD0EGbiDCYPa43 geeRKp8NpXHig7rgo98789O/x3pDmDcLgqs8s23HWQ2slxL7EIcSF cNFq5Tawdjv1SuOvYJNaVMmVT4wjMWGfpjSjj7wi8nX51pecyU25J oRjhkWPVtdEPgGQmHmuewzqJay4ereTyWEzi5LqdfA1i2row9xiQt Yhe9HUhe1H2s8w==.. Host:.ws.hekkers.net.. Cache-Control:.no-cache,.max-age=43200.. Connection:.keep-alive...
You see? Why did the Upgrade: websocket header suddenly disappear? Cause that’s what triggers the server to respond in a way that enables working with websockets!
The only conclusion I could make is that it must be my telco Vodafone to be the one to mess up the headers that are sent by my smart phone.
And I did another test, just to be sure I’m not telling any rubbish here; Lightstreamer has a ‘special‘ URL with which you can see how your browser connects to their streaming demos; either by websockets, streaming HTML or any other mechanism that allows real time streaming of data. And that webpage confirmed my conclusion: Vodafone blocks websockets, intentionally or not.
Now, this is really stupid – my wife also has a smart phone, but with a very low monthly tariff and websockets are working OK for her. And me, with my monthly Vodafone costs of three times as much, can’t use websockets.. ?
So pay attention to which mobile network operator you choose!
A tip for a Websocket friendly telco, anyone?