Websockets and mobile network operators

Something I found out very recently, is the fact that Websockets don’t always work as expected. And neither the client side nor the server side can do anything about it- in my case, it’s the telco that messes up things. Either intentionally or not, I don’t know. And to be honest, I don’t really care – it doesn’t work, that’s the only thing that matters.

There are several sources that warn about proxy servers in combination with Websockets. According to one of those sources, the chance that Websockets will work when the traffic has to pass a proxy server is about 60%.

The day after I had my first Websocket-enabled web page running, I tested it from my PC at my work location. But unfortunately, it didn’t work.. my first thought was that it must be the transparent proxy server we use over there – I installed that one myself, so I know the state it is in 😉 But hey, never mind, I still got my smart phone, an Android with Firefox Mobile which supports Websockets, right?

Wrong. It didn’t work either. First thought was that I must have been dreaming. But that same day, back at home, the page worked again. Strange, cause I don’t use the company network with my smart phone.

So what’s the difference? The distance? Well, in fact it is.. 😉  The distance from my home Wifi to be precise! That’s why I couldn’t get websockets working 50 km away from home – I needed Wifi to get things running..

Time for some serious WSI (Websocket Scene Investigation).

While at home, my websocket page worked fine. But after I disabled Wifi on the smart phone, it stopped working. But after turning Wifi back on, it all worked perfect again, as before. What is this; what’s the big difference? It’s not like I’m addressing local IP addresses while I’m at home, so firewalls and all other things that can block or otherwise mess up things can be ruled out…I got the feeling that there was some unknown entity in between that was bugging me and I just had to find out what it was.

The first thing I did was setting the log level of my Apache reverse proxy server to debug. Maybe it was my own proxy server that was misbehaving? Wifi-mode told me it couldn’t be that, but it’s always good to check and double-check the things under your own control before blaming something or someone else, so I did. I performed 2 tests with my smart phone – 1 with Wifi enabled and the other one with Wifi disabled and watched the Apache logs.

The good, with Wifi enabled:

82.95.233.76 - - [28/Nov/2012:20:44:05 +0100] "GET /mosquitto/ HTTP/1.1" 101 20 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"

Perfect. Response 101.

Now the bad (Apache error log, with Wifi disabled):

[debug] proxy_util.c(1415): [client 62.140.137.84] proxy: ws: found worker ws://localhost:80/ for ws://localhost:80//
[debug] mod_proxy.c(849): Running scheme ws handler (attempt 0)
[debug] mod_proxy_http.c(1812): proxy: HTTP: declining URL ws://localhost:80//
[debug] mod_proxy_ajp.c(524): proxy: AJP: declining URL ws://localhost:80//
[debug] mod_proxy_ftp.c(807): proxy: FTP: declining URL ws://localhost:80// - not ftp:
[debug] mod_proxy_connect.c(100): proxy: CONNECT: declining URL ws://localhost:80//
[warn] proxy: No protocol handler was valid for the URL /mosquitto/. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule

And the access log says:

62.111.137.xx - - [28/Nov/2012:20:41:15 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"
62.111.137.xx - - [28/Nov/2012:20:41:21 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"
62.111.137.xx - - [28/Nov/2012:20:41:24 +0100] "GET /mosquitto/ HTTP/1.1" 500 615 "-" "Mozilla/5.0 (Android; Mobile; rv:17.0) Gecko/17.0 Firefox/17.0"

Aha, response 500, which means Internal server error, the error code you get when there’s something going wrong. But what and why? I still didn’t get it. Same phone, same browser, same web page, .. the only differences were connection and time of day.

Digging a bit deeper was all I could think of to find the cause. tcpdump, a network traffic analysis tool I haven’t used for quite some time, was the first thing that popped up in my mind, so after re-reading the man pages I set up tcpdump to capture the port 80 traffic to and from my Apache server – one capture from my smart phone with Wifi enabled and the other one without Wifi. Here are the results:

Wifi enabled:

GET./mosquitto/.HTTP/1.1..
Host:.ws.hekkers.net..
User-Agent:.Mozilla/5.0.(Android;.Mobile;.rv:17.0).Gecko/17.0.Firefox/17.0..
Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..
Accept-Language:.nl,en-us;q=0.7,en;q=0.3..
Accept-Encoding:.gzip,.deflate..
Connection:.keep-alive,.Upgrade..
Sec-WebSocket-Version:.13..
Origin:.http://www.hekkers.net..
Sec-WebSocket-Protocol:.mqtt..
Sec-WebSocket-Key:.7ribPC7W8+/Jd6d58JElFA==..
Cookie:.__qca=P0-1711558157-1343631612547..
Pragma:.no-cache..
Cache-Control:.no-cache..
Upgrade:.websocket..

And with Wifi disabled, the headers are:

GET./mosquitto/.HTTP/1.1..
User-Agent:.Mozilla/5.0.(Android;.Mobile;.rv:17.0).Gecko/17.0.Firefox/17.0..
Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..
Accept-Language:.nl,en-us;q=0.7,en;q=0.3..
Accept-Encoding:.gzip,.deflate..
Sec-WebSocket-Version:.13..
Origin:.http://www.hekkers.net..
Sec-WebSocket-Protocol:.mqtt..
Sec-WebSocket-Key:.5OeSLgzyhItbIYt7PcZMJw==..
Cookie:.__qca=P0-1711558157-1343631612547..
Pragma:.no-cache..
X-VF-ACR:.204004DYNMVFNLACRXXYYYYYYYYYYXXXVd97865Ex9p
M2tFXP2+lEtG2MEYYX0XKm9e9LXijY/GrxAd/k9UpEIfGAbmwfFqj
GvB68WhBsSlB3s2K5F9XAGS9nSoIURdX6hNviIdtqq6uVGZsQBmMO
xVqeNiA9tmuf/pdv7Ag+dkQBzBs9XpKHB3zUJMojD0EGbiDCYPa43
geeRKp8NpXHig7rgo98789O/x3pDmDcLgqs8s23HWQ2slxL7EIcSF
cNFq5Tawdjv1SuOvYJNaVMmVT4wjMWGfpjSjj7wi8nX51pecyU25J
oRjhkWPVtdEPgGQmHmuewzqJay4ereTyWEzi5LqdfA1i2row9xiQt
Yhe9HUhe1H2s8w==..
Host:.ws.hekkers.net..
Cache-Control:.no-cache,.max-age=43200..
Connection:.keep-alive...

You see? Why did the Upgrade: websocket header suddenly disappear? Cause that’s what triggers the server to respond in a way that enables working with websockets!

The only conclusion I could make is that it must be my telco Vodafone to be the one to mess up the headers that are sent by my smart phone.

And I did another test, just to be sure I’m not telling any rubbish here;  Lightstreamer has a ‘special‘ URL with which you can see how your browser connects to their streaming demos; either by websockets, streaming HTML or any other mechanism that allows real time streaming of data. And that webpage confirmed my conclusion: Vodafone blocks  websockets, intentionally or not.

Now, this is really stupid – my wife also has a smart phone, but with a very low monthly tariff and websockets are working OK for her. And me, with my monthly Vodafone costs of three times as much, can’t use websockets.. ?

So pay attention to which mobile network operator you choose!

A tip for a Websocket friendly telco, anyone?

Bookmark the permalink.

11 Responses to Websockets and mobile network operators

  1. Martijn van Zal says:

    We recently ran into the exact same problem. For us web sockets were working perfectly on an iPhone with wifi, umts (Ben.nl/t-mobile network) and umts (Vodafone). But when using a T-mobile sim, websockets stopped working.

    We figured the same thing: there must be a transparent proxy in the way which doesn’t support websockets.

    Our solution turned out to be pretty simple: move your web socket server to a different port. Since we were afraid some firewalls would block traffic on unknown ports, we decided to use port 443 (ssl) without using encryption at all. From that moment, our system seems to work perfectly with all Dutch telco’s/Internet providers.

  2. Toby Jaffey says:

    One solution is to not use websockets at all. I’m not yet convinced that they’re ready for use in the real world, which is why I use HTTP long polling in my MQTT to HTTP server.

    https://github.com/jobytaffey/mqtt-http-server

  3. Dave Boonstra says:

    Hi,

    My opinion is not to use websockets on port 80.

    You will be running into limitations in many places. Public wifi hotspots and mobile telcos may have proxy servers. Not all servers are using the latest and greatest software. The solution is to use port 443 as you can not push everyone to upgrade their boxes for a protocol that is not yet widely adopted.

    Mobile networks often have these proxies for the customer’s benefit. They reduce bandwidth consumption by doing several smart things like just-in-time buffering of streaming video.

    I’m sure nobody is breaking WS intentionally, it is just too new a technolgy which has not been taken into account.

    Move to port 443 and your worries should be gone.

  4. Pingback: Infovore » Links for January 16th

  5. Pingback: Performance Calendar » Mobile ISP image recompression

  6. Nikos says:

    Are more carriers opening up now for websockets?

  7. Pingback: Can Websockets Work On Mobile Phones?

  8. Pingback: Can Websockets Work On Mobile Phones? - BlogoSfera

Leave a Reply

Your email address will not be published. Required fields are marked *