Bash: Geolocation, the trilogy

Three iterations of the same script, between December 2008 and October 2009, each one born from the previous one and its limits. V1 is lost with the hard drive (a Samsung 500 GB, with all my Bash, C, PHP and Perl projects, my personal music, the thing was 90% full!), v2 and v3 survive. It is the story of a script’s evolution in public.

The revenge of netcat (v2, February 2009)

After an IRC discussion on geolocation, a friend who used the same site passed me one of his scripts that in a single line did exactly what mine did but much faster thanks to netcat (no disk access). Unfortunately, following the loss of my hard drive, I lost a bunch of tools, including that famous script. So I rewrote the script, being significantly less good than my friend, it takes me more than one line.

The geolocv2.sh script queries geoiptool.com with netcat, building the HTTP headers by hand:

#!/bin/bash
BIN="/bin/nc"; BIN1="/usr/bin/awk"; BIN2="/usr/bin/cut";
HOST="www.geoiptool.com"; PORT="80"; UA="GeoIP(0.02)";

HEAD="GET /fr/?IP=$1";
H="Host:$HOST\r\nUser-Agent:$UA\r\nAccept:text/html\r\nAccept-Language:fr,fr-fr\r\nAccept-Charset:ISO-8859-1,utf-8\r\nConnection:close\r\n\r\n";

affiche(){
     page=$(echo -ne "$HEAD${H}" | $BIN $HOST $PORT);
     echo "FQDN : $(echo $page | $BIN1 'BEGIN {FS = "arial_bold"}{print $3}' | $BIN2 -d '>' -f2 | $BIN2 -d '<' -f1)";
     echo "IPv4 : $(echo $page | $BIN1 'BEGIN {FS = "arial_bold"}{print $4}' | $BIN2 -d '>' -f2 | $BIN2 -d '<' -f1)";
     echo "Country : $(echo $page | $BIN1 'BEGIN {FS = "arial_bold"}{print $5}' | $BIN2 -d '>' -f2 | $BIN2 -d '<' -f1)";
     echo "City : $(echo $page | $BIN1 'BEGIN {FS = "arial_bold"}{print $8}' | $BIN2 -d '>' -f2 | $BIN2 -d '<' -f1)";
}

affiche

The technical point: netcat opens the connection, sends the HTTP headers by hand (the GET /fr/?IP=$1 with the headers built as an array) and reads the HTML response from geoiptool.com. The parsing is done in awk with an FS on arial_bold (the site’s CSS class name) and cut commands on the angle brackets. It is HTML parsing by brute force and it works because the site has simple stable HTML.

The output:

FQDN : vhost.crawl.co.jp
IPv4 : 59.106.153.9
Country : JP (JPN)
City : Osaka

The revenge of bash (v3, October 2009)

The third iteration drops netcat: bash has a built-in /dev/tcp since bash 2.04 and this pseudo-device opens a TCP connection without installing anything. No more netcat, no more mandatory external awk (but awk stays more readable than pure bash for parsing) and the script fits in a heredoc typed in one block:

#!/bin/bash
if [[ -z $1 ]];then
  echo "Usage : geoloc <ip/host>"
else
  BIN1="/usr/bin/awk"; BIN2="/usr/bin/cut";
  page=$(exec 3<>/dev/tcp/www.geoiptool.com/80;
    echo -e "GET /?IP=$1 HTTP/1.1\r\nHost:www.geoiptool.com\r\n\r\n" >&3;
    cat <&3);
  echo "FQDN : $(echo $page|$BIN1 'BEGIN {FS = "arial_bold"}{print $3}'|$BIN2 -d '>' -f2|$BIN2 -d '<' -f1)";
  echo "IPv4 : $(echo $page|$BIN1 'BEGIN {FS = "arial_bold"}{print $4}'|$BIN2 -d '>' -f2|$BIN2 -d '<' -f1)";
  echo "COUNTRY : $(echo $page|$BIN1 'BEGIN {FS = "arial_bold"}{print $5}'|$BIN2 -d '>' -f2|$BIN2 -d '<' -f1)";
  echo "REGION : $(echo $page|$BIN1 'BEGIN {FS = "arial_bold"}{print $7}'|$BIN2 -d '>' -f3|$BIN2 -d '<' -f1)";
  echo "CITY : $(echo $page|$BIN1 'BEGIN {FS = "arial_bold"}{print $8}'|$BIN2 -d '>' -f2|$BIN2 -d '<' -f1)";
  unset page
fi

The /dev/tcp/host/port is not a file: it is a bash builtin that opens a TCP connection when redirected. The exec 3<> opens fd 3 read-write, the echo >&3 sends the HTTP request, the cat <&3 reads the response. All in a subshell ($(...)), without an external subprocess for transport: bash does the TCP natively.

The output, with the real period test:

$ ./geolocv3.sh www.toto.com
FQDN : vhost.crawl.co.jp
IPv4 : 59.106.153.9
COUNTRY : JP (JPN)
REGION : Osaka
CITY : Osaka

What the trilogy tells

V1 (lost) used netcat and disk access. V2 keeps netcat but drops disk access (the friend’s one-liner vs my ten). V3 drops netcat and replaces transport with the bash builtin. Each iteration removed a dependency: first the temp file, then the netcat.

The /dev/tcp is the kind of builtin nobody uses: it is documented in man bash under REDIRECTION, it is disabled in some restricted production shells (the set -o restricted blocks it) and it does exactly what a netcat does without installing anything. That is the v3 lesson: before installing a tool, check that the shell does not already know how to do it.

A confession to close. The version that actually survived in my .bash_functions was neither of these two, it was the dumbest: a wget -qO- http://www.geoiptool.com/?IP=$1 | html2text | grep 'Host Name\|IP Address\|City\|Country\|code\|itude', five pipes and zero finesse. The whole trilogy of hand-parsing HTML was just a pretext to play with netcat and /dev/tcp and parsing by CSS class broke at the site’s first redesign. Today the question no longer even arises: curl ipinfo.io/8.8.8.8 returns clean JSON and that is all that remains of those ten lines.