An Ang Mo Geek in Singapore

Creating a Serverless GeoIP API

Why build a GeoIP API when you can buy it directly from Maxmind or use their GeoIP Databases?

Using the downloadable database works well when used on an instance. When you need it inside Lambda or Glue, it becomes difficult to keep it up-to-date.

It becomes easier to use an external HTTP-API. Hosting it yourself reduces the response time by having it nearer to your apps and you can extend your service by combining multiples sources.

Make it serverless

To avoid maintenance and to not have to pay for idle time, you want to have it serverless. Storage is the one part that still costs something when idle, so we need something cheap. Lambda, S3, S3-Select and API Gateway are the services used for this.

GeoAPI-Serverless

GeoAPI-Serverless: Lambda, S3-Select, API Gateway, Cloudfront

Storing the data

We use Lambda to download the CSV source data when there is a new update. This job is run once a month when Maxmind’s data is updated.

This file comes as a ZIP and contains a range file and multiple GeoNames files. We keep only the range file and one language that we store back to S3.

Another Lambda is triggered once a new source file is stored, this Lambda will combine the original range file and the GeoNames files and create the source data.

Queryable range file

We need to partition the data to reduce the query size. We use the the first block of digits of the IP.

To efficiently query a range, we need to convert the IP to query to a number and find to which range it belongs.

var IpParts = IpAddress.split('.')
var IpLong = parseInt(IpParts[0])*256^3 + parseInt(IpParts[1]*256^2) + parseInt(IpParts[2])*256 + parseInt(IpParts[3])

To simplify, we will make use of the IP package.

The source data comes with ranges as CIDR notation (a.b.c.d/e), we need to convert it to start and end:

const ip = require('ip')

let network = '157.23.200.0/24' // Read from range file
let subnet = ip.cidrSubnet(network)
let network_head = parseInt(network.split('.')[0]) // 157
let rangeStart = subnet.networkAddress // 157.23.200.0
let rangeEnd = subnet.broadcastAddress // 157.23.200.255
let network_range_start = ip.toLong(rangeStart) // 2635581440
let network_range_end = ip.toLong(rangeEnd) // 2635581695
...

After combining this data with the GeoNames values and converting to JSON, we have this line:

{
  "network_head": 157,
  "network_range_start": 2635581440,
  "network_range_end": 2635581695,"%s.gz" %
  "updated_at":1525279354,
  "valid_until":1525521600,
  "data": {
    "continent": {
      "code": "EU",
      "name": "Europe"
    },
    "country": {
      "code": "FR",
      "name": "France",
      "is_eu": true
    },
    "city": {
      "name": "Paris",
      "metro_code": "",
      "postal_code": "75001"
    },
    "subdivision_1": {
      "iso_code": "IDF",
      "name": "Île-de-France"
    },
    "subdivision_2": {
      "iso_code": "75",
      "name": "Paris"
    },
    "location": {
      "geo_point": {
        "type": "Point",
        "coordinates": [
          2.3292,
          48.8628
        ]
      },
      "accuracy_radius": 20
    },
    "time_zone": "Europe/Paris"
  },
}

This line is appended to the partition db/157.

The partitions are compressed and stored in S3: s3://mybucket/db/GeoLite2-City/157.gz

Querying the data

To query the data, we use a Lambda function invoked through API Gateway with the IP address to query and the database to use as arguments:

curl https://ipdb.example.com/157.23.200.32/city
ip_request = event['pathParameters']['ip']
db_type = event['pathParameters']['dbType']
if db_type == 'city':
    db_edition = 'GeoLite2-City'

We retrieve the IP’s “network head” and convert the IP to a number:

import ipaddress
network_head = ip_request.split('.')[0]
ip_long = int(ipaddress.ip_address(ip_request))

We use S3-Select to fetch the exact data line we need:

import os
import boto3
key = "%s.gz" % os.path.join('db', db_edition, network_head)

expression = "select * from s3object s where s.network_range_start<=%s \
and s.network_range_end>=%s" % (ip_long, ip_long)

query_result = S3.select_object_content(
    Bucket='mybucket',
    Key=key,
    ExpressionType='SQL',
    Expression=expression,
    InputSerialization={'JSON': {'Type': 'LINES'}, 'CompressionType': 'GZIP'},
    OutputSerialization={'JSON': {}},
)
records = json.dumps({
    "data": {}
})
for s3event in query_result['Payload']:
    if 'Records' in s3event:
        records = s3event['Records']['Payload'].decode('utf-8')
        break

return json.loads(records)

The code

We used serverless.com to orchestrate the pieces. You can download the code from Github: https://github.com/Spuul/geoip-serverless

Protect the API

Since you are only allowed to use the Maxmind data for your own infrastructure, you need to protect the access. For that we use the x-api-key header provided by Api Gateway.

To reduce the load on the API and the amount of S3 queries, we cache the queries. To have a better control over the caching mechanism (custom cache-control headers), we won’t use API Gateway’s cache, but use our own Cloudfront distribution.

The result

First call is a bit slow, Lambda needs to spawn and the query to S3 needs to be made.

$ cat curl-timing.txt
time_namelookup:  %{time_namelookup}\n
   time_connect:  %{time_connect}\n
time_appconnect:  %{time_appconnect}\n
time_pretransfer:  %{time_pretransfer}\n
  time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
                ----------\n
     time_total:  %{time_total}\n

$ curl -w "@curl-timing.txt" -H "x-api-key: XXXXXXXX" -s -v "https://myipservice.example.com/157.23.200.32/city"
< X-Cache: Miss from cloudfront
{"continent": {"code": "EU", "name": "Europe"}, "country": {"code": "FR", "name": "France", "is_eu": true}, "city": {"name": "Paris", "metro_code": "", "postal_code": "75001"}, "subdivision_1": {"iso_code": "IDF", "name": "\u00cele-de-France"}, "subdivision_2": {"iso_code": "75", "name": "Paris"}, "location": {"geo_point": {"type": "Point", "coordinates": [2.3292, 48.8628]}, "accuracy_radius": 20}, "time_zone": "Europe/Paris"}
time_namelookup:  0.004130
   time_connect:  0.006398
time_appconnect:  0.019093
time_pretransfer:  0.019134
  time_redirect:  0.000000
time_starttransfer:  1.236681
                ----------
     time_total:  1.236967

Second call for the same IP will leverage Cloudfront cache:

$ curl -w "@curl-timing.txt" -H "x-api-key: XXXXXXXX" -s -v "https://myipservice.example.com/157.23.200.32/city"
< X-Cache: Hit from cloudfront
{"continent": {"code": "EU", "name": "Europe"}, "country": {"code": "FR", "name": "France", "is_eu": true}, "city": {"name": "Paris", "metro_code": "", "postal_code": "75001"}, "subdivision_1": {"iso_code": "IDF", "name": "\u00cele-de-France"}, "subdivision_2": {"iso_code": "75", "name": "Paris"}, "location": {"geo_point": {"type": "Point", "coordinates": [2.3292, 48.8628]}, "accuracy_radius": 20}, "time_zone": "Europe/Paris"}
time_namelookup:  0.012311
   time_connect:  0.014386
time_appconnect:  0.025544
time_pretransfer:  0.025580
  time_redirect:  0.000000
time_starttransfer:  0.027953
                ----------
     time_total:  0.028005

A call for a new IP will not use Cloudfront but will leverage the already spawn Lambda:

$ curl -w "@curl-timing.txt" -H "x-api-key: XXXXXXXX" -s -v "https://myipservice.example.com/157.23.200.33/city"
< X-Cache: Miss from cloudfront
{"continent": {"code": "EU", "name": "Europe"}, "country": {"code": "FR", "name": "France", "is_eu": true}, "city": {"name": "Paris", "metro_code": "", "postal_code": "75001"}, "subdivision_1": {"iso_code": "IDF", "name": "\u00cele-de-France"}, "subdivision_2": {"iso_code": "75", "name": "Paris"}, "location": {"geo_point": {"type": "Point", "coordinates": [2.3292, 48.8628]}, "accuracy_radius": 20}, "time_zone": "Europe/Paris"}
time_namelookup:  0.012263
   time_connect:  0.014462
time_appconnect:  0.026965
time_pretransfer:  0.027003
  time_redirect:  0.000000
time_starttransfer:  0.538435
                ----------
     time_total:  0.538682