LISP on Ubiquiti EdgeRouter Lite

In this blog post, I will explain how to compile the Open Overlay Router (OOR) software for MIPS and how to set it up on an Ubiquiti EdgeRouter Lite to participate in the Locator/ID Separation Protocol (LISP) beta network.

For the purpose of this blog post, I will assume you already have an understanding of LISP and are familiar with the administration of Ubiquiti EdgeRouters.

There has been previous interest in LISP on Ubiquiti hardware and someone’s compiled the packages before. Unfortunately, it’s not clear whether this was successful, so I thought to find out whether it would be possible to participate in the LISP beta network using an EdgeRouter.

Disclaimer:

The guide below documents how I tested LISP. If you choose to follow it, you do so at your own risk, and you should ensure you read the last section on future work.

Compiling OOR for EdgeRouter

While it’s possible to install Debian packages on EdgeRouters, OOR is currently not available as a Debian package, so we need to build it from source. Other community members have found that using a VM running in a QEMU MIPS emulator is an efficient way of achieving this, so I followed their example. Alternatively, you could probably install the build tools on your EdgeRouter. I was a little confused by not finding Debian builds for mips64 (not mips64el) wheezy builds, but aurel32’s qemu mips image README contained instructions for starting a 64bit Debian MIPS machine, so I mostly followed those:

# wget required files from https://people.debian.org/~aurel32/qemu/mips/
qemu-system-mips64 -M malta -kernel vmlinux-3.2.0-4-5kc-malta -hda debian_wheezy_mips_standard.qcow2 -m 2047 -append "root=/dev/sda1 console=ttyS0 mem=256m@0x0 mem=1791m@0x90000000" -nographic

Using -nographic forwards the system console to STDIN by default (see this link).

Once it’s booted, ensure apt can find and download the required packages, and then install them:

apt-get install debian-keyring debian-archive-keyring 
apt-key update 
apt-get update 
# NB: there are no more security updates for wheezy on mips (LTS excludes mips) 
apt-get dist-upgrade
# libzmq3 is in backports
echo "deb http://ftp.debian.org/debian wheezy-backports main" > /etc/apt/sources.list 
apt-get update
apt-get install build-essential git-core libconfuse-dev gengetopt libcap2-bin libzmq3-dev libxml2-dev

Now that all the prerequisites were fulfilled, we can compile the OOR code. Clone the OOR code and follow the instructions from the OOR’s readme file, except you don’t need to run make install:

git clone https://github.com/OpenOverlayRouter/oor.git
cd oor
make

Now we need to transfer the code out of the VM:

cd ..
tar czvf oor.tar.gz oor
scp oor.tar.gz you@somemachine

I would recommend grabbing a copy of the tarball before transferring it to the EdgeRouter, as it will disappear from your EdgeRouter with the next system upgrade (see this page for details).

Setting up OOR on EdgeRouter

First, we need to install the required libraries. Note that we don’t need the dev versions or the build tools, and I found that some packages were already installed. We still need to add both the main wheezy repositories and the wheezy-backports for libzmq3. So, based on the adding Debian packages instructions, I configured the following via the router’s Command Line Interface (CLI):

configure
package repository wheezy components "main contrib non-free"
package repository wheezy distribution wheezy
package repository wheezy url http://ftp.<countrycode>.debian.org/debian
package repository wheezy-backports components main
package repository wheezy-backports distribution wheezy-backports
package repository wheezy-backports url http://ftp.<countrycode>.debian.org/debian

Now, we can run ‘sudo su’ to log in as root and drop into bash, where we can then install the libraries:

apt-get install libconfuse0 libzmq3

The rest of the OOR setup is pretty similar to what you might do on any other Linux system, except perhaps that I don’t install OOR system-wide.

Extract the tarball:

tar zxfv oor.tar.gz
cd oor/oor

Create an OOR config file based on the example:

cp oor.conf.example /config

and edit it to suit your needs (vi is pre-installed). For my use-case, running an xTR on the LISP beta network, I configured the following:

# leave general config; turn up debug level if you need to troubleshoot
operating-mode = xTR
control-iface = eth0
encapsulation = LISP
rloc-probing { (leave defaults) }
map-resolver { (as required) }
nat_traversal_support = off
map-server {
  address = (as required)
  key-type = (as required)
  key = (as required)
  proxy-reply = off
}
proxy-etr {
  (as required)
}
database-mapping {
  eid-prefix = 153.16.X.X/28
  iid = 0
  # rloc-iface would probably work as well
  rloc-address {
    address = (your external IP)
    priority = 1
    weight = 100
  }
}
proxy-itrs = {
  # uncomment addresses
}

# Other sections can be commented.

If you’ve not already done so, now is a good time to configure your EID prefix as a local network, perhaps adding a DHCP server and firewall rules/exemptions as appropriate. For example, configure the following:

# firewall rules for WAN side
set firewall name WAN_LOCAL rule <N> action accept
set firewall name WAN_LOCAL rule <N> description 'allow LISP'
set firewall name WAN_LOCAL rule <N> destination group address-group ADDRv4_eth0
set firewall name WAN_LOCAL rule <N> destination port 4341-4342
set firewall name WAN_LOCAL rule <N> log disable
set firewall name WAN_LOCAL rule <N> protocol udp

# EID interface (VLAN subinterface shown here)
set interfaces ethernet eth<X> vif <VLAN-ID> address 153.16.X.Y/28
set interfaces ethernet eth<X> vif <VLAN-ID> description lisp
set interfaces ethernet eth<X> vif <VLAN-ID> firewall in name vlan<ID>-in
set interfaces ethernet eth<X> vif <VLAN-ID> firewall local name vlan<ID>-local
set interfaces ethernet eth<X> vif <VLAN-ID> firewall out name vlan<ID>-out
# set a very safe MTU just to test
set interfaces ethernet eth<X> vif <VLAN-ID> mtu 1400

# Open up firewall rules for testing
set firewall name vlan<ID>-isolate default-action accept
set firewall name vlan<ID>-isolate description ''
set firewall name vlan<ID>-isolate-in default-action accept
set firewall name vlan<ID>-isolate-in description ''
set firewall name vlan<ID>-isolate-out default-action accept
set firewall name vlan<ID>-isolate-out description ''

Then, run the OOR:

./oor -f /config/oor.conf

Testing OOR on EdgeRouter

I’ve found that sending pings from the EdgeRouter itself over LISP doesn’t seem to work, even when I specify the source interface by running ‘sh’ and then execute ‘ping -I ethX <dest IP>’.

So, instead, send some traffic from another device on the EID-prefix subnet through the router. Before sending any bigger packets, ensure the MTU size used by the client is appropriate.

After some minor troubleshooting, both LISP-to-LISP and LISP-to-non-LISP traffic was working for me!

The main issue I ran across was OOR occasionally asserting when deleting a map-cache entry. It’s possible to work around this by running OOR using something like this:

while true; do ./oor -f /config/oor.conf; rm -f /var/run/oor.pid; sleep 1; echo "Restarting OOR"; done

Future work

The above focuses on the minimum required to test LISP on an Ubiquiti EdgeRouter. As such, there are several topics that require further work before using this in production:

• EdgeOS is based on Debian wheezy, and the EdgeRouter is a MIPS platform. Sadly, this means that there are no official security update binaries available, as MIPS platforms are excluded from LTS. It would be best to compile OOR for MIPS on a system with the latest security patches
• Ensure MTU sizes are communicated appropriately to clients and/or that Path MTU Discovery (PMTUD) works correctly.
• Find a solution to the occasional crash on map-cache entry deletion – I will let the OOR contributors know about this.
• Run OOR as a service rather than manually running the binary.