In this blog post, I will explain how to compile the Open Overlay Router (OOR) software for MIPS and how to set it up on an Ubiquiti EdgeRouter Lite to participate in the Locator/ID Separation Protocol (LISP) beta network.
For the purpose of this blog post, I will assume you already have an understanding of LISP and are familiar with the administration of Ubiquiti EdgeRouters.
There has been previous interest in LISP on Ubiquiti hardware and someone’s compiled the packages before. Unfortunately, it’s not clear whether this was successful, so I thought to find out whether it would be possible to participate in the LISP beta network using an EdgeRouter.
Disclaimer:
The guide below documents how I tested LISP. If you choose to follow it, you do so at your own risk, and you should ensure you read the last section on future work.
Compiling OOR for EdgeRouter
While it’s possible to install Debian packages on EdgeRouters, OOR is currently not available as a Debian package, so we need to build it from source. Other community members have found that using a VM running in a QEMU MIPS emulator is an efficient way of achieving this, so I followed their example. Alternatively, you could probably install the build tools on your EdgeRouter. I was a little confused by not finding Debian builds for mips64 (not mips64el) wheezy builds, but aurel32’s qemu mips image README contained instructions for starting a 64bit Debian MIPS machine, so I mostly followed those:
# wget required files from https://people.debian.org/~aurel32/qemu/mips/
qemu-system-mips64 -M malta -kernel vmlinux-3.2.0-4-5kc-malta -hda debian_wheezy_mips_standard.qcow2 -m 2047 -append "root=/dev/sda1 console=ttyS0 mem=256m@0x0 mem=1791m@0x90000000" -nographic
Using -nographic forwards the system console to STDIN by default (see this link).
Once it’s booted, ensure apt can find and download the required packages, and then install them:
apt-get install debian-keyring debian-archive-keyring apt-key update apt-get update # NB: there are no more security updates for wheezy on mips (LTS excludes mips) apt-get dist-upgrade # libzmq3 is in backports echo "deb http://ftp.debian.org/debian wheezy-backports main" > /etc/apt/sources.list apt-get update apt-get install build-essential git-core libconfuse-dev gengetopt libcap2-bin libzmq3-dev libxml2-dev
Now that all the prerequisites were fulfilled, we can compile the OOR code. Clone the OOR code and follow the instructions from the OOR’s readme file, except you don’t need to run make install:
git clone https://github.com/OpenOverlayRouter/oor.git cd oor make
Now we need to transfer the code out of the VM:
cd .. tar czvf oor.tar.gz oor scp oor.tar.gz you@somemachine
I would recommend grabbing a copy of the tarball before transferring it to the EdgeRouter, as it will disappear from your EdgeRouter with the next system upgrade (see this page for details).
Setting up OOR on EdgeRouter
First, we need to install the required libraries. Note that we don’t need the dev versions or the build tools, and I found that some packages were already installed. We still need to add both the main wheezy repositories and the wheezy-backports for libzmq3. So, based on the adding Debian packages instructions, I configured the following via the router’s Command Line Interface (CLI):
configure package repository wheezy components "main contrib non-free" package repository wheezy distribution wheezy package repository wheezy url http://ftp.<countrycode>.debian.org/debian package repository wheezy-backports components main package repository wheezy-backports distribution wheezy-backports package repository wheezy-backports url http://ftp.<countrycode>.debian.org/debian
Now, we can run ‘sudo su’ to log in as root and drop into bash, where we can then install the libraries:
apt-get install libconfuse0 libzmq3
The rest of the OOR setup is pretty similar to what you might do on any other Linux system, except perhaps that I don’t install OOR system-wide.
Extract the tarball:
tar zxfv oor.tar.gz cd oor/oor
Create an OOR config file based on the example:
cp oor.conf.example /config
and edit it to suit your needs (vi is pre-installed). For my use-case, running an xTR on the LISP beta network, I configured the following:
# leave general config; turn up debug level if you need to troubleshoot operating-mode = xTR control-iface = eth0 encapsulation = LISP rloc-probing { (leave defaults) } map-resolver { (as required) } nat_traversal_support = off map-server { address = (as required) key-type = (as required) key = (as required) proxy-reply = off } proxy-etr { (as required) } database-mapping { eid-prefix = 153.16.X.X/28 iid = 0 # rloc-iface would probably work as well rloc-address { address = (your external IP) priority = 1 weight = 100 } } proxy-itrs = { # uncomment addresses } # Other sections can be commented.
If you’ve not already done so, now is a good time to configure your EID prefix as a local network, perhaps adding a DHCP server and firewall rules/exemptions as appropriate. For example, configure the following:
# firewall rules for WAN side set firewall name WAN_LOCAL rule <N> action accept set firewall name WAN_LOCAL rule <N> description 'allow LISP' set firewall name WAN_LOCAL rule <N> destination group address-group ADDRv4_eth0 set firewall name WAN_LOCAL rule <N> destination port 4341-4342 set firewall name WAN_LOCAL rule <N> log disable set firewall name WAN_LOCAL rule <N> protocol udp # EID interface (VLAN subinterface shown here) set interfaces ethernet eth<X> vif <VLAN-ID> address 153.16.X.Y/28 set interfaces ethernet eth<X> vif <VLAN-ID> description lisp set interfaces ethernet eth<X> vif <VLAN-ID> firewall in name vlan<ID>-in set interfaces ethernet eth<X> vif <VLAN-ID> firewall local name vlan<ID>-local set interfaces ethernet eth<X> vif <VLAN-ID> firewall out name vlan<ID>-out # set a very safe MTU just to test set interfaces ethernet eth<X> vif <VLAN-ID> mtu 1400 # Open up firewall rules for testing set firewall name vlan<ID>-isolate default-action accept set firewall name vlan<ID>-isolate description '' set firewall name vlan<ID>-isolate-in default-action accept set firewall name vlan<ID>-isolate-in description '' set firewall name vlan<ID>-isolate-out default-action accept set firewall name vlan<ID>-isolate-out description ''
Then, run the OOR:
./oor -f /config/oor.conf
Testing OOR on EdgeRouter
I’ve found that sending pings from the EdgeRouter itself over LISP doesn’t seem to work, even when I specify the source interface by running ‘sh’ and then execute ‘ping -I ethX <dest IP>’.
So, instead, send some traffic from another device on the EID-prefix subnet through the router. Before sending any bigger packets, ensure the MTU size used by the client is appropriate.
After some minor troubleshooting, both LISP-to-LISP and LISP-to-non-LISP traffic was working for me!
The main issue I ran across was OOR occasionally asserting when deleting a map-cache entry. It’s possible to work around this by running OOR using something like this:
while true; do ./oor -f /config/oor.conf; rm -f /var/run/oor.pid; sleep 1; echo "Restarting OOR"; done
Future work
The above focuses on the minimum required to test LISP on an Ubiquiti EdgeRouter. As such, there are several topics that require further work before using this in production:
- EdgeOS is based on Debian wheezy, and the EdgeRouter is a MIPS platform. Sadly, this means that there are no official security update binaries available, as MIPS platforms are excluded from LTS. It would be best to compile OOR for MIPS on a system with the latest security patches
- Ensure MTU sizes are communicated appropriately to clients and/or that Path MTU Discovery (PMTUD) works correctly.
- Find a solution to the occasional crash on map-cache entry deletion – I will let the OOR contributors know about this.
- Run OOR as a service rather than manually running the binary.
Hello,
would this also work for the Unifi USG router running EdgeOS ?
That’s a good question, I don’t know. I imagine they’re probably also Debian-based devices, but I’m not sure if it will be as “easy” to install packages. Looks like someone’s managed to at least install nano, which is a start…