Thomas Jepp

APU2 and Debian Buster as a home router

Posted on

This (terse!) guide shows:

  • how to get a working serial console with Debian on the APU2
  • setting up basic network configuration (including full 1500 MTU on PPPoE)
  • setting up firewalling using nftables
  • setting up DHCP, DNS, NTP
  • setting up QoS using CAKE

This guide doesn't cover WiFi in any form - for WiFi I recommend a separate AP that you can position for best coverage.

I don't use this set up any more as the APU2 isn't fast enough for gigabit connections with PPPoE - I moved to a Mikrotik RB5009, also purchased from linitx.com.

The router

The PC Engines APU2 is a single-board x86 computer with a reasonable 64-bit quad core CPU, up to 4GB of RAM, storage via mSATA or SD card, and up to 4 Intel NICs.

It draws <10W in normal operation - which allows it to be passively cooled, and it is fairly unusual in x86 platforms in that it doesn't have any display out - only a serial terminal.

I have the apu2c4 model - which has 4GB of ECC DDR3 and 3x Intel i210AT NICs. I've fitted a 250G mSATA SSD for persistent storage.

I got mine from linitx.com - a good UK distributor of PC Engines hardware.

On a 330/30 service, the APU2 works well. On faster services that require PPPoE it can't keep up - even with the newest firmware which allows frequency boosts to ~1.4GHz. About 400mbit was attainable for me out of the box, with 700mbit after some sysctl tweaks.

The network setup

I use Andrews & Arnold as my internet provider. They're a small ISP, but they provide a high quality service with static IPv6 and easy access to static blocks of IPv4. This guide should be reasonably useful for most UK internet connections, however.

I have a FTTP service using PPPoE that's presented as a gigabit ethernet port - but I've also used this router with a FTTC service using a separate FTTC modem with exactly the same setup. If your modem doesn't support RFC4638 (mini jumbo frames for PPPoE), then you'll need to adjust the MTU appropriately. A good modem that does is the Openreach branded Huawei HG612.

My network is connected as follows:

  • enp1s0 is connected to the FTTP ONT or FTTC modem
  • enp2s0 is connected to my main LAN - in this example, this is 192.168.1.0/24 and 2001:8b0:db8::/64
  • enp3s0 is used for a live IP range - in this example, I use 192.0.2.0/29 and 2001:8b0:db8:1::/64

Installing Debian

  1. Grab the netinstall image from https://www.debian.org/CD/netinst/. Make sure it's the amd64 image. Once downloaded, write it to a flash drive. I use dd for this:
    # replace /dev/sdX with the device node for your USB drive
    dd if=debian-10.6.0-amd64-netinst.iso bs=1M of=/dev/sdX
    
  2. Connect up your serial console, and start your terminal emulator using 115200 baud, 8 data bits, no parity, one start bit and one stop bit.
  3. Apply power to the APU2, and use the serial console to boot from USB - press F10 to enter the boot menu, then pick your flash drive from the list.
  4. When booting from the flash drive, you'll need to provide some extra boot parameters. Add console=ttyS0,115200u8 to the command line, and press enter to boot.
  5. Follow the installer through as normal.

Initial post-install configuration

  1. I set /etc/apt/sources.list to the following:

    deb http://deb.debian.org/debian/ buster main
    deb-src http://deb.debian.org/debian/ buster main
    
    deb http://deb.debian.org/debian-security/ buster/updates main
    deb-src http://deb.debian.org/debian-security/ buster/updates main
    
    deb http://deb.debian.org/debian/ buster-updates main
    deb-src http://deb.debian.org/debian/ buster-updates main
    
    deb http://deb.debian.org/debian/ buster-backports main
    deb-src http://deb.debian.org/debian/ buster-backports main
    

    And then run:

    apt update
    
  2. Install packages we'll need for this to work:

    # a kernel from buster-backports so we have up to date cake, nftables, and wireguard modules available
    apt install -t buster-backports linux-image-amd64
    
    # flashrom is used for firmware updates for the apu2
    apt install flashrom
    
    # a variety of useful tools for this setup
    apt install chrony curl ethtool irqbalance isc-dhcp-server jq mtr-tiny nftables ppp pppoe radvd unbound
    
    # if you intend to use bridges or vlans (optional)
    apt install bridge-utils vlan
    

    Reboot into the new kernel now.

Networking setup

  1. Set up basic networking - set /etc/network/interfaces to:

    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).
    
    source /etc/network/interfaces.d/*
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # WAN
    auto enp1s0
    iface enp1s0 inet manual
        description wan
        # this allows us to do full 1500 byte frames inside the PPPoE tunnel
        mtu 1508
        # this stops NIC hangs under some 5.x kernels
        up ethtool -K enp1s0 tx off rx off gso off gro off tso off
    
    auto pppoe-aaisp
    iface pppoe-aaisp inet ppp
        provider aaisp
        # ensure the underlying ethernet interface is up and give it some time for everything to settle
        pre-up ifup enp1s0
        pre-up sleep 5
    
    # LAN
    auto enp2s0
    iface enp2s0 inet static
        description lan
        address 192.168.1.1/24
        mtu 1500
        # this does some basic fair queuing
        up tc qdisc add dev enp2s0 root sfq perturb 10
        # this stops NIC hangs under some 5.x kernels
        up ethtool -K enp2s0 tx off rx off gso off gro off tso off
    
    iface enp2s0 inet6 static
        address 2001:8b0:db8::1/64
        mtu 1500
    
    # LIVE
    auto enp3s0
    iface enp3s0 inet static
        description live
        address 192.0.2.1/29
        mtu 1500
        # this does some basic fair queuing
        up tc qdisc add dev enp3s0 root sfq perturb 10
        # this stops NIC hangs under some 5.x kernels
        up ethtool -K enp3s0 tx off rx off gso off gro off tso off
    
    iface enp3s0 inet6 static
        description live
        address 2001:8b0:db8:1::1/64
        mtu 1500
    
  2. Set /etc/ppp/peers/aaisp to:

    user example@a.1
    plugin rp-pppoe.so enp1s0
    noipdefault
    defaultroute
    replacedefaultroute
    hide-password
    lcp-echo-interval 1
    lcp-echo-failure 10
    noauth
    persist
    maxfail 0
    mtu 1500
    noaccomp
    default-asyncmap
    +ipv6
    ipv6cp-accept-local
    ifname pppoe-aaisp
    
    logfile /var/log/ppp-aaisp.log
    

    This sets up PPP for connecting to A&A - it sets up a connection that will:

    • set the IPv4 default route
    • does a LCP ping every second and reconnects after 10 failures
    • uses a MTU of 1500 bytes
    • enables IPv6
    • renames the PPP interface to pppoe-aaisp
  3. Set /etc/ppp/chap-secrets to:

    # Secrets for authentication using CHAP
    # client	server	secret			IP addresses
    example@a.1	*	ExampleSecret
    
  4. Create /etc/ppp/ipv6-ip.d/0000-defaultroute with these contents:

    #!/bin/bash
    
    # add a default v6 route
    ip -6 route add default dev $1
    

    And make it executable:

    chmod +x /etc/ppp/ipv6-ip.d/0000-defaultroute
    
  5. Create /etc/sysctl.d/router.conf with these contents:

    # Enable reverse-path filter (source address spoofing protection)
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.all.rp_filter = 1
    
    # Enable syncookies
    net.ipv4.tcp_syncookies = 1
    
    # Enable IPv4 forwarding
    net.ipv4.ip_forward = 1
    
    # Enable IPv6 forwarding
    net.ipv6.conf.all.forwarding = 1
    
    # Disable ICMP redirects
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv6.conf.all.accept_redirects = 0
    
  6. Set /etc/resolv.conf to:

    nameserver 127.0.0.1
    
  7. Set /etc/nftables.conf to:

    #!/usr/sbin/nft -f
    
    flush ruleset
    
    table inet firewall {
        chain input {
            type filter hook input priority 0; policy drop;
    
            # allow localhost
            iif "lo" accept
    
            # allow from wan interface
            iifname "enp1s0" accept
    
            # allow return traffic
            ct state established,related accept
    
            # allow ICMP
            meta l4proto { icmp, ipv6-icmp } accept
    
            # allow DHCP from LAN
            iifname { "enp2s0" } udp dport 67 accept
    
            # allow DNS from LAN and live
            iifname { "enp2s0", "enp3s0" } tcp dport 53 accept
            iifname { "enp2s0", "enp3s0" } udp dport 53 accept
    
            # allow NTP
            iifname { "enp2s0", "enp3s0" } udp dport 123 accept
    
            reject with icmpx type admin-prohibited
        }
    
        chain forward-internet-to-lan {
            # allow ICMP
            meta l4proto { icmp, ipv6-icmp } accept
    
            # allow return traffic
            ct state established,related accept
    
            reject with icmpx type admin-prohibited
        }
    
        chain forward-internet-to-live {
            # allow ICMP
            meta l4proto { icmp, ipv6-icmp } accept
    
            # allow all traffic
            accept
        }
    
        chain forward {
            type filter hook forward priority 0; policy drop;
    
            # allow ICMP
            meta l4proto { icmp, ipv6-icmp } accept
    
            iifname "pppoe-aaisp" oifname "enp2s0" jump forward-internet-to-lan
            iifname "pppoe-aaisp" oifname "enp3s0" jump forward-internet-to-live
    
            # allow from lan to internet, live
            iifname "enp2s0" oifname { "pppoe-aaisp", "enp3s0" } accept
    
            # allow return traffic
            ct state established,related accept
    
            reject with icmpx type admin-prohibited
        }
    }
    
    table ip nat {
        chain postrouting {
            type nat hook postrouting priority 0;
    
            ip saddr 192.168.0.0/16 oifname "pppoe-aaisp" masquerade fully-random
        }
    }
    
  8. Enable nftables:

    systemctl enable nftables
    

The easiest way to apply this all is to reboot - once you've rebooted you should have basic routing (but no DHCP)!

Setting up network services

DNS

  1. Create /etc/unbound/unbound.conf.d/local.conf with these contents:

    server:
        interface: 0.0.0.0
        interface: ::
        interface-automatic: yes
        prefer-ip6: yes
        access-control: 192.168.0.0/16 allow
        access-control: 192.0.2.0/29 allow
        access-control: 2001:8b0:db8::/48 allow
    
  2. Restart unbound:

    systemctl restart unbound
    

NTP

  1. Edit /etc/chrony/chrony.conf and add this to the end:
    allow 192.168.0.0/16
    allow 192.0.2.0/29
    allow 2001:8b0:db8::/48
    
  2. Restart chrony:
    systemctl restart chrony
    

SLAAC for IPv6

  1. Set /etc/radvd.conf to:
    interface enp2s0
    {
            AdvSendAdvert on;
            MinRtrAdvInterval 3;
            MinRtrAdvInterval 10;
    
            AdvManagedFlag off;
            AdvOtherConfigFlag off;
    
            prefix 2001:8b0:db8::/64
            {
                    AdvOnLink on;
                    AdvAutonomous on;
            };
    
            RDNSS 2001:8b0:db8::1 {
            };
    
            DNSSL your.domain.here {
            };
    };
    
    This enables distributing DNS server and DNS search list via SLAAC, and allows devices to configure their own IP addresses from the advertised prefixes.
  2. Restart radvd:
    systemctl restart radvd
    

DHCP

  1. Edit /etc/dhcp/dhcpd.conf and add this to the end:
    # DNS settings
    option domain-name "your.domain.here";
    option domain-name-servers 192.168.1.1;
    
    # The length of the DHCP lease - 10 minutes
    default-lease-time 600;
    max-lease-time 7200;
    
    # This DHCP server is authoritative for the local network
    authoritative;
    
    subnet 192.168.1.0 netmask 255.255.255.0 {
        option routers 192.168.1.1;
        option domain-name-servers 192.168.1.1;
        option ntp-servers 192.168.1.1;
        range 192.168.1.100 192.168.1.199;
    }
    
  2. Restart the DHCP server:
    systemctl restart isc-dhcp-server
    

QoS

Setting up CAKE on both ingress and egress provides us with a responsive internet connection that provides fair bandwidth allocation where possible - not allowing any one device to abuse the connection and maintaining low-latency conditions for interactive services like SSH.

In addition, A&A provide an API that can be used to get your current line rate. This is very useful for setting up CAKE - especially if you use FTTC!

We use an ifb interface to bring ingress traffic into so we can apply CAKE to it.

  1. Create /usr/local/sbin/qos-setup with the contents:

    #!/bin/bash
    
    # Your A&A control panel credentials
    LOGIN="example@a"
    PASSWORD="ExampleControlPanelPassword"
    
    # Your line ID - this ID is the one at the end of the URL when you view circuit details.
    # for example: https://control.aa.net.uk/editline.cgi?ID=12345 has ID 12345
    SERVICE="12345"
    
    # 8 bytes for pppoe + 4 bytes for BT VLAN. This seems to be correct for FTTP, experimentally.
    # For FTTC you'll want to tweak this!
    OVERHEAD="12"
    
    # load ifb
    modprobe ifb
    
    # clear existing tc qdiscs
    tc qdisc del dev pppoe-aaisp root
    tc qdisc del dev pppoe-aaisp handle ffff: ingress
    tc qdisc del dev ifb0 root
    tc qdisc del dev ifb0 ingress
    ip link set dev ifb0 down
    
    echo "fetching service info"
    INFO="$(curl -s "https://chaos2.aa.net.uk/broadband/info?control_login=${LOGIN}&control_password=${PASSWORD}&service=${SERVICE}" | jq '.info[0]')"
    # these rates are from A&A's PoV, so tx = them transmitting to us, rx = them receiving from us
    INGRESS_RATE=$(echo "$INFO" | jq .tx_rate_adjusted -r)
    EGRESS_RATE=$(echo "$INFO" | jq .rx_rate -r)
    echo "ingress $INGRESS_RATE egress $EGRESS_RATE"
    
    # bring up ifb0
    ip link set dev ifb0 up
    
    # create ingress filter
    echo "create ingress filter"
    tc qdisc add dev pppoe-aaisp handle ffff: ingress
    
    # forward all ingress traffic to ifb0
    echo "forward all ingress traffic to ifb0"
    tc filter add dev pppoe-aaisp parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0
    
    # set up ingress cake
    echo "ingress cake"
    tc qdisc add dev ifb0 root cake bandwidth $INGRESS_RATE nat overhead ${OVERHEAD} ingress dual-dsthost diffserv4 regional
    
    # egress
    echo "egress cake"
    tc qdisc add dev pppoe-aaisp root cake bandwidth $EGRESS_RATE nat overhead ${OVERHEAD} dual-srchost diffserv4 regional
    

    And make it executable:

    chmod +x /usr/local/sbin/qos-setup
    
  2. Create /etc/ppp/ipv6-ip.d/0001-qos with these contents:

    #!/bin/bash
    
    # initialise QoS
    /usr/local/sbin/qos-setup
    

    And make it executable:

    chmod +x /etc/ppp/ipv6-ip.d/0001-qos
    
  3. Execute it once to test:

    /usr/local/sbin/qos-setup
    

Conclusion

At this point, you should have a nicely working Linux-based router, with working IPv4 and IPv6, DHCP for IPv4, SLAAC for IPv6, DNS, NTP, and QoS!

You might like to add:

  • SmokePing, for connection monitoring
  • WireGuard, for a simple, cross-platform VPN
  • Pi-hole, for blocking unwanted content at the DNS level