LXD MACVLAN Interfaces

Linux Containers now with MACVLAN interfaces

by Craig Miller

I have been running Linux Containers (LXD) for several years now, and I have found them really useful. The key advantages of Linux Containers are:

Running full Linux OS inside a container, which makes troubleshooting much easier (than Docker)
Cloud-like experience. Cloud computing has taken off, not just because it is someone else's computer, but because it has made spinning up a server easy. Linux Containers provides a similar easy experience in adding a server. Some of the capabilities are: snapshots of containers, transferring containers from one host to another, quick creation and removal of containers.
Secure and scalable. Linux containers run by default, as unprivilaged thereby protecting the host system. And because Linux Containers are light weight, I have been able to scale up webserver containers to over 30 running on a Raspberry Pi 3B+.
Excellent IPv6 Support. Containers have persistent MAC addresses, and therefore will request the same IPv4 address, and form the same SLAAC addresses after every reboot, regardless of container boot order (unlike Docker).

Connecting your Container to the Internet

As an old networking guy, in the past I have taken a network approach to getting my containers connected to the internet. I would configure a linux bridge (the front bridge) on the host, and then connect the Containers and the Host to that bridge. Like this:

Using a Linux Bridge

LXD also supports MACVLAN type of interface with eliminates the complexity of getting your container connected to a LAN, and then the Internet. However, there is a limitation, in that in the default config, the Linux Container can talk to the internet, but it can't talk to the host it is residing on.

Using MACVLAN interface

The MACVLAN technique uses the features of modern network interfaces (NICs) that support virtual interfaces. With virtual interface support, a single NIC can support not only multiple IP addresses, but several MAC (Media Access Control) addresses as well.

[Network Diagram with MACVLAN] Using a Linux MACVLAN

Creating LXD Profile for MACVLAN

LXD Containers use a profile to determine which resources to attach to, such as hard disk or network. The default LXD proflie looks like:

$ lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by: []

In the eth0 section, you can see that by default, the container will attach to the LXD linux bridge it sets up at init time, called lxdbr0. Unfortunately, that is a bridge to no where.

So we'll create another profile that connects to the Host NIC via MACVLAN. First copy the default, and then change a couple of lines, specifically the nictype and the parent. The parent is the name of the host ethernet device. On a Raspberry Pi running Pi OS, it is eth0.

$ lxc profile copy default macvlan
$ lxc profile show macvlan
config: {}
description: Default LXD profile
devices:
  eth0:
    nictype: macvlan
    parent: eth0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: enet
used_by: []

Using the MACVLAN profile

Now that you have a MACVLAN profile, using it is as simple as launching a new container with the -p macvlan option. For example, firing up a Container running Alpine OS:

$ lxc launch -p macvlan images:alpine/3.16 test
Creating test
Starting test  
$

Looking at the container running with the lxc ls command:

$ lxc ls
+---------+---------+------------------------+-----------------------------------------------+-----------+-----------+
|  NAME   |  STATE  |          IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+---------+---------+------------------------+-----------------------------------------------+-----------+-----------+
| test    | RUNNING | 192.168.215.129 (eth0) | fd73:c73c:444:0:216:3eff:fe65:2b99 (eth0)     | CONTAINER | 0         |
|         |         |                        | 2001:db8:8011:fd44:216:3eff:fe65:2b99 (eth0)  |           |           |
+---------+---------+------------------------+-----------------------------------------------+-----------+-----------+

And you can see that the new container already had IPv4 and IPv6 addresses (from my router). Let's try a ping from inside the container.

Checking connectivity from inside the MACVLAN attached Container

We'll step inside the container with the lxc exec command, and ping via IPv6 and IPv4.

$ lxc exec test sh

~ # ping -c 3 he.net
PING he.net (2001:470:0:503::2): 56 data bytes
64 bytes from 2001:470:0:503::2: seq=0 ttl=56 time=34.192 ms
64 bytes from 2001:470:0:503::2: seq=1 ttl=56 time=33.554 ms
64 bytes from 2001:470:0:503::2: seq=2 ttl=56 time=33.959 ms

--- he.net ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 33.554/33.901/34.192 ms

~ # ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=59 time=11.835 ms
64 bytes from 1.1.1.1: seq=1 ttl=59 time=11.933 ms
64 bytes from 1.1.1.1: seq=2 ttl=59 time=11.888 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 11.835/11.885/11.933 ms
~ #

As you can see we can get out to the internet without all the fuss of the old way of setting up a front bridge.

However using MACVLAN, the container can't talk to the LXD Host

As mentioned earlier, a downside of using MACVLAN interface is that the container cannot connect/ping the LXD Host. Most of the time, this isn't an issue, since the container will mostly be running some kind of service, web, audio streaming, etc. But there are occasions where communication between the host and the container is required.

A useful example is running LXD Dashboard. LXD Dashboard, a web front end for managing LXD, runs inside a container, but must talk to the LXD Host.

Two ships in the night

The reason why the container can't talk to the Host, is because of the virtual interface on the Host NIC. It has a special path out of the NIC, and the non-virtual interface on the host (e.g. eth0) doesn't see the packets.

In order for the MACVLAN attached container to talk to the host, an additional MACLAN interface on the Host must be created, along with a route to prefer that MACVLAN interface.

Creating a MACVLAN interface on the Host

In order to create a MACVLAN interface on the Host one must do the following steps:

Determine your subnet/prefix
Create a locally administered MAC address for the MACVLAN interface
Create a MACVLAN interface using the ip link command
Set the locally adminstered MAC address on the new interface
Wait for the interface to come up and get IPv4 and IPv6 addresses (via DHCPv4 and SLAAC)
Add a route to your subnet/prefix using the new MACVLAN interface, and set the preference for this route to high

1. Hands On - Creating a MACVLAN on your host

Attach your laptop using an Ethernet dongle/port to the Lab Network

Create a MACVLAN interface on your host. Assuming your ethernet is eth0 use the following commands:

# Get subnet prefix 
INTF='eth0'
ip -6 route | grep "$INTF"| grep '^2'   #make a note of the prefix
ip -4 route | grep "$INTF"              #make a note of the subnet

# Show MAC address of interface
ip link show dev "$INTF"                #make a note of the MAC address

# Add the MACVLAN interface
sudo ip link add host-shim link "$INTF" type macvlan  mode bridge

# Set the MAC address to be a locally administered MAC
#   set the first byte to 02, use the rest of your MAC address instead of xx
sudo ip link set address "02:xx:xx:xx:xx:xx" dev "host-shim"

# force the interface up
sudo ip link set up dev host-shim

# Set up a route to use the new MACVLAN interface for this subnet/prefix
sudo ip route add "192.168.136.0/24" dev host-shim metric 100 pref high
sudo ip route add "2607:c000:8011:fda2::/64" dev host-shim metric 100 pref high

Discuss the following:

What is the subnet/prefix you noted at the beginning?
What is the MAC address you noted?
Why do we use a locally administered MAC address?
Can you ping between eth0 to host-shim interfaces? Really?

Using the `lxd_add_macvlan_host.sh` script

Fortunately, there's a script to help us out. This script will automagically create a MACVLAN interface on the LXD Host. This script is called: lxd_add_macvlan_host.sh. LIke all good scripts it has help

$ ./lxd_add_macvlan_host.sh -h
    ./lxd_add_macvlan_host.sh - creates MACVLAN interface on LXD Host 
    e.g. ./lxd_add_macvlan_host.sh -a 
    -a  Add MACVLAN interface
    -4  Add MACVLAN IPv4 interface
    -r  Remove MACVLAN interface
    -i  use this interface e.g. eth0

 By Craig Miller - Version: 0.97

The script does not make any permanent changes to the host, but rather configures the MACVLAN interface on the fly. If you want this to be permanent, then invoke the script from /etc/rc.local (you may have to enable rc.local if you are using systemd).

Because the Lab network is a dual-stack network, we'll use the -4 option to add IPv6 and IPv4.

$ ./lxd_add_macvlan_host.sh -4
Requesting Sudo Privlages...
Working ....
Waiting for IPv4 DHCP ....
Interface: host-shim added
Pau

Looking at the interfaces, you will note that there is an additional host-shim interface that has been added:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:6c:02:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.136.140/24 brd 192.168.243.255 scope global dynamic noprefixroute eth0
       valid_lft 30348sec preferred_lft 23526sec
    inet6 2001:db8:8011:fd44::e79/128 scope global dynamic noprefixroute 
...
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:27:eb:39:57:dd brd ff:ff:ff:ff:ff:ff
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:82:62:46 brd ff:ff:ff:ff:ff:ff
14: host-shim@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:27:eb:6c:02:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.243.185/24 brd 192.168.243.255 scope global dynamic noprefixroute host-shim
       valid_lft 42483sec preferred_lft 37083sec
    inet6 2001:db8:8011:fd44::e79/128 scope global dynamic noprefixroute

Using LXD with MACVLAN

Now you can go forth and use MACVLAN on your Linux Containers, with the full knowledge that your container will not only be able to talk to the Internet, but also to the host-shim interface on the LXD Host.

Using MACVLAN interfaces means it just got simpler to install and use Linux Containers.

Notes:

In a managed Wifi network (which 99% are), the Access Point (AP) will only talk to the MAC address of the client which registered with the AP. A MACVLAN will try to use additional MAC addresses which will be rejected by the AP (since those addresses are not registered with the AP. That doesn't mean that you couldn't try to set up a Wireless Bridge with Wireless Distribution System (WDS), but that is beyond the scope of this article. For now, think of MACVLAN as only using your Ethernet NIC.

12 February 2023
updated 24 February 2023