Popping off with Firecracker VM

As someone who uses docker a lot I've always meant to learn more about containers, virtualization, and other isolation technologies but of course I'm pretty lazy. A new container like technology recently caught my eye though and I decided to check it out, a new VMM called Firecracker.

It's a new-ish release by amazon, made to offer much better isolation than containers as they have a need to run customers serverless functions on the same machine. Under the hood it connects to KVM. Dockers fast, but virtualization is more secure, and also a first class citizen on linux. I won't rehash the whole pitch, so check it out here.

A quick look at how the whole thing runs:

pic

There are a few things I like about it right off the bat and a couple I don't.

For one you can't simply volume in a path to a micro VM. This is deliberate as it lowers the risk of VM to host machine exploits, and makes it a bit more annoying to use if you're used to this workflow.

Each VM is given a kernel and a file system separately which allowing you to run a different kernel than what your file system was created with. This is pretty neat although I've never into need for this. A good use case would be updating your fleet of VMs by repacking a new file system and redeploying without having to build anything new.

Interacting with the VMs is very straightforward and deliberately simple. To start one up, you launch the VM host and interact with it through a rest API or a configuration file. The host process creates a socket the IPC to the VM. There are only a few possible commands which make it trivial to spin up and and monitor a VM.

Actually using it

Below is a simple script thrown together that spins up 50 VMs in about 3 seconds, not including time to boot which is around 10 seconds per VM concurrently. These use the API a firecracker VM exposes, one of the few ways to interact with them to start up 50 VMs, register a file system and kerel to each one, and boots it.

KERNEL=hello-vmlinux.bin
ROOTFS="rootfs.ext4"
SOCKET=/tmp/vms/firecracker.socket

# Toy example for spinning up multiple vms

function _start(){
  mkdir -p /tmp/vms/
  if [ -e $SOCKET ]; then
    /bin/rm -f $SOCKET
  fi

  ./firecracker --id 1 --api-sock $SOCKET &>/dev/null &
}

function _init(){
  curl --silent --unix-socket $SOCKET -i \
    -X PUT 'http://localhost/boot-source'   \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d "{
          \"kernel_image_path\": \"${KERNEL}\",
          \"boot_args\": \"keep_bootcon init=/sbin/init console=ttyS0 reboot=k panic=1 pci=off\"
     }" &>/dev/null

  rootfs_path=$(pwd)/$ROOTFS
  curl --silent --unix-socket $SOCKET -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d "{
          \"drive_id\": \"rootfs\",
          \"path_on_host\": \"${rootfs_path}\",
          \"is_root_device\": true,
          \"is_read_only\": false
     }" &>/dev/null
}

function _connect(){
  curl --silent --unix-socket $SOCKET -i \
      -X PUT 'http://localhost/actions'       \
      -H  'Accept: application/json'          \
      -H  'Content-Type: application/json'    \
      -d '{
          "action_type": "InstanceStart"
       }' &>/dev/null
}

function kill(){
  echo "Killing all servers"
  killall firecracker
  rm -rf /tmp/vms/
}

function list(){
  COUNT=$(ps | grep firecracker | wc -l)
  echo "Currently $COUNT VM(s) running"

}

function create(){
  COUNT=50
  echo "Creating $COUNT Vms..."
  for ((i=1;i<=COUNT+1;i++)); do
      SOCKET=/tmp/vms/firecracker_${i}.socket
      echo "Created $SOCKET"
      _start $i
      while [ ! -e $SOCKET ]; do sleep .025; done
      _init
      _connect
  done
  echo "Finished!"
}

$1

If you want to see crazy, amazon has run over 4000 of these on the same host here.

These don't have networking set up or logging (which firecracker has support for), but you get the idea. It would just be another API call to the VM. Above I have a kernel and an root file system laid out from their guide guide. What if you wanted your own file system?

One way to create one is to create an empty ext4 partition, mount it, and populate it. Here's a simple way to export a docker containers filesystem to a ext4 partition. This works pretty well since firecracker provides a kernel to use that the export will lack, since docker uses the hosts kernel.

dd if=/dev/zero of=rootfs.ext4 bs=1M count=100
sudo mount rootfs.ext4 /mnt/fs
# Make sure you have an init in the container, and optionally a tty if you want to use it interactively.
docker create <id> && docker export <id> -o rootfs.tar
tar -xf rootfs.tar -C /mnt/fs

Pop that sucker in as the root file system and watch it fly.

Running containers?

This seems like a lot to get a custom VM running if you are container based. What if you have already built but just want to semi-easily switch to running in firecracker? Luckily there is firecracker-containerd, a containerd implementation for firecracker which the name suggests. I would check out this talk which gives a great overview of the technology, and also gives a good overview of containerd in general.

There are some instructions for running a container image from docker.io inside of the firecracker repo, but these still are a little bespoke and lengthy for spinning up a custom VM with your already in place infrastructure. Luckily the hard works been done already from WeaveWorks.

Ignite by WeaveWorks is a fleshed out wrapper for firecracker-containerd written in go, allowing you to create, manage, and run firecracker VMs similar to dockers CLI. It also supports pushing to cloud providers.

Wrapping up

Most people love two thing about docker:

I think firecracker solves a lot of the core issues that about containers, namely running as root, not having to worry about user accounts, and limiting access to the host machine as much as possible. The technology is written in rust too which people tell me is safe. Most importantly it's not just a grab bag of linux utilities mashed together making syscalls to the host machine like crazy.

A good thing to keep in mind, the intended use case for this technology is not for a general container runtime (Although it works) but as a very lightweight VM for server less functions. Nevertheless the ability for firecracker-containerd to provide all these security benefits and more while making your existing containers easy to run is a pretty big win.