Terraform is an infrastructure-as-code tool. Officially as of August 13th 2020 there is no KVM / libvirt support. However, there is a maintained 3rd-party provider that allows terraform to interact with KVM hosts.

Installation

MacOS

To install terraform, the easiest way is to use Homebrew.

brew install terraform

The 3rd-party terraform-provider-libvirt can also be installed via brew. However, this also requires Go to build and link the provider to the libvirt libraries.

brew install go # 1.14.3
# configure GOPATH
export GOPATH="${HOME}/go"
brew install libvirt # 6.4.0
export PKG_CONFIG_PATH=/usr/local/Cellar/libvirt/6.4.0/lib/pkgconfig/
go get github.com/dmacvicar/terraform-provider-libvirt
mkdir -p ~/.terraform.d/plugins
cp $GOPATH/bin/terraform-provider-libvirt ~/.terraform.d/plugins

NOTE: Currently the libvirt provider requires mkisofs - this isn’t available on mac.

However, we can utilise the mkisofs option of xorriso to get around this problem:

brew install xorriso

Then in your path (i.e. /usr/local/bin) create an executable mkisofs script:

$ cat /usr/local/bin/mkisofs
#!/bin/bash

xorriso -as mkisofs "$@"

Docker

TODO

Libvirt permissions

Upon installation of the provider I encountered issues with permissions using the terraform libvirt provider controlling the KVM host via the APIs.

I haven’t investigated this fully, but virt-manager has no issues while using the GUI.

This was added a while ago, but subsequently removed from the README: https://github.com/dmacvicar/terraform-provider-libvirt/commit/22f096d9

On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/<FILE_NAME>': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon.

Uncommenting the security_driver configuration fixed the issue for me. However, I’m not sure of the security implications (if any)

libvirt “default” pool

Some tools such as virt-manager seem to create this for you, however on a fresh instance you may need to create the default storage pool. Found a useful reference on a github issue

$ virsh pool-define /dev/stdin <<EOF
<pool type='dir'>
  <name>default</name>
  <target>
    <path>/var/lib/libvirt/images</path>
  </target>
</pool>
EOF

$ virsh pool-start default
$ virsh pool-autostart default

libvirt “default” network

The same goes for the default network. However, configuration of openvswitch has created a default network that is not configured correctly. Assuming the network is inactive, run virsh net-edit default and update it to have a similar configuration to below:

<network>
  <name>default</name>
  <uuid>... leave this alone ...</uuid>
  <forward mode='bridge'/>
  <bridge name='ovsbr'/>
  <virtualport type='openvswitch'/>
</network>

Finally start (and autostart) the network

$ virsh net-start default
$ virsh net-autostart default

Additional packages

On-top of the basic KVM/libvirt packages I have needed to install these on the KVM host

qemu-utils=1:3.1+dfsg-8+deb10u5
netcat-openbsd=1.195-2

Linux modules

According to several sources, libvirt’s performance can be improved by allowing it to directly call the subsystem.

modprobe vhost_net

NOTE: modinfo vhost_net describes this module as Host kernel accelerator for virtio net. As we are using the virtio driver this seems sensible to enable.

Assuming this causes no-issues, then we can permanently add this to the modules to be loaded at boot by adding it to /etc/modules-load.d/modules.conf

Reference: How to use terraform to create a small scale cloud infrastructure

Terraform

Before we begin, we’ll use a pre-built VM image from Debian. Download the required version from the Debian repository.

Once you have this, create a configuration (e.g. libvirt.tf) file. This is largely based on https://computingforgeeks.com/how-to-provision-vms-on-kvm-with-terraform/.

Replace <user> and <ip> with the appropriate values for the KVM host. It assumes you have enabled passwordless SSH.

provider "libvirt" {
  uri   = "qemu+ssh://<user>@<ip>/system?socket=/var/run/libvirt/libvirt-sock"
}

resource "libvirt_volume" "debian10-qcow2" {
  name = "debian10.qcow2"
  pool = "default"
  # local reference - file is colocated with terraform file
  source = "./debian-10-genericcloud-amd64-20200511-260.qcow2"
  format = "qcow2"
}

data "template_file" "user_data" {
  template = file("${path.module}/cloud_init.cfg")
}

resource "libvirt_cloudinit_disk" "commoninit" {
  name = "commoninit.iso"
  user_data = data.template_file.user_data.rendered
}

# Define KVM domain to create
resource "libvirt_domain" "test" {
  name   = "test"
  memory = "1024"
  vcpu   = 1

  network_interface {
    network_name = "default"
  }

  disk {
    volume_id = libvirt_volume.debian10-qcow2.id
  }

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  console {
    type = "pty"
    target_type = "serial"
    target_port = "0"
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = true
  }
}

NOTE: The above configuration makes use of the cloud-init capabilities built into the Debian cloud image to configure the OS. Because this will be configured without human interaction we need a way to specify a password, this is specified in the cloud_init.cfg file which should be co-located with your terraform script.

ssh_pwauth: True
chpasswd:
  # Format is very specific, no space around ':'
  list: |
    root:password
  expire: False

This will allow you to connect to the console with user: root and password: password. This is not secure and is only for demonstration purposes. The most secure option is to use ssh auth only

Terraform usage

On first run, initialize terraform and it’s providers (it will prompt for this if you forget)

terraform init

Next, create a plan (sequence of things to do) and apply them.

terraform plan -out libvirt.plan
terraform apply libvirt.plan

Lastly, to teardown the infrastructure, destroy it:

terraform destroy

Finding your host

Search for whatever your subnet your vswitch is connected to

nmap -v -sn <subnet>/24

Observations

1. The provider doesn’t seem to play 100% nicely with openvswitch bridge. If you specify the network_name as default, it will detect it’s connected on bridge = ovsbr. But you cannot it configure this, so it will always deect differences

Updates

Originally this was configured with terraform 0.12. However, since the introduction of the terraform registry in 0.13 I had to confirm an in-house provider

Terraform 0.13

$ terraform 0.13upgrade

This creates a “versions.tf” file, edit the source to include the libvirt plugin

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
      version = "~> 0.6.3"
    }
    ...
  }
}

Install the plugin into the expected mirror directory (this is darwin for mac)

$ mkdir -p ~/.terraform/plugins.d/registry.terraform.io/dmacvicar/libvirt/0.6.3/darwin_amd64
$ ln -s $GOPATH/bin/terraform-provider-libvirt ~/.terraform/plugins.d/registry.terraform.io/dmacvicar/libvirt/0.6.3/darwin_amd64

Then run terraform init and follow any of the commands