1

I'm trying to provision a VM on qemu/kvm hypervisor using cloudinit with terraform and the libvirt provider. I can get the machine to start, but the cloudinit is not getting kicked off. I know that the user-data being used will work as I've tested it without using terraform, but instead using kvm to spin up a machine with a second terminal running a web server to server up the user-data file. All of this is being performed on Ubuntu 18.04.

I've tried this using cloud images for ubuntu and centos. Both will create the VM and boot up to the login prompt. Neither will actually provision the contents of the user-data though. I've also tried this using a lower version of the libvirt provider (of those still available) to the same results.

I've done some extensive searching trying to see if anyone else has similar issues. Most of the sites/questions I've found were from StackExchange sites, as well as github issues and bug reports, which I didn't capture to document unfortunately. All of them tend to be something in the user-data that is missing. None of them have been using the version of the libvirt provider I am, but instead almost always version 0.6.2. I tried to use that version of the provider in my main.tf file, but the init command returns an error that the version doesn't exist and can't be downloaded.

main.tf (only difference between centos and ubuntu is the cloud image file/location and prefix variable)

terraform {
    required_providers {
        libvirt = {
            source = "dmacvicar/libvirt"
            version = "0.7.0"
        }
    }
}

# instantiate the provider
provider "libvirt" {
    uri = "qemu:///system"
}

variable "prefix" {
    default = "terraform_centos"
}

data "template_file" "user_data" {
    template = file("${path.module}/cloud_init.cfg")
}

resource "libvirt_cloudinit_disk" "commoninit" {
    name = "commoninit.iso"
    user_data = data.template_file.user_data.rendered
}

resource "libvirt_volume" "qcow_volume" {
    name = "${var.prefix}.img"
    source = "https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2"
    format = "qcow2"
}

resource "libvirt_domain" "centos" {
    name = var.prefix
    vcpu = 2
    memory = 4096
    disk {
        volume_id = libvirt_volume.qcow_volume.id
    }

    cloudinit = libvirt_cloudinit_disk.commoninit.id

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  network_interface {
    network_name = "default"
  }

}

user-data file (as with main.tf, only changes between the two distros being tested is the hostname)

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: terraform_centos
    username: vagrant
    password: $6$dnWt7N17fTD$8.m3Rgf400iSyxLa/kUtunGUgE3N4foSg/y31HNnsGBUTpoMOmS3O9U/nJFvZjXpQTrLFrAcK5vok5EI0KZA90
  locale: en_US
  keyboard:
    layout: us
  ssh:
    install-server: true
    authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key
    allow-pw: true
  storage:
    layout:
      name: direct
  packages:
    - gcc
    - build-essential
  late-commands:
    - "echo 'vagrant ALL=(ALL) NOPASSWD: ALL' >> /target/etc/sudoers.d/vagrant"
    - "chmod 440 /target/etc/sudoers.d/vagrant"

Finally, the kvm command that actually gets cloudinit to work. This is after starting a quick python web server in another shell. Along with mounting the iso image into /mnt so that the kernel/initrd will be accessible.

kvm -no-reboot -m 4096 -drive file=focal.img,format=raw,cache=none,if=virtio -cdrom ~/isoImages/ubuntu-20.04.5-live-server-amd64.iso -kernel /mnt/casper/vmlinuz -initrd /mnt/casper/initrd -append 'autoinstall ds=nocloud-net;s=http://_gateway:3003/' -vnc :1

Qemu/kvm version info

$ virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1

Terraform version info

$ terraform version
Terraform v1.3.6
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.7.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Because I'm using cloud images for the two distros, there isn't a default username/password that I can log in with to check logs on the VM directly. I do have access to the console via virt-manager and the virsh console command. Both are sitting at a login prompt, at which the vagrant user returns a login incorrect message.

If any further information is needed, please let me know. I'm open to suggestions on what needs to be done to get this to work.

1
  • Was never able to figure out how to get terraform to work with cloudinit. Instead I've opted to use packer to build out a base image that cloudinit works fine with. Then use terraform to provision the servers that I need off that base image. Jan 6, 2023 at 18:58

2 Answers 2

1

As Brett-holman mentioned, the issue is most likely the cloud-init YAML you are passing in. If you monitor the console of the booting domain ( virsh console --domain your-kvm-domain ) using a un-modified cloud image (try one of the QCOW2 images from Ubuntu ) as your base_volume_id in the libvirt_volume you should see output from cloud-init's init process.

Try out the terraform in this repo: https://github.com/mattsn0w/terraform_kvm I use this as the basis for my homelab as well as air gapped environments at work.
I am currently using v0.7.1 of dmacvicar/libvirt's terraform provider successfully provisioning AL2, CentOS 7, RHEL 8 & 9, Ubuntu 18, 20, and 22 LTS. If a cloud-image has cloud-init installed, then you should be able to do quite a bit of configuration.

It's worth pointing out that you'll need to match the version of cloud-init in the image with the documentation. The latest version as of this comment is v23.3.3

1
  • 1
    The difference that fixed it for me was your cdrom-model.xsl, i.e. using scsi instead of ide for the CD-ROM.
    – DrPsychick
    Jan 24 at 23:03
0

Clarification

Please note that your yaml is actually a config for Subiquity, the Ubuntu autoinstaller, not cloud-init.

Cloud-init is used by Ubuntu's autoinstaller, though cloud-init itself does not use autoinstall yaml keys.

I've tried this using cloud images for ubuntu and centos.

Note that cloud-init supports centos, but I don't think that Ubuntu's autoinstaller does. If you wish to use cloud-init instead, you might want to see the examples and the tutorials.

Suggestion

Take a close look at these lines in your .tf file:

data "template_file" "user_data" {
    template = file("${path.module}/cloud_init.cfg")
}

Does Subiquity expect to pull its config from cloud_init.cfg? I doubt it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .