The Spark that Started it All

Like many of life’s journeys, it began with a conversation with a client about improving automation within their environment and the benefits that it would provide in the long term. During this conversation we were talking about the benefits of standardisation with regards to build processes and they mentioned they were considering moving to Proxmox from VMware ESXi.

I have been using Proxmox for several years for my home network / home lab and have been very happy with it as it “just works” and I have created several templates from Cloud Images which allow me to provision machines quickly.

I have used several tools from Hashicorp over the years and they generally work well, but Packer was not one of those that I had really given much attention to, so I decided that I would check it out and started with their version of “Hello World” and build a Docker Image which was very straight forward and worked.

First Steps: The Ubuntu Safety Net

My preferred Linux Distribution for Servers is Debian, but my client uses Ubuntu and it is well-documented, widely supported and it “just works” with Hashicorp Packer. I was able to get a build up in about 30 minutes that just worked.

The Debian Challenge: When Documentation Falls Short

Debian, my preferred server distribution, “how different could it be?”, after all Ubuntu is Debian-based. Oh how naive I was, with the first hurdle out of the gate, the preseed configuration.

Challenge #1: The Preseed Puzzle

Ubuntu’s automated installation was straightforward, but Debian’s preseed configuration felt like solving a cryptic crossword. The official documentation exists and is very thorough, but left many questions unanswered. After several failed builds and hours of debugging, it became evident that Debians philosphy of explicit configuration extends to the automated install process as well. Whilst this was initially frustrating, I must admit that I do prefer the power of this after the initial learning hurdle.

When you are building, whether this is in Proxmox, QEMU or Virtualbox, ensure that you disable headless mode and watch the console as the first time I was waiting about 15 minutes and it was at a prompt waiting for input which required changes to the preseed file.

Challenge #2: The Proxmox Dance

With the preseed file configured, the next challenge is ensuring that you have all the settings correct and special attention has to be paid with regards to Network and Storage configuration.

There are nuances between different providers, for example Proxmox and QEMU are happy with specifying disk size like 50G, but Virtual Box requires it to be a integer value in MBs.

  # VM General Settings
  vm_id          = "6000"
  vm_name        = "debian-12-minimal-template"
  template_description = "Debian 12 minimal template built with Packer"
  
  # VM OS Settings
  boot_iso {
    iso_file         = "local:iso/debian-12.7.0-amd64-netinst.iso"
    iso_storage_pool = "local"
  }

  # VM System Settings
  qemu_agent    = true
  scsi_controller = "virtio-scsi-pci"

  # VM Hard Disk Settings
  disks {
    disk_size         = "50G"
    format           = "raw"
    storage_pool     = "data_hdd"
    type             = "scsi"
  }

  # VM CPU Settings
  cores = "4"
  memory = "4096"

  # VM Network Settings
  network_adapters {
    model    = "virtio"
    bridge   = "vmbr0"
    firewall = false
  }

  # VM Cloud-Init Settings
  cloud_init = true
  cloud_init_storage_pool = "data_hdd"

These are not major obstacles, but just ensure that you understand the necessary virtual hardware configuration and make the appropriate adjustments.

Taking it Further

When I had it working with Proxmox, I decided to test it further and see if I could build it in QEMU and then VirutalBox, both of which had challenges as each of them has their own nuances which do need to be addressed, but with the foundation built from deploying on Proxmox, these were manageable.

Enter Ansible: Bringing Order to Chaos

I have been using Ansible for many years and already have a decent collection of playbooks and wanted to incorporate this into the builds to have baseline images that were fully customised and integrating Ansible with Packer was extremely straight-forward

provisioner "ansible" {
  playbook_file = "./playbooks/base.yml"
  extra_arguments = [
    "--extra-vars",
    "ansible_python_interpreter=/usr/bin/python3"
  ]
}

There is a lot more that can be done with Ansible and this is an area that I will personally be continually building out.

Lessons Learnt and Future Horizons

I am still very much in the early stages with Packer and will constantly be refining, enhancing and streamlining as my knowledge grows in this area, but these are the key lessons to date:

  1. Documentation is a Starting Point, Not the Destination: Official documentation are invaluable, nothing beats hands on implementations that will often result in going down many rabbit holes, diving into the depths of system internals and hunting down obscure errors, but damn it is fun and you learn a LOT.

  2. Platform Differences Matter: What works on one platform isn’t guaranteed to work and in some cases, may fail spectacularly in another. Understanding the differences is crucial and all part of the journey.

  3. Automation is Key: This was a lesson I learnt nearly 30 years ago and it holds up even more so today. The more something is automated, the more consistent and reliable the builds are and reduces the opportunity for error of every manual step that is automated.

  4. Community is King: Sharing knowledge and tapping into the experiences of others who have traveled the road before is important and contributing back to this knowledge base is a top priority.

Road Ahead

The current setup works well, but there’s always room for improvement and some areas that I will be exploring are:

  • Implementing a proper CI/CD pipeline for image builds
  • Adding automated testing for the built images
  • Creating a validation framework for configuration changes
  • Exploring multi-architecture support (I have tried building ARM image, but this is proving to be extremely challenging in QEMU on an Intel machine, but will keep chipping away at it)

Show me the Code!

I’ll be sharing the code at GitLab which will over time include:

  • Packer configurations for Proxmox and QEMU
  • Debian preseed file
  • Helper Scripts

Final Thoughts

The journey is as valuable as the destination and the skills you develop along the way are invaluable to all aspects of your life. Don’t be afraid to experiment, break things, enjoy the ride and most important take notes.