Tutorial: Automatic, Reproducible Preparation of Virtual Machine templates for your vSphere environment with HashiCorp’s Packer

Introduction

Let us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem.

From here on, I will assume some level of familiarity with Packer here. It has a builder called vmware-iso that looks like it can do the trick. However, you will run into problems quickly. For example, the vsphere-template post-processor only works if you run the vmware-iso builder remotely. It does not work with your local virtual machine. Because of this and other issues you have to apply some tricks to make the post-processing section work. Also, after using that builder once, you might notice that there are some short-comings in how Packer detects what IP the virtual machine has. It takes the first DHCP lease from ethernet0 and uses that to communicate with your virtual machine even though it is, by default, the network that is used for network address translation (NAT). This post will show you how to make it all work. By the end of it, you should have a working pipe-line:

  • automatically a new virtual machine is created in VMware Workstation with needed parameters
  • specified ISO is mounted and the OS is installed using a “pre-seeded” configuration
  • some optional post-processing scripts will be run on the virtual machine
  • the resulting virtual machine template will be uploaded to the specified vCenter

As you can see, this process is perfect for making golden templates of various operating systems. Let us delve into by using Ubuntu 16.04 as an example.

Step 1: starting up the virtual machine

The vmware-iso builder is perfect for doing except we will have to apply some tricks to make Packer pick up the correct IP address. By default, the first and only network is used for network address translation which means that when the virtual machine will be deployed, Packer will pick up the given DHCP lease and it will try to connect to it via SSH for provisioning. Because network address translation is happening, there is no way to reach that deployed virtual machine without any tricks.

To solve this problem, I recommend you to specify custom VMX properties which will make ethernet0 be connected to vmnet8, and ethernet1 to vmnet1vmnet8 is an internal, private network that will be used for communication between the host machine and the virtual machine. ethernet1 is connected to vmnet1, which is the network with an DHCP server and is used for network address translation which lets the virtual machine access the Internet. Such configuration of vmnet1 and vmnet8 is the default on VMWare Workstation so no changes are needed on that side.

This is achievable by specifying the following in the builders section of the JSON configuration file:

{
    "vmx_data": {
        "ethernet0.present": true,
        "ethernet0.startConnected": true,
        "ethernet0.connectionType": "custom",
        "ethernet0.vnet": "vmnet8",
        "ethernet1.present": true,
        "ethernet1.startConnected": true,
        "ethernet1.connectionType": "custom",
        "ethernet1.vnet": "vmnet1"
    }
}

You can find more information about VMX properties in this website: http://sanbarrow.com/vmx/vmx-network-advanced.html.

However, after provisioning the virtual machine, you might want to disable the second interface because it is a good default to only have one virtual NIC connected to a virtual machine template by default. If the user will want to add more networks and NICs – it is up to them. So to disable the second NIC after everything, add this to the configuration file that is passed to Packer:

{
    "vmx_data_post": {
        "floppy0.present": false,
        "ethernet1.present": false
    }
}

Note that this also disables the virtual floppy disk after provisioning. This is needed to unmount the floppy disk that has the pre-seed file. Obviously it is not needed in the final virtual machine template. This is all you need so far in this tutorial. You could run the file now and see the Ubuntu installer be automatically started. This is how my builders section looks like:

{
  "builders": [
    {
      "type": "vmware-iso",
      "iso_url": "http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.iso",
      "iso_checksum": "10fcd20619dce11fe094e960c85ba4a9",
      "iso_checksum_type": "md5",
      "ssh_username": "root",
      "ssh_password": "Giedrius",
      "vm_name": "Ubuntu_16.04_x64",
      "ssh_wait_timeout": "600s",
      "shutdown_command": "shutdown -P now",
      "output_directory": "ubuntu_1604",
      "boot_command": [
        "<enter><wait><f6><esc><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "/install/vmlinuz<wait>",
        " auto<wait>",
        " console-setup/ask_detect=false<wait>",
        " console-setup/layoutcode=us<wait>",
        " console-setup/modelcode=pc105<wait>",
        " debconf/frontend=noninteractive<wait>",
        " debian-installer=en_US<wait>",
        " fb=false<wait>",
        " hostname=ubuntu1604<wait>",
        " initrd=/install/initrd.gz<wait>",
        " kbd-chooser/method=us<wait>",
        " keyboard-configuration/layout=USA<wait>",
        " keyboard-configuration/variant=USA<wait>",
        " locale=en_US<wait>",
        " noapic<wait>",
        " preseed/file=/floppy/ubuntu_preseed.cfg",
        " -- <wait>",
        "<enter><wait>"
      ],
      "guest_os_type": "ubuntu-64",
      "floppy_files": [
        "./ubuntu_preseed.cfg"
      ],
      "vmx_data": {
        "ethernet0.present": true,
        "ethernet0.startConnected": true,
        "ethernet0.connectionType": "custom",
        "ethernet0.vnet": "vmnet8",
        "ethernet1.present": true,
        "ethernet1.startConnected": true,
        "ethernet1.connectionType": "custom",
        "ethernet1.vnet": "vmnet1"
      },
      "vmx_data_post": {
        "floppy0.present": false,
        "ethernet1.present": false
      }
    }
  ]}

Step 2: installing the operating system in the new virtual machine

There is almost no trick that has to be done here except that you have to make sure that DHCP is enabled on all network interfaces. Many operating systems’ installers only let you choose the default interface on which DHCP is enabled. This could be worked around with some custom amendments to the configuration.

For example, on Ubuntu 16.04 the easiest way to do this is to append some text to /etc/network/interfaces which will enable DHCP on the second interface as well. You can find more information on that file here: https://help.ubuntu.com/lts/serverguide/network-configuration.html. So, you should have something like this in your pre-seed file:

d-i preseed/late_command string in-target sh -c 'echo "auto ens33\niface ens33 inet dhcp" >> /etc/network/interfaces'

ens33 is the default name of the interface that it given to the second one on my set up so it is used here. As you can see, the network configuration is changed and DHCP client is enabled on the ens33 interface as well. If you are using some other operating system then refer to its manual to find out what exactly you have to change to enable the DHCP client on the second interface.

You are free to add anything you want to to the provisioners section of the Packer configuration. In my case, I am uploading some default configuration to /etc/profile.d and I am making those files executable so that they would be executed each time someone logs in.

Step 3: post processing a.k.a. uploading the actual template

This is an important trick that you have to do. The default post-processor only support this action if you use the vmware-iso builder in a remote configuration so some trickery is needed.

At first, you need to run the OVFTool to convert the *.vmx file to an *.ovf. Use the shell-local post-processor like this:

{
  "type": "shell-local",
  "inline": [
    "/c/Program\\ Files\\ \\(x86\\)/VMware/VMware\\\nWorkstation/OVFTool/ovftool.exe ubuntu_1604/Ubuntu_16.04_x64.vmx ubuntu_1604/Ubuntu_16.04_x64.ovf"
  ]
}

This will convert the resultant vmx file to an ovf one. Next, you have to use the artifice post-processor to change the list of artifacts so that the next post-processor that we are going to use would pick up the new ovf file as well.

The post-processor that we will use is called vsphere-template. It sifts through the artifact list (that we will generate with artifice) and uploads the OVF file to the vCenter, and converts it into an template. You can find it here: https://github.com/andrewstucki/packer-post-processor-vsphere-template (props to andrewstucki!). You can get all of the details in there. To make it short:

  • download the executable file into some known location
  • edit %APPDATA%/packer.config and add something like this there (change the exact path depending on your set-up):
{
  "post-processors": {
    "vsphere-template": "C:\\packer-post-processor-vsphere-template_darwin_amd64.exe"
  }
}

For the final piece, you will have to create a post-processor chain with vsphere-template and artifice like it is described here: https://www.packer.io/docs/post-processors/artifice.html. You will need to enclose the last two post-processors in square brackets. The final version of the post-processor section looks like this:

{
  "post-processors": [
    {
      "type": "shell-local",
      "inline": [
        "/c/Program\\ Files\\ \\(x86\\)/VMware/VMware\\ Workstation/OVFTool/ovftool.exe ubuntu_1604/Ubuntu_16.04_x64.vmx ubuntu_1604/Ubuntu_16.04_x64.ovf"
      ]
    },
    [
      {
        "type": "artifice",
        "files": [
          "ubuntu_1604\\*"
        ]
      },
      {
        "type": "vsphere-template",
        "datacenter": "example_datacenter",
        "datastore": "example_datastore",
        "host": "example_host",
        "password": "example_password",
        "username": "example_username",
        "vm_name": "Ubuntu_16.04_x64",
        "resource_pool": "example_resourcepool",
        "folder": "example_folder"
      }
    ]
  ]
}

Links

The Packer JSON configuration file looks like this: https://gist.github.com/GiedriusS/04b9881595882fdee61a83d3c7dd3f3b

The pre-seed file that I used for Ubuntu 16.04: https://gist.github.com/GiedriusS/9053014e39ad17a7e4669e28bc7494bc

For other machines there is not much difference except you will have to figure out what boot options to use, the format of the installer configuration, and what to pass to the network management daemon that is used in the operating system to enable DHCP on all interfaces.

What is Actually True and False in Python?

Intro

Did you know that in Python 2.x you can do the following?

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> False
False
>>> True
False
>>> not True
True
>>> True == False
True
>>> True != False
False

How can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn’t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> True = not False
>>> True
True
>>> False
False
>>> True == False
False

After this, everything will be back to normal. As you can see, you do not need to worry about anything because you can use various operators to assign a sane value to True again after you change it. Besides not you could use ==!= or any other operator which returns a boolean value.

This article will delve into the presented issue and explain why you are able to do this, first of all. Apart from that, there are some apt questions that are raised by this interesting behavior. They include:

  • What about Python 1.x or 3.x? Can you do the same?
  • How did the programming language developers miss this?
  • What could be the rationale behind these language design decisions?

I will try my best to look into and answer them. This is definitely an interesting piece of history of development of the Python programming language.

True and False in Python 1.x

The oldest major version of the Python programming language – 1.x – does not even have such a thing as False or True. You can see in this example:

try:
  print True
except NameError:
  print 'True not found'

This yields the text ‘True not found’ in the standard output:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> try:
...   print True
... except NameError:
...   print 'True not found'
... 
True not found

NameError is raised when you are using True in Python 1.x because Python tries to look up a variable called True and, obviously, that does not exist. Also, this shows that False does not exist as well:

try:
  print False
except NameError:
  print 'False not found'

And we get:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> try:
...   print False
... except NameError:
...   print 'False not found'
... 
False not found

In Python 1.x, like in some other languages, what is true or false is defined in terms of evaluation rules. Anything other than None, numeric zero of all types, empty sequences, and empty mappings are considered true (https://docs.python.org/release/1.6/ref/lambda.html):

In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: None, numeric zero of all types, empty sequences (strings, tuples and lists), and empty mappings (dictionaries). All other values are interpreted as true.

Thus, we could make our own True and False like this:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> False = 0
>>> True = not False
>>> print False, True
0 1

But, obviously, they are not protected from modification and they are not available in all Python programs so it is kind of pointless to have them unless you have large piece of software that was written in Python and you want to maintain an unified definition of what is True and False over all of it, and if you want to make it more future-proof because then only one small section will have to be changed to change the meaning of True and/or False.

A keen reader would notice that this also means that there is no dedicated boolean types. This section of the official language specification lists all of the types –  https://docs.python.org/release/1.6/ref/types.html. The official supported types in Python 1.x are these:

  • None
  • Ellipsis
  • Numbers
  • Sequences
  • Mappings
  • Callable types
  • Modules
  • Class and class instances
  • Files
  • Internal types

As you can see, there really is no dedicated type for boolean expressions. However, the situation was significantly improved in the next major version of Python – 2.x. Although not all negative aspects were fixed and they are still there in the language. Let’s talk about Python 2.x.

True and False in Python 2.x

Dedicated ‘bool’ type appeared in the 2.x version of the Python programming language as per this example:

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> type(True)
<type 'bool'>
>>> type(False)
<type 'bool'>

The type() function returns the type of the provided argument. As you can see, the type of True and False is bool. Boolean types were truly finally added in Python 2.x and they belong to the integer class of types (https://docs.python.org/2/reference/datamodel.html):

Booleans

These represent the truth values False and True. The two objects representing the values False and True are the only Boolean objects. The Boolean type is a subtype of plain integers, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings “False” or “True” are returned, respectively.

However, even though a dedicated boolean was added but the new values True and False were not defined to be keywords thus their value could be changed. Instead, they were defined as “constants” that live in the built-in namespace (https://docs.python.org/2/library/constants.html):

A small number of constants live in the built-in namespace. They are:

False
The false value of the bool type.

New in version 2.3.

True
The true value of the bool type.

New in version 2.3.

Unfortunately but the fact that they are “constants” does not constitute that they are immutable in Python 2.x. All of this verbiage essentially just means that there are some variables of certain types and certain values pre-loaded into every Python program and the program itself is free to change their meaning.  Before 2.4, you even could assign a new value to None but later they changed it to raise an SyntaxError exception if you attempted to do that. Why they did not do that for True and False as well – I don’t know. I seriously wonder what insane use-cases or existing code they were accommodating for by not making the same change for True and False as well at the same time in 2.4.

Also, notice that the Python 2.x documentation makes a separation between “true” and “false” constants. “true” constants are those to which you cannot assign a new value because it raises an exception, and “false” constants are those to which you can. The official documentation even puts those words in quotes, I am not making it up. This could really make you say: “wat”.

Wat meme

If you ask me I see it as a huge inconsistency in language design and it makes no sense to me not to make same change from Python 2.4 on-wards to make it illegal to assign new values to True and False as well, and just remove the whole “false” constants notion in general. Perhaps they were afraid of making such a backwards incompatible change and so the developers waited until 3.x?

True and False in Python 3.x

This mess was finally permanently fixed in the next major version of Python, 3.x.  True and False, and other constants like None were turned into keywords. True was defined to be equal to the number 1 and False was defined to be equal the number 0. There are no more such thing as “false” or “true” constants. You can see that from this error message:

$ python
Python 3.6.3 (default, Oct 24 2017, 14:48:20) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> type(True)
<class 'bool'>
>>> type(False)
<class 'bool'>
>>> True = False
  File "<stdin>", line 1
SyntaxError: can't assign to keyword
>>> None = False
  File "<stdin>", line 1
SyntaxError: can't assign to keyword

As you can see, True and None (and others) became keywords and thus officially they became immutable. They are listed here https://docs.python.org/3/reference/lexical_analysis.html#keywords. Boolean type is still a sub-type of the Number types: https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy. The text that describes the boolean type is identical to the 2.x version.

Defining True and False to be constants that are immutable also brings some performance improvements because implementations of Python no longer have to look up what is the value of True and False every time they were used in an infinite loop, for example. Things were finally completely fixed by version 3.x and it only took almost 15 years to properly implement True and False.

What were they thinking?

From what I can tell is that Guido never cared at the beginning to add a separate boolean type just like, for example, the C programming language never had a boolean type until the C99 version of the standard. All of the boolean operations were simply expressed in terms of the evaluation rules.

The boolean type and the True/False constants were proposed in Python Enhancement Proposal (PEP) number 0285 (https://www.python.org/dev/peps/pep-0285/). However, it seems to me that at that time Python 1.x had a bunch of those constants that were mutable and these two new constants were added which kind of floated around and had an unknown status just like the others. After a bit of time, someone noticed that it does not make much sense to override the value of None/True/False and others. At that point they were into keywords thus rendering them immutable. The fix in the version 2.4 for the None value seemed like a bit of bandage aid applied by the developers to the language but it wasn’t fixed completely. I guess that they waited until the next major version bump because it’s a backwards-incompatible change.

It’s kind of humorous because some Python developers even (https://github.com/riptideio/pymodbus/issues/43) decided to include lines like these at the beginning of their programs:

True = 1 == 1
False = 1 == 0

It is crazy that they were afraid of people using their libraries who were messing with the values of True and False. Such is the fun story of True and False in the Python world.

EDIT:

2018 January 10 – changed the words to say that in Python 3.x True and False were defined to be equal to 1 and 0 respectively, they are not the actual numbers.