Category Archives: Automation

Ansible tips’n’tricks: provision multiple machines in parallel with Vagrant and Ansible

Vagrant is a great tool that I’m regularly using for building playground environments on my laptop. I recently came across a slight inconvenience with Vagrant’s Virtualbox provider: occasionally I would like to spin up a Data Guard environment and provision both VMs in parallel to save time. Sadly you can’t bring up multiple machines in parallel using the VirtualBox provisioner according to the documentation . This was true as of April 11 2019 and might change in the future, so keep an eye out on the reference.

I very much prefer to save time by doing things in parallel, and so I started digging around how I could achieve this goal.

The official documentation mentions something that looks like a for loop to wait for all machines to be up. This isn’t really an option, I wanted more control over machine names and IP addresses. So I came up with this approach, it may not be the best, but it falls into the “good enough for me” category.

Vagrantfile

The Vagrantfile is actually quite simple and might remind you of a previous article:

  1 Vagrant.configure("2") do |config|
  2   config.ssh.private_key_path = "/path/to/key"
  3 
  4   config.vm.define "server1" do |server1|
  5     server1.vm.box = "ansibletestbase"
  6     server1.vm.hostname = "server1"
  7     server1.vm.network "private_network", ip: "192.168.56.11"
  8     server1.vm.synced_folder "/path/to/stuff", "/mnt",
  9       mount_options: ["uid=54321", "gid=54321"]
 10 
 11     config.vm.provider "virtualbox" do |vb|
 12       vb.memory = 2048
 13       vb.cpus = 2
 14     end
 15   end
 16 
 17   config.vm.define "server2" do |server2|
 18     server2.vm.box = "ansibletestbase"
 19     server2.vm.hostname = "server2"
 20     server2.vm.network "private_network", ip: "192.168.56.12"
 21     server2.vm.synced_folder "/path/to/stuff", "/mnt",
 22       mount_options: ["uid=54321", "gid=54321"]
 23 
 24     config.vm.provider "virtualbox" do |vb|
 25       vb.memory = 2048
 26       vb.cpus = 2
 27     end
 28   end
 29 
 30   config.vm.provision "ansible" do |ansible|
 31     ansible.playbook = "hello.yml"
 32     ansible.groups = {
 33       "oracle_si" => ["server[1:2]"],
 34       "oracle_si:vars" => { 
 35         "install_rdbms" => "true",
 36         "patch_rdbms" => "true",
 37       }
 38     }
 39   end
 40 
 41 end

Ansibletestbase is my custom Oracle Linux 7 image that I keep updated for personal use. I define a couple of machines, server1 and server2 and from line 30 onwards let Ansible provision them.

A little bit of an inconvenience

Now here is the inconvenient bit: if I provided an elaborate playbook to provision Oracle in line 31 of the Vagrantfile, it would be run serially. First for server1, and only after it completed (or failed…) server2 will be created and provisioned. This is the reason for a rather plain playbook, hello.yml:

$ cat hello.yml 
---
- hosts: oracle_si
  tasks:
  - name: say hello
    debug: var=ansible_hostname

This literally takes no time to execute at all, so no harm is done running it serially once per VM. Not only is no harm done, quite the contrary: Vagrant discovered an Ansible provider in the Vagrantfile and created a suitable inventory file for me. I’ll gladly use it later.

How does this work out?

Enough talking, time to put this to test and to bring up both machines. As you will see in the captured output, they start one-by-one, run their provisioning tool and proceed to the next system.

$ vagrant up 
Bringing machine 'server1' up with 'virtualbox' provider...
Bringing machine 'server2' up with 'virtualbox' provider...
==> server1: Importing base box 'ansibletestbase'...
==> server1: Matching MAC address for NAT networking...

[...]

==> server1: Running provisioner: ansible...

[...]

    server1: Running ansible-playbook...

PLAY [oracle_si] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [server1]

TASK [say hello] ***************************************************************
ok: [server1] => {
    "ansible_hostname": "server1"
}

PLAY RECAP *********************************************************************
server1                    : ok=2    changed=0    unreachable=0    failed=0   

==> server2: Importing base box 'ansibletestbase'...
==> server2: Matching MAC address for NAT networking...

[...]

==> server2: Running provisioner: ansible...

[...]
    server2: Running ansible-playbook...

PLAY [oracle_si] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [server2]

TASK [say hello] ***************************************************************
ok: [server2] => {
    "ansible_hostname": "server2"
}

PLAY RECAP *********************************************************************
server2                    : ok=2    changed=0    unreachable=0    failed=0

As always the Ansible provisioner created an inventory file I can use in ./.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory. The inventory looks exactly as described in the |ansible| block, and it has the all important global variables as well.

$cat ./.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory
# Generated by Vagrant
server2 ansible_host=127.0.0.1 ansible_port=2201 ansible_user='vagrant' ansible_ssh_private_key_file='/path/to/key'
server1 ansible_host=127.0.0.1 ansible_port=2200 ansible_user='vagrant' ansible_ssh_private_key_file='/path/to/key'

[oracle_si]
server[1:2]

[oracle_si:vars]
install_rdbms=true
patch_rdbms=true

After ensuring both that machines are up using the “ping” module, I can run the actual playbook. You might have to confirm the servers’ ssh keys the first time you run this:

$ ansible -i ./.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -m ping oracle_si
server1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
server2 | SUCCESS => {
"changed": false,
"ping": "pong"
}

All good to go! Let’s call the actual playbook to provision my machines.

$ ansible-playbook -i ./.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory provisioning/oracle.yml 

TASK [Gathering Facts] ***************************************************************
ok: [server1]
ok: [server2]

TASK [test for Oracle Linux 7] *******************************************************
skipping: [server1]
skipping: [server2]

And we’re off to the races. Happy automating!

Advertisements

Ansible tips’n’tricks: testing and debugging Ansible scripts using Vagrant

At last year’s UKOUG I presented about Vagrant and how to use this great piece of software to test and debug Ansible scripts easily. Back then in December I promised a write-up, but for various reasons only now got around to finishing it.

Vagrant’s Ansible Provisioner

Vagrant offers two different Ansible provisioners: “ansible” and “ansible_local”. The “ansible” provisioner depends on a local Ansible installation, on the host. If this isn’t feasible, you can use “ansible_local” instead. As the name implies it executes code on the VM instead of on the host. This post is about the “ansible” provisioner.

Most people use Vagrant with the default VirtualBox provider, and so do I in this post.

A closer look at the Vagrantfile

It all starts with a Vagrantfile. A quick “vagrant init ” will get you one. My test image I use for deploying the Oracle database comes with all the necessary block devices and packages needed, saving me quite some time. Naturally I’ll start with that one.

$ cat -n Vagrantfile 
     1    # -*- mode: ruby -*-
     2    # vi: set ft=ruby :
     3    
     4    Vagrant.configure("2") do |config|
     5    
     6      config.ssh.private_key_path = "/path/to/ssh/key"
     7    
     8      config.vm.box = "ansibletestbase"
     9      config.vm.define "server1" do |server1|
    10        server1.vm.box = "ansibletestbase"
    11        server1.vm.hostname = "server1"
    12        server1.vm.network "private_network", ip: "192.168.56.15"
    13    
    14        config.vm.provider "virtualbox" do |vb|
    15          vb.memory = 2048
    16          vb.cpus = 2 
    17        end 
    18      end 
    19    
    20      config.vm.provision "ansible" do |ansible|
    21        ansible.playbook = "blogpost.yml"
    22        ansible.groups = { 
    23          "oracle_si" => ["server1"],
    24          "oracle_si:vars" => { 
    25            "install_rdbms" => "true",
    26            "patch_rdbms" => "true",
    27            "create_db" => "true"
    28          }   
    29        }   
    30      end 
    31    
    32    end

Since I have decided to create my own custom image without relying on the “insecure key pair” I need to keep track of my SSH keys. This is done in line 6. Otherwise there wouldn’t be an option to connect to the system and Vagrant couldn’t bring the VM up.

Lines 8 to 18 define the VM – which image to derive it from, and how to configure it. The settings are pretty much self-explanatory so I won’t go into too much detail. Only this much:

  • I usually want a host-only network instead of just a NAT device, and I create one in line 12. The IP address maps to and address on vboxnet0 in my configuration. If you don’t have a host-only network and want one, you can create it in VirtualBox’s preferences.
  • In line 14 to 17 I set some properties of my VM. I want it to come up with 2 GB of RAM and 2 CPUs.

Integrating Ansible into the Vagrantfile

The Ansible configuration is found on lines 20 to 30. As soon as the VM comes up I want Vagrant to run the Ansible provisioner and execute my playbook named “blogpost.yml”.

Most of my playbooks rely on global variables I define in the inventory file. Vagrant will create an inventory for me when it finds an Ansible provisioner in the Vagrantfile. The inventory it creates doesn’t fit my needs though, but that is easy to change. Recent Vagrant versions allow me to create the inventory just as I need it. You see this in lines 22 to 28. The resulting inventory file is created in .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory and looks like this:

$ cat .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory 
# Generated by Vagrant

server1 ansible_host=127.0.0.1 ansible_port=2222 ansible_user='vagrant' ansible_ssh_private_key_file='/path/to/ssh/key'

[oracle_si]
server1

[oracle_si:vars]
install_rdbms=true
patch_rdbms=true
create_db=true

That’s exactly what I’d use if I manually edited the inventory file, except that I don’t need to use “vagrant ssh-config” to figure out what the current SSH configuration is.

I define a group of hosts, and a few global variables for my playbook. This way all I need to do is change the Vagrantfile and control the execution of my playbook rather than maintaining information in 2 places (Vagrantfile and static inventory).

Ansible Playbook

The final piece of information is the actual Ansible playbook. Except for the host group I’m not going to use the inventory’s variables to keep the example simple.

$ cat -n blogpost.yml 
     1    ---
     2    - name: blogpost
     3      hosts: oracle_si
     4      vars:
     5      - oravg_pv: /dev/sdb
     6      become: yes
     7      tasks:
     8      - name: say hello
     9        debug: msg="hello from {{ ansible_hostname }}"
    10    
    11      - name: partitioning PVs for the volume group
    12        parted:
    13          device: "{{ oravg_pv }}"
    14          number: 1
    15          state: present
    16          align: optimal
    17          label: gpt

Expressed in plain English, it reads: take the block device indicated by the variable oravg_pv and create a single partition on it spanning the entire device.

As soon as I “vagrant up” the VM, it all comes together:

$ vagrant up
Bringing machine 'server1' up with 'virtualbox' provider…
==> server1: Importing base box 'ansibletestbase'…
==> server1: Matching MAC address for NAT networking…
==> server1: Setting the name of the VM: blogpost_server1_1554188252201_2080
==> server1: Clearing any previously set network interfaces…
==> server1: Preparing network interfaces based on configuration…
server1: Adapter 1: nat
server1: Adapter 2: hostonly
==> server1: Forwarding ports…
server1: 22 (guest) => 2222 (host) (adapter 1)
==> server1: Running 'pre-boot' VM customizations…
==> server1: Booting VM…
==> server1: Waiting for machine to boot. This may take a few minutes…
server1: SSH address: 127.0.0.1:2222
server1: SSH username: vagrant
server1: SSH auth method: private key
==> server1: Machine booted and ready!
==> server1: Checking for guest additions in VM…
==> server1: Setting hostname…
==> server1: Configuring and enabling network interfaces…
server1: SSH address: 127.0.0.1:2222
server1: SSH username: vagrant
server1: SSH auth method: private key
==> server1: Mounting shared folders…
server1: /vagrant => /home/martin/vagrant/blogpost
==> server1: Running provisioner: ansible…

Vagrant has automatically selected the compatibility mode '2.0'according to the Ansible version installed (2.7.7).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode

server1: Running ansible-playbook...

PLAY [blogpost] *************************************************************

TASK [Gathering Facts] ******************************************************
ok: [server1]

TASK [say hello] ************************************************************
ok: [server1] => {
"msg": "hello from server1"
}

TASK [partitioning PVs for the volume group] ********************************
changed: [server1]

PLAY RECAP ******************************************************************
server1 : ok=3 changed=1 unreachable=0 failed=0

Great! But I forgot to partition /dev/sd[cd] in the same way as I partition /dev/sdb! That’s a quick fix:

---
- name: blogpost
  hosts: oracle_si
  vars:
  - oravg_pv: /dev/sdb
  - asm_disks:
      - /dev/sdc
      - /dev/sdd
  become: yes
  tasks:
  - name: say hello
    debug: msg="hello from {{ ansible_hostname }}"

  - name: partitioning PVs for the Oracle volume group
    parted:
      device: "{{ oravg_pv }}"
      number: 1
      state: present
      align: optimal
      label: gpt

  - name: partition block devices for ASM
    parted:
      device: "{{ item }}"
      number: 1
      state: present
      align: optimal
      label: gpt
    loop: "{{ asm_disks }}"

Re-running the provisioning script couldn’t be easier. Vagrant has a command for this: “vagrant provision”. This command re-runs the provisioning code against a VM. A quick “vagrant provision” later my system is configured exactly the way I want:

$ vagrant provision
==> server1: Running provisioner: ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.7.7).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode

    server1: Running ansible-playbook...

PLAY [blogpost] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [server1]

TASK [say hello] ***************************************************************
ok: [server1] => {
    "msg": "hello from server1"
}

TASK [partitioning PVs for the Oracle volume group] ****************************
ok: [server1]

TASK [partition block devices for ASM] *****************************************
changed: [server1] => (item=/dev/sdc)
changed: [server1] => (item=/dev/sdd)

PLAY RECAP *********************************************************************
server1                    : ok=4    changed=1    unreachable=0    failed=0   

This is it! Using just a few commands I can spin up VMs, test my Ansible scripts and later on when I’m happy with them, check the code into source control.

Ansible tips’n’tricks: understanding your Ansible configuration

When writing automation scripts I tend to use a local Ansible configuration file. This has certain advantages for me, such as including it in a version control system (VCS). It also is a valid option for developers without access to the global configuration file installed by the package manager. And more convenient to use than setting environment variables.

WARNING: There are some very important security considerations though, which you must be aware of before using a local configuration file.

Until now I haven’t spent a lot of time thinking about configuration variables and the order of precedence, but that is exactly what I’d like to do in this post.

When putting this short article together I used Ansible 2.7.6 on Oracle Linux 7.6

What does the documentation have to say?

The official documentation does a really good job explaining the configuration settings. You have the choice of:

  • Environment variable
  • configuration file (ansible.cfg) in your local directory
  • A playbook-independent configuration file in ~/.ansible.cfg
  • The global settings in /etc/ansible/ansible.cfg

Let’s try this

This is an example of a very minimalist project folder in my lab, deliberately omitting a local configuration file.

[vagrant@server1 ansible]$ ls -la
total 16
drwxrwxr-x. 3 vagrant vagrant   53 Feb 14 09:09 .
drwx------. 5 vagrant vagrant 4096 Feb 14 08:40 ..
drwxrwxr-x. 7 vagrant vagrant 4096 Feb 14 09:09 .git
-rw-rw-r--. 1 vagrant vagrant   17 Feb 14 09:09 .gitignore
-rw-rw-r--. 1 vagrant vagrant   94 Feb 14 08:40 hello.yml

The example is kept super-short on purpose.

Precedence

Without a local configuration file (~/.ansible.cfg or $(pwd)/ansible.cfg) and/or defined environment variables, the settings made by your friendly system administrator in /etc/ansible/ansible.cfg govern every execution. You can use ansible-config to view the non-default values:

[vagrant@server1 ansible]$ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/etc/ansible/ansible.cfg) = True
[vagrant@server1 ansible]$ 

As you can see, the global configuration file prevents the use of cowsay.

Now if I as an automation developer wanted to override a certain setting, I could use an environment variable. The documentation tricked me for a moment – when you see “DEFAULT_LOG_PATH” for example, exporting that variable does not have any effect. What you need to use is shown in the same section, but further down, with the Environment: label. To set the default log path on your shell, use ANSIBLE_LOG_PATH as in this example:

[vagrant@server1 ansible]$ export ANSIBLE_LOG_PATH=/home/vagrant/ansible/logs/my_ansible.log
[vagrant@server1 ansible]$ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/etc/ansible/ansible.cfg) = True
DEFAULT_LOG_PATH(env: ANSIBLE_LOG_PATH) = /home/vagrant/ansible/logs/my_ansible.log

Thankfully ansible-config shows me the origin for each setting.

Using a local configuration file

Now how does a local configuration file play into this? Let’s try! I keep the global configuration file as it is, but I unset the environment variable. Here is the result:

[vagrant@server1 ansible]$ unset ANSIBLE_LOG_PATH
[vagrant@server1 ansible]$ cat ansible.cfg 
[defaults]

stdout_callback = debug
log_path = /home/vagrant/ansible/logs/ansible_blogpost.log
[vagrant@server1 ansible]$ ansible-config dump --only-changed
DEFAULT_LOG_PATH(/home/vagrant/ansible/ansible.cfg) = /home/vagrant/ansible/logs/ansible_blogpost.log
DEFAULT_STDOUT_CALLBACK(/home/vagrant/ansible/ansible.cfg) = debug
[vagrant@server1 ansible]$ 

What’s interesting is that my “nocows” setting from the global configuration file in /etc/ansible/ansible.cfg isn’t merged into the configuration. Without the use of environment variables, the only settings coming into play are those of the local configuration file. The same seems to apply if a local configuration file exists in addition to ~/.ansible.cfg. The file in the current working directory always took precedence in my testing.

This did not affect environment variables, they have always been considered by ansible-config. If for example I wanted to temporarily save the logfile to a different place and was too lazy to fire up vim, I could use this approach
[vagrant@server1 ansible]$ ANSIBLE_LOG_PATH=/home/vagrant/ansible/logs/overriding.log ansible-config dump --only-changed
DEFAULT_LOG_PATH(env: ANSIBLE_LOG_PATH) = /home/vagrant/ansible/logs/overriding.log
DEFAULT_STDOUT_CALLBACK(/home/vagrant/ansible/ansible.cfg) = debug
[vagrant@server1 ansible]$ 
Happy scripting!

Building your own local Oracle Linux 7 Vagrant base box

I have been talking about Vagrant for a long time and use it extensively on my Ubuntu-powered laptop. I am using Oracle Linux 7.6 for most of my lab builds, and I like to have specific tools such as collectl, perf, and many others available when the VM boots. I als like to stay in control of things, especially when it comes to downloading otherwise unknown things from the Internet I decided to learn how to create a Vagrant box myself.

Using Vagrant with my custom images, all I need to do is run a single command and it will spin up a clean VM using the VirtualBox provider with the exact software configuration I want. I can also supply so-called provisioners to further configure my environment. I found this particularly useful when writing and testing Ansible scripts. Sometimes I just wanted to go back to my starting point but that can be tricky at times: imagine you just partitioned your block devices for use with the database and discovered you wanted to change the flow. Getting back to unpartitioned, unformatted block devices is possible, but I don’t think it’s terribly elegant. Plus I have to manually do it, and I prefer the Ansible approach.

Building a base box

The Vagrant documentation is pretty good, so this is mostly pulling together information from 2 sources: The starting point I used was Creating a Base Box with specifics for the VirtualBox driver I’m using. I don’t claim I’m an expert in this field.

Running Vagrant VMs can be inherently insecure as you will see in a bit. It’s fine for me because I’m creating/trashing short-lived VMs on a regular basis and all I do is play around with them whilst they remain perfectly isolated from the rest of the world. If you are ok with this limitation feel free to read on, otherwise please refrain from following the steps in this blog post.

The overall process isn’t too hard to follow:

  • Create your gold image
    • Install the Operating System in VirtualBox
    • Install/upgrade any software you want to have available
    • Configure the system for Vagrant specifics
  • Create a base box off your gold image
  • Add the box to your environment
  • Start the VM and enjoy

Creating the VM and installing the Operating System

The first step obviously is to create the VM and install the operating system. For quite some time now I’m creating a VM with sufficient RAM and a couple of block devices: the first one is used as the root volume group, the second block device will be used for Oracle. Plenty of articles have been written about installing Oracle Linux on VirtualBox, I won’t write the 42nd variation here ;)

There are only a few things to pay attention to. These can all be found in the documentation I referenced earlier. First of all, please ensure that your network adaptor uses NAT. You can use port forwarding to access a NAT device in VirtualBox (configured later). The documentation furthermore recommends removing any necessary components such as USB and audio from the VM. I have used a strong password for “root” as I have no intention at all of sharing my VM. Apply security hardening at this stage.

A common error is not to enable the network device to start up automatically when the system boots. Vagrant uses port-forwarding to the NAT device and SSH keys to authenticate, there doesn’t appear to be a mechanism circumventing the network stack. With the network interface down it’s quite hard to connect via SSH.

Install/upgrade software

Once the operating system is installed and the VM rebooted, it’s time to configure it for your needs. I usually end up completing the pre-requisites for an Oracle database installation. This, too, has been covered so many times that I don’t feel like adding value by telling you how to complete the steps.

Configure the system for Vagrant

At this stage your VM should be properly configured for whichever purpose you have in mind. All that remains now is the addition of the specific configuration for Vagrant. There are a few steps to this, all to be completed on the guest.

Install VirtualBox Guest Additions

Vagrant offers the option of mounting a file system from your host on the guest VM. I very much like this feature, which is enabled by default. Please refer to the Vagrant documentation for security implications of sharing file systems between guest and host.

As with every VirtualBox VM, shared folders won’t work without installing the guest additions though so that’s what I do next. This is pretty straight forward and for Oracle Linux 7 generally speaking requires tar, bzip2, gcc and kernel-uek-devel matching your current kernel-uek. If you just completed a “yum upgrade” and your kernel was upgraded you need to reboot first. After VBoxLinuxAdditions.run has completed successfully (I am using VirtualBox 5.2.x) it’s time to move on to the next step.

Add a Vagrant user

Vagrant expects a user named vagrant to be present on the VM. It uses SSH-keys when connecting to the VM. The documentation mentions a so-called insecure key-pair I decided not to use. Instead, I created my own key pair for use with the machine and added it to ~/.ssh/authorized_keys in the vagrant user’s home directory. It is a new keypair I created on the host specifically for use with Vagrant. If you are on MacOS or Linux it’s convenient to add it to the SSH agent (ssh-add …). There are similar tools for Windows users.

Creating the user is easy and should be completed now unless you already created the user during the initial installation:

# useradd -c 'vagrant user' -m -s $(which bash) vagrant 

The user should have passwordless sudo enabled as well as per the documentation. It is also recommended by the Vagrant documentation to assign a weak password to the vagrant account, which I didn’t. I never ran the passwd command to set a password for the vagrant user and so far seem to be doing ok.

Create a base box

This concludes the preparations on the VM side. Next up you need to create the base box, which you can then refer to in your own Vagrantfile. The command to do so is just one line. Be careful though: it will create a compressed file named package.box in your current working directory. This file can be rather large, so make sure you have enough space to store it.

$ vagrant package --base <your newly created VM name>

Depending on how powerful your laptop is this can take a little while.

Add the box to your environment

The previous command will complete eventually. This is the moment where you add the box to Vagrant’s local inventory as shown here:

$ vagrant box add --name blogpost /home/martin/package.box 

This command shouldn’t take too long to complete. If you see a line “box: successfully added box ‘blogpost’ (v0) for ‘virtualbox’ you are good. You can assign any name to the box you add, it will alter on show up under that designation when you run “vagrant box list”

Start the VM and enjoy

The remaining tasks are identical to using Vagrant boxes off their repository. Start off by vagrant init <your box name> and make all the changes you normally do to the Vagrantfile. As I’m using my own SSH key I have to make sure that I’m telling Vagrant where to find it using a configuration option:

config.ssh.private_key_path = "/path/to/ssh/keyfile" 

Once you start the VM using “vagrant up” you are good to go!

Installing Ansible on Oracle Linux 7 for test and development use

There are a few alternative ways of installing Ansible on Linux, and the install guide for Ansible 2.7 (the current version at the time of writing) does a great job in explaining them all in detail.  There is a potentially easier way to get to a current Ansible version if you are using Oracle Linux 7, but it comes with a very important limitation. Let’s get that out of the way first.

You need to be aware that the RPM-based installation of Ansible as described in this blog post requires you to enable Oracle’s EPEL repository. As per https://yum.oracle.com/oracle-linux-7.html, the EPEL repository is listed under “Packages for Test and Development” (bottom of the page) and these come with the following warning:  Note: The contents in the following repositories are for development purposes only. Oracle suggests these not be used in production. 

This is really important!

If you are ok with the limitation I just quoted from Oracle’s YUM server, please read on. If not, head back to the official Ansible documentation and use a different method instead. I only use Ansible in my own lab and therefore don’t mind.

Updating the repository configuration file

Back to the topic of Ansible … Before I get around to install Ansible on my machines I update my yum repository configuration file. Things are changing quickly, and I found /etc/yum.repos.d/public-yum-ol7.repo to be outdated at times. I always refresh it from yum.oracle.com just to be sure I’m not missing out on the new stuff.

# cd /etc/yum.repos.d
# mv -iv public-yum-ol7.repo public-yum-ol7.repo.$(date +%y%m%d)
# wget http://yum.oracle.com/public-yum-ol7.repo

With the new file in place, use your preferred method to enable the ol7_developer_EPEL repository. I simply edit public-yum-ol7.repo, there are other ways like yum-config-manager getting you there. But be advised: I just noticed the from December 17 at least the UEK Release 5 repository might be enabled by default leading to a potentially unwanted kernel upgrade.

Install Ansible

With the developer EPEL repository enabled, you have access to a great many Ansible versions. At the time of writing, these were available:

# yum --showduplicates list ansible
Loaded plugins: ulninfo
Available Packages
ansible.noarch                      2.3.1.0-1.el7                       ol7_developer_EPEL
ansible.noarch                      2.4.2.0-1.el7                       ol7_developer_EPEL
ansible.noarch                      2.5.0-2.el7                         ol7_developer_EPEL
ansible.noarch                      2.5.1-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.5.2-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.6.1-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.6.2-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.6.4-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.6.5-1.el7                         ol7_developer_EPEL
ansible.noarch                      2.7.0-1.el7                         ol7_developer_EPEL

# yum info ansible
Loaded plugins: ulninfo
Available Packages
Name        : ansible
Arch        : noarch
Version     : 2.7.0
Release     : 1.el7
Size        : 11 M
Repo        : ol7_developer_EPEL/x86_64
Summary     : SSH-based configuration management, deployment, and task execution system
URL         : http://ansible.com
Licence     : GPLv3+
Description : Ansible is a radically simple model-driven configuration management,
            : multi-node deployment, and remote task execution system. Ansible works
            : over SSH and does not require any software or daemons to be installed
            : on remote nodes. Extension modules can be written in any language and
            : are transferred to managed machines automatically.

Happy testing!

Terraforming the Oracle Cloud: choosing and using an image family

For a few times now I have presented about “cloud deployments done the cloud way”, sharing lessons learned in the changing world I find myself in. It’s a lot of fun and so far I have been far too busy to blog about things I learned by trial and error. Working with Terraform turned out to be a very good source for blog posts, I’ll put a few of these up in the hope of saving you a few minutes.

This blog post is all about creating Ubuntu images in Oracle Cloud Infrastructure (OCI) using terraform. The technique is equally applicable for other Linux image types though. In case you find this post later using a search engine, here is some version information that might put everything into context:

$ ./terraform version
Terraform v0.11.10
+ provider.null v1.0.0
+ provider.oci v3.7.0

I used the “null” provider to run a few post-installation commands as shown in various terraform examples for OCI. Right now I’m trying to work out if I can’t do the same in a different way. If I am successful you can expect a blog post to follow…

Creating a Ubuntu 18.04 LTS image in OCI

To create the Ubuntu image (or any other image for that matter), I need information about the image family. Documentation about image families in OCI can be found at https://docs.cloud.oracle.com/iaas/images/.

Scrolling down/selecting the entry from the left hand side I found the link to the Ubuntu 18.04 LTS image family. Each supported image has its own documentation link, containing crucial data: an OCID per location. At the time of writing, the latest Ubuntu image was named Canonical-Ubuntu-18.04-2018.11.17-0 and had an image OCID of ocid1.image.oc1.eu-frankfurt-1.aaaa...i57q7bfsa. An OCID is short for Oracle Cloud Identifier and it’s used in many places in OCI. There are different OCIDs for the image depending on location; the (shortened) OCID I just showed you was for Frankfurt.

With the OCID at hand, I can open my favourite code editor and start putting the terraform script together. I create instances in OCI using the oci_core_instance type, documented at the terraform website.

Be careful, many of the references and code examples I found about oci_core_image are written for older versions of the terraform provider. I noticed some attributes used in the examples are deprecated. It might be useful to compare the source code examples against the current documentation

Part of the definition of an oci_core_instance requires the specification of the operating system in the source_details {} section. To create the Ubuntu VM in the Frankfurt region, I have to specify – amongst other things of course – this:

resource "oci_core_instance" "docker_tf_instance" {
...
    source_details {
        source_type = "image"
        source_id   = "ocid1.image.oc1.eu-frankfurt-1.aaaa..."
...
    }
...

The actual OCID is far longer, the example above is shortened for the sake of readability. I didn’t like it wrapping around the text box and thus destroying my layout. Make sure you use the correct OCID ;)

With the information at hand I can create the Ubuntu VM and connect to it using the specified SSH key. Have fun!

Ansible tips’n’tricks: executing related tasks together

I have recently written an ansible playbook to apply one-off patches to an Oracle Home. While doing this, I hit a little snag that needed ironing out. Before continuing this post, it’s worth pointing out that I’m on:

$ ansible --version
ansible 2.6.5

And it’s Ansible on Fedora.

Most likely the wrong way to do this …

So after a little bit of coding my initial attempt looked similar to this:

$ cat main.yml 
---
- hosts: 127.0.0.1
  connection: local
  vars:
    - patchIDs:
        - 123
        - 234
        - 456

  tasks:
  - name: say hello
    debug: msg="hello world"

  - name: unzip patch 
    debug: 
      msg: unzipping patch {{ item }}
    loop: "{{ patchIDs }}"
  
  - name: check patch for conflicts with Oracle Home
    debug: 
      msg: checking patch {{ item }} for conflict with $ORACLE_HOME
    loop: "{{ patchIDs }}"
  
  - name: apply patch
    debug: 
      msg: applying patch {{ item }} to $ORACLE_HOME
    loop: "{{ patchIDs }}"
    failed_when: "item == 456"

This is a stub of course … I have stripped any non-essential code from the playbook but it should give you a first idea.

Can you spot the bug

This looks ok-ish, but there’s a (not quite so hidden) bug in there. And no, I didn’t have a failed_when condition in my playbook that would always evaluate to true with this input :) Consider this output:

$ ansible-playbook main.yml 

PLAY [127.0.0.1] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [127.0.0.1]

TASK [say hello] ***************************************************************************
ok: [127.0.0.1] => {}

MSG:

hello world


TASK [unzip patch] *************************************************************************
ok: [127.0.0.1] => (item=123) => {}

MSG:

unzipping patch 123

ok: [127.0.0.1] => (item=234) => {}

MSG:

unzipping patch 234

ok: [127.0.0.1] => (item=456) => {}

MSG:

unzipping patch 456


TASK [check patch for conflicts with Oracle Home] ******************************************
ok: [127.0.0.1] => (item=123) => {}

MSG:

checking patch 123 for conflict with $ORACLE_HOME

ok: [127.0.0.1] => (item=234) => {}

MSG:

checking patch 234 for conflict with $ORACLE_HOME

ok: [127.0.0.1] => (item=456) => {}

MSG:

checking patch 456 for conflict with $ORACLE_HOME


TASK [apply patch] *************************************************************************
ok: [127.0.0.1] => (item=123) => {}

MSG:

applying patch 123 to $ORACLE_HOME

ok: [127.0.0.1] => (item=234) => {}

MSG:

applying patch 234 to $ORACLE_HOME

failed: [127.0.0.1] (item=456) => {}

MSG:

applying patch 456 to $ORACLE_HOME

fatal: [127.0.0.1]: FAILED! => {}

MSG:

All items completed


PLAY RECAP *********************************************************************************
127.0.0.1                  : ok=4    changed=0    unreachable=0    failed=1

Whoops, that went wrong! As you would predict, the last task failed.

The simulated conflict check is performed for each patch, before the patch is applied in the next step. And this is where the bug in the code can hit you. Imagine you are on a recent PSU, let’s say 180717. The playbook:

  1. Checks patch 123 for incompatibilities with PSU 180717
  2. Followed by patch# 234 …
  3. and eventually 456.

No issues are detected. The next step is to apply patch 123 on top of our fictional PSU 180717, followed by 234 on top of 180717 plus 123, and so on. When it comes to patch 456, a conflict is detected: opatch tells you that you can’t apply 456 on top of 180717 + 234 … I have simulated this with the failed_when clause. The bug in this case is my playbook failing to detect a conflict before actually trying to apply a patch.

I need a procedure!

So what now? In a shell script, I would have defined a function, maybe called it apply_patch. It’s task include the unzipping, checking pre-requisites, and eventually the call to opatch apply. In the body of the script I would have looped over all patches to apply and called apply_patch() for each patch to be applied. In other words unzip/check prerequisites/apply are always performed for a given patch before the code advances to the next patch in sequence.

But how can this be done in Ansible? A little bit of research (and a conversation with @fritshoogland who confirmed what I thought was to be changed) later I noticed that you can include tasks and pass variables to them. So what if I rewrote my code to take advantage of that feature? Here’s the end result:

$ cat main.yml 
---
- hosts: 127.0.0.1
  connection: local
  vars:
    - patchIDs:
        - 123
        - 234
        - 456

  tasks:
  - name: say hello
    debug: msg="hello world"

  - name: include a task
    include_tasks: includedtask.yml
    loop: "{{ patchIDs }}"

Note that I’m using the loop syntax recommended from Ansible 2.5 and later (see section “Migrating from with_X to loop”).

The include file references the variable passed as {{ item }}.

$ cat includedtask.yml 
---
- name: unzip patch {{ item }}
  debug: 
    msg: unzipping patch {{ item }}

- name: check patch {{ item }} for conflicts with Oracle Home
  debug: 
    msg: checking patch {{ item }} for conflict with $ORACLE_HOME

- name: apply patch {{ item }}
  debug: 
    msg: applying patch {{ item }} to $ORACLE_HOME
  failed_when: "item == 456"

Now if I run this playbook, each task (unzip/check prerequisites/apply) is executed for each individual patch.

$ ansible-playbook main.yml

PLAY [127.0.0.1] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [127.0.0.1]

TASK [say hello] ***************************************************************************
ok: [127.0.0.1] => {}

MSG:

hello world


TASK [include a task] **********************************************************************
included: /home/martin/ansible/blogpost/including_task/better/includedtask.yml for 127.0.0.1
included: /home/martin/ansible/blogpost/including_task/better/includedtask.yml for 127.0.0.1
included: /home/martin/ansible/blogpost/including_task/better/includedtask.yml for 127.0.0.1

TASK [unzip patch 123] *********************************************************************
ok: [127.0.0.1] => {}

MSG:

unzipping patch 123


TASK [check patch 123 for conflicts with Oracle Home] **************************************
ok: [127.0.0.1] => {}

MSG:

checking patch 123 for conflict with $ORACLE_HOME


TASK [apply patch 123] *********************************************************************
ok: [127.0.0.1] => {}

MSG:

applying patch 123 to $ORACLE_HOME


TASK [unzip patch 234] *********************************************************************
ok: [127.0.0.1] => {}

MSG:

unzipping patch 234


TASK [check patch 234 for conflicts with Oracle Home] **************************************
ok: [127.0.0.1] => {}

MSG:

checking patch 234 for conflict with $ORACLE_HOME


TASK [apply patch 234] *********************************************************************
ok: [127.0.0.1] => {}

MSG:

applying patch 234 to $ORACLE_HOME


TASK [unzip patch 456] *********************************************************************
ok: [127.0.0.1] => {}

MSG:

unzipping patch 456


TASK [check patch 456 for conflicts with Oracle Home] **************************************
ok: [127.0.0.1] => {}

MSG:

checking patch 456 for conflict with $ORACLE_HOME


TASK [apply patch 456] *********************************************************************
fatal: [127.0.0.1]: FAILED! => {}

MSG:

applying patch 456 to $ORACLE_HOME


PLAY RECAP *********************************************************************************
127.0.0.1                  : ok=13   changed=0    unreachable=0    failed=1 

This way, I should be able to stop the playbook as soon as the pre-requisite conflict checker has completed.