Category Archives: DevOps

Vagrant Ansible Provisioner: working with the Ansible Inventory

Vagrant and Ansible are a great match: using Vagrant it’s very easy to work with virtual machines. Creating, updating, and removing VMs is just a short command away. Vagrant provides various provisioners to configure the VM, and Ansible is one of these. This article covers the ansible provisioner as opposed to ansible_local.

Earlier articles I wrote might be of interest in this context:

The post was written using Ubuntu 22.04 patched to 230306, I used Ansible and Vagrant as provided by the distribution:

  • Ansible 2.10.8
  • Vagrant 2.2.19

Configuring the Ansible Inventory

Very often the behaviour of an Ansible playbook is controlled using variables. Providing variables to Ansible from a Vagrantfile is quite neat and subject of this article.

Let’s have a look at the most basic Vagrantfile:

Vagrant.configure("2") do |config|
  
  config.vm.box = "debianbase"
  config.vm.hostname = "debian"

  config.ssh.private_key_path = "/home/martin/.ssh/debianbase"

  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 2
    vb.memory = "2048"
    vb.name = "debian"
  end
  
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "provisioning/blogpost.yml"
    ansible.verbose = "v"
  end
end

I frequently use a flag indicating if the Ansible script should reboot the VM after the update of all packages completed. Within the provisioning folder I store group_vars, roles, and the main playbook as per the recommendation in the docs:

$ tree provisioning/
provisioning/
├── blogpost.yml
├── group_vars
│   └── all.yml
└── roles
    └── role1
        └── tasks
            └── main.yml

All global variables I don’t necessarily expect to change are stored in group_vars/all.yml. This includes the reboot_flag flag that defaults to false. The playbook does not need to list the variable in its own vars section, in fact doing so would grant the variable a higher precedence and my way of providing a variable to Ansible via Vagrant would fail. Here is the playbook:

- hosts: default
  become: true

  tasks: 
  - debug:
      var: reboot_flag

  - name: reboot
    ansible.builtin.reboot:
    when: reboot_flag | bool

Since rebooting can be a time consuming task I don’t want to do this by default, which is fine by me as I understand that I have to reboot manually later.

Let’s see what happens when the VM is provisioned:

PLAY [default] *****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [default]

TASK [debug] *******************************************************************
ok: [default] => {
    "reboot_flag": false
}

TASK [reboot] ******************************************************************
skipping: [default] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}

PLAY RECAP *********************************************************************
default                    : ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Overriding variables

In case I want to override the flag I can do so without touching my Ansible playbook only by changing the Vagrantfile. Thanks to host_vars I can pass variables to Ansible via the inventory. Here’s the changed section in the Vagrantfile:

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "provisioning/blogpost.yml"
    ansible.verbose = "v"
    ansible.host_vars = {
      "default" => {
        "reboot_flag" => true
      }
    }
  end

All host_vars for my default VM are then appended to the inventory in .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory.

Next time I run vagrant provision the flag is changed to true, and the VM is rebooted:

PLAY [default] *****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [default]

TASK [debug] *******************************************************************
ok: [default] => {
    "reboot_flag": "true"
}

TASK [reboot] ******************************************************************
changed: [default] => {
    "changed": true,
    "elapsed": 20,
    "rebooted": true
}

PLAY RECAP *********************************************************************
default                    : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Summary

Vagrant offers a very neat way of creating an Ansible inventory on the fly. If your Ansible playbooks are written in a way that different execution paths/options are configurable via variables a single playbook is highly flexible and can be used for many things. In the age of version control it’s very convenient not having to touch the source code of an Ansible playbook as that might interfere with other projects. Variables, passed at runtime, are much better suited to create flexible automation scripts.

Advertisement

Avoiding pitfalls when using cURL in CI/CD pipelines

Continuous Integration/Continuous Delivery (or Deployment, depending on your point of view) pipelines are at the core of many successful software projects. When designing your pipelines you sooner or later end up using REST calls to perform certain tasks. cURL is a popular command line tool to invoke REST APIs, and is commonly used in pipelines. Before you start using cURL in that capacity I’d like to draw your attention to a potential pitfall you can run into.

CI/CD Pipelines

A CI/CD pipeline typically consists of a series of tasks executed after a (git) commit is pushed to the remote registry. The idea is to ensure compliance with coding standards, formatting, and code quality, amongst a great wealth of other things. A pipeline is typically sub-divided into stages such as “build”, “lint”, “deploy” or anything else you can think of. Each stage consists of one or more tasks.

Whether or not the pipeline progresses to the next stage depends on the success or failure of tasks. Return codes are usually used to determine success or failure: a return code of 0 implies success, everything else usually terminates the pipeline’s execution.

Experimenting with cURL Exit Codes

In order to use cURL effectively in a CI pipeline it’s important to understand its error codes. Consider the following simulated API using node and express.js:

import express from 'express'
const app = express()
const port = 8080
const host = '0.0.0.0'

// allow for a successful test
app.get('/', (req, res) => {
  res.set('Content-Type', 'text/plain')
  res.send('test successful')
})

// invoke this URL to provoke a HTTP 400 (bad request) error
// see https://expressjs.com/en/4x/api.html#res.set for details
app.get('/failure', (req, res) => {
  res.set('Content-Type', 'text/plain')
  res.status(400).send('Bad Request')
})

app.listen(port, host, () => {
  console.log(`Simulated API server available on ${host}:${port}!`)
})

I created a small container image with the above code using the node:lts image (that’s node 18.14.2 and express 4.18.2, the most current versions at the time of writing, Feb 25th) and ran it.

“But what about security?” I hear you ask. You will undoubtedly have noted that this isn’t production code, it lacks authentication and other security features, logging, and basically everything apart from returning a bit of text and a HTTP status code. I’m also going to use HTTP calls for the API – enabling HTTPS would have been overkill for my example. In the real world you wouldn’t run APIs without TLS protection, would you? Since this post is about HTTP status codes and cURL in CI pipelines none of the extra bells and whistles are necessary, and crucially they’d probably distract from the actual problem. If you’re coding your APIs you should always adhere to industry best practices!

Starting the container

I stated the container as follows:

podman run --rm -it --name some-api --publish 8080:8080 api:0.5

The CMD directive in the project’s Dockerfile starts node and passes the api.mjs file to it. The API is now ready for business:

Simulated API server available on 0.0.0.0:8080!

Scenario 1: normal, successful completion

Let’s start with the successful invocation of the simulated API:

$ curl http://localhost:8080
test successful
$ echo $?
0

OK, nothing to see here, moving on… This is what was expected and shown for reference ;)

Scenario 2: Bad Request

I’m pointing curl to http://localhost:8080/failure next:

$ curl http://localhost:8080/failure
Bad Request
$ echo $?
0

Hmm, so that’s odd, curl‘s return code is 0 (= success) despite the error? Let’s dig a little deeper by using the verbose option and returning the headers

$ curl -iv http://localhost:8080/failure
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
> GET /failure HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< X-Powered-By: Express
X-Powered-By: Express
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Content-Length: 11
Content-Length: 11
< ETag: W/"b-EFiDB1U+dmqzx9Mo2UjcZ1SJPO8"
ETag: W/"b-EFiDB1U+dmqzx9Mo2UjcZ1SJPO8"
< Date: Sat, 25 Feb 2023 11:34:16 GMT
Date: Sat, 25 Feb 2023 11:34:16 GMT
< Connection: keep-alive
Connection: keep-alive
< Keep-Alive: timeout=5
Keep-Alive: timeout=5

< 
* Connection #0 to host localhost left intact
Bad Request

So it’s pretty clear that the HTTP status code is 400 (Bad Request). But that’s not reflected in the return code. Let’s fix this!

Instructing cURL to fail

A look at the cURL manual page reveals this interesting option:

       -f, --fail
              (HTTP) Fail silently (no output at all) on server  errors.  This
              is  mostly done to enable scripts etc to better deal with failed
              attempts. In normal cases when an HTTP server fails to deliver a
              document,  it  returns  an HTML document stating so (which often
              also describes why and more). This flag will prevent  curl  from
              outputting that and return error 22.

              This  method is not fail-safe and there are occasions where non-
              successful response codes will slip through, especially when au‐
              thentication is involved (response codes 401 and 407).

Which looks like exactly what I need. Let’s try this option:

$ curl --fail http://localhost:8080/failure
curl: (22) The requested URL returned error: 400 Bad Request
$ echo $?
22

Well that’s better! There’s a non-zero return code now.

Summary

The --fail option in curl (or --fail-with-body if your version of curl is 7.76 or later) allows DevOps engineers to architect their pipelines with greater resilience. Rather than manually parsing the cURL output checking for errors you can now rely on the REST API call’s return code to either proceed with the pipeline or stop execution. Please note that the –fail option isn’t fail-safe, as per the above comment in the man-page. Neither does it protect you from an API returning a HTTP-200 code if in fact an error occurred. But it’s definitely something I’ll use from now on by default.