Ansible Dynamic Inventory Plugin for OCI: where are all my hosts?

I wrote about the configuration of the Ansible Dynamic Inventory Plugin for Oracle Cloud Infrastructure (OCI) in a previous post. As it seems to happen all the time, the length of the post escalated quickly. When I finished the previous post there was still a lot to say! For example, I wanted to share the answer to the question where all my hosts were hiding :)

Caveat

As I said in the previous post, the Dynamic Inventory Plugin is a great time saver, especially when used as part of build pipelines. However, as with anything that’s misconfigured and doesn’t adhere to best practices, there is a risk associated. You might run playbooks against hosts you didn’t intend to. Ensure you have proper Identity and Access (IAM) policies in place, and use the principle of last privilege throughout. And use tags to filter hosts, as you see in this and the previous example. And finally, use the ansible toolset to validate the hosts you target.

I can’t see all my cloud VMs!

After I have been able to resolve the initial problems with the Dynamic Inventory Plugin I was a bit surprised to see it report fewer hosts than expected. At the time I had used the following configuration (YAML) file after piecing the various examples together:

vagrant@ocidev:~/ansible$ cat ougdemo-compartment.oci.yml 
plugin: oracle.oci.oci

regions:
- eu-frankfurt-1

filters:
- defined_tags: { "project": { "name": "simple-app" } }

compartments:
- compartment_ocid: "ocid1.compartment...."
  fetch_hosts_from_subcompartments: true 

This should show me all the hosts of my simple-app project in the compartment I indicated. However the output did not match the number of VMs I had deployed:

vagrant@ocidev:~/ansible$ ansible-inventory -i ougdemo-compartment.oci.yml --graph

[...]

@all:
  |--@IHsr_EU-FRANKFURT-1-AD-2:
  |  |--130.61.233.113
  |--@Oracle-Tags#CreatedBy=ougdemouser:
  |  |--130.61.233.113
  |--@Oracle-Tags#CreatedOn=2020-11-11T18_01_04.484Z:
  |  |--130.61.233.113
  |--@all_hosts:
  |  |--130.61.233.113
  |--@ougdemo-department:
  |  |--130.61.233.113
  |--@project#name=simple-app:
  |  |--130.61.233.113
  |--@region_eu-frankfurt-1:
  |  |--130.61.233.113
  |--@tag_role=bastionhost:
  |  |--130.61.233.113
  |--@ungrouped:
vagrant@ocidev:~/ansible$ 

Although there is a lot of output, actually there is just 1 IP address (VM) discovered. Despite the fact I had a few more hosts created as shown by the OCI CLI:

$ oci compute instance list \
-c ${COMPARTMENT} \
--query "data[].{name:\"display-name\",state:\"lifecycle-state\",tags:\"defined-tags\"}" \
--output table
+------------+---------+---------------------------------------------------------------------------------------------------------------------------+
| name       | state   | tags                                                                                                                      |
+------------+---------+---------------------------------------------------------------------------------------------------------------------------+
| appserver1 | RUNNING | {'project': {'name': 'simple-app'}, 'Oracle-Tags': {'CreatedBy': 'ougdemouser', 'CreatedOn': '2020-11-11T18:01:03.639Z'}} |
| bastion1   | RUNNING | {'project': {'name': 'simple-app'}, 'Oracle-Tags': {'CreatedBy': 'ougdemouser', 'CreatedOn': '2020-11-11T18:01:04.484Z'}} |
| appserver2 | RUNNING | {'project': {'name': 'simple-app'}, 'Oracle-Tags': {'CreatedBy': 'ougdemouser', 'CreatedOn': '2020-11-11T18:01:03.916Z'}} |
+------------+---------+---------------------------------------------------------------------------------------------------------------------------+ 

So where are the missing hosts? I knew they were part of my private subnet so after a little bit of digging around the docs, I found this:

> ORACLE.OCI.OCI    (/home/vagrant/.ansible/collections/ansible_collections/oracle/oci/plugins/inventory/oci.py)

        Get inventory hosts from oci. Uses a .oci.yaml (or .oci.yml)
        YAML configuration file.

OPTIONS (= is mandatory):

[ more documentation ]

- hostname_format
        Host naming format to use. Use 'fqdn' to list hosts using the instance's
        Fully Qualified Domain Name (FQDN). These FQDNs are resolvable within the
        VCN using the VCN resolver specified through the subnet's DHCP options.
        Please see https://docs.us-
        phoenix-1.oraclecloud.com/Content/Network/Concepts/dns.htm for more details.
        Use 'public_ip' to list hosts using public IP address. Use 'private_ip' to
        list hosts using private IP address. By default, hosts are listed using
        public IP address.
        [Default: (null)]
        set_via:
          env:
          - name: OCI_HOSTNAME_FORMAT

[ more documentation ]

That explains it – by default only public IPs are returned. Since my app servers are in a private subnet, they don’t have public IPs, and won’t show up.

Fetching all hosts

With this knowledge at hand the solution merely implied changing the configuration file. I chose for it to display the fully qualified domain name (FQDN). Now all my hosts are fetched by the Dynamic Inventory Plugin:

vagrant@ocidev:~/ansible$ ansible-inventory -i ougdemo-compartment.oci.yml --graph

[...]

@all:
  |--@IHsr_EU-FRANKFURT-1-AD-2:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@IHsr_EU-FRANKFURT-1-AD-3:
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |--@Oracle-Tags#CreatedBy=ougdemouser:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@Oracle-Tags#CreatedOn=2020-11-11T18_01_03.639Z:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |--@Oracle-Tags#CreatedOn=2020-11-11T18_01_03.916Z:
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |--@Oracle-Tags#CreatedOn=2020-11-11T18_01_04.484Z:
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@all_hosts:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@ougdemo-department:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@project#name=simple-app:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@region_eu-frankfurt-1:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@tag_role=appserver:
  |  |--appserver1.app.simpleapp.oraclevcn.com
  |  |--appserver2.app.simpleapp.oraclevcn.com
  |--@tag_role=bastionhost:
  |  |--bastion1.bastion.simpleapp.oraclevcn.com
  |--@ungrouped: 

Happy scripting!