How to configure Ansible dynamic inventory for AWS and GCP

This basic guide tells how to use AWS and GCP plugins to retrieve compute/ec2 instances for ansible inventory.

1. Install required modules. AWS required boto3 and botocore, others are for GCP. If you intend to use this only for AWS, install only boto3 and botocore.

pip3 install requests google-auth boto3 botocore

2. Install python-requests:

yum install python3-requests

3. Edit the file /etc/ansible/ansible.cfg and add the required plugins, make sure not to remove the default ones, only append to the list.

[root@satellite ~]# cat /etc/ansible/ansible.cfg | grep -i enable_plugins
enable_plugins = host_list, virtualbox, yaml, constructed, ini, script, gcp_compute, aws_ec2

4. (AWS) Create the dynamic inventory file for AWS, aws_ec2.yaml, with your access and secret keys:

[root@satellite ~]# cat aws_ec2.yaml
--
plugin: aws_ec2
aws_access_key: XXXXXXXXXXX
aws_secret_key: XXXXXXXXXXX
keyed_groups:
 - key: tags
[root@satellite ~]#

5. (GCP) Create the dynamic inventory file for GCP, my.gcp.yaml(the file should end in .gcp.yaml), provide your GCP project ID in this file, we are authenticating to GCP using a serviceaccount and its certs are stored in a file /root/learning. This file is downloadable from GCP, this requires a service account to be created.

[root@instance-1 ~]# cat my.gcp.yaml
plugin: gcp_compute
projects:
 — <Your GCP Project ID here>
auth_kind: serviceaccount
service_account_file: /root/learning.json
keyed_groups:
  - key: labels

6. (GCP only) To create serviceaccount and download the cert file, please follow:

Create a service account
Download JSON credentials

7. Grouping of hosts is done by the list keyed_groups defined in both inventory files. We can group the hosts by anything, it could be labels, zone or project. In this example we have grouped by labels/tags. You can find all valid key values from the output of:

[root@satellite ~]# ansible-inventory — list -i my.gcp.yaml | grep -iw “labels\|project\|zone”
 “labels”: {
 “project”: “learning-314005”,
 “zone”: “asia-south1-c”,

8. Test the configuration, notice hosts grouped by labels/tags:

[root@satellite ~]# ansible-inventory -i aws_ec2.yaml --graph
@all:
  |--@_ansible_aws1:
  |  |--ec2-3-108-53-71.ap-south-1.compute.amazonaws.com
  |--@aws_ec2:
  |  |--ec2-3-108-53-71.ap-south-1.compute.amazonaws.com
  |--@ungrouped:
[root@satellite ~]#
[root@satellite ~]# ansible-inventory -i my.gcp.yaml --graph
@all:
  |--@_ansible_satellite:
  |  |--34.93.21.26
  |--@ungrouped:
[root@satellite ~]#

9. (Inventory variables): With dynamic inventories you cannot define variables in the inventory file directly, for that we need to create group_vars directory having filenames matching the ansible host groups:

[root@satellite ~]# cd playbooks/
[root@satellite playbooks]# ls -ltr *
-rw-r — r — . 1 root root 47 May 27 06:45 ping.yaml
host_vars:
total 0
group_vars:
total 8
-rw-r — r — . 1 root root 31 May 27 06:47 aws_ec2
-rw-r — r — . 1 root root 33 May 27 06:47 _ansible_satellite
[root@satellite playbooks]#
[root@satellite playbooks]# cat group_vars/aws_ec2
---
ansible_ssh_user: ec2-user
[root@satellite playbooks]# cat group_vars/_ansible_satellite
---
ansible_ssh_user: shailendra

10. Lets run ping on all hosts to check if inventory variables are working, note that we can use more than one inventory at a time:

[root@satellite playbooks]# cat ping.yaml
- hosts:
 — all
 tasks:
 — action: ping
[root@satellite playbooks]#
[root@satellite playbooks]# ansible-playbook -i ../aws_ec2.yaml -i ../my.gcp.yaml ping.yaml
PLAY [all] **************************************************************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************************************************
ok: [34.93.21.26]
ok: [ec2–3–108–53–71.ap-south-1.compute.amazonaws.com]
TASK [ping] *************************************************************************************************************************************************
ok: [ec2–3–108–53–71.ap-south-1.compute.amazonaws.com]
ok: [34.93.21.26]
PLAY RECAP **************************************************************************************************************************************************
34.93.21.26 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ec2–3–108–53–71.ap-south-1.compute.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

NOTE: This was run on Centos8 which uses python3 by default, if you intend to do this on a host running python2 by default, you might run into some problems but there is a work-around:

# python3 $(which ansible-inventory) --list -i aws_ec2.yaml
# python3 $(which ansible) -m ping -i aws_ec2.yaml all