Hi there!
Today there are endless options for configuration management. Each one with their advantages and disadvantages. As AWS Engineer and most exclusively working with Open Sources solutions, I’d like to use a tool that can fit my profile and be flexible enough to handle any other OS.
I found on Ansible this answer. Its provides a good option for configuration management due to its simplicity, agentless architecture, and ability to interact easily with your ever-changing, scaling, and dynamic AWS architecture.
There are a lot of tutorials regarding AWS and Ansible as well. But its hard to find something like a ‘read and do yourself’ document based on a common AWS environment: VPC, Public and Private subnets and Ansible orchestrating it all *dynamically*.
That’s the objective here! And let’s start…
I wont describe here the creation of a VPC, subnets, instances, etc. I’m supposing that you can do it. But if you don’t, check this.
Current environment:
- 01 VPC
- 02 subnets (Public / Private)
- 01 NAT instance
- 01 Ansible Server based on ami-1e159872 (Amazon Linux AMI)
- 01 Ansible Client based on ami-1e159872 (Amazon Linux AMI)
The Ansible Client is not really required, but I created to have a target to test at the end.
Security
- 01 Security Group allowing SSH to my IP attached to Ansible Server
- 01 Security Group allowing SSH over the VPC address only
- 01 IAM role attached to Ansible Server
- 01 keypair
Ansible is based on SSH calls, that’s why SSH ports must to be opened internally *ONLY* for instances placed on Private subnet (well.. even if it was world wide opened, no one can reach it out without public ip).
The most important thing here is the IAM role. We are going to use Ansible to manage instances but we don’t want to save our Access Keys on the instance. This is not good.
Instead, we are use IAM Role and attach the role on the Ansible Server instance.
Attention: You cannot attach an IAM Rule to a existing instance. Create it first and attach to the instance during the deploy process.
I created a Role called “Ansible” (yeah, quite predictable name) and attached the “PowerUserAccess” Policy on it:
Then create the instance attaching this Role:
I should not describe this step, but this is critical. If you miss that, all process will fall down 😉
Installing Ansible on Ansible Server
I have tried several ways to install Ansible (pip, yum, git) and honestly the only one that worked well was GIT:
sudo su
yum update
yum install -y git
cd /usr/local/src
yum -y install git python-jinja2 python-paramiko PyYAML make MySQL-python
git clone git://github.com/ansible/ansible.git
cd ansible
git submodule update --init --recursive
make install
CTRL + D
Test it simply calling “ansible-playbooks” command. If you have a list of parameters followed by “Error! You must specify a playbook file to run”, you did it right!
Ansible Dynamic Inventory Management
Ansible configuration requires a file called “hosts” that contains – obviously – the list of hosts that you’ll manage. But we are talking about AWS. Maintaining an inventory file might not be the best approach, because hosts may come and go over time, be managed by external applications, or you might even be using AWS autoscaling. So… Things can change horizontally anytime. Would be possible to manage dynamically the hosts, wouldn’t?
For this reason, we’ll use the EC2 external inventory script. To get started with dynamic inventory management, you’ll need to grab the EC2.py script and the EC2.ini config file. The EC2.py script is written using the Boto EC2 library and will query AWS for your running Amazon EC2 instances. The EC2.ini file is the config file for EC2.py, and can be used to limit the scope of Ansible’s reach.
Create the “/ansible/inventory/” directory on Ansible Server and give to it the right permissions and download the inventory files and set the permission to the script:
sudo mkdir /ansible
sudo chown -R ec2-user:ec2-user /ansible
mkdir /ansible/inventory
cd /ansible/inventory
wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini
chmod +x ec2.py
Set the variables on the system:
export ANSIBLE_HOSTS=/ansible/inventory/ec2.py
export EC2_INI_PATH=/ansible/inventory/ec2.ini
One important thing here that makes me loose several hours is the ec2.ini configuration to suit our environment setting up the following variables:
destination_variable = private_dns_name
vpc_destination_variable = private_ip_address
Without this, Ansible will find only the instances that has public ips. We don’t wanna this.
Also, download the basic configuration of Ansible and save it on /ansible. You can have my config file from here. The full config file (with parameters that we wont use now) can be downloaded here.
Setting up SSH keys
Ansible is based on SSH calls (again) due to this concept, we need to provide to our server the pem file. Transfer it using scp, winscp or any other tool to your Ansible Server and complete the setup:
cp /tmp/training.pem ~/.ssh/
chmod 600 ~/.ssh/training.pem
ssh-agent bash
ssh-add ~/.ssh/training.pem
At this stage, you should be ready to communicate with your instances. Let’s test it:
/ansible/inventory/ec2.py --list
You’ll see your Amazon EC2 inventory broken down and grouped by a variety of factors:
Lets call Ansible using the inventory script to make sure its working well:
ansible -i /ansible/inventory/ec2.py -m ping all
As you can see, its listing correctly my 03 instances (including NAT)
You can also use tags to call your instances:
ansible -i /ansible/inventory/ec2.py -m ping tag_Name_Ansible*
Using the same concept, you can use Ansible to update your instances, using remote yum calls:
ansible -i /ansible/inventory/ec2.py -m ping tag_Name_Ansible*
And this is pretty awesome! You can manage groups of instances updating, sending files, etc just using tags, which is a very good practice by the way.
Spinning up new Instances
To use Ansible, you must to be able to fully understand Playbooks.
Using Playbooks you can run multiple Tasks and provide some more advanced functionality that you’d miss out on using ad-hoc commands. I wont explain in deep all of these concepts, but I’ve found a very clear article explaining this in details.
Ansible Role
First, create a directory: “/ansible/roles/provision-ec2/tasks/”
Name the file below as “main.yml” and save it on the previous created directory:
---
- name: Provision EC2
local_action:
module: ec2
key_name: "{{ ec2_keypair }}"
group_id: "{{ ec2_security_group }}"
instance_type: "{{ ec2_instance_type }}"
image: "{{ ec2_image }}"
vpc_subnet_id: "{{ ec2_subnet_ids|random }}"
region: "{{ ec2_region }}"
instance_tags: '{"Name":"{{ec2_tag_Name}}","Type":"{{ec2_tag_Type}}","Environment":"{{ec2_tag_Environment}}"}'
assign_public_ip: no
wait: true
count: 1
volumes:
- device_name: /dev/xvda
device_type: gp2
volume_size: "{{ ec2_volume_size }}"
delete_on_termination: true
register: ec2
- debug: var=item
with_items: ec2.instances
- add_host: name={{ item.public_ip }} >
groups=tag_Type_{{ec2_tag_Type}},tag_Environment_{{ec2_tag_Environment}}
ec2_region={{ec2_region}}
ec2_tag_Name={{ec2_tag_Name}}
ec2_tag_Type={{ec2_tag_Type}}
ec2_tag_Environment={{ec2_tag_Environment}}
ec2_ip_address={{item.public_ip}}
with_items: ec2.instances
- name: Wait for the instances to boot by checking the ssh port
wait_for: host={{item.public_ip}} port=22 delay=60 timeout=320 state=started
with_items: ec2.instances
These are the instructions to create a new instance and check it before finish the task ssh’ng it.
Next, well create the variable file, that will feed the role with detailed information regarding our environment and number of instances to be created, for example.
Create the directory “/ansible/ec2-vars/” and save the file “instances.yml” into that:
ec2_keypair: "training"
ec2_security_group: "sg-718d9014"
ec2_instance_type: "t2.micro"
ec2_image: "ami-1e159872"
ec2_subnet_ids: ['subnet-bd9925e4']
ec2_region: "sa-east-1"
ec2_tag_Name: "Ansible_Client"
ec2_tag_Type: "Ansible_Instance2"
ec2_tag_Environment: "testing"
ec2_volume_size: 8
Tip: All of these variables and possible values can be found here.
And the last file is the Playbook file that will call the others. Create it on “/ansible/provision-ec2.yml” with this content:
---
- hosts: localhost
connection: local
gather_facts: false
user: root
pre_tasks:
- include_vars: /ansible/ec2-vars/{{type}}.yml
roles:
- provision-ec2
Basically Playbook will call the Role instructions and Role will get the required information from variable file. Easy.
Lets run it and see the magic happens:
ansible-playbook -vv -i localhost, -e "type=instances" /ansible/provision-ec2.yml
After couple minutes, you’ll have your new instance up and running. It can be easily checked by Ansible or by “old-school” method: AWS Console:
Successfully created and tagged. Awesome isn’t it?
Last but not least, let’s update the packages of the new instance, once it was recently created:
ansible -i /ansible/inventory/ec2.py -s -m yum -a "name=* state=latest" tag_Name_Ansible_Client2
You’ll see a big output:
If you run again, you’ll see that everything was updated successfully:
That’s all!
I’m preparing my lab to go little bit deeper on Ansible (ELB, Code deployment, etc) and I’ll share with you all in soon.
Thanks!
AWS tech 