- What is Ansible?
- Creating a Role
- HAProxy Virtual IP Failover
- Testing Roles
- Multi-machine testing
This article describes the configuration management tool Ansible and some of its basic concepts. Then it discusses how to test Ansible code. Refer to the version chart below. It assumes familiarity with the concept of configuration management, programming, and testing.
What is Ansible?
Ansible is a configuration management tool which can be used to manage software installations. Configuration management helps us to ensure that software installations are documented, consistent, and repeatable. Let’s say you want to copy the files for your web application to a server and make sure that its runtime is installed (such as Ruby or .NET Core). While there are many who might deploy manually, it is convenient to be able to do this in an automated way triggered by an event such as an update to source code. Ansible is just one of many tools that can help with this automation and is often compared to alternatives such as Chef, Puppet, and SaltStack.
We use Ansible by defining a Playbook which defines a set of tasks to perform and a set of hosts to target. When we apply the Playbook, each task will be applied to each host. Some common use cases include:
- Executing remote commands
- Copying and downloading files
- Interaction with cloud providers
- User, group, and permission management
In order to create some measure of isolation and reuse, a developer can break
a Playbook into one or many
For example, if we primarily deploy Ruby applications, we might create a role
webapp_prerequisites which will prepare a host to run a Ruby
application by installing the desired Ruby version and installing
HAProxy to use as a local HTTP proxy. (For a brief
discussion on why we would want to do this, see 2). This Role can then be
referenced by each of the Playbooks that install our individual Ruby
If you want to follow along with the examples below and setup your environment, see 7.
Creating a Role
Let’s create a simple role that installs Ruby, so that we can run a simple
Sinatra or Rails
application, and installs HAProxy to proxy requests to the application. In
the Roles directory of our Ansible installation, we will create a directory
webapp_prerequisites that contains a file
tasks/main.yml with the
--- - name: Install Ruby package: name: ruby state: present - name: Install HAProxy package: name: haproxy state: present
HAProxy Virtual IP Failover
In order to demonstrate a more complex Role, we will create a Role that sets up two HAProxy hosts. One of the hosts will be active, and one will be a hot standby. The active HAProxy host will be bound to a virtual IP (VIP), and if there is some problem with the active host, the system with failover to the second HAProxy. In this scenario, the second host will claim the VIP.
Keepalived is a daemon that provides generic failover capabilities. We can
configure it to monitor for the presence of an
haproxy process. If one
isn’t found, Keepalived peers can communicate with each other in order to
coordinate assigning the VIP.
First, we have to install HAProxy on each host. We will make the assumption we’re running on CentOS. We also want the HAProxy service to be enabled and started.
- name: Install haproxy package: name: haproxy - name: Enable haproxy systemd: name: haproxy state: started enabled: true
We typically want HAProxy to proxy another sevice. For these purposes, we’ll assume we’re proxying http://example.com/. We’ll create a configuration file to transfer to the remote host.
# haproxy.cfg defaults mode http frontend my_frontend bind *:80 default_backend my_backend backend my_backend # We need to send this request header otherwise # the upstream server won't respond. http-request set-header Host example.com server example example.com:80
We need a task to copy this configuration file to the appropriate location on the remote host:
- name: Copy haproxy.cfg copy: src: haproxy.cfg dest: /etc/haproxy/haproxy.cfg notify: reload haproxy
Whenever the configuration file is changed by our
copy task, we want to
queue a reload of the
haproxy service. 6
Having configured HAProxy, we need to install and configure Keepalived.
- name: Install keepalived package: name: keepalived - name: Setup keepalived service systemd: name: keepalived state: started enabled: true
Finally, we have to configure Keepalived with a
Jinja template file. 1 It’s somewhat involved but
the gist is we want to configure Keepalived to claim the VIP whenever HAProxy
goes down. We templatize the configuration file because it will differ
We also need an Ansible task to create this file on the hosts:
- name: Create keepalived configuration template: src: keepalived.conf.j2 dest: /etc/keepalived/keepalived.conf
That completes the role. For the full source see 8. Now that we’re done, how do we know it works? We can run it against a host we create manually but an even better way to do things is to create an automated test.
Just like other pieces of software, Ansible Roles can be tested to confirm their operation. The reasons you might want an automated test are described in a previous blog post. It’s important to test Roles in some way because they often provide the base components of your system like runtimes, HTTP servers, and file and directory structures. Especially if multiple people are working on the same Role, the risk of adding the wrong configuration to a file or breaking a Jinja configuration template is high.
The standard way to test Ansible Roles is using
Molecule. Molecule is a testing
framework designed for Ansible. It works by adding a
molecule directory to
each Role, which contains instructions for how to provision testing hosts,
Playbooks to apply the Role, and tests to run against each provisioned
testing host. If you have an existing Role, this directory can be
automatically generated by installing
molecule, then running the following
command inside the Role directory. For example, we can add molecule to the
webapp_prerequisites Role by running the following command from inside the
% molecule init scenario \ --driver-name vagrant \ --scenario-name default \ --role-name webapp_prerequisites
In the command above, we tell
molecule that we want to create a new testing
default, and to provision testing hosts using
Vagrant. Vagrant makes it easy to
programmatically provision virtual machines (VMs). 4
Afterwards, from inside the Role directory, we can run the tests using
molecule test. The
molecule test command will provision the testing
infrastructure, in this case VMs running on
VirtualBox, prepare and configure the hosts
using Ansible, run tests against it using Python’s
[testinfra](https://testinfra.readthedocs.io/en/latest/) framework, then
destroy the testing infrastructure, reporting the results of the test. This
can take quite awhile when using virtual machines, so when I am in the
process of developing a Role, I use
molecule verify, which will run all
those steps without destroying the infrastructure. I can repeatedly run
verify as I develop the Role.
Given the example Role definition above, we may want to test that HAProxy has
been properly installed by running a simple command and checking its exit
code. We can login to an instance provisioned by Molecule using
We first run a command that should return the version of HAProxy that is
installed, then we
echo the exit code stored in
We can have Molecule test this automatically by writing a test using the
testinfra Python provisioner testing framework. In this article, we’ll be
using testinfra 1.14.1. In
molecule/tests/test_default.py, we can add the
following test, that does the same thing we did manually above.
def test_haproxy_installed(host): cmd = host.run("/usr/sbin/haproxy -v") assert cmd.rc == 0
molecule verify, we get a message like
tests/test_default.py::test_haproxy_installed[ansible://instance] PASSED [100%] =========================== 1 passed in 1.95 seconds =========================== Verifier completed successfully.
testinfra has an API for retrieving hosts, running commands on those hosts,
and obtaining information about the host (facts). For example, we can run a
simple test to see if HAProxy is running and enabled on start with the
def test_services_running_and_enabled(host): service = host.service('haproxy') assert service.is_running assert service.is_enabled
We can run arbitrary commands on the host, retrieving the stdout, stderr, and
return code. This gives us a lot of flexibility in running our tests. For
example, to determine if HAProxy is serving requests, we can use
test connectivity while logged onto the host:
def test_redirection(host): cmd = host.run("curl -v http://localhost") assert "HTTP/1.1 200" in cmd.stderr
In this test, we confirm that something is responding on the default
port 80, and that it responds with a 200 status code. If it’s present, the
text will be in the
stderr of the command. We use Python’s
statement that will throw an error if the following condition is false.
The test above is useful, and you can imagine the more complicated tests we can create to verify an instance. Still more complex scenarios arise in the case of multi-host testing.
In order to spin up multiple VMs, we modify the
molecule.yml definition to
platforms: - name: host1 box: bento/centos-7.5 instance_raw_config_args: - 'vm.network :private_network, ip: "192.168.3.9"' - name: host2 box: bento/centos-7.5 instance_raw_config_args: - 'vm.network :private_network, ip: "192.168.3.10"'
We assign them static IPs
192.168.3.10 in a private network so
that they can communicate with each other. A private network limits access to
the VM to only the VM host machine 5.
When we run tests involving multiple hosts, we may need to setup the test by
performing actions across hosts.
testinfra will run the tests for each
host, so we need a way to get a reference to a host that is not the one being
tested. For this article, we will call the host currently being tested by
testinfra the current testing host.
testinfra typically passes a
instance to each test method as the first argument:
def test_something(host): # <-- the host arg is a Host instance # test definition
We can use this
Host instance to retrieve other hosts we previously defined
platforms section of
molecule.yml. We call
host.get_host with a
URI that refers to the host we want. When we are provisioning hosts with
Ansible, it looks something like this:
my_host = host.get_host("ansible://my_host_name?ansible_inventory=" + os.environ['MOLECULE_INVENTORY_FILE'])
get_host takes a URI whose scheme
my_host_name is one of the host names defined in
platforms. We tell Ansible the location of the inventory file that Molecule
has generated via the
ansible_inventory query parameter. Ansible requires a
list of hosts to operate on, which is commonly referred to as the inventory.
ansible://my_host_name?ansbile_inventory=/tmp/inventory.txt └─┬──┘ └─┬──────┘ └───────────────────────────┬┘ scheme path query parameter
You may want to only run the test from a single host. Although I don’t
know of a way to do this, we can skip tests when running on the wrong host.
For example, if we only want a test to run against
host1, then we can check
the hostname of the current testing host, and return from the test method
if it doesn’t match
def test_only_host1(host): ansible_facts = host.ansible.get_variables() if ansible_facts['inventory_hostname'] != 'host1': return
In this case, we are using Ansible, so we request the facts about the host
from Ansible. This includes the
inventory_hostname, which is the hostname
that Ansible refers to the host by. This may be something different than what
the host uses to refer to itself.
In order to test the HAProxy VIP Playbook, we want to create a test that will
confirm that the VIP still responds after the master
responding. It doesn’t matter which host is the current testing host as
long as we confirm that when the
server goes down, whichever host is associated to the VIP is still serving
requests. In the test, we will assume we AREN’T running on
def test_vip_active_after_one_node_goes_down(host): # if this is not host1, return try: service_haproxy(host, 'stop') wait_awhile() test_vip() finally: # After the test, make sure that haproxy is started again service_haproxy(host, 'start') wait_awhile()
test_vip function just does a local curl and hits the VIP:
def test_vip(): import subprocess assert 0 == subprocess.call([ "/usr/bin/curl", "-vs", "--connect-timeout", "1", "http://192.168.3.2"])
For detail on what
wait_awhile, which are somewhat
self-explantory, see 10.
In this article, we learned a little about what Ansbile is and what it can do.
We learned about how to create Roles, and how to test them. We also
learned how to write a complex multi-host testing scenario using
1 1: Jinja is the template language that Ansible uses. It uses curly braces to denote variables, and has looping and conditional constructs.
2 1: Local HTTP Proxies are useful as middleware to provide functionality that we might not want to build into our application. For example, HAProxy has features related to routing, http header injection, Lua-scripting, SSL, and throttling built into it can provide to an application without the need to make the application more complex by building it into the application. For example, if the application needed simple basic authentication, this could be provided by a local HAProxy instance proxying the application. This is made even more relevant when one uses containers. Using containers, it’s incredibly simple to add sidecar containers that add functionality.
3 1: The method we use to install Ruby and HAProxy results in versions that are dependent upon the underlying package manager and repositories installed on the host. For example, CentOS 7.5’s default repositories only include up to HAProxy 1.5, but the most recent version as of this writing is HAProxy 1.8.
4 1: Vagrant is useful for creating VMs for development. Vagrant itself is compatible with a number of local and remote VM providers such as VirtualBox, VMWare products, and Amazon Web Services (AWS).
5 1: The terminology is a bit confusing, but in this article, we call a host some
logical computer running an OS. This is different than a
VM host which
is a machine that runs Virtual Machines.
6 1: In Ansible, Handlers are used to queue actions to happen at a later time. We know we want to reload HAProxy if the configuration changes, but we also might want to reload HAProxy if we have a change in an SSL certificate. We don’t want to reload HAProxy twice, so we just queue this reload to happen later.
7 1: Setting Up Ansible
The source code at 8 provides a
requirements.txt file which lists the
Python dependencies needed to follow along with the example. We can use
virtualenv to install these
dependencies into an isolated python environment. On MacOS, we can use
homebrew to install Python 3 via
% brew install python
After making sure it is on the
PATH, we can confirm our Python version using:
% python -V Python 3.7.2
Then, we can install
virtualenv globally using
pip install virtualenv
We can create a
virtualenv environment and activate it using something like:
virtualenv .python source .python/bin/activate
Afterwards, we can install any requirements provided in the
pip install -r requirements.txt
9 1: Keepalived Jinja Configuration template: https://github.com/jamiely/ansible_haproxy_vip/blob/master/templates/keepalived.conf.j2
10 1: VIP test using