Ansible Deep Dive

25 min readJun 26, 2023

Ansible is most simplest automation tool that is written in python. We also have terraform but it is more specific to cloud related automation.

Ansible is simple as it do not have any agent concept like puppet, chef, saltstack if you have to manage 100s of servers no need to install against on all the servers. Ansible uses existing connections like in linux it uses ssh. Ansible playbooks are written in YAML format which is very easy to understand.

Ansible Setup:

We are gonna launch few EC2 instances, one will have ansible and then we will have two web servers and one database server. We will do all the setup using ansible.

First we will launch an EC2 instance with ubuntu and create new security group for this instance and allow port 22 from your ip and also create new key pairs for future ssh access of the machine:

Allow port 22:

Key pairs:

Use user data script to install ansible on EC2 instance:

Document which will help to install ansible on any OS:

https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html#installing-ansible-on-ubuntu

Ansible installation very simple:

Copy these command and paste on user data section of EC2 instance to install ansible on it:

Also create two more EC2 instances as a target servers with centos:

Allow port 22 from ansible security group to these instance so that ansible machine would access these through ssh:

So we have 4 t2.micro instances it will not cost on compute as t2.micro is available in free tier services.

Check the ansible installation on EC2 instance by doing ssh to that:

Make a new directory in Ansible instance for exercises:

Now we will see exercise_1 that is about inventory and ping module in Ansible:

We can manage multiple hosts using ansible, all the node or hosts info we will have to give in inventory file. When we install ansible there will be a default inventory file:

But it is better to create own inventory file in your project directory:

Give hosts parameters in Inventory file also copy the downloaded public key of each host instance and paste in the inventory file:

Now save this inventory file and test connections first:

Ansible provides you adhock command you can execute module on target right from the shell. Like ping module do the ssh in ansible target host.

When we do ssh of target host it ask to connecting yes or no, it is not better when we have to manage 100s of machine so type no here and edit the ansible global configuration file:

and add below line in it then check:

But what if we have tens of server we can not check the individual connection of each server, so we will create groups. Groups are good for scalability. We can also create group of groups with :children parameter.

Lets test it:’

We can also test all the host in inventory file:

We can also define host at group level with :vars parameter:

If we define user at both level (host and group) so host level will have higher priority.

So for testing purpose remove it from host level:

Now we will see what is jason, how to read it and what is YAML format because playbooks and scripts are written in YAML format in Ansible, we can create inventory also in YAML format:

Lets take a example of python dictionary which have key:values to store data:

This dictionary variable has three key value pairs, 1- Devops which has values in list, 2- Development which also has values in list, 3- ansible_facts which has values in dictionary also.

This is difficult to read we can make it in vertical so that it may easy to read:

This is an example of JSON format which is very famous these days.

Now copy this JSON format we will convert it in YAML format, in YAML we do not have opening and closing parenthesis, double quotes, square brackets.

YAML

Devops:

- AWS

- Jenkins

- Python

- Ansible

Development:

-Java

- NodeJS

- net

-ansible_facts:

-python: /usr/bin/python

-version: 2.7

This is document of Ansible on YAML format:

https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html

Ansible Adhoc:

Create a new exercixe_2 and copy all the things from exercise_1 to this new exercise directory:

We will install a package on ewb01 server using yum through ansible:

We got a failed response in JSON format because we are running it with centos user and we need root privileges to run it:

Use — become in the end of ansible command to elevate the privileges:

Conf management tools like ansible are in IDEMPOTENT in nature means if system in same state it not gonna apply the state again like if we run above command first time it change the state True because it installed the package on machine if you run the same command again it will show change : False:

To start the httpd service in web01:

Now we will create a new file and push this file using ansible adhoc:

As we have push index.html file on Web01 instance now allow port 80 on security group of web01 and check on the browser with its public ip:

Ansible playbooks are written in YAML format, playbook is like bash scripts, python scripts to automate the task.

Example of one single play with one single task:

Example of two plays: First play websrvgrp has two tasks and second play dbsrvgrp has only one task:

Now lets write our own playbook:Create a new exercise_3 directory and copy all the stuff from

Three dashes are optional in the starting of yaml file its mean yaml file starts from here:

In the above playbook example we have two plays and each play also has two tasks:

1- Install Apache and httpd service on websrv groups which is defined in inventory file and two instances in it web01 and web02, after name we define module yum which is use to install services, after define module we defined name of service to install and its current state which will be present. Another module we used is service in both tasks it function is to start and enable services.

In the plays we also use become keyword which mean this plays will run with root user privileges.

Now time to test this simple playbook:

If we got any syntax error it will be display in red color.

Before execute this playbook we will first uninstall the httpd service from web01 as we have already installed it:

Now execute the playbook:

Lets add some new task for webserver in our playbook, we wil also transfer or copy the index.html file through playbook:

Check the web01 server:

To check list of all modules of ansible type:

⇒ ansible-doc -l

We can also make dry run of our playbook with -C option first means it will show as it is executing playbook but actually not:

Now we will see how can we find our right module:

As we have install and enabled mariadb service in our YAML playbook, now we will create and add user to our database, create a database name accounts and give privileges to user that we created on our mysql database.

Go on ansible website and module section→ database module.

First we will create a new database:

Remove the webserver play just keep database play in our playbook in exercise_4 and add new task to create database:

We got an python error that it require a Mysql or PyMysql module on remote Db01 instance as it is executing the playbook on remote machine:

Lets login to the Db01 instance and search for that python package:

So got the name of package now we will install it through ansible play book:

Now again execute the playbook:

Now we will add user in our database:

First we will create a new database:

Remove the webserver play just keep database play in our playbook in exercise_4 and add new task to create database:

We got an python error that it require a Mysql or PyMysql module on remote Db01 instance as it is executing the playbook on remote machine:

Lets login to the Db01 instance and search for that python package:

So got the name of package now we will install it through ansible play book:

Now again execute the playbook:

Now we will add user in our database:

Again execute the playbook:

Ansible Configuration Section:

So far we did not made any changes in ansible configuration its mean it has default configurations. We can change default configurations like change ssh port number etc.

First we will look for global ansible.cfg file:

If you dont see the configurations in ansible.cfg file as in upper picture you can generate new file with below command:

But before take the backup of ansible.cfg file:

⇒ ansible-config init — disabled > ansible.cfg

After running this command you will see ansible with all defaults commands in same ansible.cfg global file:

We always writes -i inventory whenever we executes the ansible, instead of that we can define parameters in /etc/ansible/hosts file.
Forks = 10 means if we have 100s of machine but ansible will ssh 10 machines at a time.

Basic Exercise:

Just change the remote port=1022 in ansible.cfg file then come to the ansible exercise and check the ssh to hosts it will fail, because we ask ansible to do ssh on port 1022:

Note: remember to remove the # and ; before any command in ansible.cfg file because its a comment.

We can create our own ansible.cfg file in our project:

Now we will test the playbook:

This time we will not use -i inventory because we already define the inventory path in ansible.cfg file:

But we define the log file path at root that is /var/log/ansible.log and we are running playbook with ubuntu user so it will not be able to create that log file so first create the log file manually and assign ubuntu user to it using chown command then run the playbook:

If you want to save debug level log then use -vv when run the playbook:

We can even use -vvv or -vvvv four verbose for detailed logs and troubleshooting.

Ansible Variables:

Variables in ansible are same as any other programming language, first we will see how to define variable in our ansible playbook:

Fact Variables: setup module

When we run the the playbook the first task that executed is gathering facts that task will run the setup module and generate fact variables like when we run ansible on web01 instance it generate the fact variable “ansible_os_family” whose value will be “red hat , debian etc”.

We can use these variable by putting them in configuration file and make decision based on them.

Store Output : register module

Whenever we run ansible playbook or any command it gives the output in JSON format, we can store that output in a variable then use it.

Now let see these variable one by one.

First create a new exercise for this:

We will take example of previous playbook task in which we created accounts database so we can use variable there to make it reusable:

First we will understand how to define the variable inside the playbook:

Then at task where we are creating database use this variable there:

We can also use debug module to print the value of variables in playbook:

Now we will see inventory based variables like variable for groups and hosts.

First comment the previous define variable in playbook:

If we execute the playbook we will get error of undefined variable:

So, we will create a new directory:

Create a new file with name “all” variables of this file will be accessible by all hosts of inventory file:

Imp note: directory name and file which contains variables , name should be as it is as shown in above example, else ansible will not read variables. Directory → group_vars and file name → all.

To understand more about group and host variables we will create a new directory with fresh playbook:

Create a new ansible playbook that create a new user on host machine db01, web01, web02 with help of variable:

It created use on remote machines but did not show the username on output json format, we can use debug for that but we will use “register” module , this will store the output of our task:

It will give very detail output values store in dictionary:

We can also define these variable in globally in group_vars/all file but if same user defined in both locations i.e inside playbook and in global file precedence will be given to inside playbook variable then to global one.

As we know our inventory files has two groups 1- websrvgrp which has two host web01 and web02 and 2- dbsrvgrp which has only one user db01, instead of making group_vars/all file we can create file for groups only also:

Now we have same variable in all file as well as websrvgrp file, but priority will be goes to group variable:

As we created group_vars we can also create host_vars/{hostname} directory:

Now for web02 we have three different file with same variable but highest priority will ge giev to host_variable file:

As shown in output web02 user priority given to host_file and for web01 user as no entry define in host_file so priority given to group file and for db01 there is no entry in host as well as group file so priority given to all file.

So far this is our variable precedence in ansible. But one thing that supersedes all is command line.

Fact Variables : setup module

Now we will run setup module through adhoc command and see the fact variables:

Whenever we run ansible playbook the gathering fact task get executed it runs for all the host that defined ib playbook. It uses the setup module and it generates lot of information about target machine.

We can execute setup module by adhoc command also:

So this setup module run tools called ohai this tools collects system information and returns in JSON format:

We will write a playbook which will access the fact variables and print it:

Now accessing another fact variable to check free ram of target host but this is not string variable but dictionary:

Below output is generated through ansible adhoc command we want this info through playbook fact variable:

In playbook:

Like dictionary we can also print the list variables through our playbook:

Provisioning Server through Ansible:

Create a new exercise with new playbook:

First create a new ec2 instance (web03) with ubuntu machine as other three are centos so we will manage both and add its host details in inventory file:

Now we will write simple script to install ntp on remote hosts including ubuntu and centos:

We will write playbook if if condition utilizing state module variable that returns os type of target host to distinguish between centos and yum and centos uses yum package manager while ubuntu uses apt:

Run setup module to get ansible_distribution value of host:

Now this is our playbook:

Now start the ntpd service on all hosts:

Now we will use loops to execute tasks to save our time:

We can also use dictionary in loops instead of list when you have to add multiple users but in different groups:

Files Module in Ansible:

We will go through with some modules:

First create a new exercise directory and check the status of all hosts:

First we will push a banner file by using copy module in our previous provisioning.yaml file:

Now we will make configuration changes of installed NTP on target hosts, first copy the /etc/ntp.conf file form any target host, then we will edit it then push it to the target host using copy module:

Now create a new file in same exeercise_10 directory for ntp and paste that ntp conf and make some changes in that like change the default ntp servers entry in that file:

Comment out the old ntp server entries and add some new servers from the internet:

Also copy one file from ubuntu machine and do the same with it also: so you will have two ntp files at ansible exercise 10 directory that we will push on all hosts centos file for centos host and ubuntu file for ubuntu host:

Ntp_debian_file:

Make a new directory and move both ntp files there:

We make the template directory because in template we can have dynamic data. Like we can have variables in template file when we deploying the template file with template module is going to read the file and see is there any dynamic content then replace it with actual content, it uses jinja2 templating.

So first write the task in provisioning.yaml playbook:

Deploy and restart ntp service on remote hosts:

Change the extension of both files to j2 that is jinja 2:

But still we have all static data to make it dynamic we will store the variables in group_vars/all file:

Create three variable there and assign ntp servers details to them as mentioned in net_conf file:

So we will replace these variable in ntp_debian.conf.j2 and ntp_redhat.conf.j2 file with servers details:

So now our conf files also has variables so we can reuse it as per our requirement and future needs.

Now execute the playbook:

We have seen copy and template module now we will see file module, that is use to change ownership of files, can also use to create directories and link files etc:

Maybe we want to create directory on target host to move some data, so come in playbook and create a new task for it:

Output:

It creates the directory successfully but when we run it it runs the whole playbook and in playbook the task of restart the services also mentioned which is not required during task of directory creation, So we can do some intelligent work whenever any conf change only then restart the service otherwise not, for this ansible give us Handlers.

Handlers will only executed at the end of playbook,

Keep the “Dir for dev data” task up from the tasks “start and enable ntp” and then create a handlers task after “Dir for dev data” task:

Now execute playbook it will not run the last two task of below handlers as there is no change in the service, it will run till creation of directory:

Make some changes in conf file of both ntp_debian.conf.j2 and ntp_redhat.conf.j2 like put some ### and again run the playbook you will see how handlers works:

Ansible Roles:

Currently our ansible directory has many files, sub directories, playbooks etc it is very difficult to manage so we have ansible roles that make management easy.

Here we distributes our playbook content into different directories. Main purpose of roles is to make code reusable like if your org. Has many apache servers so you can use roles to make your code reusable for all the servers.

We will create role for our last playbook content. Currently our directory has this structure:

First we will make new exercise directory for this:

Now create a new directory for roles then run command to create roles in to it:

Now time to move our content:

So, first move all the variables in default file:

Copy these all:

And paste in default file:

Now move other content from playbook, first we willl move tasks, so copy all the tasks not handlers just tasks from the playbook:

And paste the task in tasks file.

Now copy and paste the handlers:

Now move templates:

Now time to do some cleanup.

Remove all the task and handlers from your playbook and define the role name there:

Now go to roles/post-install/tasks/main.yml file and remove the template path just put template name there ansible will auto find the path:

Before:

After:

Now remove all the files and directories from your main exercise directory that has moved to roles like templates, group_vars:

Now time to test the playbook:

Now we will see few more things about roles:

First create a new exercise for that:

Roles have reusability, as we have defines our variables in default file of roles but if this roles is require again with different variables , so we can over write variables. We can pass the variables values from playbook also:

Currently we have three variables define in default file:

ntp0, ntp1m ntp2, we can pass different values of these from our playbook also, let see:

https://galaxy.ansible.com

Here you can find alot of predefined roles for your projects.

We can download already predefined roles and use it:

We will see demo that how to use any role from ansible galaxy.

Lets take a example of java code just copy it and paste in your ansible working directory:

Now call this role from your playbook:

Now these two roles will be executed but first geerlingguy.java then post-installs.

Ansible for AWS:

Now we will see how can we use ansible to manage our AWS services like creating an EC2 instances login to EC2 instance through ssh keys, but how ansible would know what is our AWS account (authentication and authorization)? It will be done through access key and secret key:

So we need to export secret and access key of AWS to ansible first and create a new user:

Go to AWS→IAM →users→ add user

Once user is created, click on the user→ security credentials → access keys→ create access key:

Note: Keep your access key and secret key private and save, as if someone got these access it could me harmful to your AWS account as this user has administrative access:

Remember to download the .csv file (it have your keys) before generate it:

t these keys:

Open the .bashrc file of ansible instance and placed these keys there:

⇒ sudo nano .bashrc

Now execute .bashrc file so it export our variables:

We will start with the basic playbook so create key pair first:

Playbook:

When we run the script it will through error because ansible is written in python and python needs some libraries to access different resources like AWS it has module boto3 we need to install this module to remove error: