Harnessing the power of Jinja2 with Ansible

Posted in: Articles, DevOps, Open Source, Site Reliability Engineering, Technical Track

Configuration files management is the core component of every configuration management tool. Templates are one of the best methods for creating configuration files as they allow dynamic variables. Ansible has a template module that uses Jinja2 in its core. Jinja2 is written in Python and allows Python-like expressions which allows us to have Python features and flexibility to be embedded into our configuration file management strategy.

Looping and Testing

The following is a standard template usage to create multiple files using the same template. Using the loop method with_items, multiple files can be created based on the declared variable log_path.

  vars:
    set_category: true
    log_path:
      - name: "auth"
        path: "/var/log/auth.log"
        category: auth
      - name: "syslog"
        path: "/var/log/syslog"
        category: syslog
  tasks:
    - name: create config files
      template: src=./templates/sources.j2 dest=/etc/sumo/sumo.d/{{item.name}}.json
      with_items: "{{ log_path }}"

With this, variables like log_path.path can be called as item.path from the template.

Tests using “if” expressions can be used as seen in the following example. If only set_category variable is set to TRUE, the category will be updated.

{
  "api.version": "v1",
  "source": {
      "name": "{{ item.name }}",
      "sourceType": "LocalFile",
      "pathExpression": "{{ item.path  }}",
{% if set_category %}
      "category": "{{ item.category  }}"
{% endif %}
     }
}

The above code would create the following files:

/etc/sumo/sumo.d/auth.json

{
  "api.version": "v1",
  "source": {
      "name": "auth",
      "sourceType": "LocalFile",
      "pathExpression": "/var/log/auth.log",
      "category": "auth"
     }
}

/etc/sumo/sumo.d/syslog.json

{
  "api.version": "v1",
  "source": {
      "name": "syslog",
      "sourceType": "LocalFile",
      "pathExpression": "/var/log/syslog",
      "category": "syslog"
     }
}

Repeated configuration becomes an easy task and if it is required in a file – “for loop” can be used from a template.

{% for item in log_path %}
{{ item.name }} :  {{ item.path }}
{% endfor %}

This would create an entry like below:

auth :  /var/log/auth.log
syslog :  /var/log/syslog

Filters

Jinja2 provides a wide range of filters that can transform the data inside the template.

For example, join filter would take a list and concatenate it to a string with a separator. The following is an example:

{{ allhosts | join(',')}}

The above join filter would take the following list:

    allhosts:
      - localhost
      - 10.0.0.1
      - 10.0.0.2

and create the following:

localhost,10.0.0.1,10.0.0.2

json_query is a filter which would allow searching inside a data structure. The below query would search inside the log_paths variable for a name key with value auth and would print the value of associated path:

{{ log_paths | json_query('[?name==`auth`].path') }}
/var/log/auth.log

Another advantage of filters is that multiple filters can be lined up using pipe. For example, adding a basename filter to the above query would get the last name of the above file path.

{{ log_paths | json_query('[?name==`boom`].path') | basename}}
auth.log

map filter is another filter, which is very useful. It can either look up an attribute from a list or apply another filter on the objects.

For example, the following would look for all values with attribute category:

{{ log_paths | map(attribute='category') | join(', ')  }}
auth, syslog

In the following example, another filter called regex_replace is called by map filter to attach https:// to all the ip addresses which starts with 10:

{{ allhosts | map('regex_replace', '^(10.*)$', 'https://\\1') | join(', ')  }}
localhost, https://10.0.0.1, https://10.0.0.2

As we can see, Ansible Jinja2 templates use the power of Python, allowing complex data manipulation while managing diverse environments. Custom filters can also be created using Python and added to filter_plugins/ under the ansible top directory.

email

Interested in working with Minto? Schedule a tech call.

About the Author

Devops Engineer
Minto Joseph is an expert in opensource technologies with a deep understanding of Linux. This allows him to troubleshoot issues from kernel to the application layer. He also has extensive experience in debugging Linux performance issues. Minto uses his skills to architect, implement and debug enterprise environments.

4 Comments. Leave new

Hello Minto, Really useful tips and advice. I am trying to use a similar method to rename multiple jinja2 templates and rename them when it reaches the destination directory, however, could not get any other method working. Tried ‘with_fileglob’ and ‘with_filetree’ but did not get the expected results.

Below is the method I ended up with but the files do not get updated according to the variables set in vars/main.xml. They also do not end up in the target ‘files’ directory.

mkdir {tasks,templates,vars}
cat < tasks/main.yml
– name: iterate and send template properties files
template:
src: “{{ item.src }}”
dest: “{{ item.dest }}”
with_items:
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-1_properties.j2’, dest: ‘/home/zee/files/test-1.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
– {src: ‘test-2_properties.j2’, dest: ‘/home/zee/files/test-2.properties’}
EOF

cat < vars/main.yml
# Override these variables

# DIGITAL Application Artefacts
ap-core-database: ‘5.6.4’
ap-core-backoffice: ‘5.6.4’
ap-core-backend: ‘5.6.4’
ap-security-signature: ‘4.3.0’
ap-security-verification: ‘4.3.0’
ap-security-mdes-crypto: ‘4.3.0’
ap-security-cfi-crypto: ‘3.8.1’
ap-security-bah-crypto: ‘4.5.0’
services-verify-email: ‘3.4.0’
osb: ‘5.5.3’
ap-security-file-crypto: ‘4.2.0’
ap-digest-signing: ‘1.2.0’
ap-digest-verification: ‘1.0.0’
db-replication: ‘1.3.0’
ap-security-jws-crypto: ‘4.5.0’
ap-pos-nfc-rest-core: ‘1.0.1’
ap-security-jws-pba-crypto: ‘4.5.0’
ap-pos-nfc-core: ‘1.0.1’

# MDM Application Artefacts
ap-mdm-core: ‘3.6.5’
osb-mdm: ‘3.5.1’

# POSNFC Application Artefacts
ap-pos-nfc_core: ‘3.1.3’
ap-pos-nfc-backoffice-core: ‘3.1.1’
osb-pos-nfc: ‘3.1.2’

# BATCH Aplication Artefacts
ap-batch-core: ‘5.6.6’
EOF

cat < templates/test-1.properties.j2
# DIGITAL Application Artefacts
ap_core_version={{ ap-core-database }}
ap_core_backoffice_version={{ ap-core-backoffice }}
ap_core_backend_version={{ ap-core-backend }}
ap_security_signature_version={{ ap-security-signature }}
ap_security_verification_version={{ ap-security-verification }}
ap_security_mdes_crypto_version={{ ap-security-mdes-crypto }}
ap_security_cfi_crypto_version={{ ap-security-cfi-crypto }}
ap_security_bah_crypto_version={{ ap-security-bah-crypto }}
services_email_ws_version=
services_verify_email_version=(( services-verify-email }}
osb_version={{ osb }}
iosb_version=${osb_version}
ap_security_file_crypto_security_version={{ ap-security-file-crypto }}
ap_digest_signing_version={{ ap-digest-signing }}
ap_digest_verification={{ ap-digest-verification }}
db_replication_version={{ db-replication }}
ap_security_jws_crypto_version={{ ap-security-jws-crypto }}
ap_pos_nfc_rest_core_version={{ ap-pos-nfc-rest-core }}

ap_security_jws_pba_crypto_version={{ ap-security-jws-pba-crypto }}
ap_pos_nfc_rest_core_version={{ ap-pos-nfc-core }}

# MDM Application Artefacts
ap_mdm_core_version={{ ap-mdm-core }}

osb_mdm_version={{ osb-mdm }}
iosb_mdm_version=${osb_mdm_version}

# POSNFC Application Artefacts
ap_pos_nfc_core_version={{ ap-pos-nfc-core }}
ap_pos_nfc_backoffice_core_version={{ ap-pos-nfc-backoffice-core }}
osb_pos_nfc_version={{ osb-pos-nfc }}
iosb_pos_nfc_version=${osb_pos_nfc_version}
# BATCH Aplication Artefacts
# As soon as batch core will be released and uploaded to nexus, use the version from nexus.
ap_batch_core_version={{ ap-batch-core }}
EOF

Reply

One more thing, “EOF” were stripped during the copy and paste. Please replace to “cat <<EOF < tasks/main.yml", etc.

Reply

playbook:-
“cat < deploy-prop.yml”

– name: Generate properties files
hosts: localhost
vars:
ansible_connection: local
ansible_python_interpreter: “{{ ansible_playbook_python }}”
template_path: /home/zee/templates
dest_template_path: /home/zee/files
gather_facts: no

roles:
– { role: cr_properties, tags: [ ‘properties_cr’, ‘generateproperties’ ] }
EOF

Reply
Minto Joseph
June 2, 2020 11:08 am

I assume that this is a role and you want to deploy multiple config files with diff configs from a jinja2 file. 1. I do not see any reason to repeat same source and destination files again and again .. 2. You need to consider using dictionary variables like below in the the variable files.
foo:
field1: one
field2: two
bar:
field1: three
field2: four
Then they can be used to reference while creating files.
You need to update the jinja2 file and playbook to reflect this.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *