Ansible - Notify the right way

Ansible can trigger tasks based on the execution of other tasks. This feature is known as handlers or notifies. Many examples use this feature in the most minimal way possible, and therefore the power of handlers is pretty much unknown. But there is also stuff that can go wrong with the concept of a playbook or the thought process when writing it.

Let's dig into notify and handlers, how to make them more useful and thoughtful.

Promotion: Ansible Community Day

This article is based on a talk, I will give on the Ansible Community Day Berlin on 20th September 2023. The talk will be about "Ansible - The hard way", and tackles some issues I faced in the last ~10 years with Ansible.

If you like, please join me, it's as easy as registering via Eventbrite, hopping over and joining the talks, community gathering and meet me in person. If you cannot make it, you will find the material and slides in my presentation repository afterward.

Notify and Handlers

So, what are handlers now? Well, pretty simple. Handlers are tasks that are not running on a regular basis, but when triggered with a notification. These notifications are always sent, if the original task was "changed". Therefore, you can start a service after a package was installed or refresh/clear some cache only when a new web server was added or removed.

You can use handlers directly in your playbook or in roles, which makes them super convenient. Furthermore, handlers will always run at the end of your playbook. And lastly, Ansible detects if a handler was triggered multiple times and run it only once.

Convinced that this might be useful? Cool, let's dig into some simple examples.

Example Playbook

For the example, I would like to use something very easy and extend it during the article.

Let's create a pretty basic playbook for a web server. This can be done in a single playbook. You just need to create a file with the below content.

---
# webserver.yml

- name: "Install and configure web server"
  hosts: "all"

  handlers:

    - name: "Start and enable httpd Service"
      ansible.builtin.service:
        name: "httpd.service"
        state: "started"
        enabled: true
      become: true

    - name: "Start and enable mariadb Service"
      ansible.builtin.service:
        name: "mariadb.service"
        state: "started"
        enabled: true
      become: true

  tasks:

    - name: "Install httpd Packages"
      ansible.builtin.package:
        name: "httpd"
        state: "present"
      become: true
      notify:
        - "Start and enable httpd Service"

    - name: "Install mariadb Packages"
      ansible.builtin.package:
        name: "mariadb-server"
        state: "present"
      become: true
      notify:
        - "Start and enable mariadb Service"

Easy enough, right? We will install some packages and start the related services with handlers afterward.

Handling updates

What happens, if we want to update our services on the go? We just need to change state: "present" to state: "latest" for the package tasks, right? Well, no. This will trigger an update of the packages, but the services will not be restarted. The same can be said about configuration changes. This is often forgotten, since it easy to forget about "run it twice" scenarios. If you change package versions or configurations, you will need to restart the services afterward.

That shouldn't be too hard, right? Let's update the playbook accordingly.

---
# webserver.yml

- name: "Install and configure web server"
  hosts: "all"

  handlers:

    - name: "Start and enable httpd Service"
      ansible.builtin.service:
        name: "httpd.service"
        state: "started"
        enabled: true
      become: true

    - name: "Restart httpd Service"
      ansible.builtin.service:
        name: "httpd.service"
        state: "restarted"
      become: true

    - name: "Start and enable mariadb Service"
      ansible.builtin.service:
        name: "mariadb.service"
        state: "started"
        enabled: true
      become: true

    - name: "Restart mariadb Service"
      ansible.builtin.service:
        name: "mariadb.service"
        state: "restarted"
      become: true

  tasks:

    - name: "Install httpd Packages"
      ansible.builtin.package:
        name: "httpd"
        state: "latest"
      become: true
      notify:
        - "Start and enable httpd Service"
        - "Restart httpd Service"

    - name: "Install mariadb Packages"
      ansible.builtin.package:
        name: "mariadb-server"
        state: "latest"
      become: true
      notify:
        - "Start and enable mariadb Service"
        - "Restart mariadb Service"

I have added two more handlers and also added them to the notify part of our tasks. As you can see, you can trigger multiple tasks. Now, the services will be restarted if something changes (for example, a package was updated).

Handling Side effects

What is a side effect, you may ask? Here is a short story.

Two weeks later, in the middle of the day, an alert pops up on your phone. The web server seems to be down. You run the Ansible playbook, no updates available, packages installed, everything seems fine.

But, after logging in to the machine, you identify that MariaDB is down. Shouldn't the Ansible Playbook handle this?

First, let's explain what happened here.

MariaDB is down, fine. This can have hundreds of issues. Maybe the OOM killer came around, a colleague stopped it, a bit flipped, or a bug caused a crash. For the uptime of our system, this is not of the matter. We want to have it in the desired state, NOW! But why wasn't Ansible taking care of the restart?

Well, notify and the related handlers are only triggered if something changes. Since our packages are already present, we will never trigger the service tasks. Therefore, our playbook will not handle if someone disabled the service or if the service was not coming up after a crash. Therefore, we need to hop in the thought process again:

1. We want to install and update packages.
2. We want to have the service enabled and running.
3. If a package was updated, we want to update the related service.
4. If the playbook runs again, we want to ensure that our services are running, even no change was triggered.

Sounds like some refactoring. Fortunately, it takes only a couple of seconds to update our playbook to handle such side effects.

---
# webserver.yml

- name: "Install and configure web server"
  hosts: "all"

  handlers:

    - name: "Restart httpd Service"
      ansible.builtin.service:
        name: "httpd.service"
        state: "restarted"
      become: true

    - name: "Restart mariadb Service"
      ansible.builtin.service:
        name: "mariadb.service"
        state: "restarted"
      become: true

  tasks:

    - name: "Install httpd Packages"
      ansible.builtin.package:
        name: "httpd"
        state: "latest"
      become: true
      notify:
        - "Restart httpd Service"

    - name: "Start and enable httpd Service"
      ansible.builtin.service:
        name: "httpd.service"
        state: "started"
        enabled: true
      become: true

    - name: "Install mariadb Packages"
      ansible.builtin.package:
        name: "mariadb-server"
        state: "latest"
      become: true
      notify:
        - "Restart mariadb Service"

    - name: "Start and enable mariadb Service"
      ansible.builtin.service:
        name: "mariadb.service"
        state: "started"
        enabled: true
      become: true

Looking pretty, right? Easier to read, better to understand and also handling our side effect.

Conclusion

Taking care of your thought process and thinking about lifecycle management and side effects should be on your development agenda. Handlers can be powerful, but also lead to interesting issues and problems.

Let me know if you ran into such issues already. You have a more complicated example? I would like to hear about it.

And in case you already forgot: You can meet me at the Ansible Community Day Berlin. Mark the 20. September 2023, register your ticket and get 50% off with "ACDB2023-WHILETRUEDO". I would love to see you there.