Ask questionsMeasuring and improving callback plugin performance

<!-- Thank you for taking the time to create this issue. Your feedback is appreciated ! --> <!-- Consider reading the documentation on and joining us on Slack or IRC: -->

What is the idea ?

<!-- Include relevant information to help the community help you. Some examples:

  • the component that you are creating this issue about (api server, api client, web ui, ansible plugins, etc.)
  • linux distribution, version of python, version of ara and ansible
  • how is ara installed (from source, pypi, in a container, etc.) and how you are running it (database backend, wsgi server, etc.)
  • debugging logs by setting ARA_DEBUG to True and ARA_LOG_LEVEL to DEBUG
  • instructions on how to reproduce the issue -->

There is a performance overhead between running Ansible with and without ara the ara callback because for every hook, instead of doing nothing, Ansible does this: doc/source/_static/graphs/recording-workflow.png

Doing something, even if it's very fast, takes longer than doing nothing... because it adds up when the cost is multiplied by thousands of times across the execution of a larger playbook. So there will always be an overhead (sorry!) but it should be low enough that it's hopefully worth it to the humans :)

The main issue is that the overhead is exponential due to the multiplication of hosts and tasks. This means at a small scale it will be low enough to be negligible but a real performance hit when running playbooks involving hundreds of hosts or thousands of tasks.

I'm sure there are improvements we could do to reduce this overhead in the current callback, the API clients or the API server and come up with other implementations with fancy stuff like async, threads, or even dropping the events into a local message bus for lowering the latency to a maximum.

Before we start optimizing stuff, I'd like to measure the overhead even if just for the sake of data, metrics and science.

I think a good start would be:

  • raw ansible-playbook
  • ansible-playbook with 1.x ara callback enabled
  • ansible-playbook with 0.x ara callback enabled (for science and comparison purposes, 0.x doesn't have an API server)

There is a similar issue for the measure API server performance here:


Answer questions andrask

Disclaimer: I haven't checked the implementation deeply yet.

In my case I ran several playbooks in series. I used Mitogen and Ara at the same time and everything I gained with Mitogen (~31% speed increase) was diminished by Ara. I confirmed this by turning off Ara.

Yes, so the important thing is to make the callback a fire and forget event emitter. 4 is a real alternative, for sure, though we would lose the cool feature of real-time state.

With the other options, some other entity, a proxy, should be responsible for transferring the events to the API server. If we can rely on Linux and POSIX entities, then MessageQueues are one option, or UDP sockets are also viable for the IPC between the processes (and there are tons of other solutions, too). The proxy component may be started smartly, like the ControlPath in ssh.

I also looked at your threading patch. It seems like it doesn't do what you really intended. I mean like here

def v2_runner_on_ok(self, result, **kwargs):
    self.threads.submit(self._load_result(result, "ok", **kwargs))

The submit will submit the result of the _load_result, which is always None. But the _load_result is executed synchronously, so nothing changed. Changing the call to

    self.threads.submit(self._load_result, result, "ok", **kwargs)

may be better. It will cause the function to execute asynchronously. I also commented on the review.

The other thing I found though is that the GUI seems to be a little broken as search for host and task never work for me. Always returned everything. Honestly, I feel like pouring all this data directly to Elasticsearch and visualizing it with Kibana may be a better solution. It allows for much more analysis out of the box.

Github User Rank List