Ansible Redesign

[ncelebic]

Headline:
Ansible Redesign

---

Description:
Ansible encounters limitations in how it can function due to the way it was implemented in Morpheus. Efforts are underway to reorganize and redesign how Ansible functionality works in Morpheus, and I would like to get some feedback on some current ideas for more robust functionality. Here is the current list of ideas for this:

• Allow user to specify python virtualenv to pin ansible version/config
• Decouple inventory from Ansible integration
  • Inventory types:
    • Manual entry
    • Script to generate
    • Select in Morpheus based on criteria:
      • Cloud
      • OS Type
      • Tags
      • Labels
      • App / App Tier
    • Test button to preview that inventory to make playbook targeting easier
    • Inventory would be a dropdown in the task, or an option to select at runtime
• Verbosity selectable at the task level
• When running an ansible playbook against multiple instances, the history for those instances should contain the ansible output, or at the very least, a link to the main ansible run’s output.
• Allow passing of data from one playbook to another. Not sure how to do this one. I might have to write a plugin or module that calls back to morpheus to save a value to a var, and have it available in the injected vars of the next ansible task.

Possibly outside ansible:

• Preview variables available to ansible during a run in the context of the target, if that target exists. Bonus points for clicking a variable name and having a popup with the language specific variable reference.
  • ie: “{{ morpheus[‘instance’][‘name’] }}”

---

Example/Use case(s):
Too many to list, but the aimed functionality with regards to selection ability would be based on the ansible inventory plugin: GitHub - gomorpheus/ansible-collection-morpheus-core

---

[Brad_Felmey]

Oh, you poor fella. In my defense… you asked.

I’ve had Ansible in venv in customer production for a while now:

python3 -m venv /opt/ansible
. /opt/ansible/bin/activate
python3 -m pip install ansible

Now, on the host OS, somewhere in Morpheus’ path (such as /usr/bin/ansible and /usr/bin/ansible-playbook), make shell wrappers (mode 0755) for ansible and ansible-playbook which look like this:

. /opt/ansible/bin/activate
ansible "$@"

Poof - Ansible venv under Morpheus accomplished. Any additional Python modules and extras that are needed by Ansible can likewise be placed into this venv:

. /opt/ansible/bin/activate
python3 -m pip install azure-cli
python3 -m pip install jmespath

…etc.

I agree that more flexible inventory mangement in Morpheus for Ansible workflows would be a plus. A use case would be a customer with a set of VMs running interdependent microservices, and a deployment addresses all of the VMs in that environment. It would be super-convenient to simply have a predefined selection of “DEV”, “QA”, “PROD” and the like which contain all of the hosts relevant to those SDLC tiers. Make one selection, and let your workflow rip across the hosts in that selection.

Verbosity at the task level - I’d give my eyeteeth for this, but it’s not supported in Ansible itself, last I checked. There are some callback plugins which allow tasks to be tagged, though. This is outside of Morpheus’ purvey, of course: https://github.com/ansible/ansible/issues/24215

Passing of data from one play to another - this is already possible in Ansible by using hostvars from one play to another within a playbook. Crude example:

- hosts: foo
  tasks:
    - ansible.builtin.set_fact:
        myvar: fiddlefoo

- hosts: bar
  tasks:
    - ansible.builtin.debug:
        var: hostvars['foo']['myvar']

Passing values from one playbook to another can be accomplished via trickery by transparently wrapping your other playbooks, and so long as the master playbook is still in… er… play, then you can pass values from one playbook to another. Example - dynamically create a top-level play called “morpheus_being_crafty.yml”, and just stick in it the other play/playbooks:

- hosts: all
- ansible.builtin.import_playbook: customerplaybook1.yml
- ansible.builtin.import_playbook: customerplaybook2.yml

And I must point out that the customer’s code MUST be prepared for this. Extremely volatile things can happen when values are unexpectedly populated from external code.

Of course, anyone needing to do this could simply do it in their Ansible, so I’m not sure what the business justification would be to smoke your time on something your customers can already do, in this case.

I applaud the initiative. Out of the items you mention, really only one is within the Morpheus sphere - inventory management (and even that I have easily gotten around by manipulating inventory inside the play itself). The rest of this is all Ansible-space stuff. Greedily, I would much rather your very valuable focus be spent on stuff that isn’t duplicative of what Ansible already has/does.

I’ll be more than happy to elaborate, or STFU, whichever is more helpful.

[ncelebic]

Before I get into things, this is some great work!

A few things that I’d like to see is a clean upgrade path and multiple versions of ansible available for task runs. It seems like quite a manual process to upgrade ansible and there’s no way to run a different version alongside this. It can also present a problem in an HA environment where this now has to be maintained across 3 or more nodes, including adding python modules to each venv if needed.

Passing of data is a Morpheus level thing, which we support in other task types. It would have to be solved with a callback plugin, I believe, but it’s in line with what we already support. If you have a larger playbook with many plays in it, you can certainly pass data back and forth, but then you’re just creating a Morpheus workflow in an Ansible playbook. It works fine, but having many smaller tasks and the ability to remove or add others as needed right in the UI is something that’s inherent in the task/workflow idea.

Task is a rather tortured term here. The implication is that you can select a Morpheus Ansible task to be verbose, instead of the entire repository. Having just one Ansible task be verbose wasn’t the goal, although yes, it would be awesome.

If Ansible is your bread and butter, I generally don’t even hear about missing features, since you can work around a lot of the roadblocks currently there, except for the inventory selection. I wrote the dynamic inventory plugin to get around some of those, but it’s still limited by the scope of the Ansible task in Morpheus.

All that being said, that is some great work, and is really innovative. I’m probably going to implement this in my own lab and testing environments. In smaller installations, it will work perfectly!

[Brad_Felmey]

I did think of one thing that would be an OUTSTANDING addition to Morpheus itself, and I don’t think it would be very difficult to accomplish.

Currently, verbosity is controlled at the integration level. It’s a single checkbox, and it simply adds ‘-vvv’ to the ansible-playbook execution. You can’t get less than 3 v, and you can’t get more. It’s also on or off for ALL Ansible tasks/workflows under that integration.

What I’d love to see is the ability to select verbosity at the time of running the workflow/task. It could default to off, with a select list of ‘1, 2, 3, 4’ to choose from corresponding to the varying levels of Ansible verbosity given by -v -vv -vvv -vvvv, respectively. That way, if I’m trying to diagnose a specific issue, I don’t have to bog down everyone else’s runs with unneeded verbosity. Morpheus also gets very pukey when Ansible output exceeds about 57,000 lines (large playbooks).

[ncelebic]

It just occurred to me on a call, that you can already do this by adding -vvv or whatever level to the command line options in the task. At runtime is a separate idea, and a good one, but you can get what you want right now.

[Brad_Felmey]

I thought so, too, but that has not been my experience. No verbosity arguments that I have used in that field have had an effect.