Ubuntu18: Failing on RabbitMQ during morpheus-ctl reconfigure

OS: Ubuntu 18
Linux VA*** 4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
.
installed morpheus-appliance_5.4.14-1_amd64.deb
installed morpheus-appliance-supplemental_5.4.14-1_all.deb

morpheus.rb file

appliance_url ‘https://VA1DXCS10201…com’
elasticsearch[‘es_hosts’] = {‘VA1DXCS10101’ => 9200, ‘VA1DXCS10201’ => 9200, ‘VA1DXCS10301’ => 9200}
elasticsearch[‘node_name’] = ‘VA1DXCS10201’
elasticsearch[‘host’] = ‘0.0.0.0’
rabbitmq[‘host’] = ‘0.0.0.0’
rabbitmq[‘nodename’] = ‘rabbit@node02’
mysql[‘enable’] = false
mysql[‘host’] = ‘VA1DXCDB-ANSI01’
mysql[‘morpheus_db’] = '’
mysql[‘morpheus_db_user’] = '’
mysql[‘morpheus_password’] = '***’

When i issue :

morpheus-ctl reconfigure.

The Setup fails with this note

/opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid]

[2023-02-12T22:06:40+00:00] INFO: execute[/opt/morpheus/bin/morpheus-ctl start rabbitmq] ran successfully

  • execute /opt/morpheus/bin/morpheus-ctl start rabbitmq

execute[. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid] action run
[execute] Waiting for pid file ‘/var/run/morpheus/rabbitmq/rabbit@node02.pid’ to appear

Then it fails with the below details

do
action [:run]
default_guard_interpreter :execute
command “. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid”
backup 5
declared_type :execute
cookbook_name “morpheus-solo”
recipe_name “rabbitmq”
domain nil
user nil
end

System Info:

chef_version=14.14.29
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.10p210 (2022-04-12 revision 67958) [x86_64-linux]
program_name=/opt/morpheus/embedded/bin/chef-client
executable=/opt/morpheus/embedded/bin/chef-client
[2023-02-12T22:06:51+00:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2023-02-12T22:06:51+00:00] ERROR: Running exception handlers
Running handlers complete
[2023-02-12T22:06:51+00:00] ERROR: Exception handlers complete
Chef Client failed. 9 resources updated in 22 seconds
[2023-02-12T22:06:51+00:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2023-02-12T22:06:51+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-02-12T22:06:51+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid] (morpheus-solo::rabbitmq line 152) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘75’
---- Begin output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid ----
STDOUT: Waiting for pid file ‘/var/run/morpheus/rabbitmq/rabbit@node02.pid’ to appear
STDERR: Error: operation wait on node rabbit@node02 timed out. Timeout value used: 10000
---- End output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid ----
Ran . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid returned 75

Probably just need to manually do a morpheus-ctl kill rabbitmq and then a morpheus-ctl start rabbitmq from another console if it gets stuck there.

What should i do . The Goal is to use automation for install without manual intervention.

It would all just be guessing without knowing what your automation looks like. We have plenty of customers and partners automating the deploy, so it is possible.

A note, your morpheus.rb might just be renamed, but your rabbit hosts of node## seem off. Rabbit nodes should be the shortname of your vms and resolvable by dns.

change the rabbitmq[‘nodename’] = ‘s103’ which is pingable in DNS

10.x.y.z1 VA1DXCS10101 s101
10.x.y.z2 VA1DXCS10201 s102
10.x.y.z3 VA1DXCS10301 s103
*********** VA1DXCDB-ANSI01

is not helping. Its still failing
---- Begin output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid ----
STDOUT: Waiting for pid file ‘/var/run/morpheus/rabbitmq/s103.pid’ to appear
STDERR: Error: operation wait on node s103@VA1DXCS10301 timed out. Timeout value used: 10000
---- End output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid ----
Ran . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid returned 75

Please guide.