Ubuntu18: Failing on RabbitMQ during morpheus-ctl reconfigure

Infrastructure
1

OS: Ubuntu 18
Linux VA*** 4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
.
installed morpheus-appliance_5.4.14-1_amd64.deb
installed morpheus-appliance-supplemental_5.4.14-1_all.deb

morpheus.rb file

appliance_url ‘https://VA1DXCS10201…com’
elasticsearch[‘es_hosts’] = {‘VA1DXCS10101’ => 9200, ‘VA1DXCS10201’ => 9200, ‘VA1DXCS10301’ => 9200}
elasticsearch[‘node_name’] = ‘VA1DXCS10201’
elasticsearch[‘host’] = ‘0.0.0.0’
rabbitmq[‘host’] = ‘0.0.0.0’
rabbitmq[‘nodename’] = ‘rabbit@node02’
mysql[‘enable’] = false
mysql[‘host’] = ‘VA1DXCDB-ANSI01’
mysql[‘morpheus_db’] = '’
mysql[‘morpheus_db_user’] = '’
mysql[‘morpheus_password’] = '***’

When i issue :

morpheus-ctl reconfigure.

The Setup fails with this note

/opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid]

[2023-02-12T22:06:40+00:00] INFO: execute[/opt/morpheus/bin/morpheus-ctl start rabbitmq] ran successfully

  • execute /opt/morpheus/bin/morpheus-ctl start rabbitmq

execute[. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid] action run
[execute] Waiting for pid file ‘/var/run/morpheus/rabbitmq/rabbit@node02.pid’ to appear

Then it fails with the below details

do
action [:run]
default_guard_interpreter :execute
command “. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid”
backup 5
declared_type :execute
cookbook_name “morpheus-solo”
recipe_name “rabbitmq”
domain nil
user nil
end

System Info:

chef_version=14.14.29
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.10p210 (2022-04-12 revision 67958) [x86_64-linux]
program_name=/opt/morpheus/embedded/bin/chef-client
executable=/opt/morpheus/embedded/bin/chef-client
[2023-02-12T22:06:51+00:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2023-02-12T22:06:51+00:00] ERROR: Running exception handlers
Running handlers complete
[2023-02-12T22:06:51+00:00] ERROR: Exception handlers complete
Chef Client failed. 9 resources updated in 22 seconds
[2023-02-12T22:06:51+00:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2023-02-12T22:06:51+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-02-12T22:06:51+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[. /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid] (morpheus-solo::rabbitmq line 152) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘75’
---- Begin output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid ----
STDOUT: Waiting for pid file ‘/var/run/morpheus/rabbitmq/rabbit@node02.pid’ to appear
STDERR: Error: operation wait on node rabbit@node02 timed out. Timeout value used: 10000
---- End output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid ----
Ran . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/rabbit@node02.pid returned 75

2

Probably just need to manually do a morpheus-ctl kill rabbitmq and then a morpheus-ctl start rabbitmq from another console if it gets stuck there.

3

What should i do . The Goal is to use automation for install without manual intervention.

4

It would all just be guessing without knowing what your automation looks like. We have plenty of customers and partners automating the deploy, so it is possible.

A note, your morpheus.rb might just be renamed, but your rabbit hosts of node## seem off. Rabbit nodes should be the shortname of your vms and resolvable by dns.

5

change the rabbitmq[‘nodename’] = ‘s103’ which is pingable in DNS

10.x.y.z1 VA1DXCS10101 s101
10.x.y.z2 VA1DXCS10201 s102
10.x.y.z3 VA1DXCS10301 s103
*********** VA1DXCDB-ANSI01

is not helping. Its still failing
---- Begin output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid ----
STDOUT: Waiting for pid file ‘/var/run/morpheus/rabbitmq/s103.pid’ to appear
STDERR: Error: operation wait on node s103@VA1DXCS10301 timed out. Timeout value used: 10000
---- End output of . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid ----
Ran . /opt/morpheus/embedded/rabbitmq/.profile;/opt/morpheus/embedded/rabbitmq/sbin/rabbitmqctl wait /var/run/morpheus/rabbitmq/s103.pid returned 75

Please guide.