Installing Morpheus for RHEL 8 FIPS mode build-id conflicts with file from package

Infrastructure
1

While attempting to install Morpheus 6.2.3 FIPS on a RHEL 8 FIPS enabled instance that was provisioned by Morpheus, I ran into this conflict message:

[root@abc123~]# rpm -ivh morpheus-appliance-fips-6.2.3-1.el8.x86_64.rpm
Verifying… ################################# [100%]
Preparing… ################################# [100%]
file /usr/lib/.build-id/87/c0ba88d3a97835ad7cee91b4e1226bd7f44128 from install of morpheus-appliance-fips-6.2.3-1.el8.x86_64 conflicts with file from package morpheus-vm-node-fips-3.2.18-1.el8.x86_64

Any help is appreciated.

2

Looks like it is having an issue with the morpheus agent already being installed on the system. You could try removing that and trying again.

3

Can add --force to proceed with the install. Looking into what file(s) conflict

4

I removed the agent, performed a reboot, and then install proceeded successfully. Then I tried to do a reconfigure which failed due to virt-devel
++++++++++++++++
[2023-12-13T11:55:37-08:00] INFO: template[/opt/morpheus/embedded/nginx/conf/nginx.conf] sending restart action to runit_service[nginx] (delayed)
Recipe: morpheus-solo::nginx

  • runit_service[nginx] action restart (up to date)

Running handlers:
[2023-12-13T11:55:38-08:00] ERROR: Running exception handlers
Running handlers complete
[2023-12-13T11:55:38-08:00] ERROR: Exception handlers complete
Chef Client failed. 261 resources updated in 02 minutes 20 seconds
[2023-12-13T11:55:38-08:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2023-12-13T11:55:38-08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-12-13T11:55:38-08:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[enable_virt-devel_module] (morpheus-solo::guac line 147) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of yum -y module enable virt-devel ----
STDOUT: Last metadata expiration check: 0:01:17 ago on Wed 13 Dec 2023 11:54:17 AM PST.
STDERR: Error: Problems in request:
missing groups or modules: virt-devel
---- End output of yum -y module enable virt-devel ----
Ran yum -y module enable virt-devel returned 1

5

Is the RHEL 8 CodeReady repository enabled? The previous steps if it is enabled, and if it not it will enabled it, and then it is checking is virt-devel module is enabled, and enables that if it is not, but seem the codeready step didnt succeed. Is this a rhel 8 ami? Can you try the below and reconfigure again?
dnf config-manager --set-enabled codeready-builder-for-rhel-8-rhui-rpms

6

Thank you, got a lot further through the reconfigure and now I’m having trouble with guacd:
++++++++++++++
Running handlers:
[2023-12-13T13:31:21-08:00] ERROR: Running exception handlers
Running handlers complete
[2023-12-13T13:31:21-08:00] ERROR: Exception handlers complete
Chef Client failed. 207 resources updated in 09 minutes 06 seconds
[2023-12-13T13:31:21-08:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2023-12-13T13:31:21-08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-12-13T13:31:21-08:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: runit_service[guacd] (morpheus-solo::guac line 423) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/guacd ----
STDOUT: timeout: down: /opt/morpheus/service/guacd: 0s, normally up, want up
STDERR:
---- End output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/guacd ----
Ran /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/guacd returned 1

7

could run another reconfigure. I believe that is stating it timed out just waiting on guacd service to restart.

8

I ran another reconfigure which finished this time, but the guacd process is still not starting. I tried morpheus-ctl stop to bring all of them down and a morpheus-ctl start to bring them up, but guacd start still times out
+++++++++++++++
Recipe: morpheus-solo::bootstrap

  • file[/var/opt/morpheus/.bootstrap] action create (up to date)
    [2023-12-13T14:18:02-08:00] INFO: Chef Run complete in 42.360793753 seconds

Running handlers:
[2023-12-13T14:18:02-08:00] INFO: Running report handlers
Running handlers complete
[2023-12-13T14:18:02-08:00] INFO: Report handlers complete
Chef Client finished, 53/517 resources updated in 44 seconds
morpheus Reconfigured!
[root@abc123~]#
[root@abc123~]#
[root@abc123~]#
[root@abc123~]# morpheus-ctl stop
ok: down: check-server: 0s, normally up
ok: down: elasticsearch: 0s, normally up
ok: down: guacd: 0s, normally up, want up
ok: down: morpheus-ui: 0s, normally up, want up
ok: down: mysql: 1s, normally up
ok: down: nginx: 0s, normally up
timeout: run: rabbitmq: (pid 100181) 2411s, want down
[root@abc123~]#
[root@abc123~]#
[root@abc123~]# morpheus-ctl stop
ok: down: check-server: 15s, normally up
ok: down: elasticsearch: 12s, normally up
ok: down: guacd: 12s, normally up
ok: down: morpheus-ui: 12s, normally up
ok: down: mysql: 11s, normally up
ok: down: nginx: 10s, normally up
ok: down: rabbitmq: 1s, normally up
[root@abc123~]#
[root@abc123~]#
[root@abc123~]# morpheus-ctl start
ok: run: check-server: (pid 143859) 0s
ok: run: elasticsearch: (pid 143880) 1s
timeout: down: guacd: 0s, normally up, want up
ok: run: morpheus-ui: (pid 144198) 0s
ok: run: mysql: (pid 144203) 1s
ok: run: nginx: (pid 144440) 0s
ok: run: rabbitmq: (pid 144574) 1s
[root@abc123~]#
[root@abc123~]#
[root@abc123~]# morpheus-ctl status
run: check-server: (pid 145014) 177s; run: log: (pid 5931) 8803s
run: elasticsearch: (pid 143880) 196s; run: log: (pid 3389) 8857s
down: guacd: 0s, normally up, want up; run: log: (pid 90033) 3215s
run: morpheus-ui: (pid 147884) 0s; run: log: (pid 90099) 3208s
run: mysql: (pid 144203) 187s; run: log: (pid 3057) 8912s
run: nginx: (pid 144440) 186s; run: log: (pid 5829) 8808s
run: rabbitmq: (pid 144574) 186s; run: log: (pid 3693) 8839s
[root@abc123~]#

9

I found the log file for guacd
/var/log/morpheus/guacd/current
there are a bunch of permission denied messages for /usr/local/sbin/guacd
however when I check the permissions I see
755 root:root /usr/local/sbin/guacd
which is the same on our other morpheus server
the morpheus-app user belongs to the same groups on this new server and the old one
morpheus-app and morpheus-local

Looks like the guacd process is run by a user called guac-morpheus
Again this user has the same permissions on the old instance and the new instance.

10

Okay I’ve definitely given the user enough permission but still seeing the same error in the guacd logs.
I ran the following and attempted to start the guacd process at each iteration:
setfacl -m u:guac-morpheus:r-x /usr/local/sbin/guacd
setfacl -m u:guac-morpheus:r-x /usr/local/sbin/
setfacl -m u:guac-morpheus:r-x /usr/local/
setfacl -m u:guac-morpheus:r-x /usr/
then when none of the above worked I added guac-morpheus to the root group and tried again.
I’m still getting permission denied error message in the guacd logs

Last thing I’ve tried is a reboot of the system to see if that would help. Still seeing permission denied in the logs.

11

It may be easiest to open a support ticket and working through the issues you have. I know guacd is dependent on codeready repo as well to compile.

12

In case anyone else needs to troubleshoot similar issues. The temporary fix was to disable fapolicy to get guacd and the morpheus-ui running. I need to add some kind of rule to fapolicy to allow the files Morpheus uses to execute without issue.

2 Likes