We are receiving feedback from potential customers that they would like to see the ability to Reset a Server added. In their expeirence, when a OS is locked up, performing a shutdown is not reliable as it attempts to perform a graceful shutdown which can hang. Virtual Machines are hosted on VMWare Clusters. Playing with Morpheus, I believe the “Stop Server” option performs a shutdown and so could hang? Personally I have also seen issue where server hang when running shutdown.
Morpheus attempts a guest shutdown / restart. If we are unable to initiate, we do a force shutdown. If your server responds to the initial shutdown or restart, and then ends up hung I believe attempting another shutdown will cause the system to force it.
It’s hard for me to recreate as my systems cycle without issue.
Thanks for the feedback, to get a better understanding I’ve run 4 quick tests this morning to get an idea of Morpheus behaviour when server is stuffed in someway. Tests were performed on a Rocky 9 server. Based on results have some questions at end of this post.
Test 1
Step1> Stop vmtools service, e.g. systemctl stop vmtoolsd.service
Step2> Do “Stop Server”
Result > Morpheus initiates OS shutdown, this fails so Morpheus Powers off VM
Test 2
Step1> Replace /usr/sbin/shutdown with /usr/bin/vi, e.g. cp /usr/bin/vi /usr/sbin/shutdown
Step2> Do “Stop Server”
Result> Morpheus initiates OS shutdown but shutdown hangs because it runs vi, after just over 5 minutes Morpheus Powers off VM
Test 3
Step1> Stop vmtools service, e.g. systemctl stop vmtoolsd.service
Step2> Do “Restart Server”
Result> Morpheus initiates OS shutdown, this fails so Morpheus Powers off VM / Powers on VM
Test 4
Step1> Replace /usr/sbin/shutdown with /usr/bin/vi, e.g. cp /usr/bin/vi /usr/sbin/shutdown
Step2> Do “Stop Server”
Result> Morpheus initiates OS shutdown but shutdown hangs because it runs vi, after just over 5 minutes Morpheus Powers off VM / Powers on VM
Question 1
If Morpheus is able to initiate OS shutdown, but shutdown does not complete in 5 minutes will Morpheus always power off VM? Wondering what happens if server is slow to shutdown. Seen servers running larges databases etc to take alot more than 5 minutes.
Question 2
When Morpheus stop job is running, as shown in stop1.jpg, menu options suspend/stop/start/restart can be invoked. Which means you can end up with two stop jobs running in parallel. Is this expected behaviour?
Answering the two questions you have asked for everyone’s benefit .
Answer to Question 1:
The functionality you have described is accurate and working as intended. Morpheus has a hard-coded time limit of 5 minutes to allow the server to stop gracefully. If it doesn’t stop within this timeframe, it is automatically powered off.
In terms of the example you mentioned, we would expect users to leverage the ‘Worflows’ (Library > Automation > Workflows) functionality to ensure that the database service on the server is stopped before the Shutdown phase begins. A good Provisioning workflow would look like this:
Write a script that stops your database service (e.g. sudo systemctl stop postgresql) - create this as a task under Library > Automation > Tasks
Create a new provisioning workflow under Library > Automation > Workflows
Link the task in step 1 to the StopService phase of your workflow
When the workflow is triggered, Morpheus will wait for the service to stop (since the StopService phase of a provisioning workflow precedes Shutdown) before triggering that 5 minute time limit for the instance to shut down.
Answer to Question 2:
We would suggest raising an Idea to grey out these options once one of them is triggered. I appreciate the point you are trying to make - it doesn’t make a lot of sense to have multiple shutdown jobs running one after the other.
Thanks for checking, I agree that using a workflow task to shutdown database is one possible solution. However I cannot help thinking its somewhat in elegant as, a) the customer needs to know about the issue, b) involves them doing extra work and c) means we need to grant them access to workflows and d) it can delay things.
Using Linux as the example, for systemd based services you can set a timeout, for example in mysql.service (lifted from a Morpheus DB server) timeout is set to 5 mins. Seen it alot longer for some other apps.
I can’t help thinking that a much better solution would be to have multiple options, shutdown (performs graceful shutdown), power off (kills server) and reset server. Customers know what kind of operation they want to perform.
Regarding question 2:
I’ll raise an idea. There are several places where menu options should be disabled/greyed out.
.