"Rabbit service listener failed" errors after upgrading to 6.0.2 or higher

Infrastructure
, , ,
1

Hello,

Lately, we have observed multiple scenarios where customers have upgraded their HA environment to 6.0.2 or higher and have encountered errors/issues related to their RabbitMQ cluster.

In an HA setup, after an upgrade, even though the UI services are up and running, errors related to rabbit service listeners can be seen in the logs on one or more nodes. This is because of the spring security change.

Error:

    2023-05-10_05:28:34.16753 '[2023-05-10 05:28:34,163] [grails.rabbit.exchange.morpheusSessionEvents.container-5] �[1;31mERROR�[0;39m �[36mc.b.r.RabbitErrorHandler�[0;39m - Rabbit service listener failed. 
    2023-05-10_05:28:34.16753 'org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception
    2023-05-10_05:28:34.16753 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1787)
    2023-05-10_05:28:34.16754 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1677)
    2023-05-10_05:28:34.16754 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1592)
    2023-05-10_05:28:34.16754 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1580)
    2023-05-10_05:28:34.16756 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1571)
    2023-05-10_05:28:34.16757 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1515)
    2023-05-10_05:28:34.16757 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:994)
    2023-05-10_05:28:34.16757 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:941)
    2023-05-10_05:28:34.16758 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:85)
    2023-05-10_05:28:34.16758 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1319)
    2023-05-10_05:28:34.16758 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1225)
    2023-05-10_05:28:34.16760 at java.base/java.lang.Thread.run(Unknown Source)
    2023-05-10_05:28:34.16760 Caused by: org.springframework.amqp.support.converter.MessageConversionException: failed to convert serialized Message content
    2023-05-10_05:28:34.16760 at org.springframework.amqp.support.converter.SimpleMessageConverter.fromMessage(SimpleMessageConverter.java:117)
    2023-05-10_05:28:34.16761 at org.springframework.amqp.rabbit.listener.adapter.AbstractAdaptableMessageListener.extractMessage(AbstractAdaptableMessageListener.java:343)
    2023-05-10_05:28:34.16761 at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:283)
    2023-05-10_05:28:34.16761 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1673)
    2023-05-10_05:28:34.16762 ... 10 common frames omitted
    2023-05-10_05:28:34.16762 Caused by: java.lang.IllegalArgumentException: Could not deserialize object
    2023-05-10_05:28:34.16762 at org.springframework.amqp.utils.SerializationUtils.deserialize(SerializationUtils.java:93)
    2023-05-10_05:28:34.16762 at org.springframework.amqp.support.converter.SimpleMessageConverter.fromMessage(SimpleMessageConverter.java:113)
    2023-05-10_05:28:34.16764 ... 13 common frames omitted
    2023-05-10_05:28:34.16764 Caused by: java.io.InvalidClassException: org.springframework.security.web.savedrequest.DefaultSavedRequest; local class incompatible: stream classdesc serialVersionUID = 2179821680723387540, local class serialVersionUID = 570
    2023-05-10_05:28:34.16764 at java.base/java.io.ObjectStreamClass.initNonProxy(Unknown Source)
    2023-05-10_05:28:34.16766 at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
    2023-05-10_05:28:34.16766 at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source)
    2023-05-10_05:28:34.16766 at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
    2023-05-10_05:28:34.16766 at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
    2023-05-10_05:28:34.16767 at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
    2023-05-10_05:28:34.16767 at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
    2023-05-10_05:28:34.16767 at java.base/java.util.HashMap.readObject(Unknown Source)

This probably indicates a failure in processing a message from a RabbitMQ message broker. The error message suggests that there was a problem converting the serialized content of a message to an object. The root cause of the error is an InvalidClassException that occurs during the deserialization of a DefaultSavedRequest object from the Spring Security library. The version of the DefaultSavedRequest class that was used to serialize the message is probably not compatible with the version of the class that is being used to deserialize the message, most likely due to a difference in the serialVersionUID values of the two classes. This is a common problem when using serialization for message passing between systems with different class versions.

The solution is to create a cleanup policy in the RabbitMQ console which will delete the existing queues immediately.

Follow the steps below on how to create the cleanup policy:

  1. Begin by stopping all three appliance nodes: morpheus-ctl stop morpheus-ui
  2. Then, log into the RabbitMQ console UI and go to the Admin tab. Expand the All Policies section and you should see your existing policies.
  3. Now create a cleanup policy as shown in the screenshot below. Since new policies are applied to existing queues, we can create a policy that will set them to expire after 1 ms to get rid of them all.
  • Virtual host: morpheus
  • Name: cleanup
  • Pattern: .*
  • Apply to: Exchanges and queues
  • Priority: 1
  • Definition: expires => 1

Click 'Add policy"

cleanup_policy
cleanup_policy868×986 167 KB

  1. Once this policy is added, existing queues will immediately be deleted. If your queues were full to the point where you experienced a noticeable change in performance, it may take a couple of minutes to fully clear everything out. We can go to the Overview tab and actually watch the number of existing queues drop toward zero.
  2. Once finished, remove the cleanup policy to purge the queues and save changes.

Attached are some KBs related to RabbitMQ that may be useful as well.

How to enable and access the RabbitMQ management console for all-in-one or 3-node Morpheus appliances

How to purge RabbitMQ queues

Fixing a RabbitMQ Inconsistent Cluster

How to set recommended RabbitMQ policies

Hope this will be helpful. :slight_smile:

Thanks
Deepti

3 Likes