Windows Agent Failing with Category Does Not Exist

Provisioning & Self Service
,
1

Would anyone be able to help identify cause of errors in Windows agent log.

 

Customer is experiencing this issue on only a few virtual machines with the build:

Windows Server 2019 Standard

Version 10.0.17763 Build 17763

Agent version: 1.6.0 and 1.7.0

They have machines with the same agent version and same windows server 2019 builds where the agents are communicating successfully.

The specific errors in the event viewer I see after the initial handshake during agent startup are:

�!�2��:#��N�w�)Cnh�ERROR:init instance error: Category does not exist.ssa8**hC����:#���N&
 
�!� �f;#��N�w�)Cn��ERROR:Subscription process message error: Object reference not set to an instance of an object.u00�**�I�g-*;#���N&

After this the agent stops/is unable to send the stats packets back to Morpheus appliance.

Here is a longer extract:

�s!�Ä��\�à"ÿ�®I™�����wÿ)Cn��PÅ��INFO:Received Stomp Frame: CONNECTED - obs �**»�E���\�à"ÿ������§�ëN&���������������������
����!�Ä��\�à"ÿ�©I™�����wÿ)Cn��¯Å��INFO:Received Stomp Frame: CONNECTED\nversion:1.2\nheart-beat:0,0\nuser-name:d635ae92-ee6d-4990-9644-fadf3d3dd957\n\n\u0000��»�**��F���\�à"ÿ������§�ëN&���������������������
��Y!�Ä��\�à"ÿ�™I™�����wÿ)Cn��6Å��INFO:Sending Message: [""]Jobs��**Ä�G���\�à"ÿ������§�ëN&���������������������
��◊!�Ä��\�à"ÿ�´I™�����wÿ)Cn��¥Å��INFO:Sending Message: ["SUBSCRIBE\ndestination:/user/queue/statCommands\nid:0\n\n\u0000"]]Ä�**h�H���\�à"ÿ������§�ëN&���������������������
��π!�Ä��\�à"ÿ�¨I™�����wÿ)Cn��ñÅ��INFO:Sending Message: ["SEND\ndestination:/app/requestStatJobs\n\n\u0000"]00"]h�**ê�I�*xØ�à"ÿ������§�ëN&���������������������
��Á!�Ä��\�à"ÿ�≠I™�����wÿ)Cn��ƒÅ��INFO:Sending Message: ["SUBSCRIBE\ndestination:/user/queue/morpheusAgentActions\nid:1\n\n\u0000"]yê�**8�J�*xØ�à"ÿ������§�ëN&���������������������
��ã!�Ä*xØ�à"ÿ�ÆI™�����wÿ)Cn��hÅ��ERROR:init instance error: Category does not exist.u008�**h�K�zü∂�à"ÿ������§�ëN&���������������������
��π!�Ä*xØ�à"ÿ�ØI™�����wÿ)Cn��ñÅ��INFO:Sending Message: ["SEND\ndestination:/app/requestStatJobs\n\n\u0000"]\u00h�**ÿ�L��¿¸?à"ÿ������§�ëN&���������������������
��-�!�Äzü∂�à"ÿ�∞I™�����wÿ)Cn��
�Å��INFO:Sending Message: ["SEND\ndestination:/app/serverInit\ncontent-type:application/json\n\n{\"osName\":\"windows\",\"osVersion\":\"\",\"agentVersion\":\"1.6.0.0\",\"capacityInfo\":{\"maxCores\":0,\"maxStorage\":321457741824,\"maxMemory\":17178800128}}\u0000"]"]ÿ�**h�M��¿¸?à"ÿ������§�ëN&���������������������
��π!�Ä�¿¸?à"ÿ�±I™�����wÿ)Cn��ñÅ��INFO:Sending Message: ["SEND\ndestination:/app/requestStatJobs\n\n\u0000"]00"]h�**H�N��¿¸?à"ÿ������§�ëN&���������������������
��ô�!�Ä�¿¸?à"ÿ�≤I™�����wÿ)Cn��v�Å��INFO:Received Stomp Frame: MESSAGE - {\"stats\":[{\"typeCode\":\"server\",\"objectId\":\"82\",\"frequency\":60000,\"config\":{}},{\"typeCode\":\"vmware\",\"objectId\":\"39\",\"containerName\":\"39_39\",\"externalId\":null,\"frequency\":60000,\"config\":{\"maxStorage\":322122547200}}],\"serverTime\":1644942158587}H�**ê�O���@à"ÿ������§�ëN&���������������������
��Â�!�Ä�¿¸?à"ÿ�≥I™�����wÿ)Cn��¬�Å��INFO:Received Stomp Frame: MESSAGE\ndestination:/user/queue/statCommands\ncontent-type:application/json;charset=UTF-8\nsubscription:0\nmessage-id:s6io0keb1yyuxeyl9aj-3740516\ncontent-length:241\n\n{\"stats\":[{\"typeCode\":\"server\",\"objectId\":\"82\",\"frequency\":60000,\"config\":{}},{\"typeCode\":\"vmware\",\"objectId\":\"39\",\"containerName\":\"39_39\",\"externalId\":null,\"frequency\":60000,\"config\":{\"maxStorage\":322122547200}}],\"serverTime\":1644942158587}\u0000��ê�**ê�P���@à"ÿ������§�ëN&���������������������
��„!�Ä��@à"ÿ�¥I™�����wÿ)Cn��¿Å��ERROR:Subscription process message error: Object reference not set to an instance of an object.ssaê�**¯�Q�îfƒcà"ÿ������§�ëN&���������������������
��I�!�Ä��@à"ÿ�µI™�����wÿ)Cn��&�Å��ERROR:Could not process message: {"stats":[{"typeCode":"server","objectId":"82","frequency":60000,"config":{}},{"typeCode":"vmware","objectId":"39","containerName":"39_39","externalId":null,"frequency":60000,"config":{"maxStorage":322122547200}}],"serverTime":1644942158587}inat¯�**h�R�≤Ãàáà"ÿ������§�ëN&�

 

When I checked it looked like it could be an issue obtaining performance counter metrics. On instruction the customer has rebuilt the performance counter and restarted the agent but the issue is still present.

2

That’s an odd one no idea what that is

3

Does restarting the service fix it

4

I can look when I get back

5

Thank you, yes the service has been restarted by the customer. I was not on a call to explicitly see them restart the service but I do see the initial handshake, i.e “CONNECTED\nversion:1.2…” about 7 times across the full event viewer log. Each time it hits the same. issue Across the course of the event viewer log I can see the agent was upgraded from 1.6.0 to 1.7.0 as well (as the version in the initial payload changes to 1.7.0) with the same error present.

6

And morpheus doesn’t say it’s connected ?

7

It shows it’s sending stats

8

Il double check what is being saved for latest agent stats on the server record - il get on a call with them. Attaching full event viewer log here as well in the mean time.

10

is because externalId is null?

11
I trace the issue back to: "Category does not exist" which appears to come from PerformanceCounterCategory class. This prevents InitInstance from being successful in ContainerMonitorService. We then get the "Subscription process message error"  when processing message for the stats subscription because ContainerMonitorService has not initiated correctly. I am going to see if answer on https://stackoverflow.com/questions/31349050/performancecounter-category-does-not-exist-even-though-it-exists gets any further although we have already done similar 
12

externalId is null in my successful windows 2019 agent test.

13
Bumping this thread with the root cause in case this is searched for. I discovered that the virtual machines with agents that were failing to send stats back to Morpheus were failing to initialise the stats service because there was a missing performance counter that the Agent was expecting to be able to obtain. The specific error thrown is "Category does not exist" which comes from PerformanceCounterCategory class. This prevents InitInstance from being successful in ContainerMonitorService. The agent then throws "Subscription process message error" when processing message for the stats subscription because ContainerMonitorService has not initiated correctly. 
 
Resetting the VM performance counters was not a solution in this case because there were registry keys set on the VMs that specifically disabled the performance counters. In this case, the VirtualDisk performance counter was being disabled. 
 
To resolve this I did the following:
 
1) Ran the following in Powershell as an administrator to identify if there are any registry key entries that have disabled performance counters within the VM 
reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services /s /v "Disable Performance Counters"
If entries were returned:
 
2) Opened regedit

  1. Went to the registry key where the match was found above

  2. Searched for Disable Performance Counters and made sure that the value was 0. You can either change the value to 0 or delete the entry.

Note: Create a backup of the entry and contact your system administrator before making changes to the registry.

  1. Restarted the Windows virtual machine

  2. Checked that all counters in the following command can be found:

Get-Counter -ListSet Process
Get-Counter -ListSet Processor
Get-Counter -ListSet "Network Interface"
Get-Counter -ListSet Memory
Get-Counter -ListSet PhysicalDisk
 
Then checked that agent was successfully sending stats back to Morpheus. 
 
15

Wow that is a great find! Would have lost days on that one myself.