Correctly set state and change indicator on "missing" vCenter VMs

Ideas
, ,
1

Headline:
Need a reliable way to consult Morpheus and figure out of the instance is still a legit instance that has an id tying it to vCenter. Or more simply, we need a reliable way to determine if a VM has been removed in vCenter (making it an orphan in Morpheus).

Description:
I have used the pyMorpheus API, to develop some administration tools. One tool called orphans.py, is supposed to loop through instances in Morpheus, and check the state/status fields of the instance (it also checks the fields in the server objects tied to the instance). If the instance is believed to be gone, the user gets to manually decide whether to delete that instance. What is happening right now, is that even if the VM is deleted in vCenter, the status and state fields do not seem to be setting correctly in the json of the instance/server objects such that I can reliably tell if it is gone from vCenter. I would rather not have to make API calls to both Morpheus and vCenter and do a reconciliation. So I am proposing that we set a state or status field to “missing” in the instance (or server if that is a better way to go), so that we can have something more reliable. Maybe we can even color code something on the Instances page accordingly. Really, I think this might be a defect, but was told to introduce this into Ideas by the Support Team. I looked into VRA (we have another side of the house who uses VRA), and asked them if and how they knew whether a VM has had its rug pulled out in vCenter, and they told me that VRA uses a field and populates the value as “missing”. For what that’s worth. Maybe we can do something consistent to that.

Example/Use case(s):
Launch a VM with Morpheus. Go into vCenter and clone the VM. The VM will show as Discovered in Infrastructure->VMs. Convert it to Managed. Then, delete the instance in vCenter. Morpheus will continue to show the instance as though it is still alive. If, on the next sync, we did not find the instance - meaning no VM with that ID found - we set a field to “missing”, we could reliably use API tools to go in and remove those instances on some kind of a basis. I know that removing instances sounds dangerous, but, we have a process where people are provisioning using external provisioners, and then we convert them to managed. And when they pull the VMs down and redeploy, we are stuck with 120+ orphan instances sitting in Morpheus that we need to use the GUI to remove. Hence the need for this field and related API automation.

2

How do you know (using the GUI) which items to remove? If you have a list, you could use the morpheus API to delete the instances

3

I don’t (can’t) figure out which ones to remove using the GUI. I am using some Python code to loop through the instances, and trying to figure out which ones are no longer in vCenter. In the JSON for the instance, there is a “status” field. This field says “running”. Unfortunately, this field (for me at least) remains set to running, even if the VM is removed directly from vCenter and a Cloud Refresh is done.

        "configGroup": null,
        "configId": null,
        "configRole": null,
        "dateCreated": "2023-02-15T21:32:45Z",
        "lastUpdated": "2023-03-16T14:06:21Z",
        "statsEnabled": true,
        "status": "running",
        "userStatus": null,
        "environmentPrefix": null,
        "stats": {
          "usedMemory": 41943040,
          "maxMemory": 4294967296,
          "maxStorage": 8589934592,
          "usedStorage": 0,
          "userCpuUsage": 0.0,
          "running": true
        },
        "runtimeInfo": {
        },

If I go down a level deeper, into the server object, I don’t really see any fields that are useful for determining the state of the VM wrt vCenter, except for POSSIBLY this one.

    "computeServerType": {
      "id": 103,
      "code": "vmwareVm",
      "name": "VMware Linux VM",
      "managed": true,
      "externalDelete": true
    },

I am looking at this externalDelete field, trying to figure out what that field means. Does this field mean that the VM was deleted externally? Because if it is, then bingo - I have exactly what I am looking for.

4

Yeah, we need a Morpheus team member probably to answer that. I can see in the documentation that it is a protected java class, but it doesn’t say what it means or what its purpose is.

https://developer.morpheusdata.com/api/com/morpheusdata/model/ComputeServerType.html#externalDelete