Troubleshooting Morpheus VDI using VDI Gateway
In this post i want to give an overview of the process of opening a VDI console in Morpheus via a VDI Gateway. The diagram below attempts to show the connection sequence. Each step is then discussed in detail with some troubleshooting hints to help track down any potential issues. In the post I have used a Firefox browser with the standard Developer tools. You will also need access to the linux command window on the Morpheus Appliance and VDI Gateway.
Connection Sequence Diagram
Narrative
| Step | Source | Target | Action | Narrative |
|---|---|---|---|---|
| 1 | Browser | Appliance | GET https://applianceUrl/vdi/allocate/<id> | From the VDI Persona - click on the vdi Pool to open up the VDI console kicking off the connection sequence |
| 2 | Appliance | Browser | RESPONSE 200 | Morpheus appliance responds with json string {"success":true,"gatewayUrl":"https://gatewayUrl","allocationId":<allocationId>,"remoteApp":null,"preparing":false} |
| 3 | Browser | VDIGateway | GET https://gatewayUrl/vdi/terminal/<allocationId> | Client Browser connects to VDI Gateway specified in the response from step 2 using the gatewayUrl and allocationId
contained in the response. Worker Log Entry INFO c.m.worker.controller.VdiController - Checking Appliance URL: https://applianceUrl |
| 4.1 | VDIGateway | Browser | 307 TEMP REDIRECT | The VDI Gateway responds with a status 307 and specifies the redirect URL https://applianceUrl/vdi/terminal/<allocationId> in the headers |
| 4.2 | Browser | Appliance | REDIRECT https://applianceUrl/vdi/terminal/<allocationId> | Opens up a new tab in the browser to host the vdi console. |
| 5 | Appliance | Appliance | create consoleLease | The appliance creates a consoleLease in the vdi_connection_lease table. The lease has a 10 minute timeout. This action combined with step 4.3 writes a Audit entry in the appliance UI logs INFO c.m.AuditLogService - CEF:0|MorpheusData|Morpheus|6.0.6|terminal#showVdi|VDI Console Accessed|src=<clientIpAddress> suid=1 suser=user request=http://applianceUrl/vdi/terminal/<allocationId> requestMethod=GET cs1Label=Object Type cs1=instance cn1=85 cn1Label=Object ID cs2=SP60-L-1095 cs2Label=Object Name cn2=1 cn2Label=Account ID |
| 6 | Browser | VDIGateway | Open Websocket GET wss://gatewayUrl>/remote/socket?leasetoken=<consoleLease> |
Console web page uses the Guacamole object to open a websocket on the VDIGateway. Using the browser tools you can see this happening. Watch for status code 101 |
| 7 | VDIGateway | Appliance | GET https://applianceUrl/api/worker/consoleInfo/<consoleLease> | VDI Gateway verifies the consolLease specified on the web socket using an api call back to the appliance. The VDIGateway APIKey must be specified on the headers Logged on the Appliance in the UI logs INFO c.m.v.VdiGatewayService - looking for lease: <consoleLease> |
| 8 | Appliance | VDIGateway | API response. Encrypted object containing lease | Lease is verified and contains a encryption key for the tunnel. Logged in the Appliance UI logs c.m.v.VdiGatewayService - Lease Found! <consoleLease> |
| 9 | VDIGateway/td> | VDI Instance | Guacamole connection opened by VDIGateway | An connection to the instance is opened on the VDIGateway by guacamole. This is then proxied through the encrypted tunnel over the websocket opened in step 6 |
Troubleshooting
Troubleshooting VDI Gateway connection can be tricky. However, the basic requirements below must be met.
For Steps 1 and 2 the client must be able to reach the Morpheus Appliance on port 443
For step 3 the client must be able to resolve the url of the VDIGateway defined in /etc/morpheus/morpheus-worker.rb and also reach this host over port 443.
For step 4.1 and 4.2 The redirect URL is defined in the /etc/morpheus/morpheus-worker.rb file as worker[‘appliance_url’]. This should match the url of the Morpheus appliance as defined in /etc/morpheus/morpheus.rb on the appliance.
Step 5 as mentioned in the narrative, this step writes an audit entry into the Morpheus Appliance UI logs. The url in the log entry should match the REDIRECT url specified in step 4.1 and 4.2. At this point a connectionLease should also have been created in the vdi_connection_lease database table. The lease has a 10 minute timeout so the connection needs to be established within this time frame.
For Step 6 use the browser developer tools to verify the websocket upgrade. The http response should be 101 Switching Protocols. On the websocket uri the consoleLease guid is shown as the leaseToken query parameter
Steps 7 The worker tries to obtain the connectionLease by issuing an api call to the morpheus appliance. This is logged in the Appliance UI logs as detailed in the narrative.
INFO c.m.v.VdiGatewayService - looking for lease: <consoleLease>
You should also see the API call recorded in the nginx logs similar to this
<gatewayIp> - - [06/Oct/2023:12:27:12 +0000] "GET /api/worker/consoleInfo/<connectionLease> HTTP/1.1" 200 2678 "-" "Morpheus-Worker"
Step 8 contains the response to the API call in step 7 above. This is logged in the Morpheus UI logs.
INFO c.m.v.VdiGatewayService - Lease Found! <consoleLease>
Step 9 At this point the websocket should have opened up a secure encrypted Guacamole tunnel. In this final step the VDIGateway connects to the instance over the remote protocol (RDP,ssh or VNC or Hypervisor console). To acheive this the VDIGateway must be able to communicate over the remote protocol required by the VDI Instance. Connection back to the client browser is proxied through the websocket over port 443.
To see the connection being established you can run this command on the VDIGateway
sudo watch -n 1 "ss -4 -t -p -i -a | grep guacd"
With no connection active the output should look something like
Every 1.0s: ss -4 -t -p -i -a | grep guacd
LISTEN 0 5 127.0.0.1:4822 0.0.0.0:* users:(("guacd",pid=124672,fd=4))
As the VDI connection is launched this should be recorded in the output
Every 1.0s: ss -4 -t -p -i -a | grep guacd
LISTEN 0 5 127.0.0.1:4822 0.0.0.0:* users:(("guacd",pid=173447,fd=4),("guacd",pid=124672,fd=4))
ESTAB 0 0 127.0.0.1:4822 127.0.0.1:<nnnnn> users:(("guacd",pid=173447,fd=5),("guacd",pid=124672,fd=5))
ESTAB 0 0 <VDIGatewayIP>:<nnnnn> <VDIInstanceIP>:ssh users:(("guacd",pid=173447,fd=10))
Above you can see the VDI has opened an ssh session to the vdi through the guacd service.
Hope you find this guide useful for troubleshooting VDI