@mreed
We began testing the Datadog Plugin in nonprod and noticed the tab says Host Not Found. Upon inspecting the logs it looks like it may be sending the API call with the ? character URL encoded as %3F causes a 404 error. When I tested this through Postman with the same API key and App key to the same Datadog endpoint I was able to find the host and then change the ? to %3F and got the 404 error.
It looks like the API call happens around line 132 of the plugin code in the DataDogTabProvider.groovy
After performing some testing it looks like the code in the API call method used has been updated to use the literal question mark instead of encoding it. The latest version of the DataDog plugin should be installed to leverage this update (https://github.com/martezr/morpheus-datadog-instance-tab-plugin/releases/download/v0.0.8/morpheus-datadog-instance-tab-plugin-0.0.8.jar). Additionally, there have been changes to the plugin settings as well. The checkbox for the govcloud endpoint has been replaced by a text field as DataDog has continued to add additional API endpoints and a customization text area has been added to as a way to explicitly define a scalable method for acquiring the instance’s hostname beyond just the default instance name (if this is not required then the text area should be empty).
Awesome! I get the host now in the Datadog tab via the plugin. The Overview section populates with the host info and agent checks. The System Info section populates with the sys info summary as well as the platform, cpu, and memory details. The only catch is the Actions button which offers the Open Dashboard action that is just a hyperlink to the Datadog UI with a query parameter to try and display the host by hostname. There seem to be two issues with the way this works:
The subdomain on the URL is targeting https://app.[api-endpoint]/infrastructure?tags=host%3A[hostname] which may work for some customers, but in our case the subdomain “app” needs to be the account name we’re logging into. We have different account names that related to each tenant in Morpheus. So the master tenant lets call it “main” would be https://main.[api-endpoint] and our first subtenant lets say is team-bobby would be https://team-bobby.[api-endpoint]. However, there doesn’t seem to be a way to customize this Open Dashboard URL at the moment.
The next issue related to the URL is with using the query parameter “tags=host%3A[hostname]” which doesn’t appear to work for us. I tried just copy the URL from the action Open Dashboard and pasting it into the browser, editing the subdomain to match ours, and then browsing it. It does open the Infrastructure list but the query only seems to input the filter into the search field in the Datadog UI and then it shows no results in the list of hosts. This is really odd because if I clear the search field and then type the same filter (let’s say it’s hostname is abc) “host:abc” then it filters to that host in the list and the URL updates to add that query parameter in the same format you programmed it to be: https://[subdomain].[api-endpoint]/infrastructure?tags=host%3Aabc
and from what I can see in the network tab of Chrome Developer Tools it kicks off an API call behind the scenes to retrieve the host into the infrastructure list. The API call that the Datadog UI kicks off looks the same to me even if search for the host:abc in the field instead of using the query parameter, but it works from the search field. Maybe there’s a cookie I haven’t noticed or something I’m not seeing that makes the difference.
Another idea I had was when you initially query for the host to populate the Datadog tab for the plugin you should also get the ID for the host as part of that API call. The ID is unique to the host in Datadog’s DB and if you were to store that information in Morpheus as part of the plugin then you could change the URL of the action “Open Dashboard” to go to this URL instead then it can skip the infrastructure list and go directly to the dashboard: https://[subdomain].[api-endpoint]/dash/host/[id]
Updating the URL used is easy but the tricky part that I’ve been thinking about is having a good way to reference the subdomain used in this on a per tenant basis. The options that come to mind are the usual places like tags or environment variables associated with the instance. There could also be a static list in the plugin settings for a mapping of tenant to subdomain but that could get a bit unwieldy pretty quickly. I’ll give it some more thought on a scalable location that the necessary information could be stored and referenced. Additionally, there’s also the option of providing an option to disable that drop-down altogether.
I should probably clarify that the Datadog subdomains for our different accounts don’t match up exactly with our tenant names in Morpheus so some way to specify the subdomains in variables or something would be needed. So not sure the best approach.
The latest version of the DataDog plugin (Release v0.0.9 · martezr/morpheus-datadog-instance-tab-plugin · GitHub) has been updated to provide support for handling the multi-organization use case. The “Plugin Customization” field in the plugin settings has been extended to handle a mapping between a Morpheus tenant/subtenant and the DataDog organization subdomain.
Looks like the subnet customization work in the main tenant. I was attempting to test this in a subtenant as well in our non-prod Morpheus appliance, but for some reason the instance in the subtenant doesn’t seem to be able to display the Datadog tab anymore. I tried a bunch of things to get the tab to display, but nothing seems to work.
Also in the main tenant when I adjusted the subdomain using your new customization options I noticed that the link now goes to: https://[subdomain].[api-endpoint]/dash/host_name/[instance-name]
This doesn’t appear to work with Datadog’s UI.
I tried browsing to the URL and got: {"errors":["Host Dashboard with ID None not found"]}
So I think it still needs to know the ID from Datadog which will be a bunch of numbers. Ours are currently 8 digits. https://[subdomain].[api-endpoint]/dash/host/[id]
I did find that there is a new role based permission related to the Datadog plugin tab. However, even after I enabled this for the Tenant Permission Boundary, for the Impersonated Admin Role, and did the same within the subtenant for the Tenant Admin Role and the Impersonated Admin Role again nothing changed. I was still unable to see the tab in the instance view. I have the tag “datadog” set for visibility in the plugin options. I haven’t seen any messages in the logs about the plugin when I try to get it to display in the subtenant.
Correction, I had to log out and log back in after fixing the Role permissions. I’m able to see the Datadog tab now in the subtenant and I can confirm that the subdomain is showing correctly based on the plugin configuration I made. So all that is left is the issue of the link needing to go to the host ID instead of the hostname to show the dashboard in Datadog’s web UI.
@michael.dunagan I’m glad to hear that. I’ll take a look at updating the code to fetch the DataDog ID and store it for reference purposes to use for the link.
2024-04-03_21:15:45.64480 ''[2024-04-03 21:15:45,646] [http-nio-127.0.0.1-8080-exec-9] INFO c.m.t.DataDogTabProvider - DataDog Plugin: Using vmtest064 as the DataDog hostname due to no additional settings being defined
2024-04-03_21:15:45.64608 ''[2024-04-03 21:15:45,646] [http-nio-127.0.0.1-8080-exec-9] INFO c.m.t.DataDogTabProvider - DataDog Plugin: Configured API endpoint https://api.[datadog-endpoint].com
2024-04-03_21:15:45.64657 ''[2024-04-03 21:15:45,646] [http-nio-127.0.0.1-8080-exec-9] INFO c.m.c.u.RestApiUtil - calling api: https://api.[datadog-endpoint].com/api/v1/hosts?filter=vmtest064
2024-04-03_21:15:45.64663 ''[2024-04-03 21:15:46,060] [http-nio-127.0.0.1-8080-exec-9] ERROR c.m.t.DataDogTabProvider - DataDog Plugin: Error in fetching data from DataDog java.lang.NumberFormatException: For input string: "3738172kB"
The datadog API sometimes, unfortunately, returns the total memory value with a unit symbol appended so the conversion to an Integer is failing. I’ve added some input validation and use of a different data type to handle this better.
Raised a pull request so I’ll discuss with @mreed next week and update this thread if/when it’s merged
