Better error handiling in Morpheus cypher package

HI All

i belive morpheus cypher package needs to have a better error handling,.

we are running the below error almost every day when 5 to 8 jobs run at the same time, which use the morpheus cypher package. the reason might be most of our jobs, i guess 5 to 8 of them are scheduled at the same time, out of 8 jobs one the job is erroring out with below error.

Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-f85459cd-28c3-4ae8-9dc0-19b435a3b701/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 85, in __init__
    self.set_cypher_endpoint()
  File "/var/opt/morpheus/morpheus-local/workspace/py-f85459cd-28c3-4ae8-9dc0-19b435a3b701/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 103, in set_cypher_endpoint
    if LooseVersion(data['buildVersion']) < LooseVersion('5.3.3'):
                    ~~~~^^^^^^^^^^^^^^^^
KeyError: 'buildVersion'

i downloaded the source code for morpheus cypher package

at line number 103 in def set_cypher_endpoint(self) function,

current code,

    def set_cypher_endpoint(self):
        appliance_url = self.url
        url = appliance_url + "/api/ping"
        headers = {'content-type': 'application/json', 'Authorization': "BEARER %s" % self.token}
        with warnings.catch_warnings():
            warnings.filterwarnings("ignore", category=Warning)
            r = requests.get(url=url, headers=headers, verify=self.ssl_verify)
        data = r.json()
        ***if LooseVersion(data['buildVersion']) < LooseVersion('5.3.3'):***
            self.cypher_endpoint = "/api/cypher/v1/"
        else:
            self.cypher_endpoint = "/api/cypher/"

i think the code should be

def set_cypher_endpoint(self):
    appliance_url = self.url
    url = appliance_url + "/api/ping"
    headers = {'content-type': 'application/json', 'Authorization': "BEARER %s" % self.token}
    
    with warnings.catch_warnings():
        warnings.filterwarnings("ignore", category=Warning)
        r = requests.get(url=url, headers=headers, verify=self.ssl_verify)

    try:
        data = r.json()
        # Check if 'buildVersion' is present
        if 'buildVersion' in data and LooseVersion(data['buildVersion']) >= LooseVersion('5.3.3'):
            self.cypher_endpoint = "/api/cypher/"
        else:
            # Use the fallback endpoint if buildVersion is missing or < 5.3.3
            self.cypher_endpoint = "/api/cypher/v1/"
    except (ValueError, KeyError):
        # Fallback to /api/cypher/v1/ if there's a JSON error or missing 'buildVersion'
        self.cypher_endpoint = "/api/cypher/v1/"

who ever was developing that package, can you please take a look?

Tagging @ncelebic @cbunge

support ticket i opened for this issue has been closed,since this needs to be addressed here. can someone from Morpheus fix this issue please?

Hi @rmartha,

This module is not a core Morpheus product, and that’s why support won’t handle it directly. The repository is located at GitHub - tryfan/python-morpheus-cypher . I’d be happy to review a pull request or address an issue. Please note that there is no SLA for fixing this as this falls under our policy located here: Morpheus Support

Hi @ncelebic

here is the pull request

Thanks for the PR! After looking at this entire block more closely, it was causing more trouble than it was worth in testing. This was based on an API path change in version 5.3.3 of Morpheus, which is no longer relevant. I left an escape hatch open in the README, but otherwise just ripped it out.

I just pushed version 0.3.0 to PyPI. Let me know if you have any issues either in this thread, or you can raise an issue in the github repo.

Thanks!

@ncelebic now i am seeing this error

Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-772fabfd-e479-46de-9432-6b28a09530d0/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get
    raise Exception(data['msg'])
Exception: Invalid Vault Token

I am not getting this error when I run the module. This looks like a token issue in Morpheus as opposed to a module issue. I would need more detail to offer any more insight.

@ncelebic we see this error this morning too,

Traceback (most recent call last):
  File "", line 44, in 
  File "", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-04d507ad-579b-4f9d-9f14-f1a0e7c5e196/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get
    raise Exception(data['msg'])
Exception: Unauthorized. You must be authenticated to access this api

While you might not notice the error with a single execution, our scenario involves jobs that repeatedly execute this module more than ten times simultaneously.

@cbunge any suggestions on how we can get the apitoken from the below code?

self.token = morpheus['morpheus']['apiAccessToken']

That’s a run once token during an execution and immediately expires the next time that variable is called or when the execution completes.

I believe the var morpheus.executionLeaseToken allows concurrency for cypher access

@cbunge can you explan more about it? Do you have any sample code on how we cna use the var morpheus.executionLeaseToken ?

@cbunge any update on this?

@ncelebic @cbunge we are keep getting this exceptions, can one of you please help us here?

Traceback (most recent call last): File "<stdin>", line 44, in <module> File "<stdin>", line 25, in main File "/var/opt/morpheus/morpheus-local/workspace/py-605bc1f9-1e72-4027-9244-d75767499d25/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get raise Exception(data['msg']) Exception: Unauthorized. You must be authenticated to access this api

Traceback (most recent call last): File "<stdin>", line 44, in <module> File "<stdin>", line 25, in main File "/var/opt/morpheus/morpheus-local/workspace/py-a2c37c59-4866-48ff-80c7-31270aeced57/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get raise Exception(data['msg']) Exception: Invalid Vault Token

You would just use morpheus[“executionLeaseToken”] instead of morpheus[‘morpheus’][‘apiAccessToken’]

executionLeaseToken is valid during the task execution while apiAccessToken is valid until it is called again.

If you are using the cypher module you would use

from morpheuscypher import Cypher

value = Cypher(morpheus=morpheus,ssl_verify=False,token=morpheus["executionLeaseToken"]).get('secret/test')

print(value)

@ncelebic It might be good to update the code to prefer executionLeaseToken over apiAccessToken. Older versions won’t have executionLeaseToken so I could have it fall back to apiAccessToken. I will make a pull request.

Thank you somuch @Tyler_Boyd i did see the Pull request, will test and let you know once the code is merged.

You should be good to test now.

If you want to specify the version you can set: ‘morpheus-cypher==0.3.1’ in the additional packages field

Thank you @Tyler_Boyd will let you know if i run into any issues

@Tyler_Boyd did not see the error so far,.