Exception: token not specified in ENV or morpheus 'morpheus' 'apiAccessToken'

We have a Pythion script to start the RDS instances through Workflow, Recently we created few jobs to start/stop all the RDS instances using the workflow.

So our Subtenant have 13 RDS instance’s, using the Jobs and a schedule they are stopping the RDS instances everyday evening 8pm ad starting them in the morning 7am,

Since this all 13 Jobs using the same schedule most of the RDS instances are starting/stopping fine, but some of them are getting the below exception, in logs i don’t see any error,

is any one seen this error before?

Task Execution Failed on Attempt 1
Traceback (most recent call last):
File “”, line 29, in
File “/var/opt/morpheus/morpheus-local/workspace/py-55779591-8d13-4453-84ac-3aefb8aa2ab1/lib/python3.7/site-packages/morpheuscypher/init.py”, line 84, in init
raise Exception(“token not specified in ENV or morpheus[‘morpheus’][‘apiAccessToken’]”)
Exception: token not specified in ENV or morpheus[‘morpheus’][‘apiAccessToken’]

Are these all using the same task/workflow and just passing different instance arguments in?

Also, what version of Morpheus are you running into this issue? Did this exist on prior versions?

Correct, these all are using the same task/workflow and passing different instance names.

We started using this in 5.4.4 and we are in 5.4.5 right now,

i see the errors in both 5.4.4 and 5.4.5

here is the python code i use to create the Client

import boto3

import botocore.exceptions

import botocore.errorfactory

import time

import datetime

import sys

from pprint import pprint

from operator import itemgetter

from morpheuscypher import Cypher

from tabulate import tabulate

import random

from random import randint

from time import sleep

sleeptime = random.randint(5, 15)

#print(“sleeping for:”, sleeptime, “seconds”)

time.sleep(sleeptime)

#print(“sleeping is over”)

grab current time for role and DB Snapshot names

time_stamp = time.strftime("%Y%m%d-%H%M%S")

input_cloud = morpheus[‘customOptions’][‘RDSCLOUDS’]

#input_role_arn = sys.argv[1]

input_role_arn = Cypher(morpheus=morpheus).get((‘secret/AWS’ + str(input_cloud) + ‘role_arn’))

#pprint(input_role_arn)

def assume_role(role_arn, region):

"""Grabs temporary keys of assumed roles from ARN

Args:

    role_arn: ARN of the role you want to assume

    region: AWS Region based on Environment selected input_Environemnt

Returns:

    temp_key: Temporary key for assumed role

    temp_secret: Temporary secret key for assumed role

"""

# set resource and region

client = boto3.client(

    'sts',

    region_name = region

    )

response = client.assume_role(

    RoleArn=role_arn,

    RoleSessionName='RDSTask' + time_stamp

    )

temp_key = response['Credentials']['AccessKeyId']

temp_secret = response['Credentials']['SecretAccessKey']

temp_token = response['Credentials']['SessionToken']

return temp_key, temp_secret, temp_token

def rds_client(temp_key, temp_secret, temp_token, region):

"""Sends Boto3 tasks based on task input

Args:

    temp_key: Temporary key from assume_role function

    temp_secret: Temporary secret ket from assume_role function

    temp_token: Temporary session token from the assume_role function

    task: Input task to run corresponding Boto3 task

    db_name: Name of RDS to run task against

    region: AWS Region based on Environment selected input_Environemnt

Returns:

    AWS JSON message

"""

# set resource and region

client = boto3.client(

    'rds',

    region_name = region,

    aws_access_key_id = temp_key,

    aws_secret_access_key = temp_secret,

    aws_session_token = temp_token

    )

return client

def rds_service(task, db_name, client):

# run task based on task input

rds_instances = []

rdsinstances = client.describe_db_instances()

for i in rdsinstances['DBInstances']:

    rds_instances.append(i['DBInstanceIdentifier'])

if db_name in rds_instances:

    response = "response"

    if task == "start":

        response = client.start_db_instance(

            DBInstanceIdentifier=db_name

        )



    if task == "stop":

        try:

            response = client.stop_db_instance(

                DBInstanceIdentifier=db_name,

                #DBSnapshotIdentifier=db_name+time_stamp

@ncelebic Any idea why they seem to be hitting the token exception from morpheuscypher on some tasks and not others. Assuming its raising this exception if self.token is None:
raise Exception(“token not specified in ENV or morpheus[‘morpheus’][‘apiAccessToken’]”)

I was able to reproduce that error in my environment, hopefully the below will indicate the same. In a different (or the same) python task, place this in your code:
print(morpheus['morpheus']['apiAccessToken'])

Run the task and choose an instance to run it against. That should print out the apiAccessToken for the instance. Run it against a working instance and non-working instance. I assume you’ll receive a “None” back on a non-working system, whereas on a working system you’ll receive an API token back.

If this is the case, this can be due to a user being deleted in Morpheus, which may have been the one that originally deployed the instance(s). These values will be removed upon that user deletion.

This entry should return the username associated with the instance:
print(morpheus['username'])

@kgawronski Thanks for recreating the issue though, but here we are not having issues for the same RDS instances everyday, for example, if an instance start fine today and the same instance will not have the problem tomorrow, basically same RDS instance is not having issues every day.

every day different RDS instances have problem stop/start.

Here is the another error i see very as well for the same behavior

Task Execution Failed on Attempt 1
Traceback (most recent call last):
File “”, line 29, in
File “/var/opt/morpheus/morpheus-local/workspace/py-82d0ead8-12ec-44c2-8e01-251e6c0478e5/lib/python3.7/site-packages/morpheuscypher/init.py”, line 85, in init
self.set_cypher_endpoint()
File “/var/opt/morpheus/morpheus-local/workspace/py-82d0ead8-12ec-44c2-8e01-251e6c0478e5/lib/python3.7/site-packages/morpheuscypher/init.py”, line 103, in set_cypher_endpoint
if LooseVersion(data[‘buildVersion’]) < LooseVersion(‘5.3.3’):
KeyError: ‘buildVersion’

@kgawronski i have added those print commands yesterday and i am not able to see the api AccessToken or user name here is the same error i see as well

Task Execution Failed on Attempt 1

Traceback (most recent call last):
File “”, line 29, in
File “/var/opt/morpheus/morpheus-local/workspace/py-c863bc7e-1931-4ed6-bec1-b376d4f9d956/lib/python3.7/site-packages/morpheuscypher/init.py”, line 84, in init
raise Exception(“token not specified in ENV or morpheus[‘morpheus’][‘apiAccessToken’]”)
Exception: token not specified in ENV or morpheus[‘morpheus’][‘apiAccessToken’]

You may want to put those commands I mentioned in a separate task and run them, seems the Cypher module is looking for that value and throwing the error before output can happen.

That said, it might help you in a direction but it would be best to place a support ticket for a more thorough troubleshooting.

@kgawronski I have downloaded the Cypher package from morpheus-cypher · PyPI. When i look at the init.py line number 84,

Some reason the token is getting null when token accessed multiple times at the same time, i think they can put a try block or something to re-try when it is null. I also opened a support ticket with all this information, i hope they will update this package soon,

class Cypher:
def init(self, url=None, token=None, morpheus=None, ssl_verify=True):
self.url = None
self.token = None
self.cypher_endpoint = None
self.ssl_verify = ssl_verify

    if url is None:
        if "morpheus_url" in os.environ:
            self.url = os.environ.get("morpheus_url", None)
        elif morpheus is not None:
            self.url = morpheus['morpheus']['applianceUrl']
    else:
        self.url = url

    if self.url is None:
        raise Exception("url not found or specified in ENV or morpheus['morpheus']['applianceUrl']")
    else:
        self.url = self.url.rstrip('/')

    if token is None:
        if "morpheus_token" in os.environ:
            self.token = os.environ.get("morpheus_token", None)
        elif morpheus is not None:
            self.token = morpheus['morpheus']['apiAccessToken']
    else:
        self.token = token
    if self.token is None:
        raise Exception("token not specified in ENV or morpheus['morpheus']['apiAccessToken']")
    self.set_cypher_endpoint()

Ok, starting from the top.

The flow for a token is:

  • If you specify token='a87242487...' when using the module, it will use that.
  • If you have a morpheus_token environment var set, it will use that.
  • If you specify morpheus=morpheus, it will use morpheus['morpheus']['apiAccessToken'] as supplied to the python script from Morpheus.
  • Else complain that there’s no token.

The above complaint about the version is that we now require authentication to get the Morpheus version, so the module will fail if the token isn’t accepted by Morpheus in the API query.

Out of curiosity, how did you raise a ticket with the author? I certainly didn’t see anything that I remember, and there’s no github issue.