anyone know where in morpheus I reconfigure to have it use a different version of python than what is installed with the python3-virtualenv package, which is python3.6, I have the binary to python3.10 referenced in the python task but the log still complains about python3.6 site-packages libraries.
Can you post a snippet of Morpheus complaining about python3.6 site-packages? It sounds odd. We test with Python installed at the system level, so I’m curious as to how Python 3.10 was installed and what base OS it is.
The error output:
Task Execution Failed on Attempt 1
Error Initializing virtualenv…Make sure it’s installed and configured properly on the appliance…
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
import distutils.sysconfig
Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 2376, in
main()
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 705, in main
create_environment(home_dir,
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 933, in create_environment
py_executable = os.path.abspath(install_python(
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 1467, in install_python
fix_local_scheme(home_dir, symlink)
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 1550, in fix_local_scheme
if sysconfig._get_default_scheme() == ‘posix_local’:
AttributeError: module ‘sysconfig’ has no attribute ‘_get_default_scheme’. Did you mean: ‘get_default_scheme’?
Using base prefix ‘/usr/local’
New python executable in /var/opt/morpheus/morpheus-ui/workspace/py-03802947-a21b-4eef-a598-3f7b8a1f0034/bin/python3.10
Also creating executable in /var/opt/morpheus/morpheus-ui/workspace/py-03802947-a21b-4eef-a598-3f7b8a1f0034/bin/python
Running virtualenv with interpreter /usr/local/bin/python3.10
How was python 3.10 installed?
downloaded the source code and compiled it. OS is RHEL 8.
Installed the package python3-virtualenv prior to the python3.10 install. Everything worked fine with python3.6.8, is there something specific besides referencing the python3.10 binary in the python task that is needed to satisfy the virtualenv for the diff version of python?
RHEL 8 and derivatives ship with virtualenv 15.1.0 which isn’t compatible with Python 3.10. I do not recommend the following workaround for a production system because of potential conflicts with future yum updates, but upgrading the system virtualenv seems to work:
[ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
<snip>
[ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ virtualenv -p /usr/local/bin/python3.10 venv
Running virtualenv with interpreter /usr/local/bin/python3.10
<snip>
Using base prefix '/usr/local'
New python executable in /home/ncelebic/venv/bin/python3.10
Not overwriting existing python script /home/ncelebic/venv/bin/python (you must use /home/ncelebic/venv/bin/python3.10)
Traceback (most recent call last):
<snip>
AttributeError: module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?
[ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ sudo pip3 install -U virtualenv
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting virtualenv
<snip>
Installing collected packages: typing-extensions, zipp, importlib-metadata, six, platformdirs, filelock, importlib-resources, distlib, virtualenv
Successfully installed distlib-0.3.4 filelock-3.4.1 importlib-metadata-4.8.3 importlib-resources-5.4.0 platformdirs-2.4.0 six-1.16.0 typing-extensions-4.1.1 virtualenv-20.14.1 zipp-3.6.0
[ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ virtualenv -p /usr/local/bin/python3.10 venv
created virtual environment CPython3.10.4.final.0-64 in 922ms
creator CPython3Posix(dest=/home/ncelebic/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ncelebic/.local/share/virtualenv)
added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ source venv/bin/activate
(venv) [ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ which python
~/venv/bin/python
(venv) [ncelebic@nc-vmware-morphtdbuilds-3072 ~]$ python -V
Python 3.10.4
Nick,
This is exactly what I had done.
which pip3
/usr/bin/pip3
[ec2-user@ip-172-31-4-28 ~]$ pip3 -V
pip 22.0.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
[ec2-user@ip-172-31-4-28 ~]$ pip3 list | grep virtualenv
virtualenv 20.14.1
[ec2-user@ip-172-31-4-28 ~]$ virtualenv -p /usr/local/bin/python3.10 venv
created virtual environment CPython3.10.4.final.0-64 in 770ms
creator CPython3Posix(dest=/home/ec2-user/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ec2-user/.local/share/virtualenv)
added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[ec2-user@ip-172-31-4-28 ~]$ which python
/usr/bin/which: no python in (/home/ec2-user/.rvm/gems/ruby-3.1.1/bin:/home/ec2-user/.rvm/gems/ruby-3.1.1@global/bin:/home/ec2-user/.rvm/rubies/ruby-3.1.1/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.rvm/bin)
[ec2-user@ip-172-31-4-28 ~]$ source venv/bin/activate
(venv) [ec2-user@ip-172-31-4-28 ~]$ which python
~/venv/bin/python
(venv) [ec2-user@ip-172-31-4-28 ~]$ python -V
Python 3.10.4
But executing a python script in the python task in Morpheus I get the following error:
Task Execution Failed on Attempt 1
Error Initializing virtualenv…Make sure it’s installed and configured properly on the appliance…
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
import distutils.sysconfig
Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 2376, in
main()
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 705, in main
create_environment(home_dir,
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 933, in create_environment
py_executable = os.path.abspath(install_python(
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 1467, in install_python
fix_local_scheme(home_dir, symlink)
File “/usr/lib/python3.6/site-packages/virtualenv.py”, line 1550, in fix_local_scheme
if sysconfig._get_default_scheme() == ‘posix_local’:
AttributeError: module ‘sysconfig’ has no attribute ‘_get_default_scheme’. Did you mean: ‘get_default_scheme’?
Using base prefix ‘/usr/local’
New python executable in /var/opt/morpheus/morpheus-ui/workspace/py-56894519-39e4-41e5-a3bc-587080258ba3/bin/python3.10
Also creating executable in /var/opt/morpheus/morpheus-ui/workspace/py-56894519-39e4-41e5-a3bc-587080258ba3/bin/python
Running virtualenv with interpreter /usr/local/bin/python3.10
screenshot of the task in Python attached, does it need additional command arguments??
I overlooked something. Even running pip as root didn’t overwrite /bin/virtualenv, which is fine. I followed the breadcrumbs to alternatives and this workaround seems to work: sudo alternatives --install /usr/bin/virtualenv python-virtualenv /usr/local/bin/virtualenv 10000
That won’t overwrite anything, but place a higher priority alternative over the system virtualenv.
Nick,
I got this to work. Thanks for the help.
I will add that there was one additional thing. I had to add a symlink from the new virtualenv 20.14.1 back to /usr/local/bin for morpheus because the morpheus-local user in its PATH is looking for it in this PATH echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
and could not find it.
Morpheus runs python tasks just fine now. Now I will go thru an upgrade process and see what breaks.
I would add it would be nice if, since there can be multiple python envs, it would be nice if Morpheus python task had the ability to source a different venv from the task, We can specify the python binary already but the venv, it would be nice if that was seemless or we could specify from the python task at least. Enhancement request??
Tks,
Dave
Hi Dave,
It’s definitely something that’s been on my personal wishlist. If you put it in the ideas section, I would vote for it, but I’ve got my own ways of poking people here.