Permission Denied when Deploying any (VM) on KVM or MVM Cluster

I have a fresh install of Ubuntu 22.04, and a fresh install of Morpheus-Appliance 7.0.7.
It is running on baremetal, HPE Proliant 325 Gen 11 (AMD CPU).

I have:

  • Installed and configured Morpheus through the initial setup
  • Set up a Group, Cloud (Morpheus option selected), and Cluster (KVM option selected).

Problem:

When i go to “Provisioning” → “Instances” → “Add” and go through the steps, it will error out with a “Permission Denied” error.

2024-11-19_15:17:30.94819 ''[2024-11-19 15:17:30,955] [appJobHigh-4] INFO  c.m.InstanceProvisionService - provisionInstance networkPrepResults: {success=true, networkResults={success=true, containers=[], results=null}}
2024-11-19_15:17:30.95525 ''[2024-11-19 15:17:30,973] [appJobHigh-4] INFO  c.m.InstanceProvisionService - Provisioning Instance Container 10
2024-11-19_15:17:30.97328 ''[2024-11-19 15:17:31,130] [appJobHigh-4] INFO  c.m.n.NetworkConfigService - Network Config: [primaryInterface:[networkType:dhcp, id:65, subnet:null, macAddress:null, network:com.morpheus.Network : 2, vlanId:null, type:com.morpheus.ComputeServerInterfaceType : 10, doStatic:false, doStatic6:false, doDhcp:true, doDhcp6:false, ipv4Enabled:true, ipv6Enabled:false, name:eth0, externalId:br0, searchDomains:null], extraInterfaces:[], doStatic:false, doDhcp:false, doCustomizations:false, havePool:false, haveDhcpReservation:false, success:true]
2024-11-19_15:17:31.13071 ''[2024-11-19 15:17:31,378] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - Insert Image
2024-11-19_15:17:31.37845 ''[2024-11-19 15:17:31,379] [appJobHigh-4] INFO  c.m.ContainerService - provisionContainer runResult.success: false
2024-11-19_15:17:31.38003 ''[2024-11-19 15:17:31,380] [appJobHigh-4] INFO  c.m.ContainerService - still deploying container: null
2024-11-19_15:17:31.38082 ''[2024-11-19 15:17:32,572] [RxCachedThreadScheduler-6] INFO  c.m.VirtualImageService - metadata file found! null
2024-11-19_15:17:32.57245 ''[2024-11-19 15:17:32,572] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - runConfig.image: [name:Morpheus Debian 11 20240604, minDisk:4294967296, minRam:536870912, virtualImageId:209, tags:morpheus, ubuntu, imageType:qcow2, cloudFiles:[qemu/20240604/debian-11/morpheus-debian-11-amd64-20240604.qcow2, qemu/20240604/debian-11/morpheus-debian-11-amd64-20240604.raw.tar.gz, qemu/20240604/debian-11/morpheus-debian-11-qemu.morpkg], diskIds:[[name:morpheus-debian-11-amd64-20240604.qcow2, path:qemu/20240604/debian-11/morpheus-debian-11-amd64-20240604.qcow2, uniqueId:dc4cee56-4cb0-46ac-bec8-42fd16addc7b]], diskList:[[name:Morpheus Debian 11 20240604, imageType:qcow2, imageFile:qemu/20240604/debian-11/morpheus-debian-11-amd64-20240604.qcow2, minDisk:null, minRam:null, metaRow:[position:0], position:0, bootDisk:true, fileName:morpheus-debian-11-amd64-20240604.qcow2, uniqueId:dc4cee56-4cb0-46ac-bec8-42fd16addc7b]]]
2024-11-19_15:17:32.57263 ''[2024-11-19 15:17:32,586] [RxCachedThreadScheduler-6] INFO  c.m.c.KvmComputeUtility - findImage: -rw-r--r-- 1 root root 761528320 Nov 19 14:59 /var/morpheus/kvm/images/dc4cee56-4cb0-46ac-bec8-42fd16addc7b
2024-11-19_15:17:32.58664
2024-11-19_15:17:32.58665 ''[2024-11-19 15:17:32,598] [RxCachedThreadScheduler-6] INFO  c.m.c.KvmComputeUtility - findImage: -rw-r--r-- 1 root root 761528320 Nov 19 14:59 /var/morpheus/kvm/images/dc4cee56-4cb0-46ac-bec8-42fd16addc7b
2024-11-19_15:17:32.59898
2024-11-19_15:17:32.59899 ''[2024-11-19 15:17:32,598] [RxCachedThreadScheduler-6] INFO  c.m.c.KvmComputeUtility - findImage: [success:true, imageExists:true]
2024-11-19_15:17:32.59900 ''[2024-11-19 15:17:37,628] [RxCachedThreadScheduler-6] INFO  c.m.c.KvmComputeUtility - refresh virsh pool: [success:true, command:sudo bash -c 'sync ; virsh pool-refresh "morpheus-images" ; sleep 5', data:Pool morpheus-images refreshed
2024-11-19_15:17:37.62897
2024-11-19_15:17:37.62897 , error:, exitValue:0, errorData:]
2024-11-19_15:17:37.62898 ''[2024-11-19 15:17:37,628] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - imageResults: [success:true, imageExists:false, image:[path:/var/morpheus/kvm/images, name:null], imageId:/var/morpheus/kvm/images]
2024-11-19_15:17:37.62898 ''[2024-11-19 15:17:37,628] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - imageUploadTask: [success:true, imageId:/var/morpheus/kvm/images, imageMap:null]
2024-11-19_15:17:37.62899 ''[2024-11-19 15:17:37,628] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - insertImage results: [success:true, imageId:/var/morpheus/kvm/images, imageMap:null]
2024-11-19_15:17:37.62899 ''[2024-11-19 15:17:37,629] [RxCachedThreadScheduler-6] INFO  c.m.p.KvmProvisionService - imageUploadTask onComplete: [success:true, imageId:/var/morpheus/kvm/images, imageMap:null]
2024-11-19_15:17:37.62901 ''[2024-11-19 15:17:37,747] [RxCachedThreadScheduler-3] INFO  c.m.ProcessService - Event Insert Detected: Moving Finalizing to runOrder: 2
2024-11-19_15:17:37.74752 ''[2024-11-19 15:17:38,193] [RxCachedThreadScheduler-3] INFO  c.m.VirtualImageService - metadata file found! null
2024-11-19_15:17:38.19386 ''[2024-11-19 15:17:38,259] [RxCachedThreadScheduler-3] INFO  c.m.ProcessService - Event Insert Detected: Moving Finalizing to runOrder: 3
2024-11-19_15:17:38.25944 ''[2024-11-19 15:17:38,263] [RxCachedThreadScheduler-3] INFO  c.m.p.KvmProvisionService - Server Volumes: [com.morpheus.StorageVolume : 59] - com.morpheus.StorageVolume : 59
2024-11-19_15:17:38.26363 ''[2024-11-19 15:17:38,282] [RxCachedThreadScheduler-3] ERROR c.m.p.KvmProvisionService - createServerVolumes error: java.io.IOException: Permission denied
2024-11-19_15:17:38.28222 'java.io.IOException: Permission denied
2024-11-19_15:17:38.28222 	at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
2024-11-19_15:17:38.28222 	at java.base/java.io.File.createNewFile(Unknown Source)
2024-11-19_15:17:38.28222 	at java_io_File$createNewFile$7.call(Unknown Source)
2024-11-19_15:17:38.28223 	at com.morpheus.AbstractHostComputeUtility$_writeHostFiles_closure2.doCall(AbstractHostComputeUtility.groovy:70)
2024-11-19_15:17:38.28223 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28223 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28223 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28224 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28224 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28224 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28224 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
2024-11-19_15:17:38.28225 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
2024-11-19_15:17:38.28225 	at groovy.lang.Closure.call(Closure.java:412)
2024-11-19_15:17:38.28226 	at groovy.lang.Closure.call(Closure.java:428)
2024-11-19_15:17:38.28226 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2357)
2024-11-19_15:17:38.28226 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2342)
2024-11-19_15:17:38.28226 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2383)
2024-11-19_15:17:38.28228 	at org.codehaus.groovy.runtime.dgm$202.invoke(Unknown Source)
2024-11-19_15:17:38.28228 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:242)
2024-11-19_15:17:38.28228 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:51)
2024-11-19_15:17:38.28229 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
2024-11-19_15:17:38.28229 	at com.morpheus.AbstractHostComputeUtility.writeHostFiles(AbstractHostComputeUtility.groovy:68)
2024-11-19_15:17:38.28229 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28230 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28230 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28230 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28231 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28231 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28231 	at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1540)
2024-11-19_15:17:38.28232 	at org.codehaus.groovy.runtime.callsite.StaticMetaClassSite.call(StaticMetaClassSite.java:50)
2024-11-19_15:17:38.28232 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
2024-11-19_15:17:38.28232 	at com.morpheus.storage.datastoretype.QcowDatastoreService.writeVolumeConfigFile(QcowDatastoreService.groovy:279)
2024-11-19_15:17:38.28233 	at com.morpheus.storage.datastoretype.QcowDatastoreService$writeVolumeConfigFile.callCurrent(Unknown Source)
2024-11-19_15:17:38.28233 	at com.morpheus.storage.datastoretype.QcowDatastoreService.cloneVolume(QcowDatastoreService.groovy:219)
2024-11-19_15:17:38.28233 	at com.morpheus.storage.datastoretype.DatastoreTypeInterface$cloneVolume.call(Unknown Source)
2024-11-19_15:17:38.28234 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:120)
2024-11-19_15:17:38.28235 	at com.morpheus.storage.DatastoreTypeService.cloneVolume(DatastoreTypeService.groovy:40)
2024-11-19_15:17:38.28235 	at com.morpheus.storage.datastoretype.DatastoreTypeInterface$cloneVolume.call(Unknown Source)
2024-11-19_15:17:38.28235 	at com.morpheus.provision.KvmProvisionService.createServerVolumes(KvmProvisionService.groovy:704)
2024-11-19_15:17:38.28236 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28236 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28236 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28236 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28237 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28237 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28237 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
2024-11-19_15:17:38.28238 	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
2024-11-19_15:17:38.28238 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
2024-11-19_15:17:38.28239 	at com.morpheus.provision.KvmProvisionService$_insertVm_closure18.doCall(KvmProvisionService.groovy:572)
2024-11-19_15:17:38.28239 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28239 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28240 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28240 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28241 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28241 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28241 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
2024-11-19_15:17:38.28242 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
2024-11-19_15:17:38.28242 	at groovy.lang.Closure.call(Closure.java:412)
2024-11-19_15:17:38.28242 	at groovy.lang.Closure.call(Closure.java:428)
2024-11-19_15:17:38.28243 	at org.grails.orm.hibernate.GrailsHibernateTemplate.doExecute(GrailsHibernateTemplate.java:297)
2024-11-19_15:17:38.28243 	at org.grails.orm.hibernate.GrailsHibernateTemplate.execute(GrailsHibernateTemplate.java:241)
2024-11-19_15:17:38.28244 	at org.grails.orm.hibernate.GrailsHibernateTemplate.executeWithNewSession(GrailsHibernateTemplate.java:153)
2024-11-19_15:17:38.28244 	at org.grails.orm.hibernate.AbstractHibernateDatastore.withNewSession(AbstractHibernateDatastore.java:360)
2024-11-19_15:17:38.28244 	at org.grails.orm.hibernate.AbstractHibernateGormStaticApi.withNewSession(AbstractHibernateGormStaticApi.groovy:78)
2024-11-19_15:17:38.28245 	at org.grails.datastore.gorm.GormEntity$Trait$Helper.withNewSession(GormEntity.groovy:1027)
2024-11-19_15:17:38.28245 	at org.grails.datastore.gorm.GormEntity$Trait$Helper$withNewSession.call(Unknown Source)
2024-11-19_15:17:38.28245 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
2024-11-19_15:17:38.28245 	at org.grails.datastore.gorm.GormEntity$Trait$Helper$withNewSession.call(Unknown Source)
2024-11-19_15:17:38.28246 	at com.morpheus.ComputeServer.withNewSession(ComputeServer.groovy)
2024-11-19_15:17:38.28246 	at com.morpheus.ComputeServer$withNewSession$0.call(Unknown Source)
2024-11-19_15:17:38.28246 	at com.morpheus.provision.KvmProvisionService.insertVm(KvmProvisionService.groovy:461)
2024-11-19_15:17:38.28247 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28247 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28247 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28248 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28248 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28248 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28248 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
2024-11-19_15:17:38.28249 	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
2024-11-19_15:17:38.28249 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)
2024-11-19_15:17:38.28249 	at com.morpheus.provision.KvmProvisionService$_runContainer_closure12$_closure99.doCall(KvmProvisionService.groovy:275)
2024-11-19_15:17:38.28249 	at com.morpheus.provision.KvmProvisionService$_runContainer_closure12$_closure99.doCall(KvmProvisionService.groovy)
2024-11-19_15:17:38.28250 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-11-19_15:17:38.28250 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28251 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28251 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28251 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28251 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28252 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
2024-11-19_15:17:38.28252 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
2024-11-19_15:17:38.28253 	at groovy.lang.Closure.call(Closure.java:412)
2024-11-19_15:17:38.28253 	at groovy.lang.Closure.call(Closure.java:406)
2024-11-19_15:17:38.28253 	at org.grails.async.factory.rxjava2.RxPromise$_closure1.doCall(RxPromise.groovy:42)
2024-11-19_15:17:38.28254 	at jdk.internal.reflect.GeneratedMethodAccessor612.invoke(Unknown Source)
2024-11-19_15:17:38.28254 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2024-11-19_15:17:38.28254 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2024-11-19_15:17:38.28255 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
2024-11-19_15:17:38.28255 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
2024-11-19_15:17:38.28256 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
2024-11-19_15:17:38.28256 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
2024-11-19_15:17:38.28256 	at groovy.lang.Closure.call(Closure.java:412)
2024-11-19_15:17:38.28257 	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:50)
2024-11-19_15:17:38.28257 	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:112)
2024-11-19_15:17:38.28257 	at jdk.proxy3/jdk.proxy3.$Proxy223.subscribe(Unknown Source)
2024-11-19_15:17:38.28257 	at io.reactivex.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:39)
2024-11-19_15:17:38.28258 	at io.reactivex.Single.subscribe(Single.java:3666)
2024-11-19_15:17:38.28258 	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
2024-11-19_15:17:38.28258 	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:608)
2024-11-19_15:17:38.28258 	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
2024-11-19_15:17:38.28258 	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
2024-11-19_15:17:38.28259 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
2024-11-19_15:17:38.28259 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
2024-11-19_15:17:38.28260 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2024-11-19_15:17:38.28260 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2024-11-19_15:17:38.28260 	at java.base/java.lang.Thread.run(Unknown Source)
2024-11-19_15:17:38.28916 '[2024-11-19 15:17:38,289] [RxCachedThreadScheduler-3] INFO  c.m.p.KvmProvisionService - createServerVolumes: [success:false, msg:Permission denied]
2024-11-19_15:17:38.28917 ''[2024-11-19 15:17:38,298] [RxCachedThreadScheduler-3] ERROR c.m.ContainerService - Unable to deploy container: Failed to create server volumes
2024-11-19_15:17:38.29905 ''[2024-11-19 15:17:38,310] [RxCachedThreadScheduler-3] ERROR c.m.ContainerService - Unable to deploy container: Failed to create server
2024-11-19_15:17:38.31084 ''[2024-11-19 15:17:38,312] [RxCachedThreadScheduler-3] INFO  c.m.p.KvmProvisionService - create server results: [success:false, msg:Permission denied]
2024-11-19_15:17:38.31244 ''[2024-11-19 15:17:38,312] [RxCachedThreadScheduler-3] INFO  c.m.p.KvmProvisionService - runTask onComplete Failed Run Result: [success:false, server:null]
2024-11-19_15:17:38.31249 ''[2024-11-19 15:17:38,318] [RxCachedThreadScheduler-3] ERROR c.m.ContainerService - Unable to deploy container: null
2024-11-19_15:17:38.31892 ''[2024-11-19 15:18:32,295] [appJobLow-2] INFO  c.m.c.KvmComputeUtility - Refresh Pool Results for local - [success:true, command:sudo virsh pool-refresh "local" ; exit 0, data:Pool local refreshed
2024-11-19_15:18:32.29542
2024-11-19_15:18:32.29543 , error:, exitValue:0, errorData:]
2024-11-19_15:18:32.29543 ''[2024-11-19 15:18:32,391] [appJobLow-2] INFO  c.m.c.KvmComputeUtility - Refresh Pool Results for morpheus-cloud-init - [success:true, command:sudo virsh pool-refresh "morpheus-cloud-init" ; exit 0, data:Pool morpheus-cloud-init refreshed
2024-11-19_15:18:32.39209
2024-11-19_15:18:32.39210 , error:, exitValue:0, errorData:]
2024-11-19_15:18:32.39210 ''[2024-11-19 15:18:32,441] [appJobLow-2] INFO  c.m.c.KvmComputeUtility - Refresh Pool Results for morpheus-images - [success:true, command:sudo virsh pool-refresh "morpheus-images" ; exit 0, data:Pool morpheus-images refreshed
2024-11-19_15:18:32.44108
2024-11-19_15:18:32.44109 , error:, exitValue:0, errorData:]
2024-11-19_15:18:32.44109 ''[2024-11-19 15:18:32,468] [appJobLow-2] INFO  c.m.h.KvmBaseHostService - Storage Pools: [[name:local, state:running, autostart:yes, persistent:yes, capacity:[unit:bytes, value:942860439552], allocation:[unit:bytes, value:24949702656], available:[unit:bytes, value:917910736896], uuid:6bfc7ff4-f4ef-4ec6-98e0-042c0537aa28, source:, target:[path:/var/morpheus/kvm/local, permissions:[mode:0775, owner:992, group:0]], type:dir, blockDevice:null], [name:morpheus-cloud-init, state:running, autostart:yes, persistent:yes, capacity:[unit:bytes, value:942860439552], allocation:[unit:bytes, value:24949710848], available:[unit:bytes, value:917910728704], uuid:72e3a6af-8e99-4e68-8f9e-69753649b53a, source:, target:[path:/var/morpheus/kvm/cloud-init, permissions:[mode:0775, owner:992, group:0]], type:dir, blockDevice:null], [name:morpheus-images, state:running, autostart:yes, persistent:yes, capacity:[unit:bytes, value:942860439552], allocation:[unit:bytes, value:24949719040], available:[unit:bytes, value:917910720512], uuid:a2a262a4-cdc8-40a0-ab11-1cda1ae668b9, source:, target:[path:/var/morpheus/kvm/images, permissions:[mode:0775, owner:992, group:0]], type:dir, blockDevice:null]]

I have even gone and set up my cluster to use the root account as a test, and that still gets permission denied.

So particularly it fails when trying to create server volumes. I cannot find any way to find what actual directory/file it is trying to create. If I knew that much, I’d abe able to check the permissions there.

[2024-11-19 15:33:40,235] [RxCachedThreadScheduler-9] INFO  c.m.p.KvmProvisionService - Server Volumes: [com.morpheus.StorageVolume : 61] - com.morpheus.StorageVolume : 61
2024-11-19_15:33:40.23516 ''[2024-11-19 15:33:40,252] [RxCachedThreadScheduler-9] ERROR c.m.p.KvmProvisionService - createServerVolumes error: java.io.IOException: Permission denied
2024-11-19_15:33:40.25278 'java.io.IOException: Permission denied

Any ideas what the problem is? I can create and launch VM’s manually with QEMU. But no matter what I try, I can’t get Morpheus to deploy a VM.

To clarify, you are attempting KVM cluster vs MVM cluster?

Correct, I am trying to deploy in a KVM cluster.

So the sequence of events to replicate:

Instances → Provision → Add
Pick any option, lets for example choose Debian.
Group Tab:
Name: debian_vm

Configure Tab:
Networks: Selected my virsh network (named br0)
Host: selected the physical host system (only option I have)

Automation Tab:
Nothing selected

Review:
Click Complete

Morpheus begins to deploy →
qcow2 image uploaded to /var/lib/libvirt/morpheus-images/d82da7e8-f93e-4a5a-adb1-367a72992998 (aka my morpheus-images virsh pool)

VM folder created: /var/morpheus/kvm/vms/debian_vm

PERMISSION ERROR HAPPENS HERE

no files are created inside the debian_vm folder.

What is morpheus trying to do here? Is it trying to copy or create files in the debian_vm folder? Can anyone provide what the ownership and permissions look like on a working environment?

The KVM cluster type I believe expects your hosts to be already configured exactly like Morpheus expects. there is a doc here:
https://docs.morpheusdata.com/en/latest/integration_guides/Clouds/kvm/kvm.html

This feature set I believe will be deprecated soon as the announcement of MVM / HPE VME cluster type will actually deploy and manage the specific HPE flavor of KVM.

Hi @cbunge - yeah I’ve followed those steps to a tee, and it’s the same problem. I’ve rebuilt the OS, freshly installed morpheus, followed those steps, and tried it on several baremetal ProLiant servers (Intel and AMD models), tried different versions of ubuntu (20.04-24.04) and cannot make it deploy successfully, each time hitting this “Permission error”.

I’m convinced morpheus is doing something odd in the backend, like checking a folder/file for specific ownership/permissions, and trying to do an exact match, and if it’s anything outside of that it’ll fail. I have set the folders to be fully accessibly by any user account, i’ve tried multiple different accounts, different permissions on folders and files, everything, and just cannot get it to deploy at all.

Okay so I never was able to get this working, unfortunately. However I was able to get MVM working and deploying VM’s. So I’ll press ahead using MVM.

Appreciate the support!