Database deployment fails when utilizing RDS on VMware with a local vSAN Datastore on vSphere 6.5.x
search cancel

Database deployment fails when utilizing RDS on VMware with a local vSAN Datastore on vSphere 6.5.x

book

Article ID: 314078

calendar_today

Updated On:

Products

VMware Data Services Manager

Issue/Introduction

Symptoms:
  • The on premise custom availability zone (AZ) is using vSphere 6.5.x and a vSAN datastore for storage.
  • Provisioning a new database using the on premise custom AZ appears to start, but never completes.
  • Provisioning to another datastore (e.g., NFS) works without issue.
  • Errors similar to the following are seen in the logs from the VMware Database Management Essentials (VDME) Snaphsot Manager appliance:
     
    2020-03-10 02:14:35.318 ERROR - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] com.vmware.h4.jobengine.JobExecution     : Task 39af3fa7-3f3e-4a63-80cc-9c851a8021c1 (WorkflowInfo{type='setupDestination', resourceType='replication', resourceId='H4-2c567f55-b067-4e53-83fc-146e189b1356', isPrivate=false, resourceName='null'}) has failed
    com.vmware.vim.binding.vim.fault.NoPermission: Permission to perform this operation was denied.
          at sun.reflect.GeneratedConstructorAccessor394.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at java.lang.Class.newInstance(Class.java:442)
          at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
          at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83)
          at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40)
          at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
          at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274)
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
          at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
          at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179)
          at com.sun.proxy.$Proxy252.CreateDirectory(Unknown Source)
          at com.vmware.h4.common.util.DatastoreHelper.ensureDirectoriesExist(DatastoreHelper.java:66)
          at com.vmware.h4.replicator.replication.util.StorageHelper.ensureDatastoreDirectories(StorageHelper.java:84)
          at com.vmware.h4.replicator.replication.SetupDestinationJob.execute(SetupDestinationJob.java:150)
          at com.vmware.h4.replicator.replication.SetupDestinationJob.execute(SetupDestinationJob.java:74)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:171)
          at com.vmware.h4.jobengine.JobEngine.lambda$resume$0(JobEngine.java:137)
          at com.vmware.h4.jobengine.ThrottledExecutorService$1.run(ThrottledExecutorService.java:132)
          at com.vmware.h4.common.mdc.MDCRunnableWrapper.run(MDCRunnableWrapper.java:30)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
    2020-03-10 02:14:35.318  WARN - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] c.v.h.c.e.ExceptionConversionService     : Unable to convert exception. Using fallback exception instead.
    com.vmware.vim.binding.vim.fault.NoPermission: Permission to perform this operation was denied.
          at sun.reflect.GeneratedConstructorAccessor394.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at java.lang.Class.newInstance(Class.java:442)
          at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
          at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88)
          at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83)
          at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40)
          at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
          at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274)
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
          at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
          at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305)
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179)
          at com.sun.proxy.$Proxy252.CreateDirectory(Unknown Source)
          at com.vmware.h4.common.util.DatastoreHelper.ensureDirectoriesExist(DatastoreHelper.java:66)
          at com.vmware.h4.replicator.replication.util.StorageHelper.ensureDatastoreDirectories(StorageHelper.java:84)
          at com.vmware.h4.replicator.replication.SetupDestinationJob.execute(SetupDestinationJob.java:150)
          at com.vmware.h4.replicator.replication.SetupDestinationJob.execute(SetupDestinationJob.java:74)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:171)
          at com.vmware.h4.jobengine.JobEngine.lambda$resume$0(JobEngine.java:137)
          at com.vmware.h4.jobengine.ThrottledExecutorService$1.run(ThrottledExecutorService.java:132)
          at com.vmware.h4.common.mdc.MDCRunnableWrapper.run(MDCRunnableWrapper.java:30)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
    2020-03-10 02:14:35.321 DEBUG - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] c.v.h.r.r.DestroyDestinationJob          : Stopping destination for group H4-2c567f55-b067-4e53-83fc-146e189b1356
    2020-03-10 02:14:35.321 DEBUG - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] c.v.h.r.r.DestroyDestinationJob          : Starting pre-stop cleanup
    2020-03-10 02:14:35.325 DEBUG - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] c.v.h.r.r.CleanupFailoverDestinationJob  : Executing cleanup failover for group H4-2c567f55-b067-4e53-83fc-146e189b1356, auto=true
    2020-03-10 02:14:35.326 ERROR - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] com.vmware.h4.jobengine.JobExecution     : Task b2c2702a-7429-4159-91f8-3d6386aa7eb9 (WorkflowInfo{type='failoverTestCleanup', resourceType='replication', resourceId='H4-2c567f55-b067-4e53-83fc-146e189b1356', isPrivate=false, resourceName='null'}) has failed
    com.vmware.h4.manager.api.exceptions.ReplicationNotFoundException: Replication with id 'H4-2c567f55-b067-4e53-83fc-146e189b1356' was not found.
          at com.vmware.h4.replicator.repo.DestinationReplicationRepo.lambda$getDestinationReplicationEntity$20(DestinationReplicationRepo.java:506)
          at java.util.Optional.orElseThrow(Optional.java:290)
          at com.vmware.h4.replicator.repo.DestinationReplicationRepo.getDestinationReplicationEntity(DestinationReplicationRepo.java:506)
          at com.vmware.h4.replicator.repo.DestinationReplicationRepo.getDestinationReplicationInfo(DestinationReplicationRepo.java:181)
          at com.vmware.h4.replicator.repo.DestinationReplicationRepo$$FastClassBySpringCGLIB$$768ed132.invoke(<generated>)
          at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
          at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
          at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
          at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
          at com.vmware.h4.replicator.repo.DestinationReplicationRepo$$EnhancerBySpringCGLIB$$c16946e8.getDestinationReplicationInfo(<generated>)
          at com.vmware.h4.replicator.replication.CleanupFailoverDestinationJob.execute(CleanupFailoverDestinationJob.java:114)
          at com.vmware.h4.replicator.replication.CleanupFailoverDestinationJob.execute(CleanupFailoverDestinationJob.java:60)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:171)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:117)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:88)
          at com.vmware.h4.replicator.replication.SuspendableJob.childJob(SuspendableJob.java:162)
          at com.vmware.h4.replicator.replication.SuspendableJob.childJob(SuspendableJob.java:175)
          at com.vmware.h4.replicator.replication.DestroyDestinationJob.execute(DestroyDestinationJob.java:113)
          at com.vmware.h4.replicator.replication.DestroyDestinationJob.execute(DestroyDestinationJob.java:49)
          at com.vmware.h4.jobengine.JobEngine.execute(JobEngine.java:171)
          at com.vmware.h4.jobengine.JobEngine.lambda$resume$0(JobEngine.java:137)
          at com.vmware.h4.jobengine.ThrottledExecutorService$1.run(ThrottledExecutorService.java:132)
          at com.vmware.h4.common.mdc.MDCRunnableWrapper.run(MDCRunnableWrapper.java:30)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
    2020-03-10 02:14:35.329 ERROR - [36e854d7-dad8-4be0-8846-27fc54dca1c8_5I_Dy] [job-33] com.vmware.h4.jobengine.JobExecution     : Task 69032ea2-9299-4e98-bbc6-3634cc7935de (WorkflowInfo{type='destroyDestination', resourceType='replication', resourceId='H4-2c567f55-b067-4e53-83fc-146e189b1356', isPrivate=false, resourceName='null'}) has failed


Environment

VMware Database Management Essentials 1.x

Cause

An RDS on VMware role (VMwareRDS-Snapshot-Root) that is created during the onboarding process lacks sufficient privileges to complete the operation.

Resolution

This is a known issue impacting VMware Database Management Essentials 1.0 that will be addressed in a future version of the product.

The appropriate permissions can be applied to the role to correct the issue in an existing environment using the following steps:
  1. Log into the vSphere Web Client (Flash) or the vSphere Client (HTML5) with an administrative account that has sufficient privileges to perform role management.
  2. Adjust the VMwareRDS-Snapshot-Root role to include the additional permission required:
    1. Navigate to Administration and select Roles.
    2. Highlight the VMwareRDS-Snapshot-Root and click the pencil (edit) icon at the top of the table.
    3. In the Edit Role dialog box, highlight (in the HTML5 vSphere Client) Datastore or expand (in the Flash vSphere Web Client) the Datastore section.
    4. Check the Configure datastore permission.
    5. Save the changes and close the Edit Role dialog box.
  3. Confirm that the permissions are still applied to the environment:
    1. Select Hosts and Clusters or VMs and Templates using the tabs or the navigation menu.
    2. Highlight the vCenter used for the custom AZ in question.
    3. Review the permissions for the object and locate the User/Group named h4user-<cluster name>.
    4. Confirm that the user has the role VMwareRDS-Snapshot-Root applied to This object and its children.


Additional Information

For details on collecting logs from the VMware Database Management Essentials Snaphsot Manager appliance, please refer to Collecting diagnostic information for the VMware Database Management Essentials Snapshot Manager (71145) .

Powered by Wolken Software