vROps cluster fails to initialize after pak upgrade failure
Article ID: 308727
Updated On:
Products
VMware Aria Suite
Issue/Introduction
Symptoms:
-Master node was powered off during the installation process after the patch upload was completed.
-Disk full in master node.
-Admin login failures to master node
-Master node status shows failed when observed from other nodes.
-No backup of snapshot taken prior to upgrade.
-pak update pending here:
2020-03-26 16:47:08,221 [aE00002O] [pool-6-thread-1] INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:433 - candidate PAK activity=com.vmware.vcops.casa.upgrade.pak.CurrentPakActivity: pakID=vRealizeOperationsManagerEnterprise-70011287810, operationName=APPLY_SYSTEM_UPDATE, computed_elapsedTimeMs=0, step=7, totalSteps=16, targetSlices=[10.5.4.34, 10.5.4.33, 10.2.5.22]
2020-03-26 16:47:10,658 [aE00002O] [pool-6-thread-1] INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:457 - Invoking operation com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation on slices [10.5.4.34, 10.5.4.33], pakID=vRealizeOperationsManagerEnterprise-70011287810
2020-03-26 16:47:10,658 [aE00002O] [pool-290-thread-7] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-70011287810, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=10.5.4.34
2020-03-26 16:47:10,658 [aE00002O] [pool-290-thread-8] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-70011287810, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=10.5.4.33
2020-03-26 16:47:11,811 [aE00002O] [pool-290-thread-7] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-70011287810, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0
2020-03-26 16:47:11,848 [aE00002O] [pool-290-thread-8] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-70011287810, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0
-below errror found around the time:
2020-03-26 16:47:18,367 [aE00002O] [ajp-nio-127.0.0.1-8011-exec-4] ERROR web.servlet.HandlerExecutionChain:173 - HandlerInterceptor.afterCompletion threw exception
java.lang.NullPointerException
at org.apache.catalina.connector.Request.getRequestURL(Request.java:2406)
at org.apache.catalina.connector.RequestFacade.getRequestURL(RequestFacade.java:868)
at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
at com.vmware.vcops.casa.support.RestTraceManager.logIncomingEnd(RestTraceManager.java:115)
at com.vmware.vcops.casa.support.RestTraceIncomingInterceptor.afterCompletion(RestTraceIncomingInterceptor.java:56)
at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy76.afterCompletion(Unknown Source)
at org.springframework.web.servlet.HandlerExecutionChain.triggerAfterCompletion(HandlerExecutionChain.java:170)
at org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletion(DispatcherServlet.java:1297)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:976)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
-
- perform below command to init casa cluster but failed :
service vmware-casa restart
service apache2 restart
$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster
-Master node was powered off during the installation process after the patch upload was completed.
-Disk full in master node.
-Admin login failures to master node
-Master node status shows failed when observed from other nodes.
-No backup of snapshot taken prior to upgrade.
-pak update pending here:
2020-03-26 16:47:08,221 [aE00002O] [pool-6-thread-1] INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:433 - candidate PAK activity=com.vmware.vcops.casa.upgrade.pak.CurrentPakActivity: pakID=vRealizeOperationsManagerEnterprise-70011287810, operationName=APPLY_SYSTEM_UPDATE, computed_elapsedTimeMs=0, step=7, totalSteps=16, targetSlices=[10.5.4.34, 10.5.4.33, 10.2.5.22]
2020-03-26 16:47:10,658 [aE00002O] [pool-6-thread-1] INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:457 - Invoking operation com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation on slices [10.5.4.34, 10.5.4.33], pakID=vRealizeOperationsManagerEnterprise-70011287810
2020-03-26 16:47:10,658 [aE00002O] [pool-290-thread-7] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-70011287810, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=10.5.4.34
2020-03-26 16:47:10,658 [aE00002O] [pool-290-thread-8] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-70011287810, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=10.5.4.33
2020-03-26 16:47:11,811 [aE00002O] [pool-290-thread-7] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-70011287810, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0
2020-03-26 16:47:11,848 [aE00002O] [pool-290-thread-8] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-70011287810, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0
-below errror found around the time:
2020-03-26 16:47:18,367 [aE00002O] [ajp-nio-127.0.0.1-8011-exec-4] ERROR web.servlet.HandlerExecutionChain:173 - HandlerInterceptor.afterCompletion threw exception
java.lang.NullPointerException
at org.apache.catalina.connector.Request.getRequestURL(Request.java:2406)
at org.apache.catalina.connector.RequestFacade.getRequestURL(RequestFacade.java:868)
at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
at com.vmware.vcops.casa.support.RestTraceManager.logIncomingEnd(RestTraceManager.java:115)
at com.vmware.vcops.casa.support.RestTraceIncomingInterceptor.afterCompletion(RestTraceIncomingInterceptor.java:56)
at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy76.afterCompletion(Unknown Source)
at org.springframework.web.servlet.HandlerExecutionChain.triggerAfterCompletion(HandlerExecutionChain.java:170)
at org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletion(DispatcherServlet.java:1297)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:976)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
-
- perform below command to init casa cluster but failed :
service vmware-casa restart
service apache2 restart
$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster
Environment
VMware vRealize Operations Manager 8.x
Cause
casa-webapp folder missing during the pak upgrade due to disk issue, so it is not possible to start casa cluster.
Resolution
1.Take a snapshot of the vROps cluster..
2.On master, locate the casa configuration files stored by the upgrade under /storage/db/vcops/tomcat.conf.backup/casa-webapp.<date>.conf.backup.
3.Copy the files under the directory from step 2 to /usr/lib/vmware-casa/casa-webapp/conf replacing the existing files there.
4 .Perform the below command again and wait for the cluster to come online:
service vmware-casa restart
service apache2 restart
$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster
6.Perform the pak upgrade again and make sure the version is correct.
2.On master, locate the casa configuration files stored by the upgrade under /storage/db/vcops/tomcat.conf.backup/casa-webapp.<date>.conf.backup.
3.Copy the files under the directory from step 2 to /usr/lib/vmware-casa/casa-webapp/conf replacing the existing files there.
4 .Perform the below command again and wait for the cluster to come online:
service vmware-casa restart
service apache2 restart
$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster
6.Perform the pak upgrade again and make sure the version is correct.
Feedback
Yes
No
Powered by
