vROps cluster fails to initialize after pak upgrade failure

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

1. Master node was powered off during the installation process after the patch upload was completed.
   
   
2. Disk full in master node.
   
   
3. Admin login failures to master node 
   
   
4. Master node status shows failed when observed from other nodes. 
   
   

No backup of snapshot taken prior to upgrade.

-pak update pending here:
[YYYY-MM-DD HH:MM:SS] [########] [pool-6-thread-1]  INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:433 - candidate PAK activity=com.vmware.vcops.casa.upgrade.pak.CurrentPakActivity: pakID=vRealizeOperationsManagerEnterprise-70011287810, operationName=APPLY_SYSTEM_UPDATE, computed_elapsedTimeMs=0, step=7, totalSteps=16, targetSlices=[##.#.#.##, ##.#.#.##, ##.#.#.##]
[YYYY-MM-DD HH:MM:SS] [########] [pool-6-thread-1]  INFO PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator:457 - Invoking operation com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation on slices [##.#.#.##, ##.#.#.##], pakID=vRealizeOperationsManagerEnterprise-###########
[YYYY-MM-DD HH:MM:SS] [########] [pool-290-thread-7] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-###########, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=##.#.#.##
[YYYY-MM-DD HH:MM:SS] [########] [pool-290-thread-8] DEBUG PAK_MANAGER_ACTION_LOGGER upgrade.pak.UpgradeOrchestrator$1OperateOnSlice:586 - Entering OperateOnSlice thread for pakID=vRealizeOperationsManagerEnterprise-###########, operation=com.vmware.vcops.casa.upgrade.pak.ApplySystemUpdateOperation, slice=##.#.#.##
[YYYY-MM-DD HH:MM:SS] [########] [pool-290-thread-7] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-###########, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0
[YYYY-MM-DD HH:MM:SS] [########] [pool-290-thread-8] TRACE PAK_MANAGER_ACTION_LOGGER upgrade.pak.PakUpgradeOperation:40 - pakID=vRealizeOperationsManagerEnterprise-###########, expectedState=APPLIED_SYSTEM_UPDATE, pollResult=com.vmware.vcops.casa.upgrade.pak.ClusterPollResult: expectedSlices=1, operationSuccessfulCount=0

 

 

    5. Below errror found around the time:

[YYYY-MM-DD HH:MM:SS] [########] [ajp-nio-127.0.0.1-8011-exec-4] ERROR web.servlet.HandlerExecutionChain:173 - HandlerInterceptor.afterCompletion threw exception
java.lang.NullPointerException
        at org.apache.catalina.connector.Request.getRequestURL(Request.java:2406)
        at org.apache.catalina.connector.RequestFacade.getRequestURL(RequestFacade.java:868)
        at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
        at javax.servlet.http.HttpServletRequestWrapper.getRequestURL(HttpServletRequestWrapper.java:213)
        at com.vmware.vcops.casa.support.RestTraceManager.logIncomingEnd(RestTraceManager.java:115)
        at com.vmware.vcops.casa.support.RestTraceIncomingInterceptor.afterCompletion(RestTraceIncomingInterceptor.java:56)
        at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
        at com.sun.proxy.$Proxy76.afterCompletion(Unknown Source)
        at org.springframework.web.servlet.HandlerExecutionChain.triggerAfterCompletion(HandlerExecutionChain.java:170)
        at org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletion(DispatcherServlet.java:1297)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:976)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

    6. perform below command to init casa cluster but failed :
           # service vmware-casa restart
           # service apache2 restart
           # $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster

Environment

VMware vRealize Operations Manager 8.x

Cause

 "casa-webapp" folder missing during the pak upgrade due to disk issue, so it is not possible to start casa cluster.

Resolution

1. Take a snapshot of the vROps cluster..

2. On master, locate the casa configuration files stored by the upgrade under /storage/db/vcops/tomcat.conf.backup/casa-webapp.<date>.conf.backup. 

3. Copy the files under the directory from step 2 to /usr/lib/vmware-casa/casa-webapp/conf replacing the existing files there.

4. Perform below commands again and wait for the cluster to come online:
     # service vmware-casa restart
     # service apache2 restart
     # $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster

5. Perform the pak upgrade again and make sure the version is correct.