SDDC Manager upgrade fails from VMware Cloud Foundation 4.0 hp1 to subsequent releases

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides a workaround for the above upgrade failure.

Symptoms:

• LCM upgrade of SDDC Manager fails to upgrade Domain Manager with the following appearing in /var/log/vmware/vcf/lcm/lcm.log:

exception occurred while running the upgrade {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Exception while initiating VcfService upgrade.
        at com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl.initiateVcfServiceUpgrade(VcfServicePrimitiveImpl.java:1140)
        at com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl.runUpgrade(VcfServicePrimitiveImpl.java:800)
        at com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl.startOrResumeUpgrade(VcfServicePrimitiveImpl.java:723)
        at com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl.postUpgrade(VcfServicePrimitiveImpl.java:113)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveService.postUpgrade(PrimitiveService.java:255)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveService$$FastClassBySpringCGLIB$$aff213fb.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
        at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:67)
        at sun.reflect.GeneratedMethodAccessor1444.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 : [{“upgradeId”:“26d50db5-349c-44da-9d5c-64666ac3ad19",“resourceId”:“550759f0-8478-41f3-a5fb-47d75c06d99a”,“startTime”:1594286989906,“endTime”:null,“lastUpdatedTime”:null,“upgradeStatusCode”:“COMPLETED_WITH_FAILURE (1068 bytes)]

• The following error may also appear in /var/log/vmware/vcf/lcm/lcm.log:

***************************
APPLICATION FAILED TO START
***************************
Description:
Binding to target org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under '' to com.vmware.evo.sddc.lcm.adapter.inventory.impl.VersionAliasConfig failed:
    Property: .versionAliases
    Value: {DOMAIN_MANAGER=[BaseAlias{base='4.0.0-vcf4000RELEASE-16008205', alias=[4.0.0-vcf4000RELEASE-16262746]}]}
    Reason: Please check version format. Expected: x.y.z(.w)(.v)(-REL)-12345

• https://ikb.vmware.com/s/article/79235 was previously applied in the environment

Environment

VMware Cloud Foundation 4.x

Cause

This is a known issue affecting VMware Cloud Foundation 4.x.

Resolution

This issue is resolved in VCF 4.0.1.

Workaround:
In order to work around the above failure, please follow these steps:

1. Take a snapshot of the SDDC Manager VM.
2. Delete the version aliasing
   1. Initiate an SSH session to the SDDC Manager as the vcf user.
   2. Enter the command "su" and enter the root password.
   3. Take a backup copy of the /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml:
      cp /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml ~/VersionAlias.yml
   4. Open the file in vi editor: vi /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml
   5. Remove the below content for Domain Manager -
          versionAliases:
          DOMAIN_MANAGER:
              - alias:
                   - 4.0.0-vcf4000RELEASE-16262746
                       base: 4.0.0-vcf4000RELEASE-16008205
3. Revert Domain Manager to J1 version
   1. Download the RPM from: http://build-squid.eng.vmware.com/build/mts/release/bora-16008205/publish/rpm/vcf-domain-manager-4.0.0-16008205.noarch.rpm to SDDC Mgr in /tmp/ directory
   2. Install the downloaded RPM:
      rpm -Uvh /tmp/vcf-domain-manager-4.0.0-16008205.noarch.rpm --force
4. Restart LCM: systemctl restart lcm
5. Restart the upgrade from within the SDDC Manager UI.