LCM self upgrade fails with the error "In UpgradeController, failed to get completed upgrades since date,Invalid upgrade"

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

LCM self upgrade fails with the error "In UpgradeController, failed to get completed upgrades since date,Invalid upgrade"
You see errors similar to the following in the /home/vrack/lcm/logs/lcm.log file on the SDDC Manager Controller VM:

ERROR [com.vmware.evo.sddc.lcm.api.rest.controller.upgrade.UpgradeController] Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 In UpgradeController, failed to get completed upgrades since date
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
500 Internal Server Error
"code": "Internal Server Error"

2018-08-02 06:55:10.712 [ThreadPoolTaskExecutor-3] INFO [com.vmware.evo.sddc.lcm.primitive.evo.util.spi.EvoPrimitiveRestUtil] upgradeId=766e584e-1d05-4da9-9243-f8fc466600bb,resourceType=LCM,resourceId=d5287f1b-52d5-44af-84de-f027592b78c0,bundleElementId=8563ce88-57f0-4d95-8495-7e10fbbd018c In EvoPrimitiveRestUtil, getting the upgrade status from https://192.168.100.40:9443/lcm/internal/upgrades/766e584e-1d05-4da9-9243-f8fc466600bb

2018-08-02 06:55:10.717 [ThreadPoolTaskExecutor-3] ERROR [com.vmware.evo.sddc.lcm.primitive.evo.util.spi.EvoPrimitiveRestUtil] upgradeId=766e584e-1d05-4da9-9243-f8fc466600bb,resourceType=LCM,resourceId=d5287f1b-52d5-44af-84de-f027592b78c0,bundleElementId=8563ce88-57f0-4d95-8495-7e10fbbd018c In EvoPrimitiveRestUtil, exception occurred while getting upgrade status from https://192.168.100.40:9443/lcm/internal/upgrades/766e584e-1d05-4da9-9243-f8fc466600bb
org.springframework.web.client.HttpServerErrorException: 503

You see entries similar to the following in the /home/vrack/lcm/upgrades/lcm/<upgrade_id>/logs/upgrade.log file on the SDDC Manager Controller VM:

2018-07-31 02:16:27.248 [https-jsse-nio-192.168.100.40-9443-exec-2] DEBUG [com.vmware.evo.sddc.lcm.services.impl.LcmUpgradeServiceImpl] Apache-HttpClient/4.5 (Java/1.8.0_141) 07/31/2018-02:14:12 AM : Restoring property lcm.depot.adapter.proxyEnabled
2018-07-31 02:16:27.248 [https-jsse-nio-192.168.100.40-9443-exec-2] DEBUG [com.vmware.evo.sddc.lcm.services.impl.LcmUpgradeServiceImpl] Apache-HttpClient/4.5 (Java/1.8.0_141) sed: -e expression #1, char 72: unterminated `s' command
2018-07-31 02:16:27.248 [https-jsse-nio-192.168.100.40-9443-exec-2] DEBUG [com.vmware.evo.sddc.lcm.services.impl.LcmUpgradeServiceImpl] Apache-HttpClient/4.5 (Java/1.8.0_141) 07/31/2018-02:14:12 AM : Unable to restore lcm.depot.adapter.proxyEnabled, exit code: 1
2018-07-31 02:16:27.249 [https-jsse-nio-192.168.100.40-9443-exec-2] DEBUG [com.vmware.evo.sddc.lcm.services.impl.LcmUpgradeServiceImpl] Apache-HttpClient/4.5 (Java/1.8.0_141) 07/31/2018-02:14:12 AM : Unable to restore LCM properties, exit code: 1
2018-07-31 02:16:27.249 [https-jsse-nio-192.168.100.40-9443-exec-2] DEBUG [com.vmware.evo.sddc.lcm.services.impl.LcmUpgradeServiceImpl] Apache-HttpClient/4.5 (Java/1.8.0_141) 07/31/2018-02:14:12 AM : Start rolling back LCM tools

2018-07-31 02:16:27.250 [https-jsse-nio-192.168.100.40-9443-exec-2] INFO [com.vmware.evo.sddc.lcm.api.rest.controller.internal.LcmUpgradeController] Apache-HttpClient/4.5 (Java/1.8.0_141) LCM upgrade failed

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 2.2.x

Cause

This issue occurs when there are duplicate proxy settings under LCM DEPOT PROPERTIES and ESXi VIB upload and query sections in the “/home/vrack/lcm/lcm-app/conf/application-evo.properties” file on the SDDC Manager Controller VM, as seen in the following example:

<Begining of the file>
...
.....
......
################### LCM DEPOT PROPERTIES ########################
lcm.depot.adapter.host=depot.vmware.com
lcm.depot.adapter.port=443
lcm.depot.adapter.remote.rootDir=/PROD2
lcm.depot.adapter.remote.repoDir=/evo/vmw
lcm.depot.adapter.local.baseDir=${user.home}/VMware/vRack/nfs-mount/lcm/repo/depot/local
lcm.depot.adapter.enableBundleSignatureValidation=true
lcm.depot.adapter.certificateCheckEnabled=true
lcm.depot.adapter.remote.index.filename=index
lcm.depot.adapter.proxyEnabled=false
lcm.depot.adapter.proxyHost=proxy.vmware.com
lcm.depot.adapter.proxyPort=3128

....
.....
.......

################### ESXi VIB upload and query #####################
lcm.esx.vum.depot.parent.directory=/tmp/vcf-vum-depot/
spring.http.multipart.location=/tmp/

lcm.depot.adapter.proxyEnabled=true
lcm.depot.adapter.proxyHost=172.24.160.254
lcm.depot.adapter.proxyPort=3128

<End of file>

Note: The preceding entries are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

Use a text editor to remove the entry of the proxy settings that is present under ESXi VIB upload and query section and update the same proxy settings under LCM DEPOT PROPERTIES in the /home/vrack/lcm/lcm-app/conf/application-evo.properties file on the SDDC Manager Controller VM, per the following example:

################### LCM DEPOT PROPERTIES ########################
lcm.depot.adapter.host=depot.vmware.com
lcm.depot.adapter.port=443
lcm.depot.adapter.remote.rootDir=/PROD2
lcm.depot.adapter.remote.repoDir=/evo/vmw
lcm.depot.adapter.local.baseDir=${user.home}/VMware/vRack/nfs-mount/lcm/repo/depot/local
lcm.depot.adapter.enableBundleSignatureValidation=true
lcm.depot.adapter.certificateCheckEnabled=true
lcm.depot.adapter.remote.index.filename=index
lcm.depot.adapter.proxyEnabled=true
lcm.depot.adapter.proxyHost=172.24.160.254
lcm.depot.adapter.proxyPort=3128

################### ESXi VIB upload and query #####################
lcm.esx.vum.depot.parent.directory=/tmp/vcf-vum-depot/
spring.http.multipart.location=/tmp/

Issue the following command to restart the lcm service: systemctl restart lcm
Restart the upgrade workflow from SDDC manager UI.