Stalled health service causes timeout at "Perform Component Upgrade" step when upgrading VCF Automation from version 9.1 GA to any Express Patch.

Products

VCF Automation

Issue/Introduction

Symptoms:

Environment is currently on VCFA 9.1.0 (GA, EP1 or EP2) and was previously upgraded from Aria Automation 8.x
Patch was applied before the environment toggled off its plain HTTP health endpoint
```
kubectl get vaconfig prelude-vaconfig -ojson | jq .spec.health_app
...
null
```
Patch operation times out

Listing pods in prelude namespace show health-status-app stuck in ContainerCreating. There might be other health-related pods in Error state

kubectl -n prelude get pods
...
health-status-app-XYXYX                               0/2     ContainerCreating   0                21h     <none>         vcfa-ded-XYXYX   <none>           <none>
disable-health-status-external-http-29726580-XYXYX    0/1     Error               0                19m     <IP address>   vcfa-ded-XYXYX   <none>           <none>
vcfa-health-verifier-XYXYX                            0/1     Error               0                53m     <IP address>   vcfa-ded-XYXYX   <none>           <none>

A required configuration object for the health-status service is missing

kubectl -n prelude get configmap health-monitor-config
...
Error from server (NotFound): configmaps "health-monitor-config" not found

Environment

VCF Automation 9.1.0.x (Upgraded from Aria Automation)

Cause

A known issue may cause a configuration object for the health-status service to disappear which causes it to stall. This blocks further progress on the patch operation.

Resolution

This is resolved in VCFA 9.1.1

Workaround

Run the remediate-health-status-hr.sh script, which detects the problem and prevents the health-statuse service from stalling by asserting the needed configuration exists and will not disappear during patch.

This workaround can be performed both as preventive step after upgrade from Aria Automation 8.x to VCFA 9.1.0 release family and before patching VCFA 9.1.0 to EP1 or EP2.
It can also fix the problem in case the patch has failed. Once resolved, another KB 452678 must be executed in order to resume the patch workflow.

Procedure

On already-affected systems, the following steps can be run to allow the upgrade to complete:

Take a backup of VCFA via Fleet LCM

Log in to one of the VMs of VCFA and switch root

ssh vmware-system-user@<VCFA-Platform-FQDN>
sudo -i

Install the patch script on the node:

mkdir -p ~/remediate-health-status-hr && cd ~/remediate-health-status-hr

base64 -d <<< '/Td6WFoAAAFpIt42BMC6KICgASEBHAAAAAAAABO7PyjgT/8UMl0AORlJ/fllSr1UbAa8/v7kdYn9FWkvP7Vq+5HSenXHZv/0eeOgXIePa4c/ZtTtQYVc7jKNv6HF4gHfmNX2XTEwNJERNM5/BHCexeX3m2PAmeXLjml2YvwfPh9BhXGIuzquonS9JqQRU7dlFYmvSFC3eFaEs3ws0BC8oSm4MHH+brRcdIaOobVWTa885JeEN7Gd3/ynR6XIz/ayOGvbAgzaCqS2hPvo2HRXPE5Cig7H1GaVnLuLj2s1cy5IXUbu4Pizq3iWAf9ujI6aYHIHqRmhrGcGCDQ2kdvfn7qismOtJpdSFdBAXKl06H7mOUY/orvgAnKuGK6TVpkElKS9+ya1dzLN80hHyAlnmgM9yvqH3qvAFC0y62MH7DwLJnLTtiBZVKbFjKL2cqGJUaPpPfjrAC4E81TWV4NBn+7yU3GNKJk9IbH/t0rFXt7fex+2DpN5X8nkZY7lmhYUVuS2II1IDarypt37oeWdB/3khwx7G/gmcAMZHmaNw8twgJsRKoe/TjDGKEXZYxQuUzs6QLEy60k+8o3q/OAE2XdQoK+vOofynqUpTd2Bjz9SzqqYMvT8fgXwtDX566P3PwuwRw7lsLcdJPKJmZMXLJaLiY0or3i9XzZM2RLFKu09kn87eOLrQlMfnLqfk2r9qrv7HsgOBrh5YCCrA6Apbr2dhswOJ9CBfDJTvtYxt1ADU0aHvnSbQbOApfgTwaTnMB/w1FIEZGJGVC9F525TsBi8j+npUI+rc6h55S7i6D1QrhhClvU22nF2Bjd3tmj6fR+rpd+PK42F9pDpkOtmSGw5nlz6oVJXhdyYNB6QBFRrUH2ku0GbDMUxlypM1W56mgHhmqhE6ujCSxqdJxXlEop7QV1w3PtdIQZfShHFRuvVEFnypPGMsX52p/uip5FdJ5MJLOFRjVS6L06AvWkZpKt1tQVDvsivZ1pRUOwX6KVBtmQziiFIXqHH/gWOtTGZHHlGD8m2OQcD6VUHeoIbsG6MqDHJpMyYc8HfF9AFraPWdY2dHYccldDF4hgzFSj8QLhR9QgqDtGJSkrqyyBdxYIXcsZqzlXEcz+mSllsESACHT3krkWfCnDA57o/D3V1FF1Xhyw/mhyc0n2v79TyEqyk3HZs0dhuRuv4DVqb6m8EQ4whr0ec6HBgGHFM+szvE5gquNN3QZpVnEBIr0YCp8tbuZAKqsqpveetQLbFGJTYYOIfx77PB18yUmevBakClXvwVO8hPlln7nflTCgTVQ8N8EnLZUgCJ1wO3fxw5Iewnm84tEFmPS+socwm1CnXe0WyNJORn0ifSjhgXs4tbr+R8UkoLqsoehcNzDjc3Nd4Pi6idoGvQKaQaBVtCaMLGw01ZMb4JZAFw4g0YJuiSr4eOrszDxb8X9Afa4axVCLiwsy8Nvj/vq7yzgZedH8ZcEZqxpdal5PIDZf12Mu8p2u01q/FfP3/SSisyiQ4nUsbWRILoEsYGTiTIpb/4y1nS+qGrkU4VZGxBlpqVJghah5etGoPuVfY0XVcpeI4M9Df4NZ4+3RNjeFstFmBrMkhGlCMVjXLiJT5qn9ryEae+YQYIN/8J4nl6gPxP+rl8b0/INdnYhz96onFalJjMZxrpvw2e6ND9fC/w9XYq3ERlg/r9at0TCmJPByKVcxedgY1jtrqfIu1mUR8MnbHqw/369/3WInJgTefZvnTtm0hGgpBilyIxEnFsxmyowCHcWSnJCSPB/NvpIK9Fpo8qSxXBnOZPoSQpUQvfv6iCuUoWzX41Vhq2fKsuwsEpa2NVEQ19c5q9RkjXr92sao4XtXwKcUHIXb8jX6cBf8bNOaBSuFH5pdPvHWTYNpxFIIUa29ZPCRShuHXbvnb0D2Baj61xHiVkb2L8fRBaCWUnHffVJFYnaIQAMSOjcCaje1dGKjPoNwDOPsuUoGKRsEmTb7E/rwLR2rAMDaR2yr6Hx/QV9Pe9xhlQlfftHJS1eTyQbfkacsk3xKTY/wUWBfxytGS6aQAy8NWRj8qVREe1W9PGfeC6X1yl+MY/x/SNjPE85Z+xsHqGzQ+G0KSgiYoEW2aLhVZ3HuxAH5RLpzEOXc0RjmZUy5GWRGOCbHtkhE0hJwZyfDowPQOx8lxGN/Dfx4BIjaJ9ByC3KNTAEd1XJqv9jgIEdW8wOKIhgaf1+QyVf/qLt0gSLfFnxAXtAf0banFn5vrAV4MybOUTOEqY3T3j5gh0B9mUk8XQq2EWVWv14QyWoyAleFT0Dr4GxeU9eNtBCnkifIy4/vN5oM3WPxopU0W7f0flRVynd3jqYzds2SZ6/+r40r9OTXG+kv9RmdqMn5gdEoVe/8HuFwYqVZ/fekg1TLQUl1aO3E7mFTDFkOxAoT8HPVRHEhHKFmBBlQX30F+UaHJMc3f7Ebpyflj4wfo1HQEIRzA4RXmhxJ04+VYBhc/2cwt7uVEz2SqPXcfOLAFVEUxEJXeDz5Q3AOVlcd54yWgxRE7y/xcs0nwU2P539LuivNngCdL+m4f9bne7FTm5kg2E5uwlddbLybRXbz6OI7nnQaxCVTLs1HTob3qhDkt43wotUumHlYISpSLO0/xx55wy+VfuuYOe+GkPB6aXbmQY9JINv50CXrdKSPrx2mv9kJJr/GxiOPblh1RqVqY808OBILF1Ct1uLrc+dYEABtaU9z+qU1L+kSvswLoMLu89clNhNzOL4YvpT3GsztiRgu3tlh2od/r0csiu52xpFOPvl0YmDi/YlbCEwMfPYJrpM/9E191maCQDPof1Bxt3HC9PBCj0VSmkMf44I7+k+Hhr3rCLS9ZUIwRhjIwff0TnRSJl4BDw23g0ZmjEKXS+5Wp/wOB1yHCSwBYawYhiizt3JC7atyumc7nRouggELWQ6x/hlTWXTVYmNY/m08Px7DR2Jheo4KqEXD1PGUSrB6gbPYRyXfafd1xD3zkUCRe0zzx8lKDBwWWUyZ8hb2GIC7mGJLkHCnXuVWamKkciPex9piD4HNFW7gyUqa976BlJrvwEkjGf1StAEuecqn/hIL76NEtqOV7zXXtM438eeYN2ns6uu7d1j3Z5pwaPZTrsaXGCY5kQ5dWI7FSJ/qb5KejMFBciKNj3pxrCJLT6T2Pt2YwQz7rLJTQQydGQtj6rpTNcSn+Q1Qb/c69eQ5fJ5ILHwDdtdWzttn0Ij8c/4YgQelxAS31h+kTLbg2VPcBfmMMm8cb6E5Z2jy+FBDPj3Epi6uqa/DeZDxivGUHDsimfHtowr+p1BA9h/wYlEY+0ALPdt/l5BBpwTV9if65a5Ru6S44rOCRmle6bV6FZNR9hc8lTrek5SDIpb0zMLniG019qX2Gggyc6hhroZ50spuesoJhX2XPrP0HfRhrVhL6yDf+ETkTskRYE9E/ZoDMWyBsGL3I+PJk/Uhf/CchG48xzYHDNbpjFGP6leVjlN68i8vUKjodXmY0UAKBqKlbEhQLrVzPdES5yAS6zoYUyOuUPDVW9vzduAIFqCja8OZRq+sx3n9V/pDdC+inOBMSyLRypbzXwttPPcHSxJ06akEaE/Ua7bzCajx9VzXPTxdJbug+etEgEGhmkhOSTUZVitZYpy7C/Gmi4uZA+4HcO9xhiKKL4W6qU7tLiuoX6WxAT/Z86E2cadk4ipEZodU8SzM4di5n1149w6a9Hv33z+qLt5hmGjnXC05Lkow87JJTZG75ioQRUx/piTAJm1nLJ6j/4S1RsSsft79xUT9/lhlHo6WdiipcRnjPbBK9A2s4KRFSGc3WC0Q2xzUMBfRgIirpGM0CSFDp5tKJFjjdX4PjBQ3mV+jaNweZJyQ7kdmnwiMNejfje0l6H2dQ1OXtOUB9KSTSi1FDFqEKAD7Mk1Tx+mi0Ipx9Cv4d3sgY5ZJ6wAGK78Du9mTCBRyp74AwSym8OXFZzb5zWlJIEl1Nz/jbN0L/kX8a5wf7wS4AHuw82fkE3CrOLHvJAkaEfxlw8QB7n+9927f8nDCu0p+VTEvA9ct8ArnyQf2Rl2grzxIFIHHNUJnFzwfl62QrEBPBUaAAPV5h1hQl8drtvRYwl6hnImOmv+gMf2oH7RxkYy76qyMfa3VdjUeTs4RHAkcsfJ/asR8eJjWcBEuM8+VKZgjhVfHjIUpBWB+4s4wWq9xpOwoL2b+rCtkrHJceM2vnz2yoEs5CMrt7rtLuCUIDL7/SF9NlhY74Jm60zgnJBz3ZgmfZ2LowmEJO56Eap9vkLcLHafA+nwHO5W0m7Fb2HzvLqoxxL1xJvzvEMoN/nFjKDYJMwn5xD+c10bAZqQLRxYGfTmPb0/W9lTUSDP/wy6xPxTlwGKz/OpkIfvQut3jsVJSsrniiJzuj10gO3gdlTH4wZxJAZ79zBOV99FYaeJn04RcJ/WXaI6dFssLeShRvEI32EOEUtmNg+2segOFMrp+UC8e42gqeTt0skisUsVuCkXtblzEIPO1w3udfhZUhKE/uaPiARTNLHL4hAHGFXhShmRUvFc5KOqbSBCC/nmx1lsWb7cpLXEp7G0NXi5RQ4Lj/gP611nohBjHsRPAwgkmzeCAIFr3orufyyN/99FE3BL1zkhYaFtK7MtON0xNd/N0tEI73h5UkvexNzCW/bbPWMJSr+mnskXIk2/iM8elsjetdhrsjN7gHSMAVl0zkgy+d5lFPhUn90bpLlw6Tx7ZPVQcdCxPQxFiV1d+qTubYsapBxqAhlJzBduzYA6KGu30fzYAjf7KA34ZEqX4TRn6no0iMG6Moc7hDVD4Zg6HBkhqY/UQZ9JwoRmTSkvzlbv7QU99uoIw6Y4DggM7She2ZaxKG123jybSORIZwJEK312l8X+jKvPYG5oNJBMUPbc4EtZgUps/ce4KsEdl7WSB5WQo9NG53pdHtw3zdQGXoeV6KHOZHFeTYjGFxwPfc3o4VdXmmwCZTTLt2UpzbmXp3RG788UbmEcQ5Af9hII+etwzKxSxqsVGFgUitzwlkOZjFZv1od/idYKzLfNf47VUEEtfarhXFUlPjZ0R5mAtIPStEXueFpZD0f7YOV5SoV4zQ1jc0nVady9Ovnf5CADpEZfiMqodaJwnEbAbTxAWDCauWu9c4xBxXangyAO95g2b/0ci6300XX0ImRCnHgUy3Emo50TY9OXXmbvTEyIwqsy9v4Vl74YU7gJgZWk83gVFBsAYKz4OmvK7fWSUu5cTLpm4IPleG8YAbUtaD7W5NTubVRQ2RVIQNMbIheQB3ZU5GxmJBuRSga0v2KQZSU3AD3rvLI1f2cOCMWNACsSIFR3eESMuN3YcFFlvI74kERu+7aRBo2o5M9a2afMd+1/oLBn6r97qeLnmDLbdqpDzbxm1fA9xpb0AvNADC07Bjomuee3vdi/aRhITVDfxgRME8+XC7OMGywF5jh4u32NQa5MgdyLQpq2UUCj49x9za3yYB+CRvsL6l/pXu/Pz7qkDFvYaQJMcuBIJxJ2DKEC2liaULJ7fLFK4EjDsL2sQvFzmHdfGzp/Xg+USD0vcuWdzasWtRXrH7thymJ7P15oQaEsWnVLHVXogeDZIZJVJSKyUxevf+M0A5VnR76ZkbIMlmsgHHvQMsorvu/B4FkBx2bTd7ayE6/EX481DUOAPst+yRkR9qB7njYRzFgh76Ru9UA4GPfy5rtWZWxKBZpLUpXClBxSkIWID7x0PgHfCtqEHw5R9PF90y0jjdltCjN/rzQ+F6gOOgBEjyidMYXgaYsWC/Kb+QTZkNltNCLIMKusJRDsfWUmx5vWR+P+n1k4ZyQtib9bxp5rWXZz3A0zuDLrGODibSEK6iFxdKQdDgjHHPgmceMvVwpRCSbd5+IKIDwm/NjNITwWvGayOKcROk6tG7SMHARDDqqIIxM5ZLrzZhyGbyKrIALvbtqvOxVu6S/lijxtbPGLo745M+tCvdaRuNx3nQLmtYVwdSQsg5zaslXRwsnlrVAQm8JvyUe511b/099I2NZKTAtKCueL1VsRjKHRjxLH179TGlpqn3qLXuh3kTH5bYuDkn0VbHDDyA/f0daZIjnsh1pdqnykF+gmBbh/5+reVs0ZyDyUEK5uhieDl12g98zGsIJ6q22mvyiWvpEkfnD8eTDjrfnrzlVGoB6nnbd6X+hEJT5QTx5UciAkXhsXkagx1Nd6y4rgl+6nI3Fgt5rgtoyLS0C8ylfKHZRILOLZKSi5eMbjgeDUzq9DnFm7g1sLMvRDi8yUzS9MuEdG7HonE8GLVyKu460FirnaJjf0aRjNHbDTFvZq38iXLMz75FaB9cW3V4SYueme+NM5bSECzbcei4//HtPcWmDNkZYRPVYkKr0q869gpFrsBQKNs0KKQKOTfCXbC2aKyRmt20nlGyYDYoPYH/pW1zdsxJG7G5A6eyBtouLTo0HBkn0LGdnccYQi2iC+sXnLDJtj2DVA/7bkP45B6p/IxOoLYo/f6lTROnP0WNVcNrVB8LEqO6s9GmcJR+MRsVQU22fRW2pipAiRqfeJDnWmY37K6Gvz9OBsxkJ+q8h/7RB8t1i8riJI3+mhrjN0X7kJTbHh13Teyt9ySa6sGg3HD2f35BpJYvENgD3Cr4a6Q4UZchAffPEv97+zbPLs9XUUHzisY7zF4m9bjz43wiT7fXXSPUQ3IE0p2gYHAU3zeLuQDrDqi8mwEIejEii5yxUIMcsm6jWkd+Fq5N6HhFd3SWwjR2tvZXUZ/TdiavyUVNBL2G3CBvu7Y4H9mn/QCZ41OBBfMFJSlPYiCw8eq+c9kjLQ8qfK+jC5ypVMXrh1o23J5vYy/HkljERvU1XXp/bO9bTxaV9tgoguUO8D0uVu2FmwolwjZ/68CeY0/KzSwvtThdqUQ7vPyYp7Qn5mgblYZGa58u3bCnUsyDvDHH6ROWW8ceGpqyjuA6GP+EFg6WcAAAANUgy+QAAdIogKABACbXKEA+MA2LAgAAAAABWVo=' | unxz | tar -x

chmod +x remediate-health-status-hr.sh

Run the script and follow the instructions:
```
./fix-health-status-hr.sh
```

Additional Information

Each run creates a working directory in the current working directory named remediate-health-status-hr-<timestamp> containing the run log (patch.log). Provide the generated logs if support is needed.