Upgrading VCF Automation and the Migration service engine component from 9.1.x to 9.1.1 fails with a precheck error
search cancel

Upgrading VCF Automation and the Migration service engine component from 9.1.x to 9.1.1 fails with a precheck error

book

Article ID: 455559

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

When upgrading from 9.1.x to 9.1.1, the migration service engine update failed with errors similar to the below.
The upgrade task may run for an extended time (> 6 hours) before the failure is reported.
In addition, attempting to upgrade VCF Automation post the migrations service engine failure will not be successfull.

Observed error messages
Step vmsp_upgrade failed. Reference Code: XXXXXXX
VCD_MIGRATOR Precheck Workflow pre-check result status: Failed

com.vmware.vcfms.system.precheck.backup.PackageDeploymentsCompleteWaitTimeout
Backup precheck failed. Timed out waiting for in-progress package deployments to complete

 

 

Environment

VCF Automation 9.1.1

Cause

This issue is caused when Migration service engine component is upgraded before VCF Automation. This upgrade order is required in 9.1.1 due to compatibility with a new version of postgres. 
Clicking the "Upgrade All" option in the VCF Operations UI will cause this failure as this will attempt to upgrade VCD migrator and VCFA at the same time. 
In addition, manually upgrading Migrator service engine before VCFA will also result in this failure
Please see https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/vmware-cloud-foundation-9-1-1-0-release-notes.html

This issue is resolved in the 9.1.1.0100 patch release as documented here

Resolution

To resolve this issue, please see the steps below

1. SSH into the VCFA appliance and become root

ssh vmware-system-user@vcfa_fqdn

sudo -i 

2. Copy and paste the code below to the console and then execute the command.

 base64 -d <<< '/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4E//FYxdADkZSf35ZUq9VGwG5P3znvYHDaU+KcJeLtqaY05DcY2Bk1wI6IBIS+NeKyr2jrddVSpU0PXjqJRhBWKLgOHC5QU8qG1ofmFm4s5d6geyIkT+gdSPe2pL+Sm5k0suXI/wAc29yN6/lUAW3fu6EdCnK8/UDal8wf68wtL/TInq8A9BZW+aPquoJAOWOmOOtPjsHf0BxhQJRTW11hEGi4qd/isLPiYRfszrafECnHDW5V0PGO2ttUe2KQjDjw+rE7VAKyT9yzxPI7exKrlBJYYNw7KpG3UyTlkZ5Pg4AccPHh6zkLova0oDl6hj3zvCbq19Yio5hIpWrraxhmNrl8IoxM2N38RztazQ/N40HKIC7JNmIQxKWSDMva0hkrTDc7EZgQqBMgwZ1ln0Nzgy/W1SmBgduobAaJ6+zXAUTYvCv5dSgOXEsl3u1l1G7MHY4EqUyMCwNmLDnCfsN/zG1mcCyaG9Tw3P5Ky6bqWFtKqCbDojZv9q8Nc1ZYVpByHzjcnwNnGzVfgID8Ol1yPQpjEoTHJV3XbKFYJ1J4R664XgY2cOYFctCa3aTkLq7Jvu8h0KoHEb4tl8d98OLANbPskr6X09eAekNw1ZkCgRMUB+C1omJwhvT295Kk93naw84bZyjJhk0nqbSfHAd2zWuf+Y5XoktnNm0VDS8jAStR6EXnWhVTvtiociI9Jw7QBSCzWPlIUg9DSsyspcbGaSIY9wyObXqKxO76nrZHQBflfiDKr9kkWtn25LyjeWzDnaVtAt5ZqvcBf8EGX7KYOw7ICgEPDB1irZ4nZKNPyaws2H3lF5/kmOartQIUHxW6+4coTXj5RYVFSA3I63a34gOLorCOHe6hKQk/gULz1YA/6L0WbtfSlUQJmk7TfFenzja225LlMooToc7hSzxQYX+uvbc/aCXl1RkBZPBSS/VQF+J3Vvhy7uKFOxR8ZPAMvME1Nc7IHw/7+QIVrKK6QSUXY4imS+4zjJtXkye0EEG772/FFixlncEAoG7ZqTmZvfgoTyAkPBDl2xg8QjoSivGKiH0m0mx0b1zBFKFOEMjWrmRWPPgOcUAOxdvXx1R4SiDt8pepH4JZSC/m3O8G6iWWCjiric6mkbdDXA81as36BNtrSHND7yDva7bNHmZSGOFf7LiQl2OWdu1gAjOhcuTOFHo2bUNbG/v1IprwHiEq20LmQm+Wo9yKnzQh9X7IMQcSJDfgrUj1Jv9RMBnhUaP4Bx+nwxMYhfMkSTALey9tO47uljBF3Kx3G+AHJ4DnP5adivNyl//esPr4TJ+TRpI66mSCeufLpvDasPEhGPEt8zshe29SO24N9nq4WMfevSCe0IlsZuUHGtRZWbx/e4pvTeyl9CwTcib00YnxqYFZwYBWvON6FEVRFLqjUu2hdBszc7IR4OfwND8n4+47Z6OJTcKYnL+cCXlkcbT24sBeGSBjNE+icjq65h8VKLpyj1JIU03yP+qBn6fAyd8HpW6XhQ4++Y2pxfDa3K6xYvSSMxlIs+pd+KFFhG8l/wTAx2hsTTNw/PNgIt9saDL/tMV06cBCMqfQ17XjemXD0VnBwwnifpAo27Q+9iTpCW56MP8z2xgZe/tQF+zGyIf2vKF7wtHkf+jPAGkG2IYIChLvIa6/DNl4vO1mmTyAnVV8IMuVabJRBB3SfRkqntVvD6o3tEdfOuQElD10F4ebgms8XAGZf3zDun/cWK17I5XZkEqBjgAAJgyXIeKf6S6wwvc/wQZnOh6Td0KHyRAsr0bOeprVClr8tLfRGFHzvOWS3bMXivjUvJFTlMrSTtIc7qkYVnibMJ8Bo2dcq21cJ2okGfbkEoLxUevV9EyQuyiMWWUavqFUdukXegxF2Ekun3/mcx6fbwOg5imFAPsNls/qCD9PL9f/ZE9Oocay0yR1E+PIGseCuphtYXnhv5ucV+TfKVHykieuIJFhL+qIl0bS/DR9re5Drbx7C5dmqoIec+kEx3ANkmxCBVPRTNpiq+OVzZVtbaJwM6Gj1MHi7upZY9FY2X/t1QgMsXZHlihkM0XN8jmcIs4Nhcc8sPB6A6qCLr8WiBJ6xU6t3RMTjquDbAZ02LWbJPhHUSXzc9OcX++sHFi5IIKvalTBuio/GZ1+vq+aCbaPpmG7x3vT6Oymr5NAqPlhWVf+mlksTrLR5vdspyKchcAo9EeEjIxLg596OFI0ontuPyifNJBEfPC7EreshVNoNFfhiJvquH/Foz6VedItrXTm9/U5JLMGJLZFog7Cdf0NBPHAJEaGyjI23XZ+tROxZ0hJ7y8hDHonq7OGAfNLZpyIx2XlNqGMCIfFQC3NehRcGlbsztwLm+cBqdAo470Sxd5TzLkhkPBR+TklEa8QfJVv/23Fcy2EjtFWdvfbQk8QgotdVsv673YLDneT4MJuI5LoAoMvNC2K2B1o/QP8TwHIOmheu+2OBLJSkNUZuvWeneD8c3ZuCqB9K5IlTR8/8sAGY3kAc94RKZfkZcn0PIMViG8JknOCZVI9s9tJGs0+7W1wi/oX+CqckBe3jPz7A6orHM/XPsCDMIPzQlHdOeMMOF/h/DCrsungIC83i3eLPnIhVrwW7Wtsk3W25HOcCYdbKdfBnoWjgf85vVCVJPnkmmaNqNuztYyyeci+TU07dwoF5PeFUI1xJbR8I6qHFQYXCQZR7RGswwyVYrtF43+yOXml95YtfNo9rmkYkaZQXcakWduirJKSF9vEnoE6LzIZBo4b6tjkxnsZ+/yuozSP0SQnOcqn1F5qWEygYJvF9V12Mz6JeTF0ipwb8Ib9g+ZqDOA4+w+SnOMGdSRi7X77Moz0NeTEdTAXi/ltf08kF3HCGKCDK6sfgSWyHiXRqbzco4N4EI+JLnSIDvgUAdQKobQObbDk5ix+krQSw4+me/yE+2O/cQIXAFDv6Nic1n/FYjR5ktxB2UC8FwGX0A2Hz5D4qytYmUR7oJRtGKZH1ceF/jHjS/qUknq2WKhyy9yYmAp8Q9aAe/bIvhxpNAkc6nZEtJA17ipcNsgNfZ1T7wNOZIA+ZseyB1taLoUfXD2AvMMwsYIlcQqwCdKctFX/nDl440WEXuTjWPNK7cwXDaFzbNbHwdqpCaLpKmobtLCPeRsu/ClCLXcq4AmGob/0clTsBPkoENNLzRBJs9OQ3pT8cevpdKuuHOSSdjQBzz1YCaat/G0nXnSLrfGmVDWM0Ch0Y/DA3CapHJP+q9PV/QNX5SBnZdbA0LLWWKn444K0I8LlN3kS9a24UUY7FdLI4ZfnxW5Eos3UoO44Z+ktSTWAAX0ZpPQa5pt+eh7Yo0p4mgPqpfeG8YfnWtdhLdJwQxoliN0oyXwHNlCQKVI/YBrwwl2BAK0uXHKAHczvSQucp0d/6Ua5ygTLhf/kx3LcdSHusCbhZ0CL1LynMLZqL7vhv6ncn13kNsrhBze/czdlwJVw/X+ny8k9cko2u6GpTPYqAPt8yEEtiSsqgUZJp5/3CdWf4reUC3c05UvmyveP2iv9kHVMZRLH46ZMm4mGfKaNXOCxlbazjUjIQ/QYCQApRVmnF5imeRAMvTcfdFwSUxvEQV9VeVH5MqW0iBScKdQkXzIP5iGgjZm8/FstVUEcftSKhKmlL/tbdOOsfPXYVc4qXg9XqrTHt9HR28DXvb++CNV96T7Voi6mFnTy2X9jgt4w0uKWObXVS/1pwC9mILNPhvtTRGpBUofrL+Y4342/Sb6T8VvPSGmj+JGML0OUkx8k29BPDUMpH9RB5NQ+JVDcCGh0U5B9s7/fdMLzrbxM+6vgIUHn4HLoVu1LwT8yOI3ZWu4pGxOUtmmVBz8mqJQ26p0bbl+R4IxWUP4HEeQ9SZmK8fRx5nks5B1N49S1hggh2VFlgJVlxYAmNVawtr+0RtqP19O9Yj2qpDRxEtmMdxNa3b+V9qBNXlVr7a1xKg5TBiCIY8FKzrQJVw0BbIu6/M4dbfEUB+cBsJENMG4quHOdFzkaNmSvwKFOR+nY6+SsFMSRGnNwHOwaQ1KXprJNIKUMdQwBWx2q6KrENQjQvjoAxcaz5rZhUmRVdmGjj71Fb/BC0YelEO02dfdRSjz9cCXQZ7xCf2dfpEKyfGI3dk9Ft2aZQNfSFyL7OyrSwGvvhPuKYbNL1dXj8V2GinGpBm3dVG5zpHpBGXx2hbsaZA7V0oYo8QFyxWQUQc9nilPM7J9dxvz+6PBdyr90s3o3K3+ZKF9j/16ZGQhyPtZs8tNThCseqsCa2BBDe9jUmqCwBSSSMwlTO/YC+41xEv2I+SWjIDZ/qqNCwQYusCsCYe08qmgrcNsg+MdwYE360uyD5jsDrRxFcDXc9p5nEojnF9JYYB8oPHAUNJtWzBrrKRHtAxTIIOy7Ak65nGjaIBXKp0/ZQRKuMgI7FolfgXbRk7xOeWHcBtN9Ww4tP/JfFmr7qURU9f7iFv9LBZLi+D4Ftd3UaydPACW/CUkNEz11Kgqs6Eh/cVtH2KT9YlMSIgvJrHABPzVNA5u8LJ1Vdb6hL/n7bsV2JhySSxOpQu1fIoG2Ojy3dN0ObA0SdrTwjvTJc1cvr7dbcK5LYzccOlC+jVTvNicl3dP7w6lmrnlBbae092GyFP34nWi+ugkxz+B1lVjaADviNddP3DpB/dmD9VNXyImvi38GV40ESKax9M3eRbSLo/1beByhMdHVUUEzvePx1Ct8QN3WxxMJ/zzQUhi3V5libg3pMEdTAhYJ7ibiSwCujJDo4Um5Imw9JcDWbVBB1jKo0/Ch1FY961LiFdWrsWXgWwhbZDzGegVqIvLLdNF6akvG+lElw8l+kMjih2TWna1eCvxNJ/d4NK3m+fc945qrqv6TX8obRvT2MoivQm1PyiZwk7X8vp81pUyJVyOo7Aj6YsfzmcW6FSceiofwEibEjrafZLrgX9GBVnymxY+WmW2CogT0UvwtVD3CjygJZOv++/QfHqSxuY5Kxt/Iic7L+OQEWo3vnCf7ED9MF5KexSg9rzmpoM8Fukal07frHoRSA62FawVl78jicBRDHMN0gY5o/BNR6BQqPVUJgdsRzwXnwHrfzf1Jb5Jr62o1KVHcaaqkhZWEaBIeJBuFD09muyL2sMApCCp8Mwz0EnAA4ly5QV+51m0wa2N4nQspm6CYxtYkR5ZQtbn3BDWIW93dqQbfhsdWcF33YWUWKdmY2/lvvp4F0LLHkktIBmIvQyj5Gy8ubQuVvU2/gvvay0oHWCxiXA8rGV/HCnv6qCTeeOZgEGdtrHs8FuAHkUsAsfGoT12jRJtlL4DQ0jh6KlLnDZGceQ0Kw6gdxeQGQJ140WBV/XBR+MIWevkYCvZXVEgVOrOXprVpSNAjfqq9wBUJqTvk2NSLcGRnzpNQxk/6GHJOOmbgaF0PSlh4JUFImDApkptJz4ovSOb01/B5FOhHu9FnTlgtKCabnrUYzN05XvRxiHLl5A8KRO+L/JUvXZZ9sGr/oee0oxLncgCAkKl5W8XPcGcAoxusoTkOdTZdLBGvdY4Q62PkYIaYQ5cwU4O620BjtHZX1PhpfrufVR1PoT/aZvnjS5th+ePbDcWOKU5z4CZ1K+KjNCvAQcttvpd+Ftj9+SvjLfUtsQf3zkcHtXyCKJu6otpL8Pgx1zpozPvWS+gcWqcydev0yv59gsOpVanA/GkF9B/AQGJHjjGSIlUWuXh75n9O30uSWvVQFN8AltIzDE6yONFQeQlRJFbskUulfkMqvtj/a8niaVK+FizgZnvyCyFJdD4mMquAgCvdDbKbD2g/Qo5t0ni16GXEfiFLoo4N9Di6C9z+rERRpODqG4X2w965DwLSMMFlMHF50v2TNVVwFz/lDwPPvYZrFOIysSSIKgQmdAmczUlJDxRGrI6LSUKpm6I6AuZw9pz7LnlqXYYsuQ3z5Wb3ob2br1OhMhqF7wN6TsIYdjJb3gO4054wnykw46IaTqs1tIn4lslEGVjMKO0FLqNzCHMnAkS5mBhSDQNjNoEax63Z9lENIUjJ/iMo2/wrciRR9GMGxFrEMZpq0QoMdZdOSLNZmCgv1IuTv8GsU1duBNVlEStYngSbCjDKpUxOYULjL2VryRSib2SOWTcIY+MQxbNYKE3MlbkkX1p8CNJe6+a98+m3k0O/8d2qpYn2xnVdHMPfyHhcVR+ps/PfPWVhB3GTGTCOLD4/VF/nW9R3pQTqslV7CN3rCTvnjLfZfv6rbUh9SiL4zewVToEfWE4KSODuzoGu4NJ8C2x9bWyLOQ+x79Y3AmxssbZR2IxC+PYIbJD9EjQ4FtS677Dw9DHR0EgBJVwiQ1/i/Qd3nS1W3UQoaiqhpEVuaFJFjmeocuBwHe4LPzJB49zADha/uLTVS8wIhN2UtCPg1FozNxQJSbEyZowtsSnxQmLiiOIfT0E2rNyUbPMcKrFnwyPNGnes9ufkkEkuWw4+H/XYYzYAC2gXD0LaA2IlUEAyeqk/BaeUXrqSW9iWCDs/Cdi4Efj3QJBhZMlFDYnlPRHTgQv/nsZ2CDaUVpTRiJ8iT9HdhgiM31ZUdwFP86FGHDypJPbGAnlYk/t8Bp8OuduvOfdUOs1e7clPW5PW28W16BTGGeZtWzDT95tyWXXgmoGeMCUMPeY4SALd3JZC6O0rSa30oPL6PBKwsQghWWq3vcbor8fBNuLE/zQ/gsfOmTIJRo8+PHLNxOjjJzhHEqtfUC4xB9Ux6l6K1MsAoWCt0VgNmA0U9RF61DVVnvsl2vW4RcbtAfv9cM68arodhLE7xlPrSJLmZ2I2AFEny3p0PcEAHlQMGoO6XZb3KRCrCC1cUBH2L2oh1PrTi86qx3DgegfG91eLxyKzxjC0SFS6Q0eTl4yk7jEG3x5lS5tLBRptZbvPifx9jH+CtaXcnL4QFxSpPmDwypE3HZsh3Mx3wczYplYJmpP+2VbQHI8YwNHaWTw5Z4wDrJ3raff76BqbBRVNzdj/jYvgBtLN3IV/DXnTz6d/Fo04bLNcf83gje2xiuui+/yebBSvfOEdd9l5zdumgKz7KWj8/ctxrHMWx2hGQsEP8r8+OeTsu9m9FEnODZiFrf/HJOGV4L+ueRuXjDrtbJuYYrkCsCQjwTd2oixsJ2XMqOCTvLNJALWgHQcUpN8JKAcdeJE3hasIn2bz5tCRXRO6FtaQToBCP6/rPyt4hTJ1EodzCqPRFj+lNYkj1htXSPV7Ryb5IW6o52Psay4JBK4DiBxnFfNovY6nrDbgvkXrlGLJqhjlX3sLXVpfv8YslNJkY+Wuebd6ZjlH5lR57TEMvOU7jqXmmBZuuYuFmmR0YsO9wtWXipHyWFoKf6iFuGZ10mmtlPil4qirFsbciKzr4O7TpEQYCrBH1APDMAAHpapaXGFlmdAAGoK4CgAQBRgzGtscRn+wIAAAAABFla' | tar -xJO | bash

It should end with "Suspend has held for 70s".

If it says "REMEDIATION INCOMPLETE" or refuses because the failure doesn't match, stop and investigate before continuing.

3. Go to the Fleet Lifecycle Management UI and retry/re-run the "VCF Automation" component upgrade. Its backup precheck should now pass.

4. Once that finishes, go to the Fleet Lifecycle Management UI and retry the "Migration service engine" upgrade.