Patch File Checksum Validation error while upgrading SDDC Manager
Article ID: 323301
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
SDDC Manager upgrade fails with patch file checksum validation error:
The format of the error message is "Patch file {} checksum {} does NOT match with the element checksum {}" like below:
2021-05-13T12:14:32.046+0000 ERROR [3daba3c97c2b1d0a,ff43] [c.v.e.sddc.lcm.orch.PrimitiveService,ThreadPoolTaskExecutor-9] primitive post upgrade returned exception
com.vmware.evo.sddc.lcm.model.error.LcmException: patch file 6b5d5a0e-5bdc-4679-82c3-860bd660ea83/bundle-24282/sddc-manager-upgrade/vcf_pantheon-1.2.5-465.53c15fc.noarch.rpm checksum 3f7ad99c540f5d4f919ba065722563bbacf6f05844720d792133222fbe504742 does NOT match with the element checksum be0b6513f42b2857aad965e0a7930c07c801b9908be767355dec68311c019b14
at com.vmware.evo.sddc.lcm.orch.PrimitiveService.doChecksumValidation(PrimitiveService.java:1014)
at com.vmware.evo.sddc.lcm.orch.PrimitiveService.postUpgrade(PrimitiveService.java:242)
at com.vmware.evo.sddc.lcm.orch.PrimitiveService$$FastClassBySpringCGLIB$$aff213fb.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Note: This could happen after using /v1/bundles to upload a bundle or using SDDC UI to schedule download a bundle.
SDDC Manager upgrade fails with patch file checksum validation error:
The format of the error message is "Patch file {} checksum {} does NOT match with the element checksum {}" like below:
2021-05-13T12:14:32.046+0000 ERROR [3daba3c97c2b1d0a,ff43] [c.v.e.sddc.lcm.orch.PrimitiveService,ThreadPoolTaskExecutor-9] primitive post upgrade returned exception
com.vmware.evo.sddc.lcm.model.error.LcmException: patch file 6b5d5a0e-5bdc-4679-82c3-860bd660ea83/bundle-24282/sddc-manager-upgrade/vcf_pantheon-1.2.5-465.53c15fc.noarch.rpm checksum 3f7ad99c540f5d4f919ba065722563bbacf6f05844720d792133222fbe504742 does NOT match with the element checksum be0b6513f42b2857aad965e0a7930c07c801b9908be767355dec68311c019b14
at com.vmware.evo.sddc.lcm.orch.PrimitiveService.doChecksumValidation(PrimitiveService.java:1014)
at com.vmware.evo.sddc.lcm.orch.PrimitiveService.postUpgrade(PrimitiveService.java:242)
at com.vmware.evo.sddc.lcm.orch.PrimitiveService$$FastClassBySpringCGLIB$$aff213fb.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Note: This could happen after using /v1/bundles to upload a bundle or using SDDC UI to schedule download a bundle.
Cause
Life Cycle Manager does not validate if patch files are correctly uploaded to SDDC.
If patch files are partially uploaded, LCM will still mark upgrade success and only throws patch file checksum error when performing an upgrade with these patch files.
If patch files are partially uploaded, LCM will still mark upgrade success and only throws patch file checksum error when performing an upgrade with these patch files.
Resolution
This is a known issue.
Currently, there is no resolution. We plan to fix it in the following release.
Workaround:
To handle this issue, delete the corrupted bundle from LCM completely using bundle cleanup script(bundle_cleanup.py)
Steps to follow:
LOG FILE : /var/log/vmware/vcf/lcm/bundle_cleanup.log
2021-05-17 21:35:08,702 [INFO] root: Performing cleanup for bundle with IDs : ['731433e4-122e-40a9-aaba-3ebc1be133d3']
2021-05-17 21:35:08,702 [INFO] root: Stopping LCM service.
2021-05-17 21:35:08,702 [INFO] root: Execute cmd: systemctl stop lcm
2021-05-17 21:35:08,814 [INFO] root: RC: 0
2021-05-17 21:35:08,814 [INFO] root: Out:
2021-05-17 21:35:08,815 [INFO] root: Removing LCM NFS mount.
2021-05-17 21:35:08,815 [INFO] root: Execute cmd: rm -rf /nfs/vmware/vcf/nfs-mount/bundle/731433e4-122e-40a9-aaba-3ebc1be133d3
2021-05-17 21:35:09,183 [INFO] root: RC: 0
2021-05-17 21:35:09,183 [INFO] root: Out:
2021-05-17 21:35:09,183 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upload_id from bundle_upload where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,204 [INFO] root: b' b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f\n'
2021-05-17 21:35:09,204 [INFO] root: b'\n'
2021-05-17 21:35:09,204 [INFO] root: RC: 0
2021-05-17 21:35:09,204 [INFO] root: Out: b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f
2021-05-17 21:35:09,205 [INFO] root: Execute cmd: curl -s -X DELETE localhost/tasks/registrations/b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f
2021-05-17 21:35:09,943 [INFO] root: RC: 0
2021-05-17 21:35:09,944 [INFO] root: Out:
2021-05-17 21:35:09,944 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select download_id from bundledownload_by_id where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,963 [INFO] root: b'\n'
2021-05-17 21:35:09,964 [INFO] root: RC: 0
2021-05-17 21:35:09,964 [INFO] root: Out:
2021-05-17 21:35:09,964 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in bundle download table
2021-05-17 21:35:09,964 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,982 [INFO] root: b'\n'
2021-05-17 21:35:09,982 [INFO] root: RC: 0
2021-05-17 21:35:09,983 [INFO] root: Out:
2021-05-17 21:35:09,983 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in upgrade table
2021-05-17 21:35:09,983 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from bundle where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,002 [INFO] root: b' 1\n'
2021-05-17 21:35:10,002 [INFO] root: b'\n'
2021-05-17 21:35:10,002 [INFO] root: RC: 0
2021-05-17 21:35:10,002 [INFO] root: Out: 1
2021-05-17 21:35:10,002 [INFO] root: Deleting bundle & upgrade info for bundle ID : 731433e4-122e-40a9-aaba-3ebc1be133d3
2021-05-17 21:35:10,003 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from bundle where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,053 [INFO] root: b'DELETE 1\n'
2021-05-17 21:35:10,054 [INFO] root: RC: 0
2021-05-17 21:35:10,054 [INFO] root: Out: DELETE 1
2021-05-17 21:35:10,054 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from image where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,073 [INFO] root: b' 1\n'
2021-05-17 21:35:10,073 [INFO] root: b'\n'
2021-05-17 21:35:10,074 [INFO] root: RC: 0
2021-05-17 21:35:10,074 [INFO] root: Out: 1
2021-05-17 21:35:10,074 [INFO] root: Deleting bundle 731433e4-122e-40a9-aaba-3ebc1be133d3 in image table
2021-05-17 21:35:10,074 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from image where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,112 [INFO] root: b'DELETE 1\n'
2021-05-17 21:35:10,112 [INFO] root: RC: 0
2021-05-17 21:35:10,112 [INFO] root: Out: DELETE 1
2021-05-17 21:35:10,113 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from partner_bundle_metadata where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,135 [INFO] root: b' 0\n'
2021-05-17 21:35:10,135 [INFO] root: b'\n'
2021-05-17 21:35:10,136 [INFO] root: RC: 0
2021-05-17 21:35:10,136 [INFO] root: Out: 0
2021-05-17 21:35:10,136 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in partner_bundle_metadata table
2021-05-17 21:35:10,136 [INFO] root: Starting LCM service.
2021-05-17 21:35:10,136 [INFO] root: Execute cmd: systemctl start lcm
2021-05-17 21:35:10,174 [INFO] root: RC: 0
2021-05-17 21:35:10,174 [INFO] root: Out:
Currently, there is no resolution. We plan to fix it in the following release.
Workaround:
To handle this issue, delete the corrupted bundle from LCM completely using bundle cleanup script(bundle_cleanup.py)
Steps to follow:
- Login SDDC VM with root user:
vcf@sddc-manager [ ~ ]$ su
Password:
root@sddc-manager [ /home/vcf ]#
Password:
root@sddc-manager [ /home/vcf ]#
- Run the bundle cleanup script with bundle ID:
cd /opt/vmware/vcf/lcm/lcm-app/bin
root@sddc-manager [ /opt/vmware/vcf/lcm/lcm-app/bin ]# ./bundle_cleanup.py 731433e4-122e-40a9-aaba-3ebc1be133d3
LOG FILE : /var/log/vmware/vcf/lcm/bundle_cleanup.log
2021-05-17 21:35:08,702 [INFO] root: Performing cleanup for bundle with IDs : ['731433e4-122e-40a9-aaba-3ebc1be133d3']
2021-05-17 21:35:08,702 [INFO] root: Stopping LCM service.
2021-05-17 21:35:08,702 [INFO] root: Execute cmd: systemctl stop lcm
2021-05-17 21:35:08,814 [INFO] root: RC: 0
2021-05-17 21:35:08,814 [INFO] root: Out:
2021-05-17 21:35:08,815 [INFO] root: Removing LCM NFS mount.
2021-05-17 21:35:08,815 [INFO] root: Execute cmd: rm -rf /nfs/vmware/vcf/nfs-mount/bundle/731433e4-122e-40a9-aaba-3ebc1be133d3
2021-05-17 21:35:09,183 [INFO] root: RC: 0
2021-05-17 21:35:09,183 [INFO] root: Out:
2021-05-17 21:35:09,183 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upload_id from bundle_upload where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,204 [INFO] root: b' b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f\n'
2021-05-17 21:35:09,204 [INFO] root: b'\n'
2021-05-17 21:35:09,204 [INFO] root: RC: 0
2021-05-17 21:35:09,204 [INFO] root: Out: b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f
2021-05-17 21:35:09,205 [INFO] root: Execute cmd: curl -s -X DELETE localhost/tasks/registrations/b5c1b1d8-0c8a-4875-a96f-b5bd0fa4f26f
2021-05-17 21:35:09,943 [INFO] root: RC: 0
2021-05-17 21:35:09,944 [INFO] root: Out:
2021-05-17 21:35:09,944 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select download_id from bundledownload_by_id where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,963 [INFO] root: b'\n'
2021-05-17 21:35:09,964 [INFO] root: RC: 0
2021-05-17 21:35:09,964 [INFO] root: Out:
2021-05-17 21:35:09,964 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in bundle download table
2021-05-17 21:35:09,964 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:09,982 [INFO] root: b'\n'
2021-05-17 21:35:09,982 [INFO] root: RC: 0
2021-05-17 21:35:09,983 [INFO] root: Out:
2021-05-17 21:35:09,983 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in upgrade table
2021-05-17 21:35:09,983 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from bundle where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,002 [INFO] root: b' 1\n'
2021-05-17 21:35:10,002 [INFO] root: b'\n'
2021-05-17 21:35:10,002 [INFO] root: RC: 0
2021-05-17 21:35:10,002 [INFO] root: Out: 1
2021-05-17 21:35:10,002 [INFO] root: Deleting bundle & upgrade info for bundle ID : 731433e4-122e-40a9-aaba-3ebc1be133d3
2021-05-17 21:35:10,003 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from bundle where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,053 [INFO] root: b'DELETE 1\n'
2021-05-17 21:35:10,054 [INFO] root: RC: 0
2021-05-17 21:35:10,054 [INFO] root: Out: DELETE 1
2021-05-17 21:35:10,054 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from image where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,073 [INFO] root: b' 1\n'
2021-05-17 21:35:10,073 [INFO] root: b'\n'
2021-05-17 21:35:10,074 [INFO] root: RC: 0
2021-05-17 21:35:10,074 [INFO] root: Out: 1
2021-05-17 21:35:10,074 [INFO] root: Deleting bundle 731433e4-122e-40a9-aaba-3ebc1be133d3 in image table
2021-05-17 21:35:10,074 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from image where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,112 [INFO] root: b'DELETE 1\n'
2021-05-17 21:35:10,112 [INFO] root: RC: 0
2021-05-17 21:35:10,112 [INFO] root: Out: DELETE 1
2021-05-17 21:35:10,113 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from partner_bundle_metadata where bundle_id = '731433e4-122e-40a9-aaba-3ebc1be133d3';"
2021-05-17 21:35:10,135 [INFO] root: b' 0\n'
2021-05-17 21:35:10,135 [INFO] root: b'\n'
2021-05-17 21:35:10,136 [INFO] root: RC: 0
2021-05-17 21:35:10,136 [INFO] root: Out: 0
2021-05-17 21:35:10,136 [INFO] root: Bundle with ID : 731433e4-122e-40a9-aaba-3ebc1be133d3 not found in partner_bundle_metadata table
2021-05-17 21:35:10,136 [INFO] root: Starting LCM service.
2021-05-17 21:35:10,136 [INFO] root: Execute cmd: systemctl start lcm
2021-05-17 21:35:10,174 [INFO] root: RC: 0
2021-05-17 21:35:10,174 [INFO] root: Out:
- Wait for 2 minutes since the bundle cleanup script restarts the LCM at the end of the process.
- Re-upload or download this bundle through API or SDDC UI.
- Upgrade with this bundle.
Feedback
Yes
No
Powered by
