Intelligence pods become stuck in ImagePullBackOff state when projects.registry.vmware.com repository is unstable

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• Command napp-k get pods shows one or more Intelligence pods are stuck in ImagePullBackOff state.
• From root shell of affected worker node, 'crictl pull <path to pod image>' may show 'failed size validation' error. 
  
  For example:
  root@workernode# crictl pull projects.registry.vmware.com/nsx_application_platform/clustering/third-party/spark-operator:20606736 --debug
  FATA[0040] pulling image: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "projects.registry.vmware.com/nsx_application_platform/clustering/third-party/spark-operator:20606736": failed commit on ref "layer-sha256:<hash>": "layer-sha256:<hash>" failed size validation: 348127232 != 478708363: failed precondition
  
  Note: This error can also occur in cases when there are no issues with the image repository if there is a proxy server appending content to the image layer or the payload is modified in any way in the network.
  
  

Cause

Issues with projects.registry.vmware.com repository can prevent Intelligence from being able to properly receive pod images.

Resolution

Wait until issues with projects.registry.vmware.com repository are resolved. If issues are not resolved quickly, contact Broadcom support and reference this article. 

Additional Information

Impact/Risks:
The impact varies depending on which pods are affected.