SDDC Manager failed to upgrade to 5.2.2 due to domainmanager crash

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager failed to upgrade to 5.2.2 at stage "Validate Services"
  
  
  
  
• In /var/log/vmware/vcf/domainmanager/domainmanager.log, there is OutOfMemoryError as below:
  
  yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,0000000000000000,0000] [o.a.coyote.http11.Http11NioProtocol,http-nio-xxx.xxx.xxx.xxx-xxxx-exec-11]  Failed to complete processing of a request java.lang.OutOfMemoryError: Java heap space
yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.v.c.service.ResourceCacheService,dm-exec-4]  Exception happened while performing config realized check
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:xxx)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:xxx)

yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,0000000000000000,0000] [o.s.boot.SpringApplication,main]  Application run failed

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

• domainmanager service crashed due to OutOfMemoryError;
• The crashed domainmanager service leads to "Validate Services" stage returns failed

Resolution

Workaround :

• Revert to the snapshot before upgrade
• Proceed the upgrade
• When the process is at “Refresh Custom Certificates”:
  
  
  
  

   

  • SSH to the SDDC Manager VM (as vcf, then switch to root)

     

  • Take a backup of the vcf-domain-manager.conf file

    cp -rf /etc/vmware/vcf/domainmanager/conf/vcf-domain-manager.conf /root/vcf-domain-manager.conf.BAK

     

  • Check the contents of the vcf-domain-manager.conf file for the heap size set:

    cat /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

    The current heap size set in the file should be 512 Mb, i.e. it should have the below line set:
    JAVA_OPTS=-Xmx512m

    We need to change this and set the heap size as 4 GB (below steps explain the same)

     

  • vi /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

     

  • Edit the file to set JAVA_OPTS flag as below:
    
    JAVA_OPTS=-Xmx4G
    Save and quit (wq!)

     

  • Restart domainmanager
    
    systemctl restart domainmanager

• Wait for the upgrade complete.