Importing a vCenter in a VCF 9.1 environments, fails at "Perform Brownfield Environment Validations/ The Task was interrupted "

Products

• VCF Operations

Issue/Introduction

Importing a vCenter in a VCF 9.1 environments fails at "Perform Brownfield Environment Validations/ The Task was interrupted"



In the logs we'll see a Crash of domainmanager, where the date/ time coincides with the date/ time of the process stopping in the UI  

ls -althr /var/log/vmware/vcf/domainmanager/app-crash-logs
total <TBD>G
-rw------- 1 vcf_domainmanager vcf 1.5G Month day time java_pid#####1.hprof
-rw------- 1 vcf_domainmanager vcf 1.4G Month day time java_pid#####2.hprof
-rw------- 1 vcf_domainmanager vcf 1.4G Month day time java_pid#####3.hprof
-rw------- 1 vcf_domainmanager vcf 1.4G Month day time java_pid#####4.hprof
-rw------- 1 vcf_domainmanager vcf 1.5G Month day time java_pid#####5.hprof
-rw------- 1 vcf_domainmanager vcf 1.5G Month day time java_pid#####6.hprof

Furthermore, the crash is confirmed in 

└─$ cat domainmanager.out
\ \    / / ____|  ____|
 \ \  / / |    | |__
  \ \/ /| |    |  __|
   \  / | |____| |
    \/   \_____|_|
Domain Manager
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/log/vmware/vcf/domainmanager/app-crash-logs/java_pid#######6.hprof ...
Heap dump file created [############# bytes in 81.488 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space

Environment

9.1

Cause

Larger environments can cause the domainmanager process to crash, similar to https://knowledge.broadcom.com/external/article?articleNumber=431810 
However, the previous regular expressions are not found in the Logs - i.e. "OutOfMemoryError: Java heap space| OutOfMemory Error" 

Resolution

Same as https://knowledge.broadcom.com/external/article?articleNumber=431810  
To resolve this, increase the Java Heap Memory allocation for the DomainManager and for OperationsManager service.

1. Take snapshot of SDDC Manager VM
   
   
2. For the DomainManager 
   
   
3. SSH to the SDDC Manager appliance using the vcf user and elevate to root using su
   
   
4. Stop the domainmanager service
   # systemctl stop domainmanager

5. Edit the service configuration file located at /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf. Use a text editor like vi
   # vi /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf
   
   
6. Locate the line containing the maximum heap size setting (it will include -Xmx512m) and change it to -Xmx8G:
   Original: -Xmx1024m
New: -Xmx8G
Save the file and exit the text editor.
Esc > :wq!
   
   
7. Start the domainmanager service
   # systemctl start domainmanager
   
   
8. For the OperationsManager
   
   
9. Stop the operationsmanager service
   # systemctl stop operationsmanager
   
   
10. Edit the service configuration file located at /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf. Use a text editor like vi
    # vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf
    
    
11. Locate the line containing the maximum heap size setting (it will include -Xmx2G) and change it to -Xmx8G:
    Original: -Xmx2G
New: -Xmx8G
Save the file and exit the text editor.
Esc > :wq!

12. Start the operationsmanager service
    # systemctl start domainmanager
    
    
13. Retry the failed task.
    
    
14. NOTE: After the Import please revert back the heap size setting for both services back to the default values.