Refresh Custom Certificate - Failed to fetch inventory details . The system may not behave in an expected manner. Reload the SDDC Manager UI by clicking the Refresh icon on your browser. If the problem persists, contact VMWare support

Products

• VMware SDDC Manager

Issue/Introduction

SDDC Manager upgrade is failing with Refresh Custom certificate from 5.2.0 to 5.2.1.2

DomainManager and OperationsManager Services are in Error Status:

 

curl -X GET http://localhost/inventory/vcfservices | json_pp

   {
      "description" : "Domain Manager",
      "id" : "0afb7887-520c-4ede-af9f-267bd9888744",
      "name" : "DOMAIN_MANAGER",
      "serviceUrl" : "http://127.0.0.1/domainmanager",
      "status" : "ERROR",
      "version" : "5.2.0-24108578"
   },
   {
      "description" : "Operations Manager",
      "id" : "f95fa3cf-7c8c-48b2-8b04-f747fd5c1da5",
      "name" : "OPERATIONS_MANAGER",
      "serviceUrl" : "http://127.0.0.1/operationsmanager",
      "status" : "ERROR",
      "version" : "5.2.0-24108578"
   },
  {
      "description" : "Platform services",
      "id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "name" : "COMMON_SERVICES",
      "serviceUrl" : "http://localhost/commonsvcs",
      "status" : "ACTIVE",
      "version" : "5.2.0-24108578"
   },
  {
      "description" : "Sddc Manager UI App",
      "id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "name" : "SDDC_MANAGER_UI",
      "serviceUrl" : "http://127.0.0.1/sddc-manager-ui-app",
      "status" : "ACTIVE",
      "version" : "5.2.0-24108578"
   },

 

Upgrade is failing at "Refresh Custom Certificate":

SDDU UI is failing with "Failed to Fetch Inventory"

 

Permission-related errors can be found in the <service name>.out

/var/log/vmware/vcf/domainmanager/domainmanager.out: 

domainmanager.out: 
Caused by: java.io.FileNotFoundException: /etc/vmware/vcf/domainmanager/application.properties (Permission denied)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
        at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
        at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
        at org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:231)
        at org.springframework.boot.origin.OriginTrackedResource.getInputStream(OriginTrackedResource.java:61)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader$CharacterReader.<init>(OriginTrackedPropertiesLoader.java:205)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader.load(OriginTrackedPropertiesLoader.java:80)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader.load(OriginTrackedPropertiesLoader.java:66)
        at org.springframework.boot.env.PropertiesPropertySourceLoader.loadProperties(PropertiesPropertySourceLoader.java:70)
        at org.springframework.boot.env.PropertiesPropertySourceLoader.load(PropertiesPropertySourceLoader.java:49)
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:54)
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:36)
        at org.springframework.boot.context.config.ConfigDataLoaders.load(ConfigDataLoaders.java:96)
        at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:132)
        at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:87)
        ... 29 common frames omitted
(END)

 

/var/log/vmware/vcf/operationsmanager/operationsmanager.out

operationsmanager.out:
Caused by: java.io.FileNotFoundException: /etc/vmware/vcf/operationsmanager/application.properties (Permission denied)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
        at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
        at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
        at org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:231)
        at org.springframework.boot.origin.OriginTrackedResource.getInputStream(OriginTrackedResource.java:61)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader$CharacterReader.<init>(OriginTrackedPropertiesLoader.java:205)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader.load(OriginTrackedPropertiesLoader.java:80)
        at org.springframework.boot.env.OriginTrackedPropertiesLoader.load(OriginTrackedPropertiesLoader.java:66)
        at org.springframework.boot.env.PropertiesPropertySourceLoader.loadProperties(PropertiesPropertySourceLoader.java:70)
        at org.springframework.boot.env.PropertiesPropertySourceLoader.load(PropertiesPropertySourceLoader.java:49)
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:54)
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:36)
        at org.springframework.boot.context.config.ConfigDataLoaders.load(ConfigDataLoaders.java:96)
        at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:132)
        at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:87)
        ... 29 common frames omitted

 

/var/log/vmware/vcf/operationsmanager/operationsmanager.log: 

YYYY-MM-DDTHH:MM:SS.819+0000 DEBUG [vcf_om,69b79b92c04be4b9dfcfe8c109232de0,13cd] [c.v.v.p.v.u.ValidateCredentialsTranslationTaskExecutor,om-exec-1] Exception occurred during validate credentials translation task : Error creating bean with name 'liquibase': Singleton bean creation not allowed while singletons of
this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'liquibase': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

 

 

 

 

Environment

VMware Cloud Foundation 5.2.x
VMware Cloud Foundation 9.x

Cause

The ownership of /etc/vmware/vcf/domainmanager/application.properties file and /etc/vmware/vcf/operationsmanager/application.properties file is set to vcf_sos:vcf

Resolution

Refer to the following Knowledgebase article: https://knowledge.broadcom.com/external/article?articleNumber=385592