LCM service crashes immediately after start with error "Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}""

Products

• VMware Cloud Foundation

Issue/Introduction

To fix LCM service and start it

Symptoms:

• LCM service crashes immediately after start 
• /var/log/vmware/vcf/lcm/lcm-debug logs show 

2022-09-13T04:18:34.221+0000 WARN  [vcf_lcm,0000000000000000,0000] [o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.evo.sddc.lcm.orch.Orchestrator': Unsatisfied dependency expressed through field 'primitiveService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.evo.sddc.lcm.orch.PrimitiveService': Unsatisfied dependency expressed through field 'esxPrimitive'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esxVumPrimitiveImpl': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}"
2022-09-13T04:18:34.229+0000 INFO  [vcf_lcm,0000000000000000,0000] [o.s.o.j.LocalContainerEntityManagerFactoryBean,main] Closing JPA EntityManagerFactory for persistence unit 'default'
2022-09-13T04:18:34.230+0000 INFO  [vcf_lcm,0000000000000000,0000] [com.zaxxer.hikari.HikariDataSource,main] HikariPool-1 - Shutdown initiated...
2022-09-13T04:18:34.246+0000 INFO  [vcf_lcm,0000000000000000,0000] [com.zaxxer.hikari.HikariDataSource,main] HikariPool-1 - Shutdown completed.
2022-09-13T04:18:34.253+0000 INFO  [vcf_lcm,0000000000000000,0000] [o.a.catalina.core.StandardService,main] Stopping service [Tomcat]
2022-09-13T04:18:34.266+0000 INFO  [vcf_lcm,0000000000000000,0000] [o.s.b.a.l.ConditionEvaluationReportLoggingListener,main] 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-09-13T04:18:34.271+0000 ERROR [vcf_lcm,0000000000000000,0000] [o.s.boot.SpringApplication,main] Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.evo.sddc.lcm.orch.Orchestrator': Unsatisfied dependency expressed through field 'primitiveService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.evo.sddc.lcm.orch.PrimitiveService': Unsatisfied dependency expressed through field 'esxPrimitive'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esxVumPrimitiveImpl': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}"
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at com.vmware.evo.sddc.lcm.Application.main(Application.java:29)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.evo.sddc.lcm.orch.PrimitiveService': Unsatisfied dependency expressed through field 'esxPrimitive'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esxVumPrimitiveImpl': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}"
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
    ... 18 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esxVumPrimitiveImpl': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}"
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:405)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
    ... 31 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.esx.upgrade.custom.image.spec' in value "${lcm.esx.upgrade.custom.image.spec}"
    at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:178)
    at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:124)
    at org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:239)
    at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:210)
    at org.springframework.context.support.PropertySourcesPlaceholderConfigurer.lambda$processProperties$0(PropertySourcesPlaceholderConfigurer.java:175)
    at org.springframework.beans.factory.support.AbstractBeanFactory.resolveEmbeddedValue(AbstractBeanFactory.java:918)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1248)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
    ... 42 common frames omitted

Cause

'lcm.esx.upgrade.custom.image.spec' is missing in application-prod.properties file located in /opt/vmware/vcf/lcm/lcm-app/conf

Resolution

- Take snapshot of SDDC manager VM 
- Login to SDDC Manager using vcf user via SSH , switch user to root
- Navigate to /opt/vmware/vcf/lcm/lcm-app/conf
-Edit application-prod.properties file and add lcm.esx.upgrade.custom.image.spec=/nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json under ###########ESX custom image spec location#################################
-Save the file 
-Start lcm service 

systemctl start lcm
systemctl status lcm -l

Note : If esx-custom-image-upgrade-spec.json  is not present under  /nfs/vmware/vcf/nfs-mount/, create a blank file with name esx-custom-image-upgrade-spec.json as palceholder

Additional Information

Impact/Risks:
LCM service wont start , and user cannot perform lifecycle operations