vCenter upgrade from 6.0 to 6.7 fails to start content-library with error "Keystore has not initialized yet"
search cancel

vCenter upgrade from 6.0 to 6.7 fails to start content-library with error "Keystore has not initialized yet"

book

Article ID: 345457

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • vCenter upgrade from 6.0 to 6.7 fails to start content-library with error  "Keystore has not initialized yet".
  • content-library-firstboot.py_18585_stderr.log

INFO:root:Found 1 matching service. ID is ########-####-####-####-########3900
INFO:setup_vdc:vcenter_ip type/value before decode() if necessary=<class 'str'>/<IP/FQDN>
INFO:cis_register:gen_random_pwd(16) type=<class 'str'>
ERROR starting content-library rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
 
Traceback (most recent call last):
  File "/usr/lib/vmware-content-library/firstboot/content-library-firstboot.py", line 221, in Main
    vdc_fb.start_service()
  File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 273, in start_service
    service_start(self.get_eff_service_name())
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 1006, in service_start
    raise ServiceStartException(svc_name)
cis.exceptions.ServiceStartException: {
    "resolution": null,
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "args": [
                "content-library"
            ],
            "localized": "An error occurred while starting service 'content-library'"
        }
    ]
}
VdcSvc firstboot failed. Return code is 1

  • cls.log

2019-03-29T15:49:33.941Z | ERROR    | opId-########-####-####-####-########9fa6 | cls-background-executor-4 | SolutionUserKeyStore           | entry is NULL, returning
2019-03-29T15:49:47.865Z | ERROR    | opId-########-####-####-####-########419d | cls-background-executor-4 | DefaultDomainAliasMapping      | Initialize domain map failed
com.vmware.vcde.common.services.exceptions.LookupException: Unable to get authenticated SSO client.
Caused by: com.vmware.vcde.common.services.exceptions.LookupException: IllegalStateException while looking up service com.vmware.cis:cs.identity
Caused by: java.lang.IllegalStateException: Keystore has not initialized yet
2019-03-29T15:49:48.871Z | DEBUG | unset-opId| main | VapiConfigBase | vapi config: cls-vapi-config-, threads: 8, queue: 1,024, keep live seconds: 300
2019-03-29T15:49:48.904Z | ERROR | opId-########-####-####-####-########6a4f | cls-background-executor-2 | SolutionUserKeyStore           | entry is NULL, returning


Environment

VMware vCenter Server Appliance 6.7.x

Resolution

VMware Engineering is investigating this issue.

Workaround:
  • Increase the timeout for content-library service after stage-1 .
  • During upgrade/migrate to VCSA 6.7 . After Stage 1 ;  change the  “StartTimeout” from 300 to 600 in vdcs.json .
/etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json
    "StartTimeout": 300,
    "StopTimeout": 90,
    "StartupType": "AUTOMATIC",
TO
/etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json
    "StartTimeout": 600,
    "StopTimeout": 90,
    "StartupType": "AUTOMATIC",