vCenter upgrade from 6.0 to 6.7 fails to start content-library with error "Keystore has not initialized yet"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter upgrade from 6.0 to 6.7 fails to start content-library with error  "Keystore has not initialized yet".
• content-library-firstboot.py_18585_stderr.log

INFO:root:Found 1 matching service. ID is 3b43a444-daa3-4522-a8bb-2596df043900
INFO:setup_vdc:vcenter_ip type/value before decode() if necessary=<class 'str'>/<IP/FQDN>
INFO:cis_register:gen_random_pwd(16) type=<class 'str'>
ERROR starting content-library rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
 
Traceback (most recent call last):
  File "/usr/lib/vmware-content-library/firstboot/content-library-firstboot.py", line 221, in Main
    vdc_fb.start_service()
  File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 273, in start_service
    service_start(self.get_eff_service_name())
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 1006, in service_start
    raise ServiceStartException(svc_name)
cis.exceptions.ServiceStartException: {
    "resolution": null,
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "args": [
                "content-library"
            ],
            "localized": "An error occurred while starting service 'content-library'"
        }
    ]
}
VdcSvc firstboot failed. Return code is 1

• cls.log

2019-03-29T15:49:33.941Z | ERROR    | opId-a1cc5710-b098-47af-919b-3494ebf79fa6 | cls-background-executor-4 | SolutionUserKeyStore           | entry is NULL, returning
2019-03-29T15:49:47.865Z | ERROR    | opId-4881ff40-058c-4767-9035-d0026673419d | cls-background-executor-4 | DefaultDomainAliasMapping      | Initialize domain map failed
com.vmware.vcde.common.services.exceptions.LookupException: Unable to get authenticated SSO client.
Caused by: com.vmware.vcde.common.services.exceptions.LookupException: IllegalStateException while looking up service com.vmware.cis:cs.identity
Caused by: java.lang.IllegalStateException: Keystore has not initialized yet
2019-03-29T15:49:48.871Z | DEBUG | unset-opId| main | VapiConfigBase | vapi config: cls-vapi-config-, threads: 8, queue: 1,024, keep live seconds: 300
2019-03-29T15:49:48.904Z | ERROR | opId-801de22f-aa95-4c29-ab81-7ab236816a4f | cls-background-executor-2 | SolutionUserKeyStore           | entry is NULL, returning

Environment

VMware vCenter Server Appliance 6.7.x

Resolution

VMware Engineering is investigating this issue.

Workaround:

• Increase the timeout for content-library service after stage-1 .
• During upgrade/migrate to VCSA 6.7 . After Stage 1 ;  change the  “StartTimeout” from 300 to 600 in vdcs.json .

/etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json
    "StartTimeout": 300,
    "StopTimeout": 90,
    "StartupType": "AUTOMATIC",

TO

/etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json
    "StartTimeout": 600,
    "StopTimeout": 90,
    "StartupType": "AUTOMATIC",