VCH Ops-User Failing to authenticate

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You see messages similar to the following when trying to stop/start a container in the VIC management interface:

Maximum number of retries exceeded. Failure: Connection refused:VCH_FQDN/VCH_IP_ADDRESS:2376;Reason: {"message":"Connection refused:refused:VCH_FQDN/VCH_IP_ADDRESS:2376; ","statusCode":500,"details": ["SHOULD_RETRY"],"documentKind":"com:vmware:xenon:common:ServiceErrorResponse","errorCode":0}

• VCH Admin Portal at VCH_FQDN:2373 Shows the message: "Port-layer service is not responding"
• You see messages similar to the following in the VCH Port-Layer.log file:

time="2020-01-15T06:27:13Z" level=info msg="{Formatter:0xc4204fa680 Level:info Syslog:<nil>}" Jan 15 2020 06:27:16.162Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:19Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:19Z" level=info msg="{Formatter:0xc4201b7d40 Level:info Syslog:<nil>}" Jan 15 2020 06:27:23.516Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:26Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:26Z" level=info msg="{Formatter:0xc4204fdac0 Level:info Syslog:<nil>}" Jan 15 2020 06:27:29.874Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:33Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:33Z" level=info msg="{Formatter:0xc4203bb860 Level:info Syslog:<nil>}" Jan 15 2020 06:27:38.267Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:41Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:41Z" level=info msg="{Formatter:0xc4202396f0 Level:info Syslog:<nil>}" Jan 15 2020 06:27:45.622Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:48Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:48Z" level=info msg="{Formatter:0xc420669d70 Level:info Syslog:<nil>}" Jan 15 2020 06:27:53.973Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:27:57Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:27:57Z" level=info msg="{Formatter:0xc4204d6b20 Level:info Syslog:<nil>}" Jan 15 2020 06:28:01.333Z FATAL configure_port_layer ERROR: Failed to log in to vCENTER_FQDN: ServerFaultCode: Cannot complete login due to an incorrect user name or password. time="2020-01-15T06:28:04Z" level=info msg="Launching portlayer server pprof server on 127.0.0.1:6063" time="2020-01-15T06:28:04Z" level=info msg="{Formatter:0xc4202ab440 Level:info Syslog:<nil>}"

Environment

VMware vSphere Integrated Containers 1.x

Cause

The Ops-User that the VCH was deployed with is unable to authenticate to the vCenter SSO Domain

Possible causes are:

1.  Ops-User that was deployed with the VCH password is expired or account disabled
2.  [email protected] password is expired 

If the vCenter Server credentials changed after the deployment of a VCH, you must update that VCH with the new credentials. The VCH will not function until you update the credentials.

Resolution

Provide the new vCenter Server credentials in the vic-machine configure --ops-user and --ops-password options. You use the vic-machine configure --ops-user and --ops-password options to update the credentials even if you did not specify the vic-machine create --ops-user and --ops-password options during the initial deployment of the VCH. If you did not specify vic-machine create --ops-user and --ops-password during the deployment of the VCH, by default the VCH uses the values from vic-machine create --user and --password for the --ops-user and --ops-password settings, and it uses these credentials for day-to-day, post-deployment operation.

To update the VCH credentials this example specifies the --user and --password options to log into vCenter Server, and then specifies --ops-user and --ops-password to update the password for the [email protected] account in the VCH.:

vic-machine-operating_system configure
  --target vcenter_server_address
  --user [email protected]
  --password password
  --thumbprint certificate_thumbprint
  --name vch_name
  --ops-user [email protected]
  --ops-password new_admin_password

Additional Information

https://github.com/vmware/vic-product/blob/master/docs/user_doc/vic_vsphere_admin/configure_vch.md#update-vcenter-server-credentials-