"[500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator. Back to login screen" while accessing vCenter UI
search cancel

"[500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator. Back to login screen" while accessing vCenter UI

book

Article ID: 316541

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

  • Unable to start vCenter service.Failed to start sca, vpxd, vpxd-svcs, vapi-endpoint services. Error: A system error occurred
  • Unable to access the vCenter UI and while starting all the services it fails to start. 
    service-control --start --all 

    Performing start operation on service vmdird...
    Successfully started service vmdird 
    Performing start operation on service vmcad... 
    Successfully started service vmcad 
    Performing start operation on service vmware-sts-idmd... 
    Successfully started service vmware-sts-idmd 
    Performing start operation on service vmware-stsd... 
    Successfully started service vmware-stsd 
    Performing start operation on service vmdnsd... 
    Successfully started service vmdnsd 
    Performing start operation on profile: ALL...
    Successfully started service vmware-vmon 
    Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start sca, vpxd, vpxd-svcs, vapi-endpoint services. Error: A system error occurred. Check logs for details

 

  • From /var/log/vmware/vapi/endpoint/endpoint.log, we see below mentioned error

 

    YYYY-MM-DDTHH:MM:SS.955Z | INFO  | state-manager1            | DefaultStateManager            | Invoking configuration-utilities
    YYYY-MM-DDTHH:MM:SS.234Z | INFO  | state-manager1            | DefaultStateManager            | Invoking http-server   
    YYYY-MM-DDTHH:MM:SS.235Z | INFO  | state-manager1            | BaseServerBuilder              | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
    YYYY-MM-DDTHH:MM:SS.345Z | INFO  | state-manager1            | DefaultJettyServer             | Logging initialized @13986ms to com.vmware.vapi.endpoint.http.DefaultJettyServer$JettyLogWrapper
    YYYY-MM-DDTHH:MM:SS.729Z | WARN  | state-manager1            | BaseServerBuilder              | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
     java.net.SocketException: Protocol family unavailable
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513)
        at java.net.Socket.bind(Socket.java:661)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.isPortAccessible(BaseServerBuilder.java:172)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.trimInvalidEndpoints(BaseServerBuilder.java:148)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.populateEndpointSettings(BaseServerBuilder.java:184)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.createServer(BaseServerBuilder.java:233)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.buildInitial(BaseServerBuilder.java:75)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    YYYY-MM-DDTHH:MM:SS.769Z | WARN  | state-manager1            | BaseServerBuilder              | Hostname ::1 was found to be invalid and removed from the configuration
    YYYY-MM-DDTHH:MM:SS.509Z | INFO  | state-manager1            | BaseServerBuilder              | Starting endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
    YYYY-MM-DDTHH:MM:SS.786Z | INFO  | state-manager1            | DefaultJettyServer             | Starting jetty server.
    YYYY-MM-DDTHH:MM:SS.261Z | INFO  | state-manager1            | BaseServerBuilder              | Started endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346.
    YYYY-MM-DDTHH:MM:SS.261Z | INFO  | state-manager1            | DefaultStateManager            | Invoking cis-sso-settings-builder
    YYYY-MM-DDTHH:MM:SS.433Z | INFO  | state-manager1            | CertificateUtil                | Creating anonymous SSO Admin Client for URI https://<IPaddress of VCenter>/sso-adminserver/sdk/vsphere.local
    YYYY-MM-DDTHH:MM:SS.641Z | INFO  | state-manager1            | DefaultStateManager            | Invoking sts-builder
    YYYY-MM-DDTHH:MM:SS.311Z | INFO  | shutdown-hook             | ApiEndpointServer              | Start shutting down...
    YYYY-MM-DDTHH:MM:SS.312Z | INFO  | shutdown-hook             | DefaultStateManager            | shutdown
    YYYY-MM-DDTHH:MM:SS.321Z | INFO  | shutdown-hook             | ApiEndpointServer              | Shutdown.
  • From var/log/vmware/vpxd-svcs/vpxd-svcs.log

         

YYYY-MM-YYTHH:MM:SS.PSQLException: ERROR: function replace (jsonb, unknown, unknown) does not exist

Hint: No function matches the given name and argument types. You might need to add explicit type casts.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2533)
at org.postgresql.core.v3.QueryExecutorImpl.processResults (QueryExecutorImpl.java:2268)
at org.postgresql.core.v3.QueryExecutorImpl.execute (QueryExecutorImpl.java:313)
at org.postgresq1.jdbc. PgStatement.executeInternal (PgStatement. java: 448)
at org.postgresql.jdbc. PgStatement. execute (PgStatement. java: 369)
at org.postgresql. jdbc. PgStatement.executeWithFlags (PgStatement. java: 310)
at org.postgresq1. jdbc. PgStatement. executeCachedSq1 (PgStatement. java:296)
at org.postgresql. jdbc. PgStatement. executeWithFlags (PgStatement. java: 273)
at org.postgresql.jdbc. PgStatement. executeUpdate (PgStatement. java:247)
at org. apache. commons.dbcp. DelegatingStatement. executeUpdate (DelegatingStatement. java: 228)
at org. apache. commons. dbcp. DelegatingStatement. executeUpdate (DelegatingStatement. java: 228)
at com. vmware.cis. core.kv. impl. Provider. VCDBProviderFactory. updateValueColToJSONB (VCDBProviderFactory. java: 521)
at com. vmware.cis.core. kv. impl. Provider. VCDBProviderFactory. initTable (VCDBProviderFactory. java:373)
at com. vmware. cis.core.kv. impl. Provider. VCDBProviderFactory. <init> (VCDBProviderFactory. java: 86)
at com. vmware.cis.core.kv.impl. VCDBKVManagerImpl.init (VCDBKVManagerImpl.java:72)
at sun. reflect.NativeMethodAccessorImpl. invoke0 (Native Method)
at sun. reflect.NativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl. java: 62)
at sun. reflect. DelegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl.java:43)
at java. lang. reflect.Method. invoke (Method. java: 498)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. invokeCustomInitMethod (AbstractAutowireCapableBeanFactory. java: 1925)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. invokeInitMethods (AbstractAutowireCapableBeanFactory. java: 1867)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. initializeBean (AbstractAutowireCapableBeanFactory. java: 1795)

YYYY-MM-YYTHH:MM:SS. [main ERROR com.vmware.cis.core.kv.impl.Provider.VCDBProviderFactory opId=] createKvValueIndex:
org.postgresql. util. PSQLException: ERROR: relation "gin kv value" already exists
at org.postgresql. core.v3.QueryExecutorImpl. receiveErrorResponse (QueryExecutorImpl. java: 2533)
at org.postgresql.core. v3. QueryExecutorImpl.processResults (QueryExecutorImpl. java: 2268)
at org.postgresql.core.v3.QueryExecutorImpl. execute (QueryExecutorImpl. java: 313)
at org.postgresql.jdbc. PgStatement. executeInternal (PgStatement. java: 448)
at org.postgresql.jdbc. PgStatement. execute (PgStatement. java: 369)
at org.postgresql.jdbc. PgStatement.executeWithFlags (PgStatement. java: 310)
at org.postgresql. jdbc. PgStatement. executeCachedSq1 (PgStatement. java:296)
at org.postgresql. jdbc. PgStatement. executeWithFlags (PgStatement. java: 273)
at org.postgresql. jdbc. PgStatement. executeUpdate (PgStatement. java: 247)
at org.apache. commons. dbcp. DelegatingStatement. executeUpdate (DelegatingStatement. java: 228)
at org. apache. commons. dbcp. DelegatingStatement.executeUpdate (DelegatingStatement. java: 228)
at com. vmware. cis. core. kv. impl. Provider. VCDBProviderFactory. createKvValueIndex (VCDBProviderFactory. java: 501)
at com. vmware. cis.core. kv. impl. Provider. VCDBProviderFactory. initTable (VCDBProviderFactory. java: 374)
at com. vmware. cis.core.kv. impl. Provider. VCDBProviderFactory. <init> (VCDBProviderFactory. java: 86)
at com. vmware. cis.core.kv. impl. VCDBKVManagerImpl. init (VCDBKVManagerImpl.java:72)
at sun.reflect.NativeMethodAccessorImpl. invoke0 (Native Method)
at sun. reflect.NativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl.java: 62)
at sun. reflect. DelegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl.java: 43)
at java. lang. reflect.Method. invoke (Method. java: 498)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. invokeCustomInitMethod (AbstractAutowireCapableBeanFactory. java: 1925)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. invokeInitMethods (AbstractAutowireCapableBeanFactory. java: 1867)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. initializeBean (AbstractAutowireCapableBeanFactory. java: 1795)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. doCreateBean (AbstractAutowireCapableBeanFactory. java: 594)
at org. springframework.beans. factory. support.AbstractAutowireCapableBeanFactory. createBean (AbstractAutowireCapableBeanFactory. java: 516)
at org. springframework.beans. factory. support.AbstractBeanFactory. lambda$doGetBean$0 (AbstractBeanFactory. java: 324)
at org. springframework.beans. factory. support. DefaultSingletonBeanRegistry.getSingleton (DefaultSingletonBeanRegistry. java: 234)
at org. springframework.beans. factory. support.AbstractBeanFactory. doGetBean (AbstractBeanFactory. java: 322)
at org. springframework.beans. factory. support.AbstractBeanFactory.getBean (AbstractBeanFactory. java: 202)
at org. springframework.beans. factory. support.AbstractBeanFactory. doGetBean (AbstractBeanFactory. java: 311)
at org. springframework.beans. factory. support.AbstractBeanFactory.getBean (AbstractBeanFactory.java: 202)

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

  • Protocol family unavailable error indicates that vCenter is unable to bind to specific IP or specific network family due to incorrect or missing entry in hosts file.

 

Resolution

Note: If the affected vCenter is standalone, take a valid snapshot of the vCenter Server. If the affected vCenter is in ELM, take a power-off snapshot of all the vCenters.

 

  1. SSH into the vCenter

    • Use root credentials to SSH into the affected vCenter server.

  2. Check the hosts File

    • Run the following command to view the current hosts file:

      cat /etc/hosts

    • Example output:

      # VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 localhost.localdomain localhost
127.0.0.1 photon-machine
# VAMI EDIT END

  3. Edit the Hosts File

    • If the hosts file resembles the example above, it needs modification.

    • Use the following command to edit the hosts file:

      vi /etc/hosts

    • Enter insert mode by pressing the i key.

    • Add the following line at the end of the file, replacing <IP address of the vCenter> with the affected vCenter server IP address:

      <IP address of the vCenter> localhost

    • After the modification, the file should look like this:

      # VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 localhost.localdomain localhost
127.0.0.1 photon-machine
<IP address of the vCenter> localhost
# VAMI EDIT END

  4. Save the Changes

    • Press the ESC key to exit insert mode.

    • Save and exit the file by typing:

      :wq!

  5. Restart vCenter Services

    • Stop and start all vCenter services using the following command:

      service-control --stop --all && service-control --start --all

 

Additional Information

In some scenarios, the issue can also be caused by incorrect entry in /etc/hosts file.

If the reason to keep the problematic entry cannot be specified or the entry is confirmed not in use,  the entry needs to be removed/commented out.

For example, below ipv6 entry can be causing issue because vapi-endpoint service is not able to bind to this ipv6 address to test validity.

# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 <vcenter name> localhost
::1 <vcenter name> localhost ipv6-localhost ipv6-loopback    <---------------------------
# VAMI EDIT END

 

Powered by Wolken Software