SDDC Manager UI stops responding during the upgrade process
Article ID: 316868
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
- SDDC manager UI stops responding during the upgrade process.
- You see errors similar to the following in the /home/vrack/vrm/logs/evosddc-debug.log file on the SDDC manager Controller VM:
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://127.0.0.1:8080//vrm-ui/api/1.0/vrm/logical-inventory/
domains": Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectExc
eption: Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:549)
at com.vmware.evo.sddc.lcm.rest.client.Client.getObject(Client.java:41)
at com.vmware.evo.sddc.lcm.adapter.inventory.impl.LogicalInventoryClient.getDomains(LogicalInventoryClient.java:92)
... 14 common frames omitted
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.springframework.http.client.HttpComponentsStreamingClientHttpRequest.executeInternal(HttpComponentsStreamingClientHttpRequest.java:98)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
2018-08-16 05:53:51.498 [pool-12-thread-8] DEBUG [com.vmware.evo.sddc.lcm.orch.Orchestrator] Starting orchestrating scheduled upgrades which are past due
2018-08-16 05:53:51.676 [pool-12-thread-9] DEBUG [com.vmware.evo.sddc.lcm.orch.Orchestrator] Starting orchestrating in progress upgrades
2018-08-16 05:53:52.341 [pool-12-thread-5] ERROR [com.vmware.evo.sddc.lcm.services.impl.InventoryUpgradeServiceImpl] No management domain found
2018-08-16 05:53:52.341 [pool-12-thread-5] ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.
domains": Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectExc
eption: Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:549)
at com.vmware.evo.sddc.lcm.rest.client.Client.getObject(Client.java:41)
at com.vmware.evo.sddc.lcm.adapter.inventory.impl.LogicalInventoryClient.getDomains(LogicalInventoryClient.java:92)
... 14 common frames omitted
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:8080 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.springframework.http.client.HttpComponentsStreamingClientHttpRequest.executeInternal(HttpComponentsStreamingClientHttpRequest.java:98)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
2018-08-16 05:53:51.498 [pool-12-thread-8] DEBUG [com.vmware.evo.sddc.lcm.orch.Orchestrator] Starting orchestrating scheduled upgrades which are past due
2018-08-16 05:53:51.676 [pool-12-thread-9] DEBUG [com.vmware.evo.sddc.lcm.orch.Orchestrator] Starting orchestrating in progress upgrades
2018-08-16 05:53:52.341 [pool-12-thread-5] ERROR [com.vmware.evo.sddc.lcm.services.impl.InventoryUpgradeServiceImpl] No management domain found
2018-08-16 05:53:52.341 [pool-12-thread-5] ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware Cloud Foundation 2.2.x
Cause
This issue occurs if there are any bundles being downloaded or validated and one or more of SDDC manager services suddenly stop.
Resolution
- ssh to the SDDC Manager Controller VM.
- Run the following commands to check if the vcfmanager service and scs service are started
systemctl status scs
systemctl status vcfmanager
systemctl status vcfmanager
- Run the following commands to start the above services is they are stopped
systemctl start scs
systemctl start vcfmanager
systemctl start vcfmanager
- Allow SDDC manager to populate the inventory for about 20 minutes after the services are started.
- Login to the SDDC Manager UI and resume the upgrade process
Additional Information
Feedback
Yes
No
Powered by
