SDDC upgrade fails with error FAILED_TO_GET_CREDENTIAL_DATA_EXPORT

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

SDDC upgrade fails with one of the following errors,
 

FAILED_TO_GET_CREDENTIAL_DATA_EXPORT Failed to get credentials data using export api

Can't find resource for bundle java.util.PropertyResourceBundle, key FAILED_TO_GET_EXPORT.remedy

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/v1/system/credentials/export": Read timed out; nested exception is java.net.SocketTimeoutException

The following errors can be seen in the nginx and vcf-commonsvc logs.

/var/log/nginx/error.log

2022/04/05 20:11:53 [info] 861#0: *44093 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 127.0.0.1, server: localhost, request: "GET /credentials-service/export HTTP/1.1", upstream: "http://127.0.0.1:7100/credentials-service/export", host: "localhost"

 

/var/log/vmware/vcf/commonsvc/vcf-commonsvc.log

2022-02-28T19:48:29.254+0000 INFO  [common,1fe9ea3573025d3b,dbe5] [c.v.e.s.c.r.a.c.CssSupportController,http-nio-127.0.0.1-7100-exec-49] Received call to GET /export

2022-02-28T19:50:05.251+0000 WARN  [common,1fe9ea3573025d3b,dbe5] [o.s.w.s.m.s.DefaultHandlerExceptionResolver,http-nio-127.0.0.1-7100-exec-49] Failure while trying to resolve exception [org.springframework.http.converter.HttpMessageNotWritableException]
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:472)
        at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)

```2022-02-28T19:50:05.253+0000 ERROR [common,1fe9ea3573025d3b,dbe5] [o.s.c.s.i.web.ExceptionLoggingFilter,http-nio-127.0.0.1-7100-exec-49] Uncaught exception thrown
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe; nested exception is com.google.gson.JsonIOException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)

Environment

VMware Cloud Foundation 4.x

Cause

In larger vCF environments the GET CREDENTIAL EXPORT API hits a known timeout limitation.

The following command consistently takes longer than two minutes to finish which hits the timeout limit:

time curl http://localhost/credentials-service/export |json_pp > /tmp/credentials.txt

Resolution

Limitation has been documented and the fix is scheduled for vCF 4.4.1



Workaround:
Increase the vCPU on the SDDC VM to help speed up the credential export API process.