Cloud Gateway registration fails with an error: Image docker.vmc.vmware.com/agent-id could not be pulled

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

- You have configured a Proxy with a username & a password.
- VMware Cloud Gateway fails to pull images from docker.vmc.vmware.com when the proxy password has a special character.

Environment

VMware vSphere+ Cloud Services

Cause

This issue occurs when the proxy has any special character in the password.

tail -f /var/log/vmware/aap-installer/aap-installer.log:

2023-02-14T16:22:07.443Z ERROR c.v.a.s.ImageRepository [downloadWithRetry:77] [clientInboundChannel-84] - Image docker.vmc.vmware.com/agent-coordinator-agent@sha256:ca1fdecaea64bfd4e13cdfefa7c83bbc8890267925d446a4b91e668110019807 could not be pulled
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get \"https://docker.vmc.vmware.com/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
ess(Errors.java:229)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:456)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:357)
    at com.github.dockerjava.jaxrs.async.POSTCallbackNotifier.response(POSTCallbackNotifier.java:29)
    at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:50)
    at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:24)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
2023-02-14T16:22:07.443Z ERROR c.v.a.s.AgentBootstrapper [pullAgentImages:427] [clientInboundChannel-84] - Internal server error
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get \"https://docker.vmc.vmware.com/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}

 

Resolution

VMware Engineering is working on this issue to provide further updates. 

Workaround:

There are 2 possible workarounds:

1. Use a password without special characters.

2. In the interim, do not use authentication for proxy.