Unable to create a DX O2 Tenant

Products

• DX Operational Observability

Issue/Introduction

When creating a tenant, I get the below error in ClusterManagement UI:

Failed tenant job .. Deployment jobs has failed:  Reason: The timeout period of 10000ms has been exceeded while executing POST /apis/apps/v1/namespaces/moc-dxi-lab/deployments for server null, - (INTERNAL_ERROR)

In the apmservices-manager-001 log we can see the below message

ERROR 1 --- [manager] [eduler_Worker-2] com.ca.apm.manager.jobs.AbstractJob      : Job [EM_UPDATE|#####-####-####-####-###########] Failed update. Reason: The timeout period of 10000ms has been exceeded while executing POST /apis/apps/v1/namespaces/moc-dxi-lab/deployments for server null

ERROR 1 --- [manager] [eduler_Worker-2] org.quartz.core.JobRunShell              : Job Enterprise Manager.EM_UPDATE|#####-####-####-####-########### threw an unhandled Exception:

io.fabric8.kubernetes.client.KubernetesClientException: The timeout period of 10000ms has been exceeded while executing POST /apis/apps/v1/namespaces/moc-dxi-lab/deployments for server null
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)
at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1155)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOr(BaseOperation.java:320)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOr(BaseOperation.java:98)
at com.ca.apm.manager.jobs.AbstractDeployJob.k8sCreateDeployment(AbstractDeployJob.java:618)
at com.ca.apm.manager.jobs.EmDeployJob.deploy(EmDeployJob.java:315)
at com.ca.apm.manager.jobs.EmUpdateJob.deploy(EmUpdateJob.java:203)
at com.ca.apm.manager.jobs.AbstractDeployJob.executeInternal(AbstractDeployJob.java:316)
at com.ca.apm.manager.jobs.AbstractJob.executeInternal(AbstractJob.java:70)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout period of 10000ms has been exceeded while executing POST /apis/apps/v1/namespaces/moc-dxi-lab/deployments for server null

Environment

DX O2 26.1

Cause

Delay / time out

Resolution

1) oc edit deployment apmservices-manager-001

2) Add below 2 new variables

- name: KUBERNETES_REQUEST_TIMEOUT
  value: "300000"
- name: KUBERNETES_CONNECTION_TIMEOUT
  value: "300000"

Below an example

 

3) Save the file a new pod will be created

4) Revalidate to update or create a new tenant.