" Update jobs has failed: Onboard dashboards" (INTERNAL_ERROR) after upgrading to 25.1

Products

• DX Operational Observability

Issue/Introduction

During upgrade to 25.1 we enabled " Logs for Triage " as below:

The upgrade completed as expected but in "Cluster Manager" we see the reporting the error:

"Failed tenant update job (id" <your_tenant_name> . Reason: 500,1000,-: Update jobs has failed: Onboard dashboards (<your_tenant_name>), - (INTERNAL_ERROR)" 

How to fix the ERROR?

Environment

• DX O2 25.1 OnPremise

Cause

The below exception is found in the apmservices-manager pod:

....
2025-07-08T11:52:10.445Z ERROR 1 --- [manager] [quartzScheduler_Worker-3] com.ca.apm.manager.jobs.AbstractJob      : Job [TENANT_UPDATE| Failed tenant update job (id . Reason: 500,1000,-: Update jobs has failed: Onboard dashboards (), - (INTERNAL_ERROR)

 

2025-07-08T11:52:10.458Z  INFO 1 --- [manager] [quartzScheduler_Worker-3] org.quartz.core.JobRunShell              : Job .TENANT_UPDATE| threw a JobExecutionException: 

 

org.quartz.JobExecutionException: com.ca.apm.common.api.ServicesException: 500,1000,-: Update jobs has failed: Onboard dashboards (), -

at com.ca.apm.manager.jobs.TenantUpdateJob.executeInternal(TenantUpdateJob.java:374)

at com.ca.apm.manager.jobs.AbstractJob.executeInternal(AbstractJob.java:70)

at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Caused by: com.ca.apm.common.api.ServicesException: 500,1000,-: Update jobs has failed: Onboard dashboards (OSS_Enterprise), -

at com.ca.apm.manager.jobs.TenantUpdateJob.executeInternal(TenantUpdateJob.java:348)

... 4 common frames omitted

Resolution

1) PATH the tenant configuration as below:

curl --insecure --location --request PATCH 'https://apmservices-gateway.<wildcardDNS>/tenants/tenant/<dsTenantId>' \
  -H 'accept: application/json' \
  -H 'content-type: application/json' \
  -H 'Cache-Control: no-cache' \
--header 'Authorization: Bearer <token>' \
--data-raw '{"attributes":{"apm.onboarded.dashboards": true}}'

NOTE:

- apmservices-gateway.<wildcardDNS> can be obtain from connector parameters

- dsTenantId can be obtain from cluster manager by moving the mouse over the tenant name

- <token> can be obtain as below, for example in openshift:

oc rsh <apmservices-manager pod>

cat /apmservices.sec/bootstrap.properties

"apm.security.tenantServiceMasterToken" property value

 

Example:

 

2) From Cluster Manager, click configure the Tenant and click "Save"

Tenant update will complete as expected and no error will be reported

For assistance contact Broadcom Support

Additional Information

Techdocs DX 25.1 Upgrade