Error: "Internal Server Error" occurs during invoke vmware.solution-addon-landing-zone tasks after upgrade

Products

VMware Cloud Director VMware Telco Cloud Platform

Issue/Introduction

After upgrading Cloud Director(VCD), multiple failed tasks repeat regarding the Solution Add-On Management Landing Zone with the following messages:

invoke : vmware.solution-addon-landing-zone...(urn:vcloud:entity:vmware:solutions_add_on_instance:#####-#####-#####-#####-#####)or

Created vmware.autoscale-default:xxxxx...(urn:vcloud:entity:vmware:solutions_add_on_task:#####-#####-#####-#####-#####)

or

Invoked vmware.autoscale-1.4.0-242...(urn:vcloud:entity:vmware:solutions_add_on_instance:#####-#####-#####-#####-#####)

Under task details you see an error similar to the following:
Internal Server Error - ErrorMessage = {"level":"error","msg":"Tls: failed to verify certificate: x509: certificate signed by unknown authority\nvcd-ext-crypto/crypto.GetHostCertificates\n\tvcd-ext-crypto/crypto/tls.go:27\nvcd-ext/vcd.NewClient\n\tvcd-ext/vcd/client.go:131\nvcd-ext/cmd.(*VcdClientOptions).createVcdClient\n\tvcd-ext/cmd/vcd.go:80\nvcd-ext/cmd.(*VcdClientOptions).CreateVcdClientAllowCellVersionForbidden\n\tvcd-ext/cmd/vcd.go:71\nvcd-ext/cmd/trust.(*Options).Run\n\tvcd-ext/cmd/trust/trust.go:54\nvcd-ext/cmd/trust.NewCommand.func1\n\tvcd-ext/cmd/trust/trust.go:21\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.5.0/command.go:872\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.5.0/command.go:918\nvcd-ext/cmd/cli.Run\n\tvcd-ext/cmd/cli/cli.go:60\nmain.main\n\t/opt/src/vcd-ext/main.go:13\n"} ,Log=(API Access) https://<VCD_Public_Address>/cloudapi/1.0.0/entities/urn:vcloud:entity:vmware:solutions_add_on_instance:#########-####-####-####-###########/behaviors/urn:vcloud:behavior-interface:invoke:vmware:solutions_add_on_instance:1.0.0/invocations/#########-####-####-####-###########/log”

In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on the Cloud Director Cells Internal Server Error entries are present with a stack trace similar to:

Caused by: java.lang.IllegalStateException: at com.vmware.vcloud.solutions.activities.AddOnRealizeActivity$ApplyOperation.buildResult(AddOnRealizeActivity.java:602) at com.vmware.vcloud.solutions.activities.AddOnRealizeActivity$ApplyOperation.invoke(AddOnRealizeActivity.java:627)

Environment

VMware Cloud Director 10.4.x, 10.5.x, 10.6.x

VMware Telco Cloud Platform 4.0, 4.0.1

Cause

This issue will occur if the VCD cells are unable to communicate successfully to their own public address. As a consequence the Solution Add-Ons are unable to correctly process API requests.

For a period post upgrade, Cloud Director will attempt to invoke changes in the backed. If there are any environmental issues such as misconfiguration of the Cloud Director Public Address, certificates, load balancer, networking or security restrictions then it can lead to these failed tasks appearing for a short time as those backend tasks are unable to complete successfully.

Resolution

These failed tasks in Cloud Director can be safely ignored and the upgrade itself can be considered successful. The tasks will automatically stop after a period of time, approximately one hour post upgrade typically.

It is recommended to review the Cloud Director Public Addresses, Certificates, networking and load balancer configuration to validate if all are configured correctly. Ensure that each Cloud Director cell is able to successfully connect to its own Cloud Director public address.

For More information, see:

Additional Information

NOTE: If you are using Add-ons in your VCD environment please review the Using Solution Add-Ons with VMware Cloud Director