"Error communicating with TAP GUI backend. Contact your administrator." is shown on TAP-GUI with TAP multi-cluster

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Customer with TAP multi-cluster below v1.5 version may face below error in Runtime Resources of TAP-GUI, and the resource information cannot be seen.

Run the "tanzu package installed list -n tap-install" command on the View cluster, all packages in the "Reconcile succeeded" state.

Environment

Product Version: 1.3

Resolution

The error message indicates that customer may have a problem with the Kubernetes backend, either in their Build or Run cluster.

Backstage's Kubernetes backend plugin has an issue where if one of the clusters has a communication problem, the entire query will stop. This known issue is documented in the 1.3 release notes:
In a multicluster environment when one request to a Kubernetes cluster fails, backstage-kubernetes-backend reports a failure to the front end. This is a known issue with upstream Backstage and it applies to all released versions of Tanzu Application Platform GUI. For more information, see this Backstage code in GitHub. This behavior arises from the API at the Backstage level. There are currently no known workarounds. There are plans for upstream commits to Backstage to resolve this issue.

This issue has been fixed in TAP v1.5.0.

Current workaround is verifying the CLUSTER_URLs added to the tap_gui section of the tap-values yaml file on the View cluster as well as the tokens that correspond to those accounts, and checking that all configued k8s clusters are healthy.

    kubernetes:
      serviceLocatorMethod:
        type: 'multiTenant'
      clusterLocatorMethods:
        - type: 'config'
          clusters:
          # Build Cluster
            - url: CLUSTER_URL
              name: CLUSTER-NAME
              authProvider: serviceAccount
              serviceAccountToken: "CLUSTER_TOKEN"
              skipTLSVerify: true
              skipMetricsLookup: true
          # Run Cluster
            - url: CLUSTER_URL
              name: CLUSTER-NAME
              authProvider: serviceAccount
              serviceAccountToken: "CLUSTER_TOKEN"
              skipTLSVerify: true
              skipMetricsLookup: true

To verify the CLUSTER_URLs and serviceAccountToken, run below commands to discover the CLUSTER_URL and CLUSTER_TOKEN values on the Build and Run clusters.
If you’re watching a v1.23 or earlier Kubernetes cluster, run:

CLUSTER_URL=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

CLUSTER_TOKEN=$(kubectl -n tap-gui get secret $(kubectl -n tap-gui get sa tap-gui-viewer -o=json \
| jq -r '.secrets[0].name') -o=json \
| jq -r '.data["token"]' \
| base64 --decode)

echo CLUSTER_URL: $CLUSTER_URL
echo CLUSTER_TOKEN: $CLUSTER_TOKEN

If you’re watching a v1.24 or later Kubernetes cluster, run:

CLUSTER_URL=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: tap-gui-viewer
  namespace: tap-gui
  annotations:
    kubernetes.io/service-account.name: tap-gui-viewer
type: kubernetes.io/service-account-token
EOF

CLUSTER_TOKEN=$(kubectl -n tap-gui get secret tap-gui-viewer -o=json \
| jq -r '.data["token"]' \
| base64 --decode)

echo CLUSTER_URL: $CLUSTER_URL
echo CLUSTER_TOKEN: $CLUSTER_TOKEN