Installation of policy controller fails with error "Failed to initialize TUF client from : updating local metadata and targets: error updating to TUF remote mirror: tuf: invalid key"

Products

• VMware Tanzu Application Platform

Issue/Introduction

After TAP 1.3.0 is installed, TAP fails to start and TAP-GUI is inaccessible. By checking package installation status, it shows "Reconcile failed" for both policy-controller and tap packages.

$ tanzu package installed list -n tap-install | grep -i "error"
  policy-controller   policy.####.#####.#####.com  1.1.2  Reconcile failed: Error (see .status.usefulErrorMessage for details)
  tap                 tap.#####.######.com          1.3.0  Reconcile failed: Error (see .status.usefulErrorMessage for details)

Use command kubectl get pkgi policy-controller -n tap-install -o yaml to check further on  .status.usefulErrorMessage.

$ kubectl get pkgi policy-controller -n tap-install -o yaml
apiVersion: packaging.carvel.dev/v1alpha1
...
  usefulErrorMessage: |-
    kapp: Error: waiting on reconcile deployment/webhook (apps/v1) namespace: cosign-system:
      Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "webhook-7f9d967995" has timed out progressing.))
  version: 1.1.2

By looking into the webhook pod logs, error "Failed to initialize TUF client from : updating local metadata and targets: error updating to TUF remote mirror: tuf: invalid key" is reported. 

{"level":"info","ts":1667302809.1961625,"logger":"fallback","caller":"webhook/main.go:89","msg":"Initializing TUF root from => https://sigstore-tuf-root.storage.googleapis.com"} {"level":"panic","ts":1667302810.7514682,"logger":"fallback","caller":"webhook/main.go:91","msg":"Failed to initialize TUF client from : updating local metadata and targets: error updating to TUF remote mirror: tuf: invalid key\nremote status:{\n\t\"mirror\": \"https://sigstore-tuf-root.storage.googleapis.com\",\...

Environment

Product Version: 1.3

Resolution

The root cause
Policy controller tries to initialize TUF keys during installation. Due to a breaking change in go-tuf the initialization fails when using the Official Sigstore TUF root.

Solution
The policy controller’s dependency on go-tuf must be updated to the later version. Permanent fix version is yet released. To track the update on the permanent fix, check the release notes if needed.

Workarounds
Excluding Policy Controller during install or using a self-deployed Sigstore Stack.

1. Exclude policy controller package in all profile installations and reinstall TAP.

profile: <profile vaule>
excluded_packages:
- policy.apps.#####.######.com

2. Install Sigstore Stack and use the generated TUF system as the mirror and root of the Policy Controller. For more infomation, see Install Sigstore Stack .