Defender job on diego_cell instance failed to start

Products

• VMware Tanzu Application Service

Issue/Introduction

When "Prisma Cloud for TAS" tile is installed on a TAS foundation, a BOSH runtime config called twist-#### will be deployed which is to add a defender job to diego_cell instances. For example,

$ bosh configs | grep twist
17*	runtime	twistlock-19144####c22f61c3d96-twistlock	-	2024-09-10 05:50:18 UTC

$ bosh config --type runtime --name twistlock-19144###c22f61c3d96-twistlock
Using environment '#.#.#.#' as client 'ops_manager'

ID          17
Type        runtime
Name        twistlock-19144###c22f61c3d96-twistlock
Created At  2024-09-10 05:50:18 UTC
Content     ---
            addons:
            - include:
                jobs:
                - name: garden
                  release: "-"
              jobs:
              - cpu: 1
              ......
                  tw_password: "((/opsmgr/twistlock-19144####c22f61c3d96/creds_selector/basic_auth_option/creds.password))"
                  tw_username: testuser
                release: release
                static_ip: 0
                templates:
                - name: defender
                  release: release
              name: twistlock
......

When a user tried to recreate diego_cell instance, it might fail with starting the injected defender job as show below.

L Error: 'diego_cell/009df4b5-####-####-####-dd3c41d9f9ca (60)' is not running after update. Review logs for failed jobs: defenderTask 4656399 | 02:08:33 | Error: 'diego_cell/009df4b5-####-####-####-dd3c41d9f9ca (60)' is not running after update. Review logs for failed jobs: defender

And the defender job logs indicated it never finished downloading install script from remote site with configured credential.

$ tail -n 3 defender.ctl.log
Downloading defender install script with username: testuser
Mon Sep  9 03:02:27 UTC 2024 Starting defender
Downloading defender install script with username: testuser

Cause

As shown by defender job logs, the failure was probably because the configured username/password on Prisma tile Setings page was no longer valid for accessing remote site. However the user might claim that the tile had been updated a new valid credential (copied from another working environment) followed by "Apply Changes" against Prisma tile only.

Since the credential was included in the BOSH runtime config with properties tw_username/tw_password, any update to this credential would only be applied to diego_cell instances when "Apply Changes" is run against tiles containing diego_cell (TAS/TAS [Windows]/Isolation Segment). 

Another observation was that the password directly copied from file "/var/vcap/jobs/defender/bin/ctl" in other working environment didn't match the correct value stored in Credhub (path: /opsmgr/twistlock-19144####c22f61c3d96/creds_selector/basic_auth_option/creds). For example,

the credential in file "/var/vcap/jobs/defender/bin/ctl" on a working diego_cell instance may look like 

diego_cell/03f16990-####-####-####-7f202c96f1ad:~# egrep "tw_username=|tw_password=" /var/vcap/jobs/defender/bin/ctl
		tw_username=testuser
		tw_password=test\\password\=

While the correct credential stored in Credhub was like

$ credhub get -n /opsmgr/twistlock-19144e####c22f61c3d96/creds_selector/basic_auth_option/creds
id: 27fa98f0-####-####-####-d1ca71ca65f5
name: /opsmgr/twistlock-19144####c22f61c3d96/creds_selector/basic_auth_option/creds
type: json
value:
  identity: testuser
  password: test\password=
version_created_at: "2024-09-10T07:25:50Z"

So the character '\' and '=' in the real password will be appended with a '\' when being pushed onto diego_cell instance. This is Prisma tile behaviour for accessing remote site. 

Resolution

1) When updating credentials on Prisma tile, "Apply Changes" should also be run against TAS, TAS[Windows] and Isolation Segment tiles.

2) Always retrieve the correct credential from Credhub instead of copying it from other working environment. Refer to the document about how to access BOSH Credhub with CLI.