Redis Instance Failing After PCF AV Tile Removed

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

You may see the following symptoms in your environment:

• The PCF AV tile is removed from our production (prod) platform.
• Errand jobs are skipped during the deployment of this new configuration.
• Redis service instances failing and can't recreate them via BOSH. 
• Redis deployment still looking for references to antivirus, after the tile had been deleted.

You also may receive the following error:

Task 410400 | 17:12:58 | Error: Unable to render instance groups for deployment. Errors are:
  - Unable to render jobs for instance group 'redis-instance'. Errors are:
    - Unable to render templates for job 'antivirus'. Errors are:
      - Failed to find variable '/opsmgr/p-antivirus-GUID/antivirus_rsa_creds' with id '########-####-####-####-############' from config server: HTTP Code '404', Error: 'The request could not be completed because the credential does not exist or you do not have sufficient authorization.'
      - Failed to find variable '/opsmgr/p-antivirus-GUID/antivirus

In this example, if you run "bosh instances --ps", you see clamd and freshclam processes running on the VMs. These processes are deployed by the antivirus tile.

 

Resolution

Running the upgrade-all-service-instances errand for the Redis tile gets rid of the clamd and freshclam services on each VM and eliminates any references to the antivirus tile.

bosh -d <redis_deployment> run-errand update-all-service-instances

This errand successfully gets rid of the antivirus processes. 

You may see the Shared VM smoke test for Redis fail because it hits the defined limit on the number of SI's (75). If you experience this, you can bump this limit up, for example to 100. After that, Apply Changes and re-run the smoke tests; they should pass.