BBR backup failure after upgrading tiles using Jammy stemcell

Products

• Operations Manager

Issue/Introduction

Customer reported the BBR failure after upgrading TAS or other tiles which using Jammy  stemcells.

Checking the bbr log, error message as below -

failed to find instances for deployment cf-xxxxx: failed to check os: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: ssh: handshake failed: ssh: no common algorithm for host key; client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256 ecdsa-sha2-nistp256 ssh-ed25519]
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Environment

Product Version: 2.10

Resolution

This is due to customer updated Stemcell from Xenial to Jammy. Based on the BBR release note , bbr v1.9.37 support the Jammy stemcell. Please update BBR binary v1.9.37+ should resolve the issue.

Additional Information

BBR can also be affected by the jammy-compliance release (https://github.com/vmware/dod-compliance-and-automation/tree/master). We can see a similar error arise when attempting to take a BBR backup on a deployment that is using this release: 

/tmp/build/#########/pcf-backup # ../binary/bbr deployment --target "192.168.x.x"  --username bbr_client --password $BBR_CLIENT_PASSWORD  --deployment $PAS_DEPLOYMENT  --ca-cert ../bosh-ca-cert  pre-backup-check 

[bbr] 2024/10/15 20:48:38 INFO - Looking for scripts
[20:48:40] Deployment 'cf-############' cannot be backed up.
  1 error occurred:
  error 1:
  failed to find instances for deployment cf-#############: failed to check os: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: ssh: handshake failed: ssh: no common algorithm for key exchange; client offered: [curve25519-sha256 [email protected] ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group14-sha256 diffie-hellman-group14-sha1 ext-info-c [email protected]], server offered: [diffie-hellman-group-exchange-sha256 [email protected]]

If you are using the Jammy compliance release on your deployments and getting an error seen above despite upgrading to the latest version of BBR, the workaround below may allow one to get around this issue. 

With regard to the jammy-compliance release causing this issue, there appeared to be a portion of the pre-start.sh.erb script that interfered with the ability of BBR to take a backup of deployments using the jammy-compliance release. This has since been fixed here. 

 

1. SSH into the Ops Manager VM per these instructions

2. Git clone dod-compliance:

git clone https://github.com/vmware/dod-compliance-and-automation.git

3. Switch to dod-compliance-and-automation/tas/jammy-compliance-release directory: 

cd dod-compliance-and-automation/tas/jammy-compliance-release

4. Create a new release with a new version name for jammy compliance. As example, we will use the version name 0+dev.3, but you can use any name: 

bosh cr --version=0+dev.3

5. Upload the release:

bosh ur

6. Confirm new release is part of releases output: 

bosh releases | grep jammy-compliance

7. Edit the runtime-config-cis.yml file inside of dod-compliance-and-automation/tas/jammy-compliance-release and replace the version with the new release version that you set in step 3: 

addons:
- name: cis
  include:
    stemcell:
    - os: ubuntu-jammy
  jobs:
  - name: cis
    release: jammy-compliance
    properties:
      set_useradd_inactive: true
      set_password_max_days: true
      set_user_shell_timeout: true
      restirct_su: true
      make_audit_rules_immutable: true
releases:
- name: jammy-compliance
  version: 0+dev.3 <----------------------------------------------------- REPLACE THIS VERSION WITH VERSION WE SET IN STEP 3

 8. After making the edit to runtime-config-cis.yml, update runtime config:  

bosh urc --name cis-compliance runtime-config-cis.yml

 9. Apply changes on any tiles using the jammy-compliance release i.e TAS, compliance scanner, Healthwatch, AppMetrics, etc, and try the BBR backup once more.