Apply Changes fails on Credhub VM after upgrading TAS to v6.0.17 with JVM crash in 'libbc-fips-avx.so'
search cancel

Apply Changes fails on Credhub VM after upgrading TAS to v6.0.17 with JVM crash in 'libbc-fips-avx.so'

book

Article ID: 410519

calendar_today

Updated On:

Products

VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

You upgrade to TAS v6.0.17 and Apply Changes fails when starting Credhub VM. The affected VM logs show similar error as the following:

A fatal error has been detected by the Java Runtime Environment:
SIGILL (0x4) at pc=12345, pid=38, tid=39
JRE version: OpenJDK Runtime Environment (21.0.7+9) (build 21.0.7+9-LTS)
Java VM: OpenJDK 64-Bit Server VM (21.0.7+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
Problematic frame:
   C  [libbc-fips-avx.so+0x5024]  init_bytearray_ctx+0x4

 

Note: This issue may present itself in one or more Availability Zones. For the purposes of this KB article, we will assume this issue is only happening in one of three AZ's.

Environment

TAS / TPCF v6.0.17

Cause

The error indicates that the JVM attempted to run a AVX  (Avanced Vector Extension) CPU instructions that is not supported by the underlying infrastructure, for this scenario: vSphere. 

In the affected AZ:

  • Running command 'lscpu | grep -i avx' on the failing credhub vm will show no output. This confirms this AZ does not support AVX/AVX2 instructions. 
  • Running the same command on the other Credhub VM's that are located in other AZ's, returns AVX and AVX2 listed in the output confirming it is available/supported.

The discrepancy was traced back to the VM compatibility level:

  • Affected Credhub VM's are running on ESXi v6.5 hosts without exposing the needed AVX flags. 
  • Healthy Credhub VM's in other AZ's are running fine with AVX capabilities available. 

 

Resolution

    1. First, update the VM compatibility level and recreate the affected VM to ensure AVX instructions are available. See procedure below:

      1. In vSphere Client, locate the Credhub VM in the affected AZ.
      2. Right-Click the VM, select 'Compatibility > Upgrade VM Compatibility'.
      3. Choose "ESXi 7.0 and Later" as the compatibility level.
      4. Recreate the Credhub VM. (ie. bosh delete credhub vm, then perform Apply Changes on TAS tile/cf-deployment).
      5. Once the new VM is available, confirm AVX flags are set as desired by running command 'lscpu | grep -i avx'. You should see AVX and AVX2 listed. 
      6. *If you used a different method to recreate the Credhub VM in step 4 above (ie. bosh recreate <VM-NAME> --no-converge>, then Apply Changes to TAS tile will be necessary. Confirm with step 5 as final step.

      * If the above steps to do not resolve the issue, users can take the following actions to resolve the issue:

      1. Enable EVC Mode if there are hosts with varying cpu types and refer to the compatibility guides as credhub uses the BouncyCastle Fips library which will requires the CPU to support AVX or VAES cpu instructions.
      2. Apply Changes to TAS tile.

       

      Important Note:

      • After using the above steps, you may run into a related failure (with different error) on your next Apply Changes. Please see the details and steps to resolve described in this KB Article 411063. 

       

      *If this does not resolve the issue then it is suggested to open a ticket with TANZU Support. 

Additional Information

Powered by Wolken Software