After upgrade, IDP based service accounts assigned Space Developer role on large number of spaces fail cf login

Products

• VMware Tanzu Platform Core
• VMware Tanzu Application Service

Issue/Introduction

• After upgrading to TAS 10.4.0 - 10.4.3, users may encounter cf login failures for IDP accounts that are assigned Space Developer role on a large number of spaces (~30 or more spaces).
• This has predominantly been observed on Service Accounts used for deployments from code repos like Github or Bamboo.
• This failure can be reproduced with a manual cf login command with the impacted user when targeting an ORG and SPACE in the login command.
• If users login without targeting an ORG or SPACE, the failure will not occur.
• Failures from concourse deployments will appear like:
  
  [FAILED] API endpoint: https://<CF_API_FQDN> |  |  | Authenticating... | OK |  | API endpoint:   https://<CF_API_FQDN> | API version:    3.217.0 | user:           <USER_ACCOUNT> | No org or space targeted, use 'cf8 target -o ORG -s SPACE' | FAILED
  
  
• Failures from direct cf login attempts targeting ORG And SPACE will appear like:

./cf8 login -a https://<CF_API_FQDN> -u <USER_ACCOUNT> -p $password -o <ORG_NAME> -s <SPACE_NAME>
API endpoint: https://<CF_API_FQDN>

  Authenticating...
OK

  API endpoint:   https://<CF_API_FQDN>
API version:    3.217.0
user:           <USER_ACCOUNT>
No org or space targeted, use 'cf8.exe target -o ORG -s SPACE'
Organization '<ORG_NAME>' not found.
FAILED

 

Environment

• Tanzu Application Service (TAS) 10.4.0 - 10.4.3
• MySQL v8.4

Cause

This defect was traced to MySQL bug #116741.

The problem is caused when a query uses in-memory temporary tables and those tables are converted to on-disk temporary tables due to memory pressure. For certain query patterns, this leads the MySQL executor to skip evaluation of a materialized subquery - in this case, effectively looking like organization role membership is empty.

Resolution

MySQL components are fixed in version 8.4.11. The below workaround is present in the 10.4.4 patch release of EAR and should resolve login failures. A fix containing the new MySQL 8.4.11 will be provided in a future release of the EAR/TAS tile. 10.2.x line of EAR uses MySQL v8.0, which defaults to using the "temptable_use_mmap = on" option and is therefore unaffected by this issue.

 

Workaround:

There are two workaround options: Ephemeral, and Persistent. Both workarounds involve increasing the memory and temporary disk limits for temptables in the database configuration.

 

Ephemeral Workaround This setting takes effect immediately but reverts after a MySQL restart. Perform this on the primary node:

1. 1. Connect to the MySQL instance.
   2. Run the following command to connect to the mysql DB (reference How to connect to the VMware Tanzu Application Service (TAS) for VMs internal MySQL database if needed):

mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf
      
      
   3. Use the CCDB:
      
      use ccdb;
      
      
   4. Run the following command:

SET GLOBAL temptable_use_mmap = ON, temptable_max_mmap = 1024*1024*1024 /*1GiB*/;

 

Persistent Workaround To persist this configuration across MySQL restarts, patch the tile metadata from Opsman VM:

1. 1. SSH into the Ops Manager VM: 
      
      ssh $OPSMAN_VM
      
      
   2. Identify the metadata file: 
      
      METADATA_FILE=$(sudo find /var/tempest/workspaces/default/metadata -name "*.yml" -exec grep -l "^name: cf" {} \; | grep -v product)
      
      
   3. Create a backup of the metadata file: 
      
      sudo cp -a $METADATA_FILE $METADATA_FILE.orig
      
      
   4. Patch the file with the necessary configuration:
      
      sudo sed -i -E 's/^([[:space:]])engine_config:[[:space:]]$/&\n\1 additional_raw_entries: { mysqld: { temptable_use_mmap: "ON", temptable_max_mmap: "1G" } }/' $METADATA_FILE
      
      
   5. Deploy the changes via the "Apply Changes" button in the Ops Manager UI. This performs a rolling restart to apply the configuration.

 

 

NOTE FOR PERSISTENT WORKAROUND FAILURES: The persistent workaround may fail on Step 3 (METADATA_FILE backup step) with errors like: 

cp: target '/var/tempest/workspaces/default/metadata/product-template20260630-1952-8s9nsn.yml.orig' is not a directory

 

This failure occurs because there are multiple deployment manifests for this tile in the /var/tempest/workspaces/default/metadata folder and the command expects a single file to be returned for the METADATA_FILE environment variable used in Step 2. Correct this with the following steps:

1. Identify these duplicates with the following command:
   
   sudo find /var/tempest/workspaces/default/metadata -name "*.yml" -exec grep -l "^name: cf" {} \; | grep -v product
   
   
2. View each file, specifically the product_version line, which should show the older version and the newer version. Select the newer version file (for example 10.4.2) by using the following command in Step 2 of the persistent workaround, then proceed with steps 3-5:
   
   sudo find /var/tempest/workspaces/default/metadata -name "*.yml" -exec grep -l "^name: cf" {} \;| grep -v product |sudo xargs  grep -l "product_version: 10.4.2"
   
   Example:
   
   METADATA_FILE=$(sudo find /var/tempest/workspaces/default/metadata -name "*.yml" -exec grep -l "^name: cf" {} \;| grep -v product |sudo xargs  grep -l "product_version: 10.4.2")

 

Additional Information

It is possible the 1GiB limit applied in the Workaround options may require a larger size. This isn't a per-query allowance, but a shared pool that every connection to the database draws from at the same time.

As more users hit the system at once, or as the data itself grows, that shared pool gets used up faster, so the same 1GB (now 2GB) padding doesn't go as far.

The current allocations can be checked via a database query under MySQL v8.4 to determine if this limit requires further increase:

mysql> SELECT
EVENT_NAME,
CURRENT_NUMBER_OF_BYTES_USED / 1024 / 1024 AS current_mb,
HIGH_NUMBER_OF_BYTES_USED / 1024 / 1024 AS peak_mb
FROM performance_schema.memory_summary_global_by_event_name
WHERE EVENT_NAME LIKE 'memory/temptable/%';

+--------------------------------+-------------+-------------+
| EVENT_NAME | current_mb | peak_mb |
+--------------------------------+-------------+-------------+
| memory/temptable/physical_disk | 0.00000000 | 0.00000000 |
| memory/temptable/physical_ram | 18.00054932 | 27.00082397 |
+--------------------------------+-------------+-------------+
2 rows in set (0.14 sec)

 

 

• This change persists until the next tile upgrade (e.g., to v10.4.3).
• For instructions on retrieving logs, refer to How to retrieve TAS logs.
• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.