Piplines Fails to Start with InvalidSecretException: Decryption error: Tag mismatch! When input contains secret:v1 Variables

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The Codestream pipeline remains in a status=STARTED, stage=STARTED state, failing to proceed with the vRO workflow execution.
• The codestream.log (typically found on the Code Stream appliance) contains error messages similar to the following:

• 2025-09-29T09:24:29.040Z ERROR codestream [host='codestream-app-76dbcf49-tg98n' thread='parallel-1' user='' org='' trace='' parent='' span=''] reactor.core.scheduler.Schedulers.error:324 - Scheduler worker in group main failed with an uncaught exception
          com.vmware.symphony.tango.encryption.InvalidSecretException: Decryption error: Tag mismatch!
                  at com.vmware.symphony.tango.encryption.SecretEncryptorImpl.decrypt(SecretEncryptorImpl.java:214) ~[tango-encryption-3.1.6.48.4.jar:3.1.6.48.4]
                  at com.vmware.symphony.tango.encryption.SecretEncryptorImpl.decryptValue(SecretEncryptorImpl.java:156) ~[tango-encryption-3.1.6.48.4.jar:3.1.6.48.4]
                  at com.vmware.symphony.tango.encryption.SecretEncryptor.decryptValue(SecretEncryptor.java:54) ~[tango-encryption-3.1.6.48.4.jar:3.1.6.48.4]
                  at com.vmware.codestream.service.impl.EncryptionServiceImpl.decrypt(EncryptionServiceImpl.java:129) ~[codestream.jar:na]
                  at com.vmware.codestream.service.impl.EncryptionServiceImpl.lambda$decrypt$1(EncryptionServiceImpl.java:115) ~[codestream.jar:na]
                  at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
  
          Caused by: javax.crypto.AEADBadTagException: Tag mismatch!

• The pipeline's input contains secure variables explicitly formatted like ((secret:v1:AAAAAAAAAAAAAAAA)).

Environment

Aria Automation 8.18.1

Aria Automation Pipelines

Cause

The problem arises from Code Stream's internal data parsing mechanism when it encounters secure variables in the secret:v1 format embedded within an input payload (such as a JSON object).

Engineering analysis has confirmed that this specific variable format within the payload can cause the Code Stream encryption service to fail during decryption, resulting in an InvalidSecretException: Decryption error: Tag mismatch!.

Resolution

As this issue is a known limitation within the current Code Stream architecture, and considering the deprecation of VMware Aria Automation products (as detailed in Broadcom KB Broadcom is announcing Deprecation of VMware Cloud Foundation (VCF) Automation Pipelines from VMware Cloud Foundation Automation 9), a code fix will not be provided.

A workaround is required to avoid embedding secret:v1 encrypted strings directly within the complex input payload.

Modify the Code Stream pipeline to remove the direct secret:v1 content from the input payload or create variable for the same secret and map in the pipeline input.