Smarts: Is it possible to disable the RC4 cypher in TLS? If so, how is this done? How to check what cipher suite Smarts is using for TLS communication?

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


This KB explains whether TLS RC4 should be disabled or not.
Is it possible to disable the RC4 cypher in TLS? If so, how is this done? How to check what cipher suite Smarts is using for TLS communication?
 

Some customers cannot use RC4 cipher in TLS. They need to be able to disable it at Smarts Broker to Domain level.

Environment

VMware Smart Assurance - SMARTS

Resolution

Smarts uses the RSA BSAFE software for TLS communication which does support encryption using the RC4 algorithm. However, Smarts does not specify any particular algorithm, and lets the BSAFE software choose the most secure algorithm. By default, all TLS communication between Smarts servers and clients are done using AES encryption.

Disable the RC4 algorithm by specifying what "cipher suites" you actually want to support. This can be specified using the SM_TLS_SUITE_LIST environment variable. This accepts a colon separated list of cipher suites, and only those will be supported by Smarts.

For instance, setting the parameter as follows will force the usage of AES256-GCM-SHA384 cipher suite for all TLS communication
SM_TLS_SUITE_LIST=AES256-GCM-SHA384
 
To verify the actual cipher suite being used in a running Smarts domain, run the following dmctl command:

dmctl -s <DomainName> exec dmdebug --flows --output=flows.txt

The output file will be created in <InstallDir>/smarts/local/logs/ with details on all the connections (incoming & outgoing).

Each entry will have details on the cipher suite being used. For example:

------Begin Known Flows------
CI_FlowTCP_U [HTTP acceptor, port 8086 (v4)] PHYSICAL|ACCEPTOR @0x000000000e707020
*:v4:8086* KS N/A, KR N/A
Open fd=12, conn January 7, 2015 1:28:09 AM EST, disc N/A,
0.0.0.0:8086 -> {Unconnected}, tmo 0 00:00:15 N/S 1/0
 
CI_FlowBufferedHead_U [client 1 remotepid 6197 dmctl] HEAD|BUFFERED @0x000000000e7147d0
Read buffer, 0 bytes available of 2145
?3?6465627567000000 072D2D666C6F7773 ^|6B65722048545450 2F312E310D0A436F
Write buffer, 0 bytes written of 2048
?3?[^025923E940000000 07646D6465627567 0000000752554E4E 494E470000001945
->CI_FlowVirtual_U [client 1 remotepid 6197 dmctl] IN_FLOW|VFLOW @0x000000000e71c8d0
VFlowID:1 Consuming:0x000000000e74c7b0 Pos:41 Size:41 Pending:0 tmo 8412 20:45:28
Sent (Frames/Payload): 2/175   Recv (Frames/Payload): 2/648
->CI_FlowUniplexer_U [client 1 remotepid 6197 dmctl] IN_FLOW|MUX @0x000000000e727fb0
[Buffer] SZ:2052 P:0 L:0 KS January 7, 2015 1:28:39 AM EST +745ms, KR January 7, 2015 1:28:39 AM EST +745ms F:0
Sent (Frames/Payload): 2/175   Recv (Frames/Payload): 2/648
->CI_FlowTLS_U [client 1 remotepid 6197 dmctl] IN_FLOW|ENCRYPTOR|TLS @0x000000000e715780
TLS/1.2 AES256-GCM-SHA384 TLSv12 Kx=RSA Au=RSA Enc=AES-GCM(256) Mac=SHA384
Sent (Flow/BIO): 284/1450   Recv (Flow/BIO): 724/1279
->CI_FlowTCP_U [client 1 remotepid 6197 dmctl] IN_FLOW|PHYSICAL|LOCAL @0x000000000e720420
*:v4:8086 KS N/A, KR N/A
Open fd=13, conn January 7, 2015 1:28:39 AM EST, disc N/A,
127.0.0.1:8086 -> 127.0.0.1:55428, tmo 8412 20:45:28 N/S 2/3
------End Known Flows------
 
The line in bold indicates that the connection uses the AES256-GCM-SHA384 cipher suite :
TLS/1.2 AES256-GCM-SHA384 TLSv12 Kx=RSA Au=RSA Enc=AES-GCM(256) Mac=SHA384

This means

• RSA algorithm for Key-Exchange
• RSA algorithm for Signature
• AES-GCM(256) for encryption
• SHA384 for message digest.

 

Additional Information

TLS can be disabled by setting the following variable in the in runcmd_env.sh file (located at <InstallDir>/smarts/local/logs/) and restarting the domain:
SM_NO_TLS=TRUE

The communication will then be done using the legacy Smarts communication protocol which uses AES for encryption.  The security level in the absence of TLS is controlled by using environment variables SM_INCOMING_PROTOCOL and SM_OUTGOING_PROTOCOL.
There are four levels to choose from:

0 = No encryption, clear text.

1 = DH-AES Encryption.

2 = Encryption based on site secret. This level is the default value.

3 = DH-AES and site secret.

For more information regarding environment variables SM_INCOMING_PROTOCOL and SM_OUTGOING_PROTOCOL, please review VMware Smart Assurance Security Configuration Guide