IPSEC Certificate Expiration Quick Recovery

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This Knowledge Base (KB) article details a method that quickly restores an OpsManager environment's communication that is impacted due to expired IPSEC certificates with policy_optional = false. 

The IPSEC addon includes a setting named policy_optional that determines if using IPSEC for communication between the specified IP CIDRs is optional. If policy_optional is set to false, then it is mandatory that communication be via IPSEC among the specified IP CIDRs in the IPSEC manifest. This can present a unique scenario while rotating the expired IPSEC certificates for Highly Available (HA) clusters such as Tanzu Application Service's deployment MySQL cluster. To workaround this, we can temporarily make IPSEC optional via BOSH CLI and then proceeding to rotating the certificates.

Environment

Product Version: 4.0

Resolution

The workaround includes:
1 - BOSH SSH into instance
2 - monit stop ipsec
3 - Update IPSEC file to make policy_optional to true in the IPSEC config file strongswan.conf
4 - Run IPSEC lifecycle scripts
5 - Start IPSEC once

This logic can be scripted.

Please reference the following command for an example of how to make IPSEC optional temporarily to reestablish communication between impacted VMs in a TAS deployment with deployment name "cf-1631c63accc0c5ca075c".

Please note to make any necessary adjustments to how BOSH CLI is used.
 

bosh -d cf-1631c63accc0c5ca075c ssh -c "  
sudo monit stop ipsec  
sudo sed -i 's/policy_optional = no/policy_optional = yes/g' /var/vcap/jobs/ipsec/etc/strongswan.conf  
sudo /var/vcap/jobs/ipsec/bin/post-stop  
sudo /var/vcap/jobs/ipsec/bin/pre-start  
sleep 10  
sudo monit start ipsec"

Once communication is reestablished - then rotation of the IPSEC certificates can take place.

If there are many deployments that impacted it may be scripted such as the following:
 

bosh deps --column=name > bosh-deps.txt
 
for i in `cat bosh-deps.txt`
do
bosh -d $i ssh -c "  
sudo monit stop ipsec  
sudo sed -i 's/policy_optional = no/policy_optional = yes/g' /var/vcap/jobs/ipsec/etc/strongswan.conf  
sudo /var/vcap/jobs/ipsec/bin/post-stop  
sudo /var/vcap/jobs/ipsec/bin/pre-start  
sleep 10  
sudo monit start ipsec"
done