How to Re-Apply IPsec Settings if IPsec is lost

Products

• Operations Manager

Issue/Introduction

IPsec settings are generally recommended to be applied during the time of initial install. These settings, however, may be lost in the rare event that BOSH run-time configuration is overwritten (see "Cause" section for details). If IPsec settings are lost from the VM's, then the VM's will not be able to communicate and start failing. The purpose of this article is to describe how to remedy this scenario of IPsec settings getting lost.

Settings are not present when you run the command:

bosh runtime-config

If IPsec settings are lost, then installation will run until the point where the router is updated, at which point it will get a timeout pinging the router VM:

Started updating job router > router/1 (1c0a4237-467b-44fb-81a8-e9c6401c4034). 
Failed: Timed out pinging to 1245ec6f-6837-4d1f-8ec1-5537b8a4a117 after 600 seconds (00:16:17)
Error 450002: Timed out pinging to 1245ec6f-6837-4d1f-8ec1-5537b8a4a117 after 600 seconds

 

Environment

Cause

Certain OSS procedures involve manually updating bosh runtime-config:

# bosh update runtime-config runtime-config.yml

This command will overwrite existing runtime config and cause IPsec settings to be lost. The configuration should *append* existing config with updates when running this command, otherwise, the existing settings such as IPsec will be removed from the VM's.

 

Resolution

When IPsec is version 1.6 or later, follow step 12 of the Pivotal Cloud Foundry IPsec Add-On document to resolve this issue.

Prerequisites: 

• Acquire the IPsec manifest: ipsec-addon.yml
• An administrator should have created ipsec-addon.yml when originally deploying the IPSec tile, as documented in the link above. See the Notes section below if you do not have this configuration file.

Steps:

1. Set the optional flag to true
2. Navigate to your "Installation Dashboard" in Ops Manager
3. Click "Apply Changes"
4. Wait for the installation to complete
5. Set the optional flag to false
6. Update the runtime config

$ bosh update runtime-config PATH/ipsec-addon.yml

1. After updating runtime config, verify that the settings are correct

$ bosh runtime-config

1. Navigate to your "Installation Dashboard"
2. Click "Apply Changes"

If IPsec is version 1.5 or lower:

There is no optional flag prior to 1.5, so we simply need to perform these steps:

1. In Ops Manager, revert changes to existing installation which failed
2. Update the runtime config

$ bosh update runtime-config PATH/ipsec-addon.yml

1. After updating runtime config, verify that the settings are correct

$ bosh runtime-config

1. Navigate to your "Installation Dashboard"
2. Click "Apply Changes

Once applying changes, the IPsec settings will be re-applied to any VM's missing this setting. Once VM's are updated, they will be able to communicate again and this will resolve the failing state issue of the VM's.

 

Additional Information

If IPsec settings are lost then you will see following in the installation.log

    addons:
- - name: ipsec-addon  
-   jobs:
-   - name: ipsec
-     release: ipsec
-   properties:
-     ipsec:
-       ipsec_subnets:
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       no_ipsec_subnets:
-       - ""
-       - ""
-       - ""
-       instance_certificate: ""
-       instance_private_key: ""
-       ca_certificates:
-       - ""
-       prestart_timeout: ""

All the information to recreate the runtime config can be found in the directory `/var/vcap/jobs/ipsec/etc` on any VM that did not get updated. This could be used if you do not have the original IPsec manifest to update runtime-config.