How to Change the IP Address on AON Platform Nodes in a Cluster deployment

Products

• VCF Operations for Networks

Issue/Introduction

This article describes the procedure for changing cluster network configuration on AON/vRNI Platform. It also outlines the required steps for performing a planned Disaster Recovery (DR) activity using the cluster_network_change.json and cluster_network_change.py scripts attached to this KB.

File Checksums:

MD5
0215b4919ef941e148cb8ffbe1a30841 — cluster_network_change.json

ad0ab86d33c47a29b8d3486cc42e5941 — cluster_network_change.py

SHA256
fa866a0e9296194dbcd0555be68328eb6dcef1c9a13bfa4b31e498788f5b3ba0 — cluster_network_change.json

bd2131feda3bbb26dcf4285cf390a26f4da6e0c5b36bcb1694fb3d09da3431b7 — cluster_network_change.py

Environment

VCF Operations for Networks

Resolution

This procedure applies to planned DR activities.

• Unplanned DR: Skip Step 1.2 (Shutdown the cluster at Site1) and continue with the remaining steps.

• Note: Some services may become corrupted after recovery at the DR site. Additional troubleshooting or service restoration may be required.

Prerequisites:

• Access to all cluster nodes at both sites (support/ubuntu and console users as required).
  
  
• cluster_network_change.py and cluster_network_change.json available (e.g., in `/home/support` on a reference node).
  
  
• vrni-cluster-shutdown-script.sh available (see Broadcom KB 314428).
  
  
• Platform VM and collector nodes identified.

Phase 1: PROD Site1 (Shutdown)

1.1 Reduce NTP Retry Window on All Nodes (PROD Site1)

On P1 node at PROD Site1, reduce the NTP failure counter so the monitor script does not wait the 36 retries during DR. Run as a user with sudo:

sudo bash /home/ubuntu/run_all.sh "sudo sed -i '/try for 6min/,+1 s/MAX_FAILCOUNTER=36/MAX_FAILCOUNTER=3/' /home/ubuntu/build-target/infra-base/monitor_vrni_services.sh"

1.2 Shutdown the Cluster at Site1 

*Skip this step for unplanned DR (not applicable).*

Execute the cluster shutdown script as described in the KB 314428:

Example (run from the platform node; use `127.0.0.1` when running on the platform):

./vrni-cluster-shutdown-script.sh shutdown 127.0.0.1 "/home/ubuntu/vrni-cluster-shutdown-script$(date +%s).log"

Phase 2: Site2 (Network Change & Execution)

2.1 Change network settings (1 to N, console user)

At **Site2**, from the **console user**:

1. Run **change-network-settings** (or the equivalent N to 1 procedure for your deployment).
2. **Important:** Wait until the platform VM has **rebooted successfully** before running the next steps.
3. Verify using show-config for provided network configuration

2.2 Copy `cluster_network_change.py,cluster_network_change.json` to P1 (Platform Node 1) in /home/support

Execute `chmod 644 cluster_network_change.py;chmod 644 cluster_network_change.json` for P1 

*Note: Users are advised to edit the JSON file locally and then SCP it to the P1 support home. Editing the file on the P1 node via the web console can lead to issues with hidden special characters.*

2.3 Execute below commands on **platform1** at Site2 and select yes when prompted:

Replace <NTP_IPADDRESS1> and <NTP_IPADDRESS2> with the appropriate NTP server IP addresses for the environment.

sudo bash /home/ubuntu/run_all.sh "sudo python3 /home/support/cluster_network_change.py"
sudo bash /home/ubuntu/run_all.sh "sudo bash -c \"grep -vE '^server|^pool' /etc/ntpsec/ntp.conf > /etc/ntpsec/ntp.conf.tmp && echo -e 'server <NTP_IPADDRESS1>\nserver <NTP_IPADDRESS2>' >> /etc/ntpsec/ntp.conf.tmp && mv /etc/ntpsec/ntp.conf.tmp /etc/ntpsec/ntp.conf\""

2.3 Start cluster services at Site2

After apply is done on all nodes, start cluster services from ubuntu user. Run from the platform node (use `127.0.0.1` when on the platform):

./vrni-cluster-shutdown-script.sh start-services 127.0.0.1 "/home/ubuntu/vrni-cluster-shutdown-script$(date +%s).log"

2.4 Finalise IP change (Platform1, console user)

As **console user** on **platform1** at Site2, run:

finalize-IP-change

(or the exact command name from your product CLI).

Phase 3: Restore monitor script and collector pairing

3.1 Reduce NTP retry window on all nodes (PROD Site1)

On P1 node at Site2:

From ub user (ubuntu user)
sudo bash /home/ubuntu/run_all.sh "sudo sed -i '/try for 6min/,+1 s/MAX_FAILCOUNTER=3/MAX_FAILCOUNTER=36/' /home/ubuntu/build-target/infra-base/monitor_vrni_services.sh"

3.2 Verify connectivity and re-pair collectors to new platform IP

1. On **all collectors**, run the connectivity status command and look for **Platform_VM_IP/URL** to see which platform each collector is using:

   - Run: **show-connectivity-status**
   - Identify the platform IP/URL shown for each collector.

2. If the platform IP has changed at Site2, re-pair each collector to the new platform:

   - On **each collector**, run:

     pair-collector set-platform --ip-or-fqdn <platform-newIP>

   - Replace `<platform-newIP>` with the new platform VM IP or FQDN at Site2.

*Ensure you have backups and a rollback plan before running this procedure in production.*

Please note that this script applies only to the AON Platform node in a cluster deployment. It does not apply when changing the IP address on a collector node or on a standalone Platform node environment.