Replacing a management switch:
- Login to the SDDC Manager virtual machine as the root user.
- Ensure that there is a backup of the management switch configuration under the /var/tmp folder.
- There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
- In each folder there should exist a folder named switch.
- In the switch folder, there should be a .tgz file for the management switch.
- Identify the most recent .tgz file for the management switch (192.168.3.254). If one does not exist and the management switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
- Use a file transfer utility to copy the .tgz file off of the SDDC Manager Controller virtual machine.
- Use the /home/vrack/bin/lookup-passwords command to find the current password for the cumulus user and the switch ID on the management switch. Output similar to the following will be returned:
MANAGEMENT: quanta lb9
identifiers: 192.168.3.254, R1S0
workload: hardware
username: cumulus
password: EvoSddc!2016
type: SSH
Note: In this example, the password is EvoSddc!2016 and the switch ID is R1S0. Make a note of both of these values.
- Unplug the current management switch . Make a note of all the current connections to the TOR/Spine switches and hosts in the rack.
Note: Several critical alerts will likely be generated in SDDC Manager at this point.
- Plugin the new Management switch and ensure that it is in Onie mode by issuing the cl-img-select -i and reboot commands.
- Image the new management switch with the VMware Imaging Appliance (VIA). Follow the steps in the VIA User's Guide under, Image New Management Switch.
- Once imaging of the management switch is complete, use a file transfer utility to copy the configuration backup noted in Step 2 to the /home/cumulus folder on the management switch.
- Log in to the management switch as the cumulus user with the password "EvoSddc!2016".
- Switch to the root user account by issuing the sudo su - command.
- Issue commands similar to the following to restore the configuration backup:
cd /
tar zxvf /home/cumulus/cumulus-R1S0-192.168.3.254.tgz
service networking restart
- Change the cumulus user's password from EvoSddc!2016 to the password noted in Step 3 with the passwd cumulus command.
Note: Complete Step 12 only if the Management, Datacenter, VSAN, VMotion, or VXLAN networks are using VLANs in the 3000-3999 range, or if the VMware Cloud Foundation version is 2.3 or higher. Otherwise, skip to step 13.
- On the management switch, as the root user, issue the following commands to update the reserved VLAN range:
sed -i.orig -e 's/#resv_vlan_range/resv_vlan_range/' -e 's/3300-3999/<START>-<END>/' /etc/cumulus/switchd.conf
Note: Replace the <START> with the beginning ID of the VLAN range to be reserved and <END> with the end ID of the VLAN range to be reserved
service switchd restart
service networking restart
- Log in to the SDDC Manager Controller virtual machine as the root user. Switch to the vrack user by issuing the su - vrack command.
Note: If the SDDC Manager Controller virtual machine is not accessible on the network, wait a few minutes and try again as the previous step may require additional time to complete background tasks.
- Issue the following command to populate the /home/vrack/.ssh/known_hosts file with the new ssh key from the management switch:
Note: Enter the cumulus user password when prompted.
Note: Type exit to log out of the management switch and return to the SDDC Manager Controller virtual machine.
- Rotate the management switch SSH key using the commands noted below (run as the vrack user):
curl -H "Content-Type: application/json" -X PUT --data "{\"keyGenAlgorithm\":\"RSA\", \"keyLength\":2048}" http://localhost:8080/hms-local/api/1.0/hms/napi/switches/<switch-id>/sshkeys/rotate
Note: Replace <switch-id> with the name value noted in the Step 3 (R1S0 for this example).
Note: The new ssh key value will be the output of the above command. Make a note of this new value.
- Open the .ssh/known_hosts file with a text editor.
- Locate the entry that begins with 192.168.3.254.
- Replace the ssh key value after ssh-rsa with the value returned in Step 15.
- Save and close the file.
- Verify the management switch configuration information via the following command:
curl -X GET http://localhost:8080/hms-local/api/1.0/hms/switches/<switch-id>
Note: Replace <switch-id> with the name value noted in the Step 3 (R1S0 for this example).
Note: Output similar to the following will be returned:
{"fruId":"1862331373","componentIdentifier":{"description":null,"manufacturer":"quanta","product":"lb9","partNumber":"1LB9BZZ0STR","manufacturingDate":"2014/12/23","serialNumber":"QTFCA65100034"},"location":null,"rackId":"93dfd957-b561-489c-ae63-103cf79c19ed","switchId":"R1S0","switchPorts":["eth0","swp1","swp2","swp3","swp4","swp5","swp6","swp7","swp8","swp9","swp10","swp11","swp12","swp13","swp14","swp15","swp16","swp17","swp18","swp19","swp20","swp21","swp22","swp23","swp24","swp25","swp26","swp27","swp28","swp29","swp30","swp31","swp32","swp33","swp34","swp35","swp36","swp37","swp38","swp39","swp40","swp41","swp42","swp43","swp44","swp45","swp46","swp47","swp48","swp49","swp50","swp51","swp52"],"ipAddress":"192.168.3.254","mangementPort":null,"managementMacAddress":"2c:60:0c:45:89:b6","operational_status":"true","osName":"Cumulus Linux","osVersion":"2.5.8","firmwareName":"ONIE","firmwareVersion":"2014.05.01-b23b0ab","adminStatus":null,"role":"MANAGEMENT","validationStatus":null,"powered":true,"discoverable":true}
- Log in to the SDDC Manager UI.
- Navigate to the Status page and click the View Details link in the Alerts pane.
- Click the Edit link at the top of the page.
- Select any alerts related to the management switch.
- Click the Clear Selected Alerts link at the top of the page.
- Navigate to the Dashboard page and then click the View Details link next to Physical Resources.
- Click the Rack and then click on the management switch. Verify that the switch is healthy and the configuration is accurate.
Replacing a Cisco Top of Rack (TOR) Switch:
- Login to the SDDC Manager virtual machine as the root user.
- Ensure that there is a backup of the TOR switch configuration under the /var/tmp folder.
- There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
- In each folder there should exist a folder named switch.
- In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
- Identify the most recent .gz file for the Cisco TOR switch (<switch-id>-<IP address>-cisco-running-config-<date>.gz). If one does not exist and the TOR switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
- Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the TOR switch. Output similar to the following will be returned:
TOR: Cisco Systems, Inc. N9K-C9372PX-E
identifiers: 192.168.0.112, R1S1
workload: hardware
username: admin
password: EvoSddc!2016
type: SSH
Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.112 and the switch ID is R1S1. Make a note of all of these values.
- Unplug the TOR switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.
Note: Several critical alerts will likely be generated in SDDC Manager at this point.
- Plugin the new TOR switch. Clear any existing configuration by issue the write erase and reload commands.
- Exit out of POAP (Power of Auto Provisioning mode). Follow the instruction on the switch console screen:
Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]:yes
- When prompted, set a password for the admin user. Please assign an easy to remember password and make a note of the password.
- Configure the IP address noted in Step 3 on the mgmt 0 interface, using a /21 subnet mask. Configure VRF to the mgmt 0 interface.
configure Terminal
interface mgmt 0
ip address 192.168.0.112/21
vrf member management
no shut
end
Note: Replace 192.168.0.112 with the appropriate IP address.
- Verify network connectivity on the new TOR switch with the ping <other TOR switch IP address> vrf management command.
- Copy the configuration backup noted in Step 2 to the TOR switch:
copy scp: bootflash: vrf management
Enter source filename: /var/tmp/<backup folder>/<name of TOR switch .gz file noted in Step 2>
Enter hostname for the scp server: 192.168.100.40
Enter username: root
[email protected] password: <SDDC Manager Controller VM root password>
Note: Output similar to the following will be displayed when the copy starts:
<name of TOR switch .gz file noted in Step 2> 100% 1891 1.9KB/s 00:00
Copy complete, now saving to disk (please wait)
-
Decompress and Install the restored configuration in the TOR switch's running configuration:
gunzip bootflash:/// <name of TOR switch .gz file noted in Step 2>
copy <extracted file from previous step> startup-config
copy startup-config running-config
-
Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.
-
Download the attached fru_switch_2.3.zip file. Extract the contents.
-
Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp/ folder on the SDDC Manager Controller virtual machine.
-
SSH to the SDDC Manager Controller virtual machine as the root user.
-
Execute the following command to run the fru_switch_2.3.py script:
python /tmp/fru_switch_2.3.py
Note: Enter the replacement TOR switch's IP address when prompted.
-
Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):
curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>
curl -X GET http://localhost:8080/hms-local/api/1.0/hms/switches/<switch-id>
Note: Replace <switch-id> with the name value noted in the Step 3 (R1S1 for this example).
Replacing an Arista Top of Rack (TOR) Switch:
- Login to the SDDC Manager virtual machine as the root user.
- Ensure that there is a backup of the TOR switch configuration under the /var/tmp folder.
- There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
- In each folder there should exist a folder named switch.
- In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
- Identify the most recent .gz file for the Arista TOR switch (<switch-id>-<IP address>-arista-running-config-<date>.gz). If one does not exist and the TOR switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
- Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the TOR switch. Output similar to the following will be returned:
TOR: Arista Networks DCS-7280SE-72-F
identifiers: 192.168.0.108, R1S1
workload: hardware
username: admin
password: EvoSddc!2016
type: SSH
Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.108 and the switch ID is R1S1. Make a note of all of these values.
- Unplug the TOR switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.
Note: Several critical alerts will likely be generated in SDDC Manager at this point.
- Plugin the new TOR switch. Clear any existing configuration by issue the write erase and reload commands.
- Exit out of POAP (Power of Auto Provisioning mode) by issuing the zerotouch cancel command.
- Configure the IP address noted in Step 3 on the management 1 interface, using a /22 subnet mask.
configure Terminal
interface management 1
ip address 192.168.0.108/22
Note: Replace 192.168.0.108 with the appropriate IP address.
- Verify network connectivity on the new TOR switch with the ping <other TOR switch IP address> command.
- Copy the configuration backup noted in Step 2 to the TOR switch:
bash
cd /mnt/flash
scp [email protected]:/var/tmp/<backup folder>/<name of TOR switch .gz file noted in Step 2> .
exit
Note: Enter the management switch cumulus user's password when prompted.
Note: Output similar to the following will be displayed when the copy starts:
<name of TOR switch .gz file noted in Step 2> 100% 1835 1.8KB/s 00:00
- Verify that the new file exists with the dir flash command.
- Decompress and Install the restored configuration in the TOR switch's running configuration:
bash
cd /mnt/flash
gunzip <name of TOR switch .gz file noted in Step 2>
exit
copy <extracted file from previous step> startup-config
copy startup-config running-config
Note: The password will now be updated to what was noted in Step 3.
-
Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.
-
Download the attached fru_switch_2.3.zip file. Extract the contents.
-
Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.
-
SSH to the SDDC Manager Controller virtual machine as the root user.
-
Execute the following command to run the fru_switch_2.3.py script:
python /tmp/fru_switch_2.3.py
Note: Enter the replacement TOR switch's IP address when prompted.
- Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):
curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>
Note: Replace <switch-id> with the name value noted in the Step 3 (R1S1 for this example).
Replacing a Cisco Rack-Interconnect Switch:
- Login to the SDDC Manager virtual machine as the root user.
- Ensure that there is a backup of the Rack-Interconnect switch configuration under the /var/tmp folder.
- There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
- In each folder there should exist a folder named switch.
- In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch.
- Identify the most recent .gz file for the Cisco Rack-Interconnect switch (<switch-id>-<IP address>-cisco-running-config-<date>.gz). If one does not exist and the Rack-Interconnect switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
- Copy the Cicso Rack-Interconnect Switch backup file noted in Step 2 to the Management Switch:
scp <backup filename> [email protected]:/tmpNote: Enter the Management Switch's
cumulus user password when prompted.
- Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the Rack-Interconnect switch. Output similar to the following will be returned:
SPINE: Cisco Systems, Inc. N9K-C9332PQ
identifiers: 192.168.0.30, R1S3
workload: hardware
username: admin
password: EvoSddc!2016
type: SSH
Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.30 and the switch ID is R1S3. Make a note of all of these values.
- Unplug the Rack-Interconnect switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.
Note: Several critical alerts will likely be generated in SDDC Manager at this point.
- Plugin the new Rack-Interconnect switch. Clear any existing configuration by issue the write erase and reload commands.
- Exit out of POAP (Power of Auto Provisioning mode). Follow the instruction on the switch console screen:
Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]:yes
- When prompted, set a password for the admin user. Please assign an easy to remember password and make a note of the password.
- Configure the IP address noted in Step 3 on the mgmt 0 interface, using a /21 subnet mask. Configure VRF to the mgmt 0 interface.
configure Terminal
interface mgmt 0
ip address 192.168.0.30/21
vrf member management
no shut
end
Note: Replace 192.168.0.30 with the appropriate IP address.
- Verify network connectivity on the new Rack-Interconnect switch with the ping <other Rack-Interconnect switch IP address> vrf management command.
- Copy the configuration backup on the Management Switch (noted in Step 3) to the Rack-Interconnect switch:
copy scp: bootflash: vrf management
Enter source filename: /tmp/<name of Rack-Interconnect switch .gz file noted in Step 3>
Enter hostname for the scp server: 192.168.3.254
Enter username: cumulus
[email protected] password: <Management Switch's cumulus user password>
Note: Output similar to the following will be displayed when the copy starts:
<name of Rack-Interconnect switch .gz file noted in Step 3> 100% 1891 1.9KB/s 00:00
Copy complete, now saving to disk (please wait)
-
Decompress and Install the restored configuration in the TOR switch's running configuration:
gunzip bootflash:/// <name of Rack-Interconnect switch .gz file noted in Step 3>
copy <extracted file from previous step> startup-config
copy startup-config running-config
-
Reboot the Rack-Interconnect switch with the reload command.
Note: Wait until a system ready message is displayed before proceeding.
- Verify the configuration on the Rack-Interconnect switch.
Note: The following is an example of from a dual rack installation.
- Spine ports 1,2,3,4 should be up, and have "duplex full" setting. Verify this with the show run interface e1/<1-4> commands. Output similar to the following should be returned
interface Ethernet1/1
description Rack-1 Link
switchport
switchport mode trunk
speed 40000
no negotiate auto
duplex full
mtu 9216
channel-group 1
no shutdown
interface Ethernet1/2
description Rack-1 Link
switchport
switchport mode trunk
speed 40000
no negotiate auto
mtu 9216
channel-group 1
no shutdown
interface Ethernet1/3
description Rack-2 Link
switchport
switchport mode trunk
speed 40000
no negotiate auto
mtu 9216
channel-group 2
no shutdown
interface Ethernet1/4
description Rack-2 Link
switchport
switchport mode trunk
speed 40000
no negotiate auto
duplex full
mtu 9216
channel-group 2
no shutdown
- If ports 3 and 4 are not in the channel-group 2 port channel group (per output above), the following commands can be used to place them in the correct port channel group:
int e1/3-4
channel-group 2 force
-
Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.
-
Download the attached fru_switch_2.3.zip file. Extract the contents.
-
Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.
-
SSH to the SDDC Manager Controller virtual machine as the root user.
-
Execute the following command to run the fru_switch_2.3.py script:
python /tmp/fru_switch_2.3.py
Note: Enter the replacement TOR switch's IP address when prompted.
- Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):
curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>
Note: Replace <switch-id> with the name value noted in the Step 4 (R1S3 for this example).
Replacing an Arista Rack-Interconnect Switch:
- Login to the SDDC Manager virtual machine as the root user.
- Ensure that there is a backup of the Rack-Interconnect switch configuration under the /var/tmp folder.
- There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
- In each folder there should exist a folder named switch.
- In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
- Identify the most recent .gz file for the Arista TOR switch (<switch-id>-<IP address>-arista-running-config-<date>.gz). If one does not exist and the Rack-Interconnect switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
- Copy the Arista Rack-Interconnect Switch backup file noted in Step 2 to the Management Switch:
scp <backup filename> [email protected]:/tmpNote: Enter the Management Switch's
cumulus user password when prompted.
- Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the Rack-Interconnect switch. Output similar to the following will be returned:
SPINE: Arista Networks DCS-7050QX
identifiers: 192.168.0.30, R1S3
workload: hardware
username: admin
password: EvoSddc!2016
type: SSH
Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.30 and the switch ID is R1S3. Make a note of all of these values.
- Unplug the Rack-Interconnect switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.
Note: Several critical alerts will likely be generated in SDDC Manager at this point.
- Plugin the new Rack-Interconnect switch. Clear any existing configuration by issue the write erase and reload commands.
- Exit out of POAP (Power of Auto Provisioning mode) by issuing the zerotouch cancel command.
- Configure the IP address noted in Step 4 on the management 1 interface, using a /22 subnet mask.
configure Terminal
interface management 1
ip address 192.168.0.30/22
Note: Replace 192.168.0.30 with the appropriate IP address.
- Verify network connectivity on the new Rack-Interconnect switch with the ping <other Rack-Interconnect switch IP address> command.
- Copy the configuration backup on the Management Switch (noted in Step 3) to the Rack-Interconnect switch:
bash
cd /mnt/flash
scp [email protected]:/tmp/<name of Rack-Interconnect switch .gz file noted in Step 3> .
Note: Enter the management switch cumulus user's password when prompted.
exit
Note: Output similar to the following will be displayed when the copy starts:
<name of Rack-Interconnect switch .gz file noted in Step 3> 100% 1835 1.8KB/s 00:00
- Verify that the new file exists with the dir flash command.
- Decompress and Install the restored configuration in the TOR switch's running configuration:
bash
cd /mnt/flash
gunzip <name of TOR switch .gz file noted in Step 3>
exit
copy <extracted file from previous step> startup-config
copy startup-config running-config
Note: The password will now be updated to what was noted in Step 4.
-
Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.
-
Download the attached fru_switch_2.3.zip file. Extract the contents.
-
Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.
-
SSH to the SDDC Manager Controller virtual machine as the root user.
-
Execute the following command to run the fru_switch_2.3.py script:
python /tmp/fru_switch_2.3.py
Note: Enter the replacement TOR switch's IP address when prompted.
- Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):
curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>
Note: Replace <switch-id> with the name value noted in the Step 4 (R1S3 for this example).