How to replace network switches in VMware Cloud Foundation 2.2 and greater
search cancel

How to replace network switches in VMware Cloud Foundation 2.2 and greater

book

Article ID: 336770

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

The instructions for replacing VMware Cloud Foundation 2.2 network switches included in Administering VMware Cloud Foundation - VMware Cloud Foundation 2.2 are incorrect. This article will provide the correct instructions.

Environment

VMware Cloud Foundation 2.2.x

Resolution

Replacing a management switch:

  1. Login to the SDDC Manager virtual machine as the root user.
  2. Ensure that there is a backup of the management switch configuration under the /var/tmp folder.
  • There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
  • In each folder there should exist a folder named switch.
  • In the switch folder, there should be a .tgz file for the management switch.
  • Identify the most recent .tgz file for the management switch (192.168.3.254). If one does not exist and the management switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
  • Use a file transfer utility to copy the .tgz file off of the SDDC Manager Controller virtual machine.
  1. Use the /home/vrack/bin/lookup-passwords command to find the current password for the cumulus user and the switch ID on the management switch. Output similar to the following will be returned:
MANAGEMENT: quanta lb9
        identifiers: 192.168.3.254, R1S0
        workload: hardware
            username: cumulus
            password: EvoSddc!2016
                type: SSH

Note: In this example, the password is EvoSddc!2016 and the switch ID is R1S0. Make a note of both of these values.
  1. Unplug the current management switch . Make a note of all the current connections to the TOR/Spine switches and hosts in the rack.                            

Note: Several critical alerts will likely be generated in SDDC Manager at this point.

  1. Plugin the new Management switch and ensure that it is in Onie mode by issuing the cl-img-select -i and reboot commands.
  2. Image the new management switch with the VMware Imaging Appliance (VIA). Follow the steps in the VIA User's Guide under, Image New Management Switch. 
  3. Once imaging of the management switch is complete, use a file transfer utility to copy the configuration backup noted in Step 2 to the /home/cumulus folder on the management switch.
  4. Log in to the management switch as the cumulus user with the password "EvoSddc!2016".
  5. Switch to the root user account by issuing the sudo su - command.
  6. Issue commands similar to the following to restore the configuration backup:
cd /
tar zxvf /home/cumulus/cumulus-R1S0-192.168.3.254.tgz
service networking restart
  1. Change the cumulus user's password from EvoSddc!2016 to the password noted in Step 3 with the passwd cumulus command.
Note: Complete Step 12 only if the Management, Datacenter, VSAN, VMotion, or VXLAN networks are using VLANs in the 3000-3999 range, or if the VMware Cloud Foundation version is 2.3 or higher. Otherwise, skip to step 13.
  1. On the management switch, as the root user, issue the following commands to update the reserved VLAN range:
sed -i.orig -e 's/#resv_vlan_range/resv_vlan_range/' -e 's/3300-3999/<START>-<END>/' /etc/cumulus/switchd.conf

Note: Replace the <START> with the beginning ID of the VLAN range to be reserved and <END> with the end ID of the VLAN range to be reserved 

service switchd restart
service networking restart
  1. Log in to the SDDC Manager Controller virtual machine as the root user. Switch to the vrack user by issuing the su - vrack command.
Note: If the SDDC Manager Controller virtual machine is not accessible on the network, wait a few minutes and try again as the previous step may require additional time to complete background tasks.
  1. ​Issue the following command to populate the /home/vrack/.ssh/known_hosts file with the new ssh key from the management switch:
ssh -oHostKeyAlgorithms='ssh-rsa' [email protected]

Note: Enter the cumulus user password when prompted.
Note: Type exit to log out of the management switch and return to the SDDC Manager Controller virtual machine.

  1. Rotate the management switch SSH key using the commands noted below (run as the vrack user):

curl -H "Content-Type: application/json" -X PUT --data "{\"keyGenAlgorithm\":\"RSA\", \"keyLength\":2048}" http://localhost:8080/hms-local/api/1.0/hms/napi/switches/<switch-id>/sshkeys/rotate

Note: Replace <switch-id> with the name value noted in the Step 3 (R1S0 for this example).
Note: The new ssh key value will be the output of the above command. Make a note of this new value.

  1. Open the .ssh/known_hosts file with a text editor.
  2. Locate the entry that begins with 192.168.3.254.
  3. Replace the ssh key value after ssh-rsa with the value returned in Step 15.
  4. Save and close the file.
  5. Verify the management switch configuration information via the following command:
curl -X GET http://localhost:8080/hms-local/api/1.0/hms/switches/<switch-id>

Note: Replace <switch-id> with the name value noted in the Step 3 (R1S0 for this example).
Note: Output similar to the following will be returned:

{"fruId":"1862331373","componentIdentifier":{"description":null,"manufacturer":"quanta","product":"lb9","partNumber":"1LB9BZZ0STR","manufacturingDate":"2014/12/23","serialNumber":"QTFCA65100034"},"location":null,"rackId":"93dfd957-b561-489c-ae63-103cf79c19ed","switchId":"R1S0","switchPorts":["eth0","swp1","swp2","swp3","swp4","swp5","swp6","swp7","swp8","swp9","swp10","swp11","swp12","swp13","swp14","swp15","swp16","swp17","swp18","swp19","swp20","swp21","swp22","swp23","swp24","swp25","swp26","swp27","swp28","swp29","swp30","swp31","swp32","swp33","swp34","swp35","swp36","swp37","swp38","swp39","swp40","swp41","swp42","swp43","swp44","swp45","swp46","swp47","swp48","swp49","swp50","swp51","swp52"],"ipAddress":"192.168.3.254","mangementPort":null,"managementMacAddress":"2c:60:0c:45:89:b6","operational_status":"true","osName":"Cumulus Linux","osVersion":"2.5.8","firmwareName":"ONIE","firmwareVersion":"2014.05.01-b23b0ab","adminStatus":null,"role":"MANAGEMENT","validationStatus":null,"powered":true,"discoverable":true}
  1. Log in to the SDDC Manager UI.
  2. Navigate to the Status page and click the View Details link in the Alerts pane.
  3. Click the Edit link at the top of the page.
  4. Select any alerts related to the management switch.
  5. Click the Clear Selected Alerts link at the top of the page.
  6. Navigate to the Dashboard page and then click the View Details link next to Physical Resources.
  7. Click the Rack and then click on the management switch. Verify that the switch is healthy and the configuration is accurate.

 

Replacing a Cisco Top of Rack (TOR) Switch:

  1. Login to the SDDC Manager virtual machine as the root user.
  2. Ensure that there is a backup of the TOR switch configuration under the /var/tmp folder.
  • There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
  • In each folder there should exist a folder named switch.
  • In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
  • Identify the most recent .gz file for the Cisco TOR switch (<switch-id>-<IP address>-cisco-running-config-<date>.gz). If one does not exist and the TOR switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
  1. Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the TOR switch. Output similar to the following will be returned:
TOR: Cisco Systems, Inc. N9K-C9372PX-E
        identifiers: 192.168.0.112, R1S1
        workload: hardware
            username: admin
            password: EvoSddc!2016
                type: SSH

Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.112 and the switch ID is R1S1. Make a note of all of these values.
  1. Unplug the TOR switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.                            

Note: Several critical alerts will likely be generated in SDDC Manager at this point.

  1. Plugin the new TOR switch. Clear any existing configuration by issue the write erase and reload commands.
  2. Exit out of POAP (Power of Auto Provisioning mode). Follow the instruction on the switch console screen:
Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]:yes 
  1. When prompted, set a password for the admin user. Please assign an easy to remember password and make a note of the password.
  2. Configure the IP address noted in Step 3 on the mgmt 0 interface, using a /21 subnet mask. Configure VRF to the mgmt 0 interface.
configure Terminal
interface mgmt 0
ip address 192.168.0.112/21
vrf member management
no shut 
end 

Note: Replace 192.168.0.112 with the appropriate IP address.
  1. Verify network connectivity on the new TOR switch with the ping <other TOR switch IP address> vrf management command.
  2. Copy the configuration backup noted in Step 2 to the TOR switch:
copy scp: bootflash: vrf management
  Enter source filename: /var/tmp/<backup folder>/<name of TOR switch .gz file noted in Step 2>
  Enter hostname for the scp server: 192.168.100.40
  Enter username: root
   [email protected] password: <SDDC Manager Controller VM root password>

Note: Output similar to the following will be displayed when the copy starts:
 
<name of TOR switch .gz file noted in Step 2>    100% 1891     1.9KB/s   00:00
   Copy complete, now saving to disk (please wait)

  1. Decompress and Install the restored configuration in the TOR switch's running configuration:

      gunzip bootflash:/// <name of TOR switch .gz file noted in Step 2>
      copy <extracted file from previous step> startup-config
      copy startup-config running-config

  2. Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.

    1. Download the attached fru_switch_2.3.zip file. Extract the contents.

    2. Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp/ folder on the SDDC Manager Controller virtual machine.

    3. SSH to the SDDC Manager Controller virtual machine as the root user.

    4. Execute the following command to run the fru_switch_2.3.py script:

      python /tmp/fru_switch_2.3.py

      Note: Enter the replacement TOR switch's IP address when prompted.
       

  3. Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):

    curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>
    curl -X GET http://localhost:8080/hms-local/api/1.0/hms/switches/<switch-id>

    Note: Replace <switch-id> with the name value noted in the Step 3 (R1S1 for this example).

 

Replacing an Arista Top of Rack (TOR) Switch:

  1. Login to the SDDC Manager virtual machine as the root user.
  2. Ensure that there is a backup of the TOR switch configuration under the /var/tmp folder.
  • There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
  • In each folder there should exist a folder named switch.
  • In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
  • Identify the most recent .gz file for the Arista TOR switch (<switch-id>-<IP address>-arista-running-config-<date>.gz). If one does not exist and the TOR switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
  1. Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the TOR switch. Output similar to the following will be returned:
        TOR: Arista Networks DCS-7280SE-72-F
        identifiers: 192.168.0.108, R1S1
        workload: hardware
            username: admin
            password: EvoSddc!2016
                type: SSH

Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.108 and the switch ID is R1S1. Make a note of all of these values.
  1. Unplug the TOR switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.                            

Note: Several critical alerts will likely be generated in SDDC Manager at this point.

  1. Plugin the new TOR switch. Clear any existing configuration by issue the write erase and reload commands.
  2. Exit out of POAP (Power of Auto Provisioning mode) by issuing the zerotouch cancel command.
  3. Configure the IP address noted in Step 3 on the management 1 interface, using a /22 subnet mask. 
configure Terminal
interface management 1
ip address 192.168.0.108/22

Note: Replace 192.168.0.108 with the appropriate IP address.
  1. Verify network connectivity on the new TOR switch with the ping <other TOR switch IP address> command.
  2. Copy the configuration backup noted in Step 2 to the TOR switch:
bash
  cd /mnt/flash
  scp [email protected]:/var/tmp/<backup folder>/<name of TOR switch .gz file noted in Step 2> .
  exit

Note: Enter the management switch cumulus user's password when prompted.
Note: Output similar to the following will be displayed when the copy starts:

<name of TOR switch .gz file noted in Step 2>    100% 1835     1.8KB/s   00:00
  1. Verify that the new file exists with the dir flash command.
  2. Decompress and Install the restored configuration in the TOR switch's running configuration:
bash
  cd /mnt/flash
  gunzip <name of TOR switch .gz file noted in Step 2>
  exit
copy <extracted file from previous step> startup-config
copy startup-config running-config

Note: The password will now be updated to what was noted in Step 3.

  1. Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.

    1. Download the attached fru_switch_2.3.zip file. Extract the contents.

    2. Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.

    3. SSH to the SDDC Manager Controller virtual machine as the root user.

    4. Execute the following command to run the fru_switch_2.3.py script:

      python /tmp/fru_switch_2.3.py

      Note: Enter the replacement TOR switch's IP address when prompted.

  2. Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):

curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>

Note: Replace <switch-id> with the name value noted in the Step 3 (R1S1 for this example).



Replacing a Cisco Rack-Interconnect Switch:

  1. Login to the SDDC Manager virtual machine as the root user.
  2. Ensure that there is a backup of the Rack-Interconnect switch configuration under the /var/tmp folder.
  • There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
  • In each folder there should exist a folder named switch.
  • In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch.
  • Identify the most recent .gz file for the Cisco Rack-Interconnect switch (<switch-id>-<IP address>-cisco-running-config-<date>.gz). If one does not exist and the Rack-Interconnect switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
  1. Copy the Cicso Rack-Interconnect Switch backup file noted in Step 2 to the Management Switch:
scp <backup filename> [email protected]:/tmp

Note: Enter the Management Switch's cumulus user password when prompted.
  1. Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the Rack-Interconnect switch. Output similar to the following will be returned:
SPINE: Cisco Systems, Inc. N9K-C9332PQ
        identifiers: 192.168.0.30, R1S3
        workload: hardware
            username: admin
            password: EvoSddc!2016
                type: SSH

Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.30 and the switch ID is R1S3. Make a note of all of these values.
  1. Unplug the Rack-Interconnect switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.                            

Note: Several critical alerts will likely be generated in SDDC Manager at this point.

  1. Plugin the new Rack-Interconnect switch. Clear any existing configuration by issue the write erase and reload commands.
  2. Exit out of POAP (Power of Auto Provisioning mode). Follow the instruction on the switch console screen:
Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]:yes 
  1. When prompted, set a password for the admin user. Please assign an easy to remember password and make a note of the password.
  2. Configure the IP address noted in Step 3 on the mgmt 0 interface, using a /21 subnet mask. Configure VRF to the mgmt 0 interface.
configure Terminal
interface mgmt 0
ip address 192.168.0.30/21
vrf member management
no shut 
end 

Note: Replace 192.168.0.30 with the appropriate IP address.
  1. Verify network connectivity on the new Rack-Interconnect switch with the ping <other Rack-Interconnect switch IP address> vrf management command.
  2. Copy the configuration backup on the Management Switch (noted in Step 3) to the Rack-Interconnect switch:
copy scp: bootflash: vrf management
  Enter source filename: /tmp/<name of Rack-Interconnect switch .gz file noted in Step 3>
  Enter hostname for the scp server: 192.168.3.254
  Enter username: cumulus
   [email protected] password: <Management Switch's cumulus user password>

Note: Output similar to the following will be displayed when the copy starts:
 
<name of Rack-Interconnect switch .gz file noted in Step 3>    100% 1891     1.9KB/s   00:00
   Copy complete, now saving to disk (please wait)

  1. Decompress and Install the restored configuration in the TOR switch's running configuration:

      gunzip bootflash:/// <name of Rack-Interconnect switch .gz file noted in Step 3>
      copy <extracted file from previous step> startup-config
      copy startup-config running-config

  2. Reboot the Rack-Interconnect switch with the reload command.

    Note: Wait until a system ready message is displayed before proceeding.

  3. Verify the configuration on the Rack-Interconnect switch.

Note: The following is an example of from a dual rack installation.

  1. Spine ports 1,2,3,4 should be up, and have "duplex full" setting. Verify this with the show run interface e1/<1-4> commands. Output similar to the following should be returned

interface Ethernet1/1

  description Rack-1 Link
  switchport
  switchport mode trunk
  speed 40000
  no negotiate auto
  duplex full
  mtu 9216
  channel-group 1
  no shutdown

interface Ethernet1/2

  description Rack-1 Link
  switchport
  switchport mode trunk
  speed 40000
  no negotiate auto
  mtu 9216
  channel-group 1
  no shutdown

interface Ethernet1/3

  description Rack-2 Link
  switchport
  switchport mode trunk
  speed 40000
  no negotiate auto
  mtu 9216
  channel-group 2
  no shutdown

interface Ethernet1/4

  description Rack-2 Link
  switchport
  switchport mode trunk
  speed 40000
  no negotiate auto
  duplex full
  mtu 9216
  channel-group 2
  no shutdown

  1. If ports 3 and 4 are not in the channel-group 2 port channel group (per output above), the following commands can be used to place them in the correct port channel group: 

int e1/3-4
channel-group 2 force

  1. Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.

    1. Download the attached fru_switch_2.3.zip file. Extract the contents.

    2. Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.

    3. SSH to the SDDC Manager Controller virtual machine as the root user.

    4. Execute the following command to run the fru_switch_2.3.py script:

      python /tmp/fru_switch_2.3.py

      Note: Enter the replacement TOR switch's IP address when prompted.

  2. Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):

curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>

Note: Replace <switch-id> with the name value noted in the Step 4 (R1S3 for this example).

 

Replacing an Arista Rack-Interconnect Switch:

  1. Login to the SDDC Manager virtual machine as the root user.
  2. Ensure that there is a backup of the Rack-Interconnect switch configuration under the /var/tmp folder.
  • There should be multiple folders present, with names similar to backup-<date>, backup-bringup-<date> and scheduled-backup-<date>.
  • In each folder there should exist a folder named switch.
  • In the switch folder, there should be a .gz file for each TOR and Rack-Interconnect switch
  • Identify the most recent .gz file for the Arista TOR switch (<switch-id>-<IP address>-arista-running-config-<date>.gz). If one does not exist and the Rack-Interconnect switch to be replaced is still accessible, run /opt/vmware/sddc-support/sos --backup to generate one.
  1. Copy the Arista Rack-Interconnect Switch backup file noted in Step 2 to the Management Switch:
scp <backup filename> [email protected]:/tmp

Note: Enter the Management Switch's cumulus user password when prompted.
  1. Use the /home/vrack/bin/lookup-passwords command to find the current password, IP Address and the switch ID on the Rack-Interconnect switch. Output similar to the following will be returned:
        SPINE: Arista Networks DCS-7050QX
        identifiers: 192.168.0.30, R1S3
        workload: hardware
            username: admin
            password: EvoSddc!2016
                type: SSH

Note: In this example, the password is EvoSddc!2016, the IP address is 192.168.0.30 and the switch ID is R1S3. Make a note of all of these values.
  1. Unplug the Rack-Interconnect switch . Make a note of all the current connections to the TOR/Rack-Interconnect/management switches and hosts in the rack.                            

Note: Several critical alerts will likely be generated in SDDC Manager at this point.

  1. Plugin the new Rack-Interconnect switch. Clear any existing configuration by issue the write erase and reload commands.
  2. Exit out of POAP (Power of Auto Provisioning mode) by issuing the zerotouch cancel command.
  3. Configure the IP address noted in Step 4 on the management 1 interface, using a /22 subnet mask. 
configure Terminal
interface management 1
ip address 192.168.0.30/22

Note: Replace 192.168.0.30 with the appropriate IP address.
  1. Verify network connectivity on the new Rack-Interconnect switch with the ping <other Rack-Interconnect switch IP address> command.
  2. Copy the configuration backup on the Management Switch (noted in Step 3) to the Rack-Interconnect switch:
bash
  cd /mnt/flash
  scp [email protected]:/tmp/<name of Rack-Interconnect switch .gz file noted in Step 3> .

  Note: Enter the management switch cumulus user's password when prompted.

  exit

Note: Output similar to the following will be displayed when the copy starts:

<name of Rack-Interconnect switch .gz file noted in Step 3>    100% 1835     1.8KB/s   00:00
  1. Verify that the new file exists with the dir flash command.
  2. Decompress and Install the restored configuration in the TOR switch's running configuration:
bash
  cd /mnt/flash
  gunzip <name of TOR switch .gz file noted in Step 3>
  exit
copy <extracted file from previous step> startup-config
copy startup-config running-config

Note: The password will now be updated to what was noted in Step 4.

  1. Regenerate the VMCA signed certificate for this IP Address and configure the switch with the certificate.

    1. Download the attached fru_switch_2.3.zip file. Extract the contents.

    2. Use a file transfer utility to copy the extracted fru_switch_2.3.py file to the /tmp folder on the SDDC Manager Controller virtual machine.

    3. SSH to the SDDC Manager Controller virtual machine as the root user.

    4. Execute the following command to run the fru_switch_2.3.py script:

      python /tmp/fru_switch_2.3.py

      Note: Enter the replacement TOR switch's IP address when prompted.

  2. Run the following commands from the SDDC Manager Controller VM to verify that a 200 OK response is returned from the Hardware Monitoring Service (HMS):

curl -X GET http://localhost:8448/api/1.0/hms/switches/<switch-id>

Note: Replace <switch-id> with the name value noted in the Step 4 (R1S3 for this example).


Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.

Attachments

fru_switch_2.3.zip get_app
Powered by Wolken Software