Virtual network Interface inconsitencies iobserved in newly created virtual machines during new instance deployment

Products

• VMware Integrated OpenStack
• VMware Telco Cloud Platform

Issue/Introduction

• After a successful instance creation or scale out operation, the guest operating system does not list all the interfaces.
• Some of the interfaces do not have the associated static IP addresses or MAC address assigned inside the Guest OS.
• We can see all the virtual interfaces attached in the virtual machine configuration.
• ip link show in the Guest OS shows that ethernet adapters does not have the expected naming convention and is displayed as eth<mac id>. 

  eth############ (MAC: ##:##:##:##:##:##)

   

Environment

VIO 7.0U3

TCP 3.0 

Cause

• When OpenStack boots a VM, nova generates a file called network_data.json. This file contains the instructions to add the desired network configuration such as MAC addresses, Interface Names, IPs, and Routes to the network interfaces. 

• When the Linux Guest OS boots, a service inside the VM such as cloud-init or a vendor-specific VNF script fetches the configuration from the network_data.json  file and translates it into udev rules to name the interfaces and network scripts to apply the IPs.

• The Linux kernel detects the new hardware added during the instance creation, but the operating systems network scripts do not name the ethernet interfaces correctly.

• Hence, the linux udev device manager panics and appends the associated MAC addresses to the ethernet adapter as suffix.

 

Resolution

• To confirm that this is not a openstack issue verify the network.json file used by openstack to create the virtual machine has the correct information.
  
  
  1. Identify the config drive which is a virtual CD-ROM used by openStack to write the network_data.json during the VM boot using the below commands.
     
     
     • lsblk
     • sudo blkid | grep -i config-2
     • sudo find / - type f -name "network_data.json" 2>/dev/null
       
       
  2. Mount the Drive
     
     sudo mount -o ro <path obained from Step 1> /mnt
     
     
  3. View the network_data.json File
     
     cat /mnt/openstack/latest/network_data.json | grep "ethernet_mac_address"
     
     
  4. If the output lists all the ethernet interfaces or associated mac ids the failure is entirely on the Guest OS side, as the cloud-init service failed to mount and read the drive after the scale-out event.
  5. If the network.json file does not have the expected number if ethernet interfaces or mac addresses the file a ticket with Creating and managing Broadcom support cases
  6. Unmount the drive
     
     sudo umount /mnt