How to hide the speculative-execution control mechanism for virtual machines in a VMware Cloud Foundation environment

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides instructions for hiding the speculative-execution control mechanism for virtual machines in a VMware Cloud Foundation environment.

Resolution

Note: Using the following steps will employ the resolution noted in Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004.

1. Download and extract the appropriate attached file: 

​52381_213-sos-esx-microcode-patch.zip for VMware Cloud Foundation 2.1.3 environments
52381_22_23-sos-esx-microcode-patch.zip for VMware Cloud Foundation 2.2 or 2.3 environments.

2. Copy the extracted sos-fix folder to the /tmp folder on the SDDC Manager Controller virtual machine.
3. SSH to the SDDC Manager Controller virtual machine (or the VRM virtual machine if a 2.1.3 environment) as the root user.
4. Issue the following commands to patch the sos executable:

cd /tmp/sos-fix
sh patch.sh

Note: You see output similar to:

SOS Version :  2.3.0-7506102
Backup all existing files which we are going to update under /opt/vmware/sddc-support
Patch /opt/vmware/sddc-support with new binaries
Patch completed successfully.
Execute /opt/vmware/sddc-support/sos --help to see Intel Sightings remediate options

5. Issue the following command to query the management workload domain:

/opt/vmware/sddc-support/sos --verify-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.2 or 2.3

/opt/vmware/evosddc-support/sos --verify-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.1.3

Note: You see output similar to:

Welcome to Supportability and Serviceability(SoS) utility!
NOTE : SoS may fail if workflow is undergoing password rotation operation, run SoS when no workflows are in flight.

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-28-03-2222
Verify ESXi Microcode update has been applied as stated per https://kb.vmware.com/s/article/52085
ESXi Status :
-------------

+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
| SL# | NODE |              CPU               | Microcode  | IBRPresent | IBPBPresent | STIBPresent | HypervisorAssistedGuestAffected | IntelSighting |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
|  1  | R1N8 | Intel(R) Xeon(R) Gold 6130 CPU | 0x0200003a |    True    |     True    |     True    |              False              |      True     |
|     |      |           @ 2.10GHz            |            |            |             |             |                                 |               |
|  2  | R1N6 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  3  | R1N5 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  4  | R1N3 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+

Virtual Machine Status :
------------------------

+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+
| SL# |         NODE         |                       VM Name                       | IBRPresent | IBPBPresent | STIBPresent |  vHW   | HypervisorAssistedGuestAffected |
+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+
|  1  | r1n8.vcf.vmware.corp |                   vRealize-Edge-0                   |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      |                    vrops-replica                    |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                  vrops-data-node-1                  |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                        psc-2                        |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      | NSX_Controller_ba6de5ba-444e-4210-82a1-6ffef2a58501 |    True    |     True    |     True    | vmx-10 |              False              |
|  2  | r1n6.vcf.vmware.corp |                   loginsight-vm-1                   |    True    |     True    |     True    | vmx-09 |              False              |
|     |                      |                        psc-1                        |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      | NSX_Controller_d7fcbc0d-01db-41db-8fae-afb4eb3f7559 |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      |               SDDC Manager Controller               |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      |                   vRealize-Edge-1                   |    True    |     True    |     True    | vmx-10 |              False              |
|  3  | r1n5.vcf.vmware.corp |                   loginsight-vm-2                   |    True    |     True    |     True    | vmx-09 |              False              |
|     |                      |                 SDDC Manager Utility                |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      | NSX_Controller_d4508e19-33d5-409c-80ac-0063f520c173 |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      |                        vrslcm                       |    True    |     True    |     True    | vmx-10 |              False              |
|     |                      |                      vcenter-1                      |    True    |     True    |     True    | vmx-10 |              False              |
|  4  | r1n3.vcf.vmware.corp |                   loginsight-vm-3                   |    True    |     True    |     True    | vmx-09 |              False              |
|     |                      |                    nsx-manager-1                    |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                     vrops-master                    |   False    |    False    |    False    | vmx-08 |               N/A               |
+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+

The value of IntelSighting can contain four potential values:
--------------------------------------------------------------

   *      True = ESXi host contains microcode update is affected by Intel Sighting, you
          will need to apply the workaround as outlined in
          https://kb.vmware.com/s/article/52345
   *      False = CPU not affected by Intel Sighting it is currently recommended to only
          apply one of the ESXi patches (until Intel provides a microcode update fix),
          please refer to https://kb.vmware.com/s/article/52345 for full details
   *      AffectedOncePatched = CPU is affected by Intel Sighting, but does not , need the
          work around unless it is patched or has a BIOS update
   *      N/A = CPU is not Intel

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-28-03-2222
Log file : /var/tmp/intel-sightings-Patch-2018-01-18-16-28-03-2222/sos.log​

Note: Make a note of any virtual machines that have a False status under the HypervisorAssistedGuestAffected column as they will need to be power cycled in a subsequent step.

6. If any hosts are affected as noted in the previous command, issue the following command to enable the Hypervisor-Assisted Guest Mitigation fix on those hosts:

/opt/vmware/sddc-support/sos --apply-esxi-microcode-patch --domain-name MGMT - for 2.2 or 2.3
/opt/vmware/evosddc-support/sos --apply-esxi-microcode-patch --domain-name MGMT - for 2.1.3

Note: You see output similar to:
 

Welcome to Supportability and Serviceability(SoS) utility!
NOTE : SoS may fail if workflow is undergoing password rotation operation, run SoS when no workflows are in flight.

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-47-07-16969

Patch Intel Sightings workaround on ESXi as outlined by https://kb.vmware.com/s/article/52345
ESXi Status :
-------------

+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
| SL# | NODE |              CPU               | Microcode  | IBRPresent | IBPBPresent | STIBPresent | HypervisorAssistedGuestAffected | IntelSighting |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
|  1  | R1N8 | Intel(R) Xeon(R) Gold 6130 CPU | 0x0200003a |    True    |     True    |     True    |              False              |      True     |
|     |      |           @ 2.10GHz            |            |            |             |             |                                 |               |
|  2  | R1N6 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  3  | R1N5 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  4  | R1N3 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+

Patching Intel Sightings workaround on above affected ESXi..

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-47-07-16969
Log file : /var/tmp/intel-sightings-Patch-2018-01-18-16-47-07-16969/sos.log

7. In the vSphere Web client, power off and back on any virtual machines with a False status under the HypervisorAssistedGuestAffected column in Step 3.

       8. Issue the following command to verify the status of the ESXi hosts and virtual machines in the management workload domain:

/opt/vmware/sddc-support/sos --verify-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.2 or 2.3
/opt/vmware/evosddc-support/sos --verify-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.1.3

Note: You see output similar to:

 

Welcome to Supportability and Serviceability(SoS) utility!
NOTE : SoS may fail if workflow is undergoing password rotation operation, run SoS when no workflows are in flight.

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-55-04-2867
Verify ESXi Microcode update has been applied as stated per https://kb.vmware.com/s/article/52085
ESXi Status :
-------------

+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
| SL# | NODE |              CPU               | Microcode  | IBRPresent | IBPBPresent | STIBPresent | HypervisorAssistedGuestAffected | IntelSighting |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+
|  1  | R1N8 | Intel(R) Xeon(R) Gold 6130 CPU | 0x0200003a |    True    |     True    |     True    |              False              |      True     |
|     |      |           @ 2.10GHz            |            |            |             |             |                                 |               |
|  2  | R1N6 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  3  | R1N5 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
|  4  | R1N3 |  Intel(R) Xeon(R) CPU E5-2660  | 0x0b000025 |    True    |     True    |     True    |              False              |      True     |
|     |      |          v4 @ 2.00GHz          |            |            |             |             |                                 |               |
+-----+------+--------------------------------+------------+------------+-------------+-------------+---------------------------------+---------------+

Virtual Machine Status :
------------------------

+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+
| SL# |         NODE         |                       VM Name                       | IBRPresent | IBPBPresent | STIBPresent |  vHW   | HypervisorAssistedGuestAffected |
+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+
|  1  | r1n8.vcf.vmware.corp |                   vRealize-Edge-0                   |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      |                    vrops-replica                    |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                  vrops-data-node-1                  |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                        psc-2                        |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      | NSX_Controller_ba6de5ba-444e-4210-82a1-6ffef2a58501 |   False    |    False    |    False    | vmx-10 |              True               |
|  2  | r1n6.vcf.vmware.corp |                   loginsight-vm-1                   |   False    |    False    |    False    | vmx-09 |              True               |
|     |                      |                        psc-1                        |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      | NSX_Controller_d7fcbc0d-01db-41db-8fae-afb4eb3f7559 |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      |               SDDC Manager Controller               |   False    |    False    |    False    | vmx-10 |               True              |
|     |                      |                   vRealize-Edge-1                   |   False    |    False    |    False    | vmx-10 |              True               |
|  3  | r1n5.vcf.vmware.corp |                   loginsight-vm-2                   |   False    |    False    |    False    | vmx-09 |              True               |
|     |                      |                 SDDC Manager Utility                |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      | NSX_Controller_d4508e19-33d5-409c-80ac-0063f520c173 |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      |                        vrslcm                       |   False    |    False    |    False    | vmx-10 |              True               |
|     |                      |                      vcenter-1                      |   False    |    False    |    False    | vmx-10 |              True               |
|  4  | r1n3.vcf.vmware.corp |                   loginsight-vm-3                   |   False    |    False    |    False    | vmx-09 |              True               |
|     |                      |                    nsx-manager-1                    |   False    |    False    |    False    | vmx-08 |               N/A               |
|     |                      |                     vrops-master                    |   False    |    False    |    False    | vmx-08 |               N/A               |
+-----+----------------------+-----------------------------------------------------+------------+-------------+-------------+--------+---------------------------------+

The value of IntelSighting can contain four potential values:
--------------------------------------------------------------

   *      True = ESXi host contains microcode update is affected by Intel Sighting, you
          will need to apply the workaround as outlined in
          https://kb.vmware.com/s/article/52345
   *      False = CPU not affected by Intel Sighting it is currently recommended to only
          apply one of the ESXi patches (until Intel provides a microcode update fix),
          please refer to https://kb.vmware.com/s/article/52345 for full details
   *      AffectedOncePatched = CPU is affected by Intel Sighting, but does not , need the
          work around unless it is patched or has a BIOS update
   *      N/A = CPU is not Intel

Logs : /var/tmp/intel-sightings-Patch-2018-01-18-16-55-04-2867
Log file : /var/tmp/intel-sightings-Patch-2018-01-18-16-55-04-2867/sos.log

9. Repeat Steps 5 through 8 against any other workload domains in the VMware Cloud Foundation environment by replacing MGMT with the other workload domain names in the previous commands.
10. If there are any hosts that are not part of a workload domain, issue the following command to enable the Hypervisor-Assisted Guest Mitigation fix on them:

/opt/vmware/sddc-support/sos --apply-esxi-microcode-patch --unassigned-hosts - for 2.2 or 2.3
/opt/vmware/evosddc-support/sos --apply-esxi-microcode-patch --unassigned-hosts - for 2.1.3
 

Notes:

• If the environment in question is a mutli-rack, 2.1.3 environment, the previous steps will need to be performed on all VRM virtual machines.
• Once a proper microcode patch is released by Intel and applied to the affected hosts, the following command can be run to revert the change made in Step 4:

/opt/vmware/sddc-support/sos --revert-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.2 or 2.3
/opt/vmware/evosddc-support/sos --revert-esxi-microcode-patch --domain-name MGMT --get-vms-status - for 2.1.3

 

• The preceding command should also be run against any additional workload domains that were addressed in Step 9.
• After reverting the change, Virtual Machines will need to be power cycled again, per Step 5.

Additional Information

For more information, see:

Hypervisor-Assisted Guest Mitigation for branch target injection
Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004