Using Ruby vSphere Console (RVC) for vSAN operations/troubleshooting

Products

VMware vSAN

Issue/Introduction

The purpose of the article is to:

act as a usage example for Ruby vSphere Console (RVC) to perform vSAN operations
provide information regarding log-in, navigation, some commands, and their outputs

Environment

VMware vSAN (All Versions)

Resolution

Logging into RVC

vCenter Server Appliance:

Log in to the VCSA as root via SSH.
Execute rvc given the command structure below and click Enter. Enter 'y' if prompted.
Format your login string as <user>@<domain>@<localhost>.
Example: [email protected]@localhost
Note: If the SSO login hasn't changed from [email protected] you can access RVC simply with rvc localhost

rvc localhost
[DEPRECATION] The trollop gem has been renamed to optimist and will no longer be supported. Please switch to optimist as soon as possible.
Install the "ffi" gem for better tab completion.
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.
*********************************************************************
[DEPRECATED] RVC will be deprecated. It will not be available in the
next major release.
*********************************************************************
The authenticity of host 'localhost' can't be established.
Public key fingerprint is 8fb8862046060e3af01ed6abb894cb7ca1f8e0a03073b396d98c06866bd0d38d.
Are you sure you want to continue connecting (y/n)?
Note: At this time you can safely ignore the DEPRECATED messages, RVC won't be going away any time soon.
Type y to continue

Warning: Permanently added 'localhost' (vim) to the list of known hosts
Using default username "[email protected]".
password:
Enter the password for [email protected]

Navigating RVC:

Once RVC is opened, navigate it as a pseudo-shell.
All standards *nix navigation concepts apply (ls, cd, “.” is the current directory, “..” is the parent directory, special characters/spaces must be escaped, etc.).
Navigate to the vSAN cluster by changing directories in the following manner once connected:
It should default to the top directory, showing:

Welcome to RVC. Try the 'help' command.
0 /
1 localhost/
>

Type cd 1 and enter to go into localhost.
Type ls and enter to view the next listing of directories.
Type cd <datacenter number or name> and enter. For example, cd 0 or cd VSAN-DC.
Type ls and enter.
Type cd 1 to go into "computers [host]".
Type ls and enter and then cd into the number or name corresponding to the cluster of interest.
From there you can run commands using a period "." as the current directory/cluster.
All vSAN operations in RVC are performed against a host, a cluster, or a virtual machine.

vSAN commands:

All vSAN commands in Ruby vSphere Console (RVC) begin with "vsan.".
This document goes over these commands and provides sample output where applicable.
This document does not go over commands that modify vSAN – only those to query vSAN.
You must specify an object against which to run the commands. Take input of an entire path, or change to the applicable directory and invoke it with a “.”

Cluster-level vSAN commands:

To use these commands, navigate to the cluster.

For example:

0 /
1 localhost/
> cd 1
/localhost> ls
0 VSAN-DC (datacenter)
/localhost> cd 0
/localhost/VSAN-DC> ls
0 storage/
1 computers [host]/
2 networks [network]/
3 datastores [datastore]/
4 vms [vm]/
/localhost/VSAN-DC> cd 1
/localhost/VSAN-DC/computers> ls
0 VSAN-Cluster (cluster): cpu 114 GHz, memory 47 GB
/localhost/VSAN-DC/computers> cd 0
/localhost/VSAN-DC/computers/VSAN-Cluster>

vsan.cluster_info – Dump information about this VSAN cluster includes node and disk information.

For example (truncated for brevity):

/localhost/VSAN-DC/computers/VSAN-Cluster> vsan.cluster_info .

Host: ds-lab-vsan06.dsl.vmware.com
  VSAN enabled: yes
  Cluster info:
    Cluster role: agent
    Cluster UUID: 525f3eda-73d6-c1e3-e316-############
    Node UUID: 5256f254-e121-3df6-6103-############
    Member UUIDs: ["524ad2f9-5a4b-5068-0f67-############", "52581e66-b006-23f4-2962-############", "524b42ae-44d0-eb0a-0daf-############", "53177cda-f4d6-3d64-7f55-############", "5256f254-e121-3df6-6103-############"]
  Storage info:
    Auto claim: no
    Disk Mappings:
      SSD: Local ATA Disk (naa.500253825#######) - 93 GB
      MD: Local SEAGATE Disk (naa.5000c5005#######) - 279 GB
      MD: Local SEAGATE Disk (naa.5000c5005#######) - 279 GB
      MD: Local SEAGATE Disk (naa.5000c5005#######) - 279 GB
      MD: Local SEAGATE Disk (naa.5000c5005#######) - 279 GB
  NetworkInfo:
    Adapter: vmk1 (172.###.###.206)

vsan.resync_dashboard – Display all resync activity for the cluster. This can be run against a host, but it is less useful.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster> vsan.resync_dashboard .

2014-03-06 13:32:42 -0700: Querying all VMs on VSAN ...
2014-03-06 13:32:43 -0700: Querying all objects in the system ...
2014-03-06 13:32:44 -0700: Got all the info, computing table ...
+-----------+-----------------+---------------+
| VM/Object | Syncing objects | Bytes to sync |
+-----------+-----------------+---------------+
+-----------+-----------------+---------------+
| Total | 0 | 0.00 GB |
+-----------+-----------------+---------------+
/localhost/VSAN-DC/computers/VSAN-Cluster>

In this case, there is nothing resyncing. If there were an active resync, the number of objects and bytes left to resync would populate here.

vsan.disks_stats – Display all disk statistics, distribution, etc. This can be run against a host, but it is less useful.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster> vsan.disks_stats .
Fetching VSAN disk info from esxi-vsan08 (this may take a moment) ...
Fetching VSAN disk info from esxi-vsan09 (this may take a moment) ...
Fetching VSAN disk info from esxi-vsan10 (this may take a moment) ...
Fetching VSAN disk info from esxi-vsan07 (this may take a moment) ...
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| | | | Num | Capacity | | | | Device |
| DisplayName | Host | isSSD | Comp | Total | Used | Reserved | Used | Latencies |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| naa.500253825####### | esxi-vsan06 | SSD | 0 | 65.21 GB | 0 % | 0 % | 0r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan06 | MD | 1 | 279.25 GB | 1 % | 1 % | 0r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan06 | MD | 1 | 279.25 GB | 1 % | 1 % | 0r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan06 | MD | 4 | 279.25 GB | 0 % | 0 % | 0r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan06 | MD | 1 | 279.25 GB | 0 % | 0 % | 0r/0w | 0d/0q/0k |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| naa.500253825####### | esxi-vsan07 | SSD | 0 | 65.21 GB | 0 % | 0 % | 1r/0w | 1d/0q/0k |
| naa.5000c5005####### | esxi-vsan07 | MD | 0 | 279.25 GB | 0 % | 0 % | 1r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan07 | MD | 1 | 279.25 GB | 0 % | 0 % | 1r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan07 | MD | 0 | 279.25 GB | 0 % | 0 % | 1r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan07 | MD | 6 | 279.25 GB | 0 % | 0 % | 1r/0w | 0d/0q/0k |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| naa.500253825####### | esxi-vsan08 | SSD | 0 | 65.21 GB | 0 % | 0 % | 17r/23w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan08 | MD | 8 | 279.25 GB | 14 % | 2 % | 16r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan08 | MD | 12 | 279.25 GB | 14 % | 12 % | 16r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan08 | MD | 34 | 279.25 GB | 9 % | 2 % | 16r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan08 | MD | 6 | 279.25 GB | 3 % | 1 % | 16r/0w | 0d/0q/0k |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| naa.500253825####### | esxi-vsan09 | SSD | 0 | 65.21 GB | 0 % | 0 % | 57r/25w | 0d/0q/1k |
| naa.5000c5005####### | esxi-vsan09 | MD | 28 | 279.25 GB | 4 % | 3 % | 2r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan09 | MD | 13 | 279.25 GB | 14 % | 0 % | 2r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan09 | MD | 10 | 279.25 GB | 13 % | 12 % | 5r/0w | 1d/0q/1k |
| naa.5000c5005####### | esxi-vsan09 | MD | 9 | 279.25 GB | 17 % | 0 % | 2r/0w | 0d/0q/0k |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
| naa.500253825####### | esxi-vsan10 | SSD | 0 | 65.21 GB | 0 % | 0 % | 42r/46w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan10 | MD | 9 | 279.25 GB | 8 % | 2 % | 9r/0w | 3d/0q/3k |
| naa.5000c5005####### | esxi-vsan10 | MD | 9 | 279.25 GB | 17 % | 0 % | 2r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan10 | MD | 13 | 279.25 GB | 15 % | 2 % | 2r/0w | 0d/0q/0k |
| naa.5000c5005####### | esxi-vsan10 | MD | 28 | 279.25 GB | 20 % | 1 % | 2r/0w | 0d/0q/0k |
+----------------------+------------------------------+-------+------+-----------+------+----------+---------+-----------+
/localhost/VSAN-DC/computers/VSAN-Cluster>

vsan.object_info – Displays layout of any given object must be executed against the cluster level, but specify a VSAN DOM object UUID.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster> vsan.object_info . 5d44c352-38c8-3a72-61b0-############
DOM Object: 5d44c352-38c8-3a72-61b0-############(owner: ds-lab-vsan05.dsl.vmware.com, policy: forceProvisioning = 0, hostFailuresToTolerate = 1, spbmProfileGenerationNumber = 0, proportionalCapacity = [0, 100], spbmProfileId = d0aeb587-760c-41b0-ad7e-############, stripeWidth = 1)
  Witness: 5e44c352-3418-9ebf-ca3f-############(state: ACTIVE (5), host: esxi-vsan08, md: naa.5000c5005#######, ssd: naa.500253825#######)
  RAID_1
    Component: 5e44c352-b497-9dbf-00a1-############(state: ACTIVE (5), host: esxi-vsan09, md: naa.5000c5005#######, ssd: naa.500253825#######)
    Component: 5e44c352-7cf1-9cbf-42dc-############(state: ACTIVE (5), host: esxi-vsan10, md: naa.5000c5005#######, ssd: naa.500253825#######)
/localhost/VSAN-DC/computers/VSAN-Cluster>

Host-level vSAN commands:

To use these commands, navigate to the host.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster> ls

0 hosts/
1 resourcePool [Resources]: cpu 77.22/77.22/normal, mem 126.35/126.35/normal
/localhost/VSAN-DC/computers/VSAN-Cluster> cd 0
/localhost/VSAN-DC/computers/VSAN-Cluster/hosts> ls
0 esxi-vsan06 (host): cpu 2*8*2.20 GHz, memory 34.00 GB
1 esxi-vsan08 (host): cpu 2*8*2.20 GHz, memory 34.00 GB
2 esxi-vsan09 (host): cpu 2*8*2.20 GHz, memory 34.00 GB
3 esxi-vsan10 (host): cpu 2*8*2.20 GHz, memory 34.00 GB
4 esxi-vsan07 (host): cpu 2*8*2.20 GHz, memory 34.00 GB
/brm-storage-pvn/Broomfield Storage/computers/vSAN General/hosts> cd 0
/brm-storage-pvn/Broomfield Storage/computers/vSAN General/hosts/esxi-vsan06>

vsan.disks_info – Displays eligibility of all disks attached to the host. If the disk is already claimed by VSAN, that is reflected. The example output is not provided, as its format is too wide to be easily viewed here.

VM-level vSAN commands:

To use these commands, navigate to the virtual machine. All virtual machines are under resourcePool and the applicable pool. This is true even if no resource pools have been defined.
For example:

/localhost/VSAN-DC/computers/VSAN-Cluster> cd 1
/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool> cd 0
/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms> cd 0
/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms/vsan-demo>

vsan.vm_object_info – Displays the object layout of the VM – RAID types, witnesses, states, etc. It shows which hosts and which disks contain the components, as well as which host is the DOM owner for the objects (namespace/VM home, disks, snaps, etc.). Swap object is omitted. This contains the same information as vsan.object_info for all objects comprising the virtual machine.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms/vsan-demo> vsan.vm_object_info .
Fetching VSAN disk info from esxi-vsan04 (this may take a moment) ...
Fetching VSAN disk info from esxi-vsan05 (this may take a moment) ...
VM vsan-demo:
  Namspace directory
    DOM Object: 5d44c352-38c8-3a72-61b0-############(owner: esxi-vsan09, policy: forceProvisioning = 0, hostFailuresToTolerate = 1, spbmProfileGenerationNumber = 0, proportionalCapacity = [0, 100], spbmProfileId = d0aeb587-760c-41b0-ad7e-############, stripeWidth = 1)
      Witness: 5e44c352-3418-9ebf-ca3f-############(state: ACTIVE (5), host: esxi-vsan08, md: naa.5000c5005#######, ssd: naa.500253825#######)

      RAID_1
        Component: 5e44c352-b497-9dbf-00a1-############(state: ACTIVE (5), host: esxi-vsan09, md: naa.5000c5005#######, ssd: naa.500253825#######)
        Component: 5e44c352-7cf1-9cbf-42dc-############(state: ACTIVE (5), host: esxi-vsan10, md: naa.5000c5005#######, ssd: naa.500253825#######)

  Disk backing: [vsanGeneral] 5d44c352-38c8-3a72-61b0-############/vsan-demo.vmdk
    DOM Object: 6344c352-ea57-8d81-a994-############(owner: esxi-vsan09, policy: hostFailuresToTolerate = 1, forceProvisioning = 1)
      Witness: 6444c352-2484-1ac6-2c51-############(state: ACTIVE (5), host: esxi-vsan08, md: naa.5000c5005#######, ssd: naa.500253825#######)

      RAID_1
        Component: 6444c352-acc3-19c6-5561-############(state: ACTIVE (5), host: esxi-vsan10, md: naa.5000c5005#######, ssd: naa.500253825#######)
        Component: 6444c352-88d0-18c6-f723-############(state: ACTIVE (5), host: esxi-vsan09, md: naa.5000c5005#######, ssd: naa.500253825#######)
/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms/vsan-demo>

vsan.vm_perf_stats – Displays IOPs, throughput, latency for the virtual machine and does not work on linked-clones.

For example:

/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms/vsan-demo> vsan.vm_perf_stats .
2014-03-06 14:36:33 -0700: Querying info about VMs ...
2014-03-06 14:36:33 -0700: Querying VSAN objects used by the VMs ...
2014-03-06 14:36:34 -0700: Fetching stats counters once ...
2014-03-06 14:36:34 -0700: Sleeping for 20 seconds ...
2014-03-06 14:36:54 -0700: Fetching stats counters again to compute averages ...
2014-03-06 14:36:54 -0700: Got all data, computing table
+-----------+-------------+---------------+--------------+
| VM/Object | IOPS | Tput (KB/s) | Latency (ms) |
+-----------+-------------+---------------+--------------+
| vsan-demo | 132.9r/2.8w | 4170.3r/16.3w | 0.3r/1.4w |
+-----------+-------------+---------------+--------------+
/localhost/VSAN-DC/computers/VSAN-Cluster/resourcePool/vms/vsan-demo>

Additional Information

RVC is not used much these days for troubleshooting as the majority of troubleshooting can be done via esxcli or PowerCLI, see the below links:

RVC main uses for troubleshooting:

Procedure to Enable TRIM/UNMAP to reclaim space on vSAN datastore. (Only for 7.x, as of 8.x and newer you can enable trim/unmap via vCenter UI)
Procedures for identifying Unassociated vSAN objects. (326864)
VMs are showing path (/vmfs/volumes/vsan...) instead of friendly names on UI (326853)
Configuring Stretched Cluster Results in "A specified parameter was not correct - Witness host already configured in stretched cluster" (326817)
Disable vSAN Performance Service in 8.0
vsan.check_state -re . to refresh/reload VMs that are in an orphaned/inaccessible state

How to Login into RVC?

Also see VMware Ruby vSphere Console Command Reference For vSAN attached to this KB

Attachments

VMware-Ruby-vSphere-Console-Command-Reference-For-Virtual-SAN.pdf get_app