Minimum information to identify and troubleshoot VeloCloud Orchestrator (VCO) issues

Products

• VMware SD-WAN by VeloCloud

Issue/Introduction

The purpose of this article is the following: 

• Provide you with guidance to determine whether a case is VCO or not.
• Describe the minimum information a technical support engineer or end user has to gather depending on the type of issue you found in the VeloCloud Orchestrator (VCO).

Environment

VMware SD-WAN by VeloCloud

Cause

How to identify if a support case is a VCO issue:

A ticket is a VCO case if it meets at least one of the following criteria:

• An unexpected behavior is seen from the UI, including inability to save changes that are known to be correct and/or failures of the UI when displaying information.
• The issue can't be troubleshooted with DP tools.
• Any API question or issue pertaining to any kind of use of VCO's API (e.g. provisioning automation)
• Issues related to login into VCO's UI or CLI
• Issues with alerts or notifications sent from the VCO (emails, sms, webhooks)
• Issues pertaining to the underlying OS of the VCO (e.g. disk space usage, CPU and memory consumption)
• Issues with performance navigating in the UI
• Customer request for VCO Upgrade  (hosted and On-Prem)

• Customer request for VCO migration, one or several enterprises or MSP.

If you have a case and you are not sure if it matches the criteria above, please contact the VCO support team for clarification.

Resolution

 

API and SDK problems: 

Issues with the API:

This section describes those issues that relate to API problems. It later talks about SDK problems specifically as well. The examples covered are the ones such as getting an error when configuring an edge via API or simple documentation questions. 

Data to collect:

• The API method being used (for example: enterprise/getEnterprise). 
• The parameters being sent to the server.
• The expected output and the response that you are getting (any error, no error, no response...)
• The VCO being queried, make sure to check that API call is supported in that version by going to https://code.vmware.com/apis, searching for Velocloud, selecting the first option and filtering the VCO version with the dropdown as the one below.

Issues with the SDK:

1. As of today, the Swagger documentation for the VeloCloud Orchestrator API is publicly available in vmware's developer portal, https://code.vmware.com/apis, alongside the API docs for VMware’s other product suites. This supersedes the docs that had previously been bundled with our developer SDK distribution, partners/customers must use the document available there.
2. Various client-side code samples are now available on VMware Sample Exchange. For those not aware, Sample Exchange is a repository of VMware and community-contributed code samples.  Please note that VMware – and in particular our API/SDK and Support teams – “do not guarantee the samples; they are provided ‘AS IS’” to help users bootstrap their own scripts and apps.
3. We do not generally recommend the legacy code-generated SDK client libraries for any new development unless you have an existing body of code that depends upon them; the libraries are known to be error-prone in a few respects and we are somewhat hamstrung in our ability to address key design issues because some partners and/or customers expect a degree of backward-compatibility that makes a redesign infeasible. If you do require access to a particular version of the legacy SDK distribution, please reach out to [email protected] with specifics.

User interface errors:

The VCO is composed by two separate components: the frontend and the backend. Any error here, such as "An unexpected error has occurred" whenever you are configuring a service, should be first considered a user interface issue first until narrowed down. This section provides all the necessary data that a technical support engineer will need in order to make this decision. First, some background:
 

1. The Frontend (or User interface - UI) is the part that you see and interact whenever you are configuring or monitoring the edges or gateways. This is the client-facing part of the VCO. 
2. The Backend is where our REST API is hosted. The backend is in charge of responding to API calls and 'talks' JSON (JavaScript Object Notation). All the information that you see from the VCO, such as all the edges in a customer, are populated by making an API call to the backend, to which it responds by fetching the data from different services running inside it based on the parameters that the API call sent. 
 

Data to collect:

1. The specific error that's shown, such as "An unexpected error has occurred".
2.  A screenshot from the Chrome Developer console, such as the one below and also the logs shown. We could save the data seen below by right clicking in a whitespace inside the console, then selecting 'save as' and save it in your PC. 

3. Also save all the API calls that were made as a .HAR file. To do so, please follow the steps below:

3.1 If you have to perform several steps to make the error happen, such as changing tabs inside the VCO or saving content, please make that before starting you tick the box called 'Preserve log' inside the network tab. More details here.

3.2 Go to the network tab in the console.
3.3 Refresh your browser tab in the browser (with the console open)
3.4 Make the same change, if any, to get the error. If the error is automatically shown there's no need to make any changes.
3.5 After seeing the error, right-click any of the '/portal' calls that's seen in the console and select 'Save as HAR with content'. 

4. The upload_exceptions.log, upload_velocloud.log, portal_velocloud.log, portal_exceptions.log, nginx_error.log, nginx_access.log, mysql-error.log, mysql-slow.log and lastly backend_exception.log and backend_velocloud.log. The size of these may be big, so you may filter them with a particular range of time when the issue is shown. If you don't have access to these, please let support know how to reproduce the problem and they'll gather them.

5. Describe in detail what happened before the error was seen, if any change was done, specify the time in UTC, if it's the first time this happens or if it has happened in the past and lastly if it was working before. 

Disaster Recovery problems:

Whenever an issue is reported for the disaster recovery feature in the VCO, the below information is necessary to be able to diagnose the problem. This information should be gathered by the partner/customer and handed over to the technical support engineer working on the case.

Data to collect:

1. A VCO diag bundle.
2. The output from the 'show secondary status' from the secondary VCO. 
3. A detailed description about  what happened before the error was seen, if any change was done, specify the time in UTC, if it's the first time this happens or if it has happened in the past and lastly if it was working before.

Upgrade related problems:

For those customers that have an on premise deployment of the VeloCloud Orchestrator, the following information is necessary in order to help in the problem analysis. 

Data to collect:

1. A VCO diag bundle.
2. The vco_software_update.log file that's saved under /var/log on the VCO.
3. A detailed description about what happened before the error was seen, if any change was done, specify the time in UTC, if it's the first time this happens or if it has happened in the past and lastly if it was working before.

 

Issues with the remote diagnostics tab:

Whenever an issue is seen in the remote diagnostics tab from the VeloCloud orchestrator, there are several factors to consider. The first one, and most important, is that this specific tab gets the data from the edge itself and isn't something that the VCO is making. The VCO for this cases is just a proxy displaying the edge side calls that are made. 

Based on that, if the error seen is similar to "An unexpected error has occurred" or related, please refer to the User Interface Errors section. Any other case should be considered an edge related problem and a sanity of the edge should be done first.

Data to collect:

1. Depending on where the issue lies, an edge diag bundle, a gateway diag bundle or both.
2. A VCO diag bundle might help as well.