VS reporting VS Fault: Virtual Service (vip_id 1) not ready on Service Engine. Reason:Virtual Service IP is not reachable in the Cloud

Products

• VMware Avi Load Balancer

Issue/Introduction

• VSs reporting VS Fault: Virtual Service (vip_id 1) not ready on Service Engine. Reason:Virtual Service IP is not reachable in the Cloud.
  

Environment

• All environments.

Cause

• This issue is primarily caused because of a delay in the SE processing updates from the controller.
• When a VS is placed on the SE by the controller, the controller pushes the config to the SE and the SE processes these.
• Once the SE has downloaded the config push, it sends a "ScaleOut Ready" message to the controller to let it know that the VS has been successfully accepted.
• There is a time within which the controller expects to receive the "ScaleOut Ready" from the SE.
• This issue is hit when the controller receives this from the SE after the timeout.
• The delay from the SE in downloading the configs can be because of various reasons including but not limited to:
  • Frequent network drops between the SE and the Controller.
  • SE going headless multiple times. 
  • SE processes running out of CPU cycles. This results in hung processes.
  • SE is busy downloading other files like IPReputation files. 
  • Non-VS traffic being redirected to the SE causing the KNI path to be overwhelmed.
  • To verify this, you can use the following command:
    

    1) Login to the CLI.
    2) Execute: 
    [admin:cntlr]:> show serviceengine <se-name> interface | grep tx_kni_errs

    You can get the se-name from the fault on the UI.

Resolution

• To fix the VS fault, you can Scale-In the VS from the affected SE and then do a Scale-Out again.
• If Scale-In is not permitted, Migrate the VS to another SE and then move it back to the desired SE.
• If this is also not an option, as a last resort, a disable/enable of the VS will fix this issue.