Ingress Creation Fails when Assigning a Custom BotDetection Policy

Products

• VMware Avi Load Balancer

Issue/Introduction

Customer is using AKO (Avi Kubernetes Operator) with AVI Load Balancer in a multi-tenant environment. 

While creating a new Ingress in a non-admin tenant and specifying a custom Bot Detection Policy (BotDetectionPolicy), the Virtual Service is successfully created on the AVI Load Balancer. However, the specified Bot Detection Policy is not applied to the Virtual Service.

AKO Error logs:

kubectl logs ako-0 -n namespace > filename.txt

grep "2 BotDetectionPolicy objects found" filename.txt

####-##-##T######Z        WARN    rest/dequeue_nodes.go:657       key: admin/my-cluster--###-###-###, msg: there was an error sending the macro Error during POST: Encountered an error on POST request to URL https://##.##.##.##//api/virtualservice: HTTP code: 400; error from Controller: map[error:2 BotDetectionPolicy objects found!]

Note: This issue can occur in admin tenant as well if there are multiple Bot Detection Policies present in AVI controller

Environment

AVI Load Balancer Version : 30.x, 31.x, 22.x 

AKO (Avi Kubernetes Operator)  Version: All AKO versions

 

Cause

If there are more than One Bot Detection policies present in AVI Load Balancer controller this issue will occur.

Verification:

1.  Login to AVI Load balancer controller UI.
2. Select "All Tenants" Option on the Top-Right Side
3. Navigate to Templates >> Bot >> Bot Management.  Verify if there are more than One Bot Detection policy is present 

 

Controller Log verification:

1. SSH to Leader Controller node and navigate to folder /var/lib/avi/log/
2. Check for below log in file "apiserver.INFO" 

grep "2 BotDetectionPolicy objects found" apiserver.INFO

####-##-##T##:##:##.##Z        E  4038         models/avi_utils.go:800 Unable to resolveRefersUri - 2 BotDetectionPolicy objects found!
####-##-##T##:##:##.##Z        E  4038         apihandlers/api_utils.go:376    custom marshalling failed for model virtualservice with error 2 BotDetectionPolicy objects found! 

AKO log verification:

Collect AKO pod logs using command. This will store the AKO pod logs in a file

kubectl logs ako-0 -n namespace > filename.txt

grep "2 BotDetectionPolicy objects found" filename.txt

####-##-##T######Z        WARN    rest/dequeue_nodes.go:657       key: admin/my-cluster--###-###-###, msg: there was an error sending the macro Error during POST: Encountered an error on POST request to URL https://##.##.##.##//api/virtualservice: HTTP code: 400; error from Controller: map[error:2 BotDetectionPolicy objects found!]

 

Resolution

Please remove the custom BotDetectionPolicy from the AVI Controller UI and use only System Defined Bot detection Policy in Ingress configuration.

This is a BUG from AVI side and It will be fixed in future releases.