NetOps flow data not being seen within Portal after data was being sent via Load balancer
Article ID: 405751
Updated On:
Products
Issue/Introduction
Netops flow stopped processing new data after data was being ingested via a loadbalancer.
Environment
DX NetOps flow 24.3.9 release
Cause
Load balancer was masking the device IP address due to which the packets were not being processed by NetOps flow.
allocateLoad BalancerNodePorts: true
cluster IP: Existing Cluster IP
cluster IPs:
- Primary Cluster IP
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
IPv4
ipFamilyPolicy: SingleStack
loadBalancerIP: [Your Load Balancer IP]
Resolution
To resolve this issue and ensure the preservation of the client source IP, the flow-proxy Kubernetes Service configuration was modified by setting externalTrafficPolicy to Local. This configuration change directs external traffic to node-local endpoints, which in turn preserves the client's original IP address.
Configuration Change Applied:
The flow-proxy Service manifest was updated as follows (key changes highlighted):
apiVersion: v1
kind: Service
metadata:
name: flow-proxy-loadbalancer
# ... other metadata
spec:
# ... other spec fields
allocateLoadBalancerNodePorts: true
clusterIP: [Existing Cluster IP] # Non-routable example, replaced with actual if needed
clusterIPs:
- [Primary Cluster IP]
externalTrafficPolicy: Local # <-- CRITICAL CHANGE: Changed from 'Cluster' to 'Local'
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
loadBalancerIP: [Your Load Balancer IP]
# ... other fields
type: LoadBalancer
# Citrix Load Balancer USIP annotation update
service.citrix.com/servicegroup: '{"9996-udp":{"usip":"yes"}, "4739-udp":{"usip":"yes"}}'
# Note: The port for Citrix Load Balancer USIP annotation was changed to use 9996 for the UDP service.
Explanation and Rationale:
Kubernetes Services, by default, route external traffic with externalTrafficPolicy: Cluster. This obscures the client source IP and may route traffic through an extra hop, but offers good load-spreading.
By setting externalTrafficPolicy: Local, the Service is configured to preserve the client's source IP. This ensures that the application within the pod sees the actual IP address of the client rather than the IP of an intermediate node or the load balancer. While this avoids a second network hop, it's important to note that it might lead to potentially imbalanced traffic spreading if not all nodes have healthy pods for the service.
The Kubernetes documentation further explains this behavior:
Preserving the client source IP
By default, the source IP seen in the target container is not the original source IP of the client. To enable preservation of the client IP, the following fields can be configured in the .spec of the Service:
- .spec.externalTrafficPolicy: denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type Services, but risks potentially imbalanced traffic spreading.
- .spec.healthCheckNodePort: specifies the health check node port (numeric port number) for the service. If you don't specify healthCheckNodePort, the service controller allocates a port from your cluster's NodePort range. You can configure that range by setting an API server command line option, --service-node-port-range. The Service will use the user-specified healthCheckNodePort value if you specify it, provided that the Service type is set to LoadBalancer and externalTrafficPolicy is set to Local.
Reference:
For further details on preserving client source IP with Kubernetes Load Balancers, refer to the official documentation:
Feedback
