Unable to communicate from the pod to itself when trying access the service IP that backs the service

Products

• VMware Cloud PKS

Issue/Introduction

This article explains about how to resolve communication issue from pod to itself when accessed through Service IP/DNS name.

Symptoms:

• Pod is unable to communicate to itself when it access the service IP that it backs.

• Service is reachable from Node and other pod but not from the pod that backs it.

Environment

VMware PKS 1.x

Cause

This type of communication is defined as Hair pinning and it is disabled by default in Kubernetes.

When a service is created with the type ClusterIP/LoadBalancer pod won’t be able to access itself through its Service IP/DNS name.

Resolution

This is an expected behavior with Kubernetes and the NCP plugin.

Workaround:

To workaround this issue, create service with the type Headless and access the service using the name from inside the pod.

Headless service can be used when we don’t need load-balancing and a single service IP. In this case, you can create “headless” services by specifying "None" for the cluster IP (.spec.clusterIP). For such Services, a cluster IP is not allocated, kube-proxy does not handle these services, and there is no load balancing or proxying done by the platform for them. How DNS is automatically configured depends on whether the service has selectors defined. For headless services that define selectors, the endpoints controller creates Endpoint records in the API, and modifies the DNS configuration to return A records (addresses) that point directly to the Pods backing the Service
 
You can deploy Headless service just the way ClusterIP/LoadBalancer is created, sample deployment file for reference.

==============
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
==============
Note: Modify the above Application name and the ports as per the requirement.
 

1. Create the deployment in a .yml or .yaml extension as below and paste the contents:
   vi nginx_svc.yml

2. Once the deployment file is created, create the service resource using the below command:
   kubectl create -f nginx_svc.yml

3. Check the Service resource using the below command:
   kubectl get svc -n <namespace>
   
   For example: root@pks-client:~/deployments/nginx# kubectl get svc
   NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
   kubernetes   ClusterIP   10.100.200.1   <none>        443/TCP   2d9h
   nginx        ClusterIP   None           <none>        80/TCP    3s

4. Make sure the service is created without any ClusterIP as highlighted above.

5. Check the accessibility from inside the pod using the below command:
   kubectl exec -it  <pod-name> sh
   curl <service-name>