NSX-T NCP troubleshooting and debug logging
Article ID: 327341
Updated On:
Products
VMware NSX
Issue/Introduction
This KB article describes NCP troubleshooting steps and how to enable debug logging
Symptoms:
NCP troubleshooting steps and enable debug logging
Symptoms:
NCP troubleshooting steps and enable debug logging
Environment
VMware NSX-T Data Center
Resolution
For PKS setup , NCP runs as a service, hence login PKS master node
steps can be found in Collect NSX-T NCP logs in PKS environment
If NCP runs as a POD such as openshift setup, SSH into the NCP pod
#bosh logs -d [service-instance_uuid]
#oc logs [ncp-pod-name]
#kubectl logs [ncp-pod-name]
NCP configuration file (ncp.ini) in PKS
ncp.ini located under /var/vcap/jobs/ncp/config
settings can be adjusted .
content looks like this:
[DEFAULT]
use_stderr = False
[coe]
enable_snat = true
cluster = pks-60c7bef1-####-####-####-##########07
[ha]
enable = True
[k8s]
apiserver_host_ip = master.example.internal
apiserver_host_port = 8443
ca_file = /etc/nsx-ujo/certs/k8s/ca.crt
client_cert_file = /etc/nsx-ujo/certs/k8s/client.crt
client_private_key_file = /etc/nsx-ujo/certs/k8s/client.key
ingress_mode = nat
...
In PKS master node root mode or NCP pod , execute nsxcli
nsxcli command path may different in various setup
I.E. in PKS
# /var/vcap/data/jobs/ncp/45626f1c68983490975053434143ed2eff8b138b/bin/nsxcli
81b91b91-####-####-####-##########8f> get version
2.3.1
81b91b91-####-####-####-##########8f> get ncp-nsx status
NSX Manager status: Healthy
81b91b91-####-####-####-##########8f> get ncp-master status
This instance is the NCP master
Current NCP Master id is c9a754fc-####-####-####-##########0b
Current NCP Instance id is c9a754fc-####-####-####-##########0b
Last master update at Mon Apr 1 20:11:46 2019
81b91b91-####-####-####-##########8f> get ncp-k8s-api-server status
Kubernetes ApiServer status: Healthy
To Debug NCP , change logging level from Info to Debug
81b91b91-####-####-####-##########8f> set ncp-log-level DEBUG
NCP log level is changed to DEBUG
To Stop /Start / restart NCP in PKS master node
# monit restart ncp
# monit stop ncp
# monit start ncp
To disable NCP HA
change ncp.ini and restart ncp
[ha]
enable = True to False
To change NCP HA HB and timeout timer
By default below values are hardcoded with default values, If we need change these timers then add below parameters and restart ncp.
ncp.ha.master_timeout = 9
ncp.ha.heartbeat_period = 3
[ha]
master_timeout = 18
heartbeat_period = 6
update_timeout = 6
verify if the ncp is running on the master node
#monit summary | grep ncp
Also check Hyperbus status on ESXi host
vESXi-1.nsx-lab.local> get hyperbus connection info
VIFID Connection Status
acbffb45-####-####-####-##########af 169.254.1.10:2345 HEALTHY
steps can be found in Collect NSX-T NCP logs in PKS environment
If NCP runs as a POD such as openshift setup, SSH into the NCP pod
#oc project nsx-system
# oc get pods
NAME READY STATUS RESTARTS AGE
nsx-ncp-snn56 1/1 Running 1 3d
nsx-node-agent-fk9xl 2/2 Running 331 3d
nsx-node-agent-lzxmn 2/2 Running 8 3d
nsx-node-agent-smjlw 2/2 Running 8 3d
nsx-node-agent-tgj8m 2/2 Running 18 3d
nsx-node-agent-w9dsr 2/2 Running 1444 3d
nsx-node-agent-zb4h6 2/2 Running 20 3d
#kubectl exec -it nsx-ncp-snn56 -- /bin/bash
#bosh logs -d [service-instance_uuid]
#oc logs [ncp-pod-name]
#kubectl logs [ncp-pod-name]
NCP configuration file (ncp.ini) in PKS
ncp.ini located under /var/vcap/jobs/ncp/config
settings can be adjusted .
content looks like this:
[DEFAULT]
use_stderr = False
[coe]
enable_snat = true
cluster = pks-60c7bef1-####-####-####-##########07
[ha]
enable = True
[k8s]
apiserver_host_ip = master.example.internal
apiserver_host_port = 8443
ca_file = /etc/nsx-ujo/certs/k8s/ca.crt
client_cert_file = /etc/nsx-ujo/certs/k8s/client.crt
client_private_key_file = /etc/nsx-ujo/certs/k8s/client.key
ingress_mode = nat
...
In PKS master node root mode or NCP pod , execute nsxcli
nsxcli command path may different in various setup
I.E. in PKS
# /var/vcap/data/jobs/ncp/45626f1c68983490975053434143ed2eff8b138b/bin/nsxcli
81b91b91-####-####-####-##########8f> get version
2.3.1
81b91b91-####-####-####-##########8f> get ncp-nsx status
NSX Manager status: Healthy
81b91b91-####-####-####-##########8f> get ncp-master status
This instance is the NCP master
Current NCP Master id is c9a754fc-####-####-####-##########0b
Current NCP Instance id is c9a754fc-####-####-####-##########0b
Last master update at Mon Apr 1 20:11:46 2019
81b91b91-####-####-####-##########8f> get ncp-k8s-api-server status
Kubernetes ApiServer status: Healthy
To Debug NCP , change logging level from Info to Debug
81b91b91-####-####-####-##########8f> set ncp-log-level DEBUG
NCP log level is changed to DEBUG
To Stop /Start / restart NCP in PKS master node
# monit restart ncp
# monit stop ncp
# monit start ncp
To disable NCP HA
change ncp.ini and restart ncp
[ha]
enable = True to False
To change NCP HA HB and timeout timer
By default below values are hardcoded with default values, If we need change these timers then add below parameters and restart ncp.
ncp.ha.master_timeout = 9
ncp.ha.heartbeat_period = 3
[ha]
master_timeout = 18
heartbeat_period = 6
update_timeout = 6
verify if the ncp is running on the master node
#monit summary | grep ncp
Also check Hyperbus status on ESXi host
vESXi-1.nsx-lab.local> get hyperbus connection info
VIFID Connection Status
acbffb45-####-####-####-##########af 169.254.1.10:2345 HEALTHY
Feedback
Yes
No
Powered by
