Cannot access internal servers with ZTNA managed devices

Products

• Symantec ZTNA

Issue/Introduction

ZTNA solution purchased to replace existing VPN solutions for remote users.

Remote users accessing internal corporate servers via ZTNA segment applications using WSS Agents.

ZTNA admin successfully deployed a new connector in one GEO location with access to a number of internal servers.

Users accessing some of these internal applications getting network errors despite successfully authenticating with SAML.

ZTNA forensic logs do not seem to include any information about the user even though they authenticated.

Environment

WSS Agent.

ZTNA segment applications.

SAML authentication.

 

Cause

Firewall between the connector and internal server blocking access to the resources.

Resolution

Make sure that any firewall sitting between the ZTNA connector and the internal servers all traffic in both directions.

This is a common problem with segment applications and the troubleshooting steps below can help identify root cause.

Additional Information

When troubleshooting connectivity issues with segment based applications, the Symdiag from the WSS Agent will confirm that the traffic is going into the ZTNA service via Cloud SWG (check in-tunnel PCAP and verify requests to the internal server users are accessing are visible, and include the reserved flag in the IP headers).

Of more importance is the ability to take PCAPs on the ZTNA connectors. When the opportunity arises, tcpdump can be used to verify connectivity to the back end (do not just a run a PING request which uses ICMP, as firewalls may allow this but block UDP/TCP applications instead). ZTNA connectors typically run within containers and there may be a need to manually install the tcpdump package; alternatively one could install curl/openssl/telnet/netcat to check on connectivity from this host to secure applications.

In the above scenario, the connector PCAP showed the TCP handshake completed but the SSL handshake did not. As soon as the TLS client_hello was sent, no TCP ACK for the request was received leading to multiple retransmissions before an eventual failure. This would typically indicate an issue between the connector and secure server and the focus was on understanding what devices site in between these two nodes that may be looking at the SSL handshake.

As soon as we looked at the firewall logs, we could see discards for our failing user sessions. Adding an allow rule for the communication fixed the issue.

Note that

• when connectivity fails like this, the forensic logs do not include details on this. A request to improve this logging has been made to the ZTNA development team.
• COnnectivity errors such as this may also show up TCP RST in response to an outgoing request. In such cases and when PCAPs are available, check the TTL in the IP header of the TCP RST and see if the value is different to that seen with the ACK/SYN  from the original TCP handshake. In cases where it is different, it is likely that the TCP RST is coming from another device, and not the back end server.