Windows Authentication Agent - Named Pipes Troubleshooting

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

A crucial component of the Windows authentication method is the SSO client's ability to open a named pipe to the Windows Authentication Agent.

Named pipes use the NetBIOS session layer and may be communicated over NetBEUI or TCP/IP. Additionally, named pipes rely on the client's ability to connect to the named pipe server's inter-process communications resource, IPC$.

Connectivity diagnosis

To test NetBIOS name resolution of the authentication agent from the client, use the NBTSTAT utility:

nbtstat -a SERVERNAME

This command will iterate through your network adaptors, attempting to resolve the NetBIOS name.

Once NetBIOS name resolution has been confirmed, you may test the client workstation's ability to connect to the Windows authentication agent's IPC resource by performing a 'net use' command:

net use \\SERVERNAME\IPC$

If this command completes successfully, all is well and the SSO client should be able to use the Windows authentication method just fine.

If this command fails (or prompts you for user name and password), all is not well -- Named Pipes rely on the IPC resource to communicate.

Transport protocol

To check whether NetBIOS over TCP/IP is enabled on your machine:

1. Open the Properties of your LAN connection from Network Connections in your Control Panel.
2. View the Properties of Internet Protocol (TCP/IP).
3. Go to the Advanced dialog and select the WINS tab.
4. The NetBIOS group box will indicate your current setting.

Named Pipes Test utility

A simple test utility, nptest, can help you confirm named pipe connectivity between two machines.

On the machine that will act as the authentication agent, create a pipe named foo like so:

nptest server \\.\pipe\foo

On the machine that will act as the SSO client, open the server's pipe like so:

nptest client \\SERVERNAME\pipe\foo 

...substituting SERVERNAME for the hostname of the first machine. (Remember to escape the backslashes if using Cygwin).

Any standard input to the client should be printed to standard out on the server. (It is not currently bi-directional.) Any errors that occur will print the corresponding Windows system message to standard error. Enter an empty line to end the pipe. The nptest.exe utility is available below - it is provided on an unsupported "as seen" basis.

<Please see attached file for image>

Aliased Hosts

Windows named pipes cannot connect to an 'aliased' host. This includes HOSTS/LMHOSTS file entries and any 'smart DNS' that may be in use (including CSS/GSS).

Whether the client uses NetBIOS over TCP/IP (NetBT) or SMB, the server will check the hostname in the connection request, see that is does not match its own hostname and reject the client with the Windows error code ERROR_DUP_NAME (52d,34h). Even 'localhost' is considered an invalid alias for a client on the same machine.

To allow an alias to be used on the server, edit the registry key: 
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\lanmanserver\parameters to add a REG_MULTI_SZ string value OptionalNames. Set this to the name of the server's alias and restart the system (do not include a domain name).

Note, this is an undocumented registry key and editing this is performed at your own risk.

Relevant Windows error codes

Dec	Hex	System message	Diagnosis
2	2	The system cannot find the file specified.	The Named Pipe is not open on the server, or The server could not be found. (You may get this error instead of 53 or a connection error. To the client, named pipes are just another file.) As SSO r8 attempts to connect (in order) to a pipe on the authentication agent, the nearest domain controller, and then the local machine, you may see this error if the authentication agent was not found, but the nearest domain controller was.
52	34	You were not connected because a duplicate name exists on the network. Go to System in Control Panel to change the computer name and try again.	This error text is slightly misleading. This error occurs when the server's hostname does not match the hostname that the client used to connect to it. This will most likely arise in the following conditions: The hostname is omitted, resulting in the client attempting to connect to pipe: \\\pipe\<pipename> (the local machine) rather than \\<hostname>\pipe\<pipename>. 2. The hostname being used by the client is an alias (e.g. a HOSTS file entry or if "smart DNS" being used). The NetBIOS server always checks the hostname in the connection request against its own. If it does not match, this will be the error returned.
53	35	The network path was not found.	The NetBIOS name resolution failed. If you think this is incorrect, run nbtstat to update the NetBIOS cache and try again.
86	56	The specified network password is not correct.	Similar to 1326, check that your client workstation and authentication agent server are on the same Windows domain.
87	57	The parameter is incorrect.	Most likely the pipe name was omitted (e.g, the client tried to connect to \\<hostname>\pipe\ .)
161	A1	The specified path is invalid.	Most likely the pipe UNC name was badly formatted. e.g., the client tries to connect to <hostname>\pipe\<pipename> or \\<hostname>\foo\<pipename>. In the SSO client, this error is most likely to occur due to a bug that left leading backslashes on the domain controller name (leading the client to attempt to open a pipe to \\\\<hostname>\pipe\<pipename> ). This bug was fixed in SSO Client 8.0.0.491.
1231	4CF	The network location cannot be reached. For information about network troubleshooting, see Windows Help.	Name resolution succeeded, but the transport was unavailable. Ensure that your TCP/IP NetBIOS Helper service is running. It may be that installing the NetBEUI protocol can rectify this, although this should be unnecessary as NetBIOS should be able to travel just fine over the present TCP/IP. Different flavours of Windows may have different preferences for transporting NetBIOS.
1326	52E	Logon failure: unknown user name or bad password.	Access to IPC$ is denied. (Not the authentication agent's pipe in particular, but the IPC resource itself.) This is expected if your client is trying to connect to a server that is not in the client's Windows domain.
1355	54B	The specified domain either does not exist or could not be contacted.	The client will likely receive this from a DsGetDcName() call (when trying to find the nearest domain controller) if the machine is not a member of the Windows domain.

Related Information

http://support.microsoft.com/kb/q101150/

Environment

Release:
Component: SMPLC