It is possible to receive message "XCOMN0298E Unable to allocate remote transaction program: Txpi 211: Socket connect error return value = 10061" when you perform a loopback transfer using the Windows DNS name. The transfer is successful if you use the actual IP address or IP 127.0.0.1 for the same transfer.
You may also receive this message sending or receiving files from this server.
Other possible causes:
2) Invalid Host name
3) Incorrect port
4) resolver cache has old information
XCOM r11.6 for Windows
1) It is possible that the DNS name being used is resolved as an IPV6 address instead of an IPV4 address. You would have to double check with your network team if that is the case.
If that is the case, then you want to make sure that you are using the IPV6 Port of 8046, which is the default, and also check the value of "Choose listeners" in your TCPIP section of your Global Parameters tab via the GUI. You may want to set it to "All listeners". Keep in mind that you will have to stop/start the XCOM service after making any changes to the Global Parameters.
2)Verify the host name corresponds to the IP address - Does it work if you specify the IP address?
3) Check for a firewall
4) Incorrect port
5) Try flushing the name resolver cache in the local box
See Microsoft Reference Windows Sockets Error Codes