Joining second connector to IDP fails with the error: java.net.SocketException: Broken pipe (Write failed)

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Unable to join second node to IDP cluster.
• In the /var/log/vmware/horizon/horizon.log file, you see stack trace similar to:

2018-06-28 07:12:09,954 WARN (Replication Thread) [;;;] net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator - Unable to send message to remote peer. Message was: error marshalling arguments; nested exception is:
 java.net.SocketException: Broken pipe (Write failed)
java.rmi.MarshalException: error marshalling arguments; nested exception is:
 java.net.SocketException: Broken pipe (Write failed)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:157)
 at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
 at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
 at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
 at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
 at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.SocketException: Broken pipe (Write failed)
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
 at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
 at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1915)
 at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
 at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:290)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:152)
 ... 5 more

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
 

Environment

VMware vRealize Automation 7.3.x

Resolution

To resolve this issue, replace the following files with the 56635_ehcache-731.zip file attached to this Knowledge Base article in all the vRA nodes.

Important: Ensure to backup the old files before replacing.

1. To enable the ehcache port 40003 in vRA 7.3.1 firewall rules, replace the 03-vidm-cluster-access-iptables file:

/etc/bootstrap/everyboot.d/03-vidm-cluster-access-iptables

2.  Run these commands to ensure that the permission are same as previous file:

chmod 700 /etc/bootstrap/everyboot.d/03-vidm-cluster-access-iptables
chown root:root /etc/bootstrap/everyboot.d/03-vidm-cluster-access-iptables

3. If hostname points to loopback address, it can cause ehcache replication issue. ​To avoid deletion of entry in etc/hosts, replace the setenv.sh script to set a property: 

/opt/vmware/horizon/workspace/bin/setenv.sh

4. Run these commands to ensure that the permission are same as previous file:

chmod 700 /opt/vmware/horizon/workspace/bin/setenv.sh
chown horizon:www /opt/vmware/horizon/workspace/bin/setenv.sh

5. ​Restart the horizon-workspace and reboot the appliance.