TCA 2.2 to 2.3.0 upgrade fails during migration due to MongoSocketReadTimeoutException

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:
Upgrade from TCA 2.2.0 to 2.3.0 fails while migrating from MongoDB to PostgreSQL due to com.mongodb.MongoSocketReadTimeoutException

NOTE: After this error, upgrade does not move forward and more importantly TCA 2.2 system is not impacted.

Log Snippet from /common/logs/upgrade/upgrade.log :
 

com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:563)
at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:448)
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:299)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:259)
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:450)
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:226)
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343)
at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334)
at com.mongodb.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:731)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725)
at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:725)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:89)
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196)
at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:143)
at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:92)
at com.mongodb.client.gridfs.GridFSFindIterableImpl.iterator(GridFSFindIterableImpl.java:89)
at com.vmware.vchs.hybridity.migration.MigrationHelper.migrateObjectstoreFilesFromMongoDBToPostgres(MigrationHelper.java:273)
at com.vmware.vchs.hybridity.migration.MigrationHelper.migrateDataForGivenMongoCollectionAndValidate(MigrationHelper.java:122)
at com.vmware.vchs.hybridity.migration.MigrationManager.executeMigrationAndValidation(MigrationManager.java:129)
at com.vmware.vchs.hybridity.migration.MigrationManager.doMigration(MigrationManager.java:84)
at com.vmware.vchs.hybridity.migration.MigrationManager.main(MigrationManager.java:50)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:109)
at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:580)
at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:445)
... 26 common frames omitted

Environment

VMware Telco Cloud Automation 2.2

Cause

This happens if MongoDB to PostgreSQL migration application is unable to read from MongoDB under default socket timeout limit of 2 minutes, due to various environment related reasons .
Example: MongoDB under heavy load.

Resolution

There is no expected fix for this problem since migration from MongoDB to PostgreSQL during upgrade is a one-time effort from TCA release 2.2.0 to 2.3.0.

Workaround:
Option 1:

a. Retry Upgrade should likely fix this problem.

Option 2:

Update the MONGODB_SOCKET_TIMEOUT using the following steps:

    a. Customer access shell on the system where migration failed due to above symptom as a root user.

    b. Manually download the upgrade bundle tar ball.

    c. Extract the upgrade bundle,

    

tar -xzf VMware-Telco-Cloud-Automation-upgrade-bundle-2.3.0-21500099.tar.gz

    d. Open doMigration.sh using an editor and search for MONGODB_SOCKET_TIMEOUT=120.
    e. Currently MongoDB socket timeout for migration app is set as 120 seconds default.

MONGODB_SOCKET_TIMEOUT=120 

    f.  Increase the socket timeout parameter MONGODB_SOCKET_TIMEOUT from 120 to 240 .
       

MONGODB_SOCKET_TIMEOUT=240

    g. Retry upgrade through shell using the following command:
       

./vsm-upgrade.sh image/VMware-Telco-Cloud-Automation-image-2.3.0-21500099.img.dist