Troubleshooting hung DevTest IAM process or other components

Products

• Service Virtualization

Issue/Introduction

Troubleshooting a hung DevTest IAM service or other components.

Environment

All supported DevTest releases.

Cause

A customer recently had Linux patches applied to their DevTest staging environment servers.

Ever since the patches were applied and the machines rebooted, IAM will not start.

The problem was the IAM process was in a hung state after the server re-boot.

Resolution

Steps to troubleshoot:

1. First ran this command to see what DevTest components were running:  ps -aef | grep java

2. The only service running was the Enterprise Dashboard.

3. Next cd /tmp folder.  The /tmp folder is a temporary landing place for files on LINUX.  When DevTest starts a service it will create a file which holds a process ID in the operating system /tmp directory.  This file will have have a name of the form:

prefix: "i4jdaemon_"
suffix: the absolute path to the start script with path part separators replaced with underscores
Example: "/tmp/i4jdaemon__opt_lisa_CA_IdentityAccessManager_bin_IdentityAccessManagerService"

4. We could see the above file in the /tmp folder, so we knew IAM was in a hung state because we could see it was not running, but LINUX thinks it is.

5. We removed the /tmp/i4jdaemon__opt_lisa_CA_IdentityAccessManager_bin_IdentityAccessManagerService file.

6. There was also a file for the Portal Service, so removed that file too.

7. Brought down the Enterprise Dashboard Service.

8. Cleaned out all the log files for both DevTest and IdentityAccessManager.

9. Started IAM with no errors.

10. Started Enterprise Dashboard, Registry, Portal and VSE services with no errors.

11. From the Browser brought up IAM, Enterprise Dashboard and Portal with no issues.

Commands we used for troubleshooting

ps -aef | grep java   (lists java processes running)
rm -fr *                    (deletes all files in a folder)

Additional Information

This kind of troubleshooting can be done for every DevTest service, this example was for the IAM in a hung state. 

for example:

i4jdaemon__opt_ca_lisa_DevTest_bin_EnterpriseDashboardService

i4jdaemon__opt_ca_lisa_DevTest_bin_RegistryService