PAM appliances have root disk in readonly

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

For some unknown reason, a PAM appliance has become inaccessible through http, even though it is up and running and it can be pinged. The appliance has Remote debugging on and the ssh debug patch installed, and is therefore accessible through ssh access

After issue was reported to Broadcom Support, they did ssh to the appliance experiencing problems, and upon logging in they were met with error messages about filesystem being in read only mode

On issuing mount -l, the following was reported for the /dev/loop5 filesystem

/dev/loop5 on / type ext3 (ro,noatime,errors=panic,data=ordered) [ROOT]

As a result no logs were being written, no db replication was taking place in case this was a cluster

Running uptime revealed the machine had rebooted some minutes ago

Cause

This is a standard protection mechanism in UNIX, so this is not caused by PAM. In the event of a catastrophic situation, like for instance losing access to the OS disk, or in the case of virtualization, the filesystem on which the PAM appliance disk is sitting becomes read only due to an internal process (for instance vMotion), or if the machine has been forcefully shut down in the middle of operation, the OS may detect at boot time or even during regular operation, that the mounted root filesystem is in danger of inconsistency or has errors which prevent proper operation.

In these circumstances Linux/UNIX will boot the machine with the root filesystem in read only mode. It is the customer's responsibility to remediate the situation that caused this to happen and eventually bring the filesystem consistent again. In most cases this is achieve by just rebooting the machine after having solved the underlying problem (for instance, if there was an outage which disconnected the disk from the OS itself, after connection has been restored)

Resolution

Reboot the CA PAM appliance and see if the problem goes away.

If it does not, and since the CA PAM appliance cannot be mounted from the outside and no recovery environment can be provisioned, it will be necessary to redeploy it