service apiportal stop did not stop tomcat process.

Products

• CA API Developer Portal
• CA API Gateway

Issue/Introduction

When running the command :
    service apiportal stop
The command finishes and the ps -ef | grep tomcat shows the tomcat process is stil running.

 

Environment

API Portal 3.5 

Cause

1) For vanilla install on it is working fine. 

service apiportal start 

service apiportal stop 

    is working ok to stop the tomcat process. 

2) So there must be something specific in the environment that is the cause. 
Since the service was not working, in this environment the tomcat startup/shutdown scripts were being run directly (and run as root user) : 

In the environment currently starting / stopping tomcat via scripts in :
    /opt/Deployments/lrs/server/bin
   ./startup.sh
    ./shutdown.sh

Are being used to start and stop the tomcat server. 

We also see files in the tomcat deployment area are owned by ;
      root:root 
This is not what we expect. 

For the apiportal script used in service start and stop.  It is in /etc/init.d/apiportal 

Reviewing the script we see : 
------ 
a) apiportal script will not stop tomcat, unless tomcat was started via the same apiportal script. 
mainly this is because the script stores the <pid> of the running file in : 

the pid to stop is stored here : 

PID_FILE=$PORTAL_BASEDIR/server/var/run/apiportal.pid 

so unless started with the script, it won;t know which process to stop.. 

------ 
b) Run user 
As we saw when looking at your environment, there were some files owned by l7portal and some files owned by root 

# API Portal user and group 
PORTAL_USER=l7portal 
PORTAL_USERGROUP=portalusers 

When started via the apiportal script, the run user will be l7portal and so files will be created with that user.

When started directly via scripts (as root user) new files and log files will be created / owned by root user.

c) File ownership

Once some files (log files for example) are owned by root:root then it's not possible for l7portal user to modify them.

So there will be log files, but we also saw various deployment files as well that were owned by root user.

So once tomcat is run as root user - then it probably stops l7portal user from being able to start tomcat until the file permissions are restored.   Since l7portal user will not have access to the files. 

----- 


 

Resolution

Recovery. 

a) stop tomcat 
./shutdown.sh or other way if running via root. 

b) backup 

tar up the /opt/Deployments or similar to be sure we have original copy. 

c) Change root files to be owned by l7portal 

cd /opt/Deployments 

# check how many are owned by root : 
find . -user root -exec ls -l {} \;

Note: you may also want to compare file permissions to check they are the same as similarly places files - root user may have more restrictive file permissions that other users. 

# we can change the user for those files via : 
find . -type f -user root -exec chown l7portal:portalusers {} \; 

# A similar command can be used to change file permissions (but beware, you will need to keep a list of those files,
# after doing the above they will not be identifiable via the file owner anymore)

d) Then startup using the api portal script 

service apiportal start
# check if tomcat is running. 

service apiportal stop 

# check if tomcat is stopped. (it should also give check for exiting message as well).