MySql command not found errors seen in Portal Topology upgrade logging

Products

• Network Observability
• CA Performance Management

Issue/Introduction

When running an Upgrade Automation (UA) based Portal Topology upgrade we're seeing failures. We use the following command.

ansible-playbook -i inventory -K topology-upgrade.yaml

 

When running the following command we're generating additional logging to the CLI (-vvv), and redirecting it to a log file we're calling upgrade.txt (> upgrade.txt).

ansible-playbook -vvv -i inventory -K topology-upgrade.yaml > upgrade.txt

 

We see message entries like the following in the resulting upgrade.txt file.

"stderr": "/bin/sh: mysql: command not found"

 

The playbook is failing during the Grant privileges for topologyuser on each host (IP and Hostname) task. If you look closely at the raw execution output for the MySQL host (<MySqlHost_IP>), Ansible returns that specific error.

Why is this happening? To grant the necessary privileges, Ansible connects to the MySQL server (<MySqlHost_IP>) via SSH and attempts to run a local command-line query: mysql -u root -p... -e "CALL mysql.init_portal_user_for_host(...);".

The error means the mysql command-line client tool is either not installed on the <MySqlHost_IP> server, is not in the system PATH for the topology SSH user, or the SSH user doesn't have access to the mysql binary.

Environment

All supported Network Observability DX NetOps Portal Topology installations

Cause

The Portal MySql directories permissions were modified from default. The $PORTAL_HOME/MySql directory and all directories below it were set with 700 (rwx------) permissions.

Resolution

Correct the permissions to the minimum required 755 (rwxr-xr-x) for the $PORTAL_HOME/MySql directory and all directories below it.

All directories and executables under $PORTAL_HOME/MySql need to be 755 for this to work.

We can run this to ID all directories under $PORTAL_HOME/MySql that will require this change to permissions.

1. Go to the $PORTAL_HOME/MySql directory.
2. Run "find . -type d -print"

Run this to modify those directories from 700 to 755 for permissions.

• find . -type d -print -exec chmod 755 {}/:

Lastly run this to change the top level directory permissions.

• chmod 755 $PORTAL_HOME/MySql

Once completed, with $PORTAL_HOME/MySql and all directories under it changed from 700 to 755 for permissions, run the upgrade again.