IOError in atexit.py while Exiting from gpssh in Interactive Mode

Products

• VMware Tanzu Greenplum
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite
• Greenplum
• Pivotal Data Suite Non Production Edition

Issue/Introduction

One of the scenario is OS upgrade from RHEL7 to RHEL8 after which this issue can pop up.

Customers may encounter an error when exiting from gpssh in interactive mode, specifically when attempting to leave the shell session. The error message appears as :

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/greenplum-db-6.25.2/ext/python/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
IOError: [Errno 22] Invalid argument

 

This error occurs in the atexit module during the exit process of the gpssh session. The issue is related to the writing process that occurs when attempting to record the session history, in the .gshist file.

 

 

Environment

All greenplum versions.

Cause

The issue error lies within the atexit module, which tries to write the history of the commands executed in the session to the .gshist file located in the user's home directory (/home/gpadmin/.gshist). This file stores the history of the gpssh session commands.

• When a gpssh session is active, commands are not immediately written to the .gshist file.
• However, when you exit the session, the system attempts to write all executed commands to this file.
• During this process, an IOError (Errno 22, "Invalid argument") is triggered, typically indicating an issue with the file write operation.

Resolution

1. The IOError is often caused by a hardware or disk-related problem, particularly if the home directory resides on a device with issues. Review the system logs for any hardware-related errors. Check the following logs:

• Run dmesg to identify any disk-related errors.
• Review /var/log/messages for any unusual messages related to the home disk device.

2. Ensure that the .gshist file is writable. If there is an issue with the file’s permissions, you may encounter the IOError when trying to write to it.

ls -l /home/gpadmin/.gshist

 

If file is not writable, correct the permissions:

chmod 644 /home/gpadmin/.gshist

 

3. The file itself may be corrupted or locked. You can try renaming the .gshist file to see if the problem persists:

mv /home/gpadmin/.gshist /home/gpadmin/gshist.old
gpssh -f hostfile
uptime
exit