Kubernetes SpectroSERVER MySQL fail: io_setup() failed with EAGAIN.

Products

• Network Observability

Issue/Introduction

While spinning up multiple SpectroSERVER pods the installations may fail after the first two with the following

$SPECROOT/Install-Tools/LOGS/<version>_<date>/postinst.xx.xx

********** Started: <date-time> ********** running Custom Script Install-Tools/CUS/mysql.cus  Error(s) occurred. Please see /usr/Spectrum/Install-Tools/LOGS/<version>_<date>/mysql.log. Script Install-Tools/CUS/mysql.cus has FAILED ********** Completed: <date-time> **********

The mysql.log shows

$SPECROOT/Install-Tools/LOGS/<version>_<date>/mysql.log

<date-time> 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. <date-time>  1 [Warning] [MY-012582] [InnoDB] io_setup() failed with EAGAIN. Will make 5 attempts before giving up. <date-time>  1 [Warning] [MY-012583] [InnoDB] io_setup() attempt 1. <date-time>  1 [Warning] [MY-012583] [InnoDB] io_setup() attempt 2. <date-time>  1 [Warning] [MY-012583] [InnoDB] io_setup() attempt 3. <date-time>  1 [Warning] [MY-012583] [InnoDB] io_setup() attempt 4. <date-time>  1 [Warning] [MY-012583] [InnoDB] io_setup() attempt 5. <date-time>  1 [ERROR] [MY-012584] [InnoDB] io_setup() failed with EAGAIN after 5 attempts. <date-time>  1 [ERROR] [MY-012954] [InnoDB] Cannot initialize AIO sub-system <date-time>  1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Generic error. You may need to delete the ibdata1 file be fore trying to start up again.

Environment

Kubernetes

Cause

The error EAGAIN typically indicates that the number of asynchronous I/O (AIO) events exceeded the maximum number allowed on the Kubernetes worker node.

Resolution

Increase the default number of allowed asynchronous events allowed by adjusting the /proc/sys/fs/aio-max-nr kernel parameter on the worker node host.


Temporary Change (default 65536)

    sudo sysctl -w fs.aio-max-nr=1048576

Check 

  cat /proc/sys/fs/aio-max-nr


Verify that the value reflects the newly changed value and the new pod comes up cleanly. 


NOTE: This change will not persist reboots. Please have your linux admin modify the /etc/sysctl.conf file to make the change permanent.