Aria Config Functionalities fail after Redis OS patch / Upgrade
search cancel

Aria Config Functionalities fail after Redis OS patch / Upgrade

book

Article ID: 407542

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • When the Aria Config / saltstack infra has the Redis node OS patched / upgraded, it is observed that
    • The compliance Vulnerability assessment data doesn't load. Prompted alert - "You have no compliance policies."
    • Other Activities and Job data failed to load - No Job data
    • Redis Service is running as expected. 
    • Raas logs : /var/log/raas/raas show error "connection to redis server lost" 

Environment

Aria Automation Config 8.x

Cause

  • The ACL user info used to access raas service was reset on the redis server, with the RHEL OS patch / upgrade and thus the user in the conf for raas is no longer usable. 
  • The Redis ACL users are not held in the redis.conf and with the os patching, the ACL user info is reset.
  • When attempting to connect to the redis service using the username and password at hand, failure encountered. 
    redis-cli -h <redis-host> -p 6379 --user <username> --pass <password> PING
    Is expected to return PONG, however, returns error:
    (error) NOPERM this user has no permissions to run the 'auth' command or does not exist

Resolution

Pre OS Upgrade safeguards:

  • ACL File Backup:
  • Identify the location of the ACL file. Typically, this is specified in the redis.conf (/etc/redis.conf  or /etc/redis/redis.conf) file using the aclfile directive.
    or use command:
    redis-cli CONFIG GET aclfile 
  • Locate and backup the ACL file (/var/lib/redis/users.acl). This file can be used to restore the users post patching. 

Post Upgrade remediation:

  • If the users are lost post upgrade, the user may need to be recreated using:
    ACL SETUSER <redis_user_to_be_used_by_raas> ><theSecurePassword> on allkeys +@a;; ~* &*

    Where:

    • ACL SETUSER <redis_user_to_be_used_by_raas>
      Creates or updates a Redis ACL user (this will be the account RaaS uses to connect to Redis).
    • ><theSecurePassword>
      The > means add a new password for this user.
      Example: >StrongPass123 sets the password.
      If you wanted to remove all old passwords, you’d use ! before adding a new one.
    • on
      Enables the user (a newly created user is disabled by default).
    • allkeys
      Grants the user access to all keys in Redis.
      Equivalent to ~* (but included for clarity).
    • +@a
      Grants the user all commands in the admin category (@a).
      Categories in Redis ACL group commands (e.g. @read, @write, @keyspace, @admin).
      +@a gives powerful rights like config, save, flush, etc.
    • ; ; (the double semicolon)
      Separator between command categories and key patterns.
      It’s Redis ACL syntax (you can omit if not switching between sections, but some automation scripts add it).
    • ~*
      Pattern that allows access to all key names (* = wildcard).
      Without this, the user wouldn’t be able to access any keys, even if they have command rights.
    • &*
      Pattern that allows access to all Pub/Sub channels.
      (& is the pub/sub ACL selector).
    • Sample:
      ACL SETUSER my_redis_raas_user >VMware1234 on allkeys +@a;; ~* &*

  • Ensure the redis.conf on the redis server has the password mentioned :, if not add it:
    grep requirepass /etc/redis.conf  or grep requirepass /etc/redis/redis.conf
    Output:
    requirepass theSecurePassword
  • Restart Redis Service and attempt to restart raas service. 
  • Login to the UI and validate that all functionalities run as expected.

Other Post Upgrade Checks checks on redis server :

  • Check SELinux Enforcement:
    getenforce
    # If Enforcing, check audit logs for denials 
    // You may wish to set the SELinux enforcement to permissive if required. This will need a reboot of the server to apply. 
    ausearch -m avc --start today
  • Check Filrewall configuration:
    firewall-cmd --list-all
    firewall-cmd --add-port=6379/tcp --permanent
    firewall-cmd --reload
  • RHEL updates may have replaced /etc/redis.conf with a new default or added stricter defaults. Verify:
    • bind is still allowing the RaaS server IP (not just 127.0.0.1)
    • protected-mode is set correctly (no if remote RaaS access is needed)
    • requirepass is still set correctly (if authentication is enabled)
    • tls-port / port values haven’t changed
    • Example:
      bind 0.0.0.0
      protected-mode no

Additional Information

Powered by Wolken Software