HTTP 504 Gateway Timeouts in GLPI Application due to Database I/O Saturation and Agent Storm

Products

• VMware Data Services Manager

Issue/Introduction

Users experience the following issues when attempting to access the GLPI application or managing the underlying database:

• The GLPI web interface is unresponsive and returns persistent HTTP 504 Gateway Time-out errors.

• OpenShift pod logs for the GLPI application display continuous [proxy_fcgi:error] (70007)The timeout specified has expired messages.

• The PHP-FPM worker pools are completely exhausted, with logs showing WARNING: [pool www] seems busy (you may need to increase pm.start_servers...).

• The VMware Data Services Manager (DSM) UI incorrectly reports a constant 20-50 "Slow Queries per Second" even when the application is shut down.

• The MySQL database container in DSM unexpectedly restarts with Exit Code 137 (OOM Killed).

Environment

• Application: GLPI (running on OpenShift / Kubernetes, utilizing PHP-FPM and Nginx/HAProxy Ingress)

• Database Platform: VMware Data Services Manager (DSM) 2.2.2

• Database Engine: MySQL 8.0.39 (Group Replication enabled)

Cause

This issue is the result of a two-stage cascading failure between the database infrastructure and the application layer:

1. Database Storage I/O Contention & OOM Crash:

   • The database sidecar default-full-backup-binlog-collector experiences intermittent network connection refusals when attempting to reach the external S3 backup target. This failure prevents near-real-time streaming and causes a massive backlog of local binary log (binlog) files to accumulate on the secondary DB node.

   • When connectivity is restored, an event-driven binlog catch-up/purge loop triggers. If this intensive, multi-hour purge loop overlaps with a scheduled full hot backup (Percona XtraBackup), the simultaneous processes cause a severe storage I/O and memory saturation event.

   • This saturation causes the primary MySQL node to hang during disk flushes and hit an Out-of-Memory (OOM) condition, momentarily severing application database connections.

   • Note on False Slow Queries: If log_queries_not_using_indexes = ON is configured in DSM, sub-millisecond internal health checks (e.g., telegraf-metrics) flood the slow query log, creating phantom spikes in the DSM UI.

2. Application-Layer Agent Storm (Self-Inflicted DDoS):

   • During the brief database outage, thousands of endpoint GLPI-Agent clients queue their XML/JSON inventory payloads locally.

   • Once the database recovers and the application comes back online, an uninterrupted storm of POST /front/inventory.php requests from the agents instantly hits the OpenShift pods.

   • This flood completely starves the PHP-FPM worker pool. Legitimate web UI requests from human users sit in the proxy queue until they hit the Nginx/Apache timeout limit, resulting in the 504 Gateway Timeout.

Resolution

To fully resolve the issue, actions must be taken at both the application and infrastructure layers.

Phase 1: Immediate Application Recovery (Clear the 504 Errors) Choose one of the following methods to clear the GLPI-Agent queue and restore web access:

• Option A (Scale Pods): Temporarily scale up the OpenShift glpi-app deployment replicas (e.g., from 8 to 20+ pods). This provides enough PHP-FPM workers to absorb and process the queued inventory payloads. Once the agent backlog clears and pod memory drops to normal idle baselines (~100-150 MB), scale the deployment back down to standard production limits.

• Option B (Ingress Block): Apply a temporary 403 Forbidden rule at the Nginx/HAProxy Ingress layer specifically for requests matching the URI POST /front/inventory.php or the User-Agent GLPI-Agent. This will drop the traffic before it hits PHP-FPM, immediately freeing workers for human users.

Phase 2: Database and Infrastructure Remediation To prevent the underlying database crash and I/O storm from recurring:

1. Fix S3 Backup Target Connectivity: Investigate and resolve the network routing, firewall rules, or DNS resolution issues between the DSM database subnet and the external S3 backup target. Continuous connectivity prevents binlogs from backlogging, completely eliminating the grueling catch-up purge loops.

2. Adjust Backup Scheduling: Reschedule the daily full hot backup (Percona XtraBackup) to off-peak business hours (e.g., 02:00 AM) to ensure it does not compete with application read/write traffic or unexpected administrative tasks.

3. Disable Index Logging: In the VMware DSM UI, navigate to the Database Options and set log_queries_not_using_indexes = OFF to stop internal health checks from flooding the slow query log and consuming unnecessary disk I/O.