High IOP on VM's
search cancel

High IOP on VM's

book

Article ID: 411169

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

VMs running on ESXi hosts experience very high storage latency. 

The servers in use are Cisco UCS blade servers.

Additionally the following fnic abort loglines are observed in the ESXi host vmkernel logs:
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu50:2101896)nfnic: <2>: INFO: fnic_taskMgmt: 2184: TaskMgmt abort sc->cdb: 0x28 sllid: 0xffffffffffffffff
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu50:2101896)nfnic: <2>: INFO: fnic_abort_cmd: 3862: Abort cmd called for Tag: 0x488  issued time: 180491 ms CMD_STATE: FNIC_IOREQ_CMD_PENDING CDB Opcode: 0x28  sc:0x########0c40 flags: 0x3 lun: 254 target: 0x######
2025-09-09T10:37:28.381Z Wa(180) vmkwarning: cpu50:2101896)WARNING: nfnic: <2>: fnic_abort_cmd: 3877: Abort for cmd tag: 0x488 in pending state
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu192:2099112)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1858: io_req: 0x########d98 sc: 0x########c40 tag: 0x488 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x$
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu192:2099112)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2384: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-09-09T10:37:28.381Z Wa(180) vmkwarning: cpu192:2099112)WARNING: nfnic: <2>: fnic_fcpio_itmf_cmpl_handler: 2415: abort reject received id: 0x488
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu192:2099112)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2456: io_req: 0x########d98 sc: 0x#######c40 id: 0x488 CMD_FLAGS: 0x73 CMD_STATE: FNIC_IOREQ_ABTS_PENDING hdr status: FCPIO_ITMF_REJECTED ABTS cmpl received
2025-09-09T10:37:28.381Z Wa(180) vmkwarning: cpu50:2101896)WARNING: nfnic: <2>: fnic_abort_cmd: 4015: Tag: 0x488 ABTS_STATUS: 0x47, FLAGS: 0x273, STATE: 0x3
2025-09-09T10:37:28.381Z In(182) vmkernel: cpu50:2101896)nfnic: <2>: INFO: fnic_device_reset: 4511: Device reset: tport: 0x######## lun_data: 0x########

Environment

VMware ESXi 8.x

Cause

This is a result of hardware/cabling/sfp faults between the UCS server chassis and the storage array. 

Resolution

Please consult your server hardware and storage vendors to continue troubleshooting this physical layer issue.

Additional Information

Powered by Wolken Software