Not enough memory was available for the PacketCapture process. This can happen regardless of the memory available, because PacketCapture is a single process, limited by 32-bit memory space.
One way to remedy the memory is to reduce the number of packets kept in memory. Reducing this number too low can cause packets to be dropped if traffic is high.
From the Advanced Server Settings, set:
PacketCapture.NUMBER_BUFFER_POOL_PACKETS to 400000 default is 600000.
For more details, see KB 42606 Packet Capture Buffer and Memory Usage.