Multicast traffic between two virtual machines is failing when VDS is using multicast mode as IGMP.
net-dvs-l | grep -i "multicastFilter" <--command on host, shows VDS mode as IGMP snooping
com.vmware.vswitch.multicastFilter = snooping
VMware ESXI
The IGMP/MLD Snooping mode requires virtual machines to send IGMP join/membership reports so the vDS can populate its forwarding table. If VM's in this multicast mode do not send these reports. Consequently, the vDS does not know which ports should receive the traffic and drops the multicast frame to prevent network flooding
To diagnose missing IGMP Membership Reports (Joins) originating from a virtual machine, performing a packet capture on the ESXi host switchport using command:
pktcap-uw --switchport ######## --capture VnicTx,VnicRx -o /vmfs/volumes/<DATASTORE_PATH>/vm.switchport.########.pcapng
For packet capture refer KB using-the-pktcapuw-tool-in-esxi
If the packet capture confirms that IGMP signaling is not reaching the switchport, the issue likely resides within the VM itself.
Inspect the Guest Operating System (OS) and verify the following:
Local OS Firewalls: Ensure there are no rules dropping outbound multicast/IGMP traffic.
Network Adapter Bindings: Confirm the application is bound to the correct network interface.
Multicast Routing Services: Verify that the necessary services are active, correctly configured, and successfully generating IGMP signaling onto the network stack.
For more information refer TechDocs: multicast-filtering