Click on the Category button to get more articles regarding that product.
Packet loss with VMXNET3, NIC VMWARE,
14.09.2022, 1.9.11.0, Server 2016 ON ESXi 8.0.2, 238255
Did you update the NIC driver when migrating to ESX V8?
We recently encountered issues on a server running WSUS and Trellix ePO, which normally generate a lot of traffic in all directions.
For several days, I observed sporadic packet loss to Google’s 8.8.8.8 DNS, and WSUS clients were also failing to report back.
Since some clients were connecting over Wi-Fi access points, we initially checked those. However, we also experienced issues from VMware ESXi 8.0.2 (build 238255) and Windows Server 2016, with outgoing ICMP to Google DNS showing sporadic packet loss.
We investigated MTU path settings, CRC errors on interface, but none of these were the definitive root cause.
For now, updating the VMXNET 3 NIC driver from VMware (version 1.9.11.0, dated 14.09.2022) to the latest version available via Windows Update Online Check from the Windows Update Catalog (Broadcom version 1.9.19.0, dated 25.07.2024) seems to have stabilized the issue.
Attention: There may be a potential issue with Exchange DAG clusters when using the new NIC version (Broadcom). This could affect the cluster’s keepalive interface, as those typically have special configurations from A to Z, which might be lost if a VMware engineer does not properly track these settings.
Therefore, it may not be advisable to update all larger servers or those under heavy load, such as internal SQL/Oracle database servers or clusters. As a precautionary measure, consider scripting a dump of all NIC settings to a file before updating or taking screenshots of each NIC’s configuration.
https://www.catalog.update.microsoft.com/Search.aspx?q=1.9.19.0
VMXNET3 NIC Driver
OLD: NIC VMWARE, 14.09.2022, 1.9.11.0 (other server)
NEW: NIC BROADCOM, 25.07.2024, 1.9.19.0 (WSUS/EPO > Nach Update via Windows Update suchen Online)
Performance Best Practices for vSphere 8.0 Update 1
https://www.vmware.com/docs/vsphere-esxi-vcenter-server-80u1-performance-best-practices
READ: ESXi Networking Considerations
ESXi (Updated On: 02-05-2025)
https://knowledge.broadcom.com/external/article/324556/large-packet-loss-in-the-guest-os-using.html
List of Vmware/Broadcom Drivers which are ONLINE with Windows Update Catalog:
https://knowledge.broadcom.com/external/article/313145/vmware-tools-drivers-published-to-window.html
Possible Exchange DAG Cluster problem with NEW NIC Version (If you upgrade to Broadcom)
This may be the cluster’s communication interface, I guess, because those normally have special settings from A to Z, and these may get lost if a VMware engineer does not properly track those special settings.
HPE (2024-02-20) VNIC
https://support.hpe.com/hpesc/public/docDisplay?docId=a00137862en_us&docLocale=en_US
On HPE platforms configured with any of the HPE Broadcom-based network adapters listed in the Scope section below; the network adapter may experience network link loss on Microsoft Windows Virtual Machines after upgrading the host Operating to VMware ESXi 8.0 (or later), or after updating the network driver/firmware combination in VMware ESXi 7.x that includes bnxtnet driver version 224.0.x.x (or later).
Windows Virtual Machines(VMs) may suddenly lose connectivity to all or some network destinations; and connectivity is restored by disconnecting and reconnecting the vNIC, or migrating the Virtual Machine to another VMware ESXi host. During these operations, the vNIC may generate a message in the VMware ESXi kernel logs similar to the following:
“Vmxnet3: 21700: vmname.eth0,xx:xx:xx:xx:xx:xx, portID(xxxxxxxx): Hang detected,numHangQ: 1, enableGen: 1049”
This issue has only been reported for Windows Virtual Machines using vmxnet3 as vNIC; however, it could affect other guest Operating Systems and other vNIC types.
The affected async driver version 224.0.x.x (or later) has an issue that can miss the TX packet completion under certain circumstances. This could block the vNIC TX queues of the Virtual Machine, and thus block some or all packets leaving the vNIC.
|
On the ESXi Server itself you can check drops with: |
|
Access the ESXi Shell: • You need to be logged into the ESXi host directly or via SSH. You might need to enable the ESXi Shell or SSH if it’s not already enabled. Launch esxtop: • Type esxtop in the ESXi shell and press Enter. This command opens the esxtop utility, which provides real-time performance monitoring. Switch to Networking View: • Press the n key to switch to the networking view in esxtop. This view shows network performance statistics. Monitor Packet Drops: • Look for the columns labeled %DRPTX and %DRPRX: %DRPTX shows the percentage of packets dropped on transmission. %DRPRX shows the percentage of packets dropped on reception. |
The problem looks like this:
Download of 300MB file from MS to Server over old NIC driver:
High traffic times with a lot of incoming WSUS clients.
If its regarding MS/M365 you can always shortly use the web based version of their health tool so you can be sure that it is not their side like with the azure.com Paket Loss above.
Microsoft 365 network health status
https://connectivity.office.com/status
