VM Has 8 RxQueues Only 4 Are Used
Greetings, I've been chasing an issue for over a week that I can't seem to find any answer for and was hoping somebody out there has heard of/experienced something similar.
​
We have a VM running Windows Server 2016 Datacenter and SQL 2016 Enterprise with 24 vCPU and 160GB RAM. Underlying host is a HP BL460c Gen10 with 2 Xeon 6142 (16 cores per socket) and a total of 192GB RAM. Hyperthreading is enabled on the host. We're running ESXi 6.5 P03. This host is dedicated to this VM.
​
When the server hits peak time, we start experiencing slow downs and observe that the first 4 vCPUs are pegged while the remaining 20 are busy, but still have capacity. After running through perfmon counters, we could clearly see the CPU time was being monopolized by VMXNET3 interrupts. We checked and verified that Receive Side Scaling was enabled in Windows and enabled for the network adapter. When we do Get-NetAdapterRSS, we see what appears to be 8 Receive Queues assigned to process the traffic using NUMAStatic profile (the default profile).
​
Checking the ESXi shell for network stats for the RxQueues for that adapter, we again can see there are 8 Queues available, but digging into the stats of each, only queues 0-3 have received and processed packets while queues 4-7 have no received traffic. We're not experiencing drops.
​
This VM is different from anything else in our environment since it's spread over 2 NUMA nodes (12 CPU 80GB RAM per node). It's currently configured as 1 CPU per socket x 24 sockets. The VM was created in ESXi 6.5, so it hasn't been migrated from a previous version of ESXi. Both Windows and SQL recognize that it has 2 NUMA nodes exposed as well and SQL is leveraging all 24 vCPU.
​
We have a few other VMs with 8 vCPU on different hosts that also have RSS enabled, but they appear to be using all 8 receive queues with no issues and no extra configuration required to achieve this result.
​
\*\*\* UPDATE 2/4/2019 \*\*\*
Finally performed some meaningful tests with our equipment and have come up with the following observations
1. No difference in behavior between bl460c Gen9 and bl460c Gen10 hardware. Both are using the HPE Flexfabric 20GB 2-port 650FLB adapters. 2. We can recreate the issue of having 8 receive queues, but only use 4 vCPUs despite indirection table indicating an array of 8 vCPUs listed. The issue occurs when the VM spans NUMA nodes and Windows RSS maps the first 4 CPUs from each node to process network traffic.In our example, we have a VM with 24 vCPU that's spread between 2 NUMA nodes (CPU 0-11 in Node 0 and CPU 12-23 in Node 1). RSS indirection table attempts to use CPU 0-3 and 12-15, but only ends up using CPU 0-3 to handle the received traffic. 3. To work around this issue, we manually overrode the default indirection table in Windows by issuing a **Set-NetAdapterRSS -Name \[adaptername\] -BaseProcessorGroup 0 -MaxProcessorGroup 0 -BaseProcessorNumber 12 -MaxProcessorNumber 23**. This relegates RSS to using CPUs from only the 2nd NUMA node, but does effectively allow the use of 8 CPUs.
Indirection Table Before: 0:0 0:12 0:1 0:13 0:2 0:14 0:3 0:15
Indirection Table After: 0:12 0:13 0:14 0:15 0:16 0:17 0:18 0:19
It seems like more receive queues may be possible, but this appears to be dictated by underlying hardware. In our case, we have not been able to increase our receive queues beyond 8 using the Windows GUI or PowerShell with the Set-NetAdapterRSS cmdlet
At this point we're not sure if RSS should be able to span NUMA. I've read nothing that suggests it can't, but the results we've seen have made it hard to believe that it does. VMs with 8vCPU that fit into a single NUMA node use all 8vCPU without any additional config. Adding additional vCPU leaves us capped at 8 receive queues in Windows. The amount of receive queues appears to be dictated by hardware, so we're investigating if the HPE FlexFabric 20GB 2-port 650FLB can present more than 8 receive queues.
​
\*\*\* UPDATE 2/20/2019 \*\*\*
8 receive queues appears to be the maximum allowed by the VMXNET3 adapter and is not dictated by underlying network hardware. This is a driver limitation which is documented in [Performance Best Practices forVMware vSphere 6.5](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/Perf_Best_Practices_vSphere65.pdf) in the last paragraph on pg.55
> In order to obtain the maximum performance with your specific workloads and resource availability you can try out different values for the number of receive queues (**which must be set to a power of 2 and can be a maximum of 8 or the number of vCPUs, whichever is lower**), and you can try configuring the driver to use multiple transmit queues (except in Linux, where the number of transmit queues will already match the number of receive queues). These settings are changed on the advanced driver configuration tab within the guest operating system.
This is really unfortunate for VMs that use more than 8 vCPUs and process a lot of network traffic. We haven't heard any official response about why 8 vCPUs is the limit or if there are any plans to increase this cap.
The only other option we haven't explored is implementing PCIe passthrough or possibly using SR-IOV. It seems like both of these options may allow us to directly present the network card to the OS and leverage the Windows Emulex driver directly, bypassing the hypervisor. The Emulex driver may allow for more receive queues than the VMXNET3 driver, but we haven't been able to test this.
I'll continue to post updates, but bottom line from what we can see is
1. VMXNET3 is limited to 8 receive queues (CPUs) 2. If your VM spans NUMA, it appears that you must manually dictate the indirection table behavior to avoid losing out on half of your receive queues. 3. If your VM uses 8 vCPU and does not span NUMA, then RSS on VMXNET3 will use all vCPUs to process received network traffic.
​