|
Adapter Installation Suggestions:
- Install the Network adapter in a slot that matches or exceeds the bus width of the adapter.
- Example 1: if you have a 32 bit PCI adapter put it in a 32 or 64 bit PCI or PCI-X slot.
- Example 2: if you have a 64 bit PCI-X adapter put it in a 64 bit PCI-X slot.
- Example 3: if you have a x4 PCIe adapter put it in a x4, x8, or x16 PCIe slot.
Note: Some PCIe slots are physically wired with fewer channels than the dimensions of the slot would indicate. In that case a slot that matches a x8 dimensions would have the functionality of a x4, x2 or x1 slot. Check with your system manufacture.
- For PCI and PCI-X, install the Network Adapter in the fastest available slot.
- Example 1: if you have a 64 bit PCI adapter put it in a 66 MHz 64 bit PCIslot.
- Example 2: if you have a 64 bit PCI-X adapter put in a 133 MHz (266 or 533 if available) 64 bit PCI-X slot.
Note: The slowest board on a bus dictates the maximum speed of the bus. i.e. If you install a 66MHz and a 133 MHz add-in card in a 133 MHz bus then all devices on that bus will function at 66 MHz.
- Try to install the adapter in a slot on a bus by itself. If add-in cards share a bus they will compete for bus bandwidth.
- For Intel® PRO/10 GbE Network Adapters confirm If your BIOS has an MMRBC (Maximum Memory Read Byte Count) adjustment, change it from its default (usually 512) to 4096 (maximum). If your BIOS does not support changing or modifying MMRBC, refer to the "Getting the Most From Your 10 GbE Adapter" section and "Setting MMRBC" topic in the User Guide for additional information. (After opening the User Guide, you can use the keyword MMRBC to quickly find this topic.)
Note: Some systems may set the adapter's MMRBC to 4096 by default. Driver Configuration Suggestions:
- Reduce Interrupt Moderation Rate to Low, Minimal, or Off
- The default is "Adaptive"
Note: Decreasing Interrupt Moderation Rate will increase CPU utilization.
- Enable Jumbo Frames to the largest size supported across the network (4KB, 9KB,or 16KB)
- The default is "Disabled"
Note: Enable Jumbo Frames only if devices across the network support them and are configured to use the same frame size.
- Disable Flow Control.
- The default is "Generate & Respond"
Note: Disabling Flow Control may result in dropped frames.
- Increase the Transmit Descriptors buffer size.
- The default is "256". Maximum value is 2048.
Note: Increasing Transmit Descriptors will increase system memory usage.
- Increase the Receive Descriptors buffer size.
- The default is "256". Maximum value is 2048.
Note: Increasing Receive Descriptors will increase system memory usage. TCP Configuration Suggestions:
- Tune the TCP window size
Note: Optimizing your TCP window size can be complicated as every network is different. There are many documents that are available on the Internet that describe the considerations and formulas that you should use in determining your window size. Using your preferred search engine you can find a significant amount of information on TCP window size tuning by searching for "TCP tuning".
Teaming Considerations and Suggestions:
When teaming multiple adapter ports together to maximize bandwidth, the switch needs to be considered. Typically, dynamic or static 802.3ad link aggregation is the preferred teaming mode. However, this teaming mode demands multiple contiguous ports on the switch. Consideration should be given to port groups on the switch. Typically, a switch has multiple ports grouped together which are serviced by one PHY. This one PHY may have a limited shared bandwidth for all the ports which it supports. An example would be a 16 port switch with 4 groups of 4 ports and each group of ports supported by a single PHY with a 2 Gbps bandwidth shared across the 4 ports. This is not the backbone speed of the switch which is usually much higher.
If teaming 4 ports on Intel(R) gigabit network adapters or LANs on motherboard together in an 802.3ad static or dynamic teaming mode, and the switch has shared bandwidth across contiguous ports, the performance gain may be limited to the bandwidth which is shared (in the above example, 4 gigabit ports would share a total PHY bandwidth of 2 Gbps. Whether switch ports are grouped together is dependent on the switch manufacturer and model and varies from switch to switch.
If such a performance limitation is encountered after teaming adapters, network administrators may consider alternative teaming modes, for instance ALB (Adaptive Load Balancing including Receive Load Balancing by its default setting). ALB has no demands on the switch and does not need to be connected to contiguous switch ports. If the link partner has port groups, an ALB team can be connected to any port of the switch, distributing the connections across the available port groups on the switch and increasing the overall network bandwidth which can be achieved.
Performance Testing Considerations:
- When copying a file from one system to another (1:1) throughput will be significantly lower than performing simultaneous transfers to/from multiple clients. Low throughput performance on 1:1 networks is due to latency inherent in a single TCP/IP session.
- When copying a file from/to one system to another, the hard drives of each system may be a significant bottle neck. Consider using high RPM, higher throughput hard drives, striped RAIDs, or RAM drives in the systems under test.
- It's recommended that the systems under test connect through a full-line rate, "non-blocking" switch.
- Theoretical Maximum Bus Throughput
- PCI Express (PCIe) Theoretical Bi-Directional Bus Throughput
| PCI Express Implementation |
Encoded Data Rate |
Unencoded Data Rate |
| x1 |
5 Gb/sec |
4 Gb/sec (0.5 GB/sec) |
| x4 |
20 Gb/sec |
16 Gb/sec (2 GB/sec) |
| x8 |
40 Gb/sec |
32 Gb/sec (4 GB/sec) |
| x16 |
80 Gb/sec |
64 Gb/sec (8 GB/sec) | | Note:There are 1,024 Megabits (Mb) in 1 Gigabit (Gb). There are 8 bits in 1 Byte.
- PCI & PCI-X Bus Theoretical Bi-Directional Bus Throughput
| Bus and Frequency |
32-Bit Transfer Rate |
64-Bit Transfer Rate |
| 33-MHz PCI |
1,064 Mb/sec |
2,128 Mb/sec |
| 66-MHz PCI |
2,128 Mb/sec |
4,256 Mb/sec |
| 100-MHz PCI-X |
Not applicable |
6,400 Mb/sec |
| 133-MHz PCI-X |
Not applicable |
8,192 Mb/sec | | Note: There are 1,024 Megabits (Mb) in 1 Gigabit (Gb). There are 8 bits in 1 Byte.
Operating System:
This applies to:
|
