MTU Change on a vSphere Environment

A jumbo frame is an Ethernet frame with a payload greater than the standard maximum transmission unit (MTU) of 1,500 bytes.

Jumbo frames allow ESXi to send larger frames out onto the physical network . Jumbo Frames can significantly improve network performance and decrease the amount of CPU utilization on a server especially when using ISCSI storage.

Jumbo frames are becoming more and more popular as they are heavily used in VSAN and NSX environments where the later requires a minimum MTU value of 1600.

The network must support jumbo frames end-to-end. Jumbo frames up to 9kB (9000 bytes) are supported. Before enabling Jumbo frames, check with your hardware vendor to ensure that your physical network adapter supports jumbo frames.

MTU should be changed end-to-end over the environment so that you can benefit from its value without any unexpected behavior. The below guideline should be followed to do so:

Change MTU to 9K end-to-end on the physical network. This include switches, physical routers and firewalls and IPS devices if available through the communication data path between servers.
Change MTU on VDS distributed switches.
Change MTU on ESXi vmKernel ports (MGMT, vMotion, …).
You are done.

You can test MTU change by issuing vmkping commands (between vmKernel interfaces) with “don’t fragment” option and MTU size 8972 bytes leaving 20 bytes for IP header and 8 bytes for ICMP header.

SSH to one of the ESXi servers and use the below command:

vmkping -I vmk<x> -d -s 8972 <IP Address of destination vmKernel Port>

where:

vmk<x>:stands for the source vmkernel port ID.

-d: Don’t Fragment option.

-s: used to specify the size of the packet to be transmitted (8972 in our case).

Below are results before/after MTU change:

Before MTU Change:You can notice that with an MTU 1500, you can’t ping between vmKernel interfaces with a packet size 8972 without defragmenting.
After MTU Change: