Module 4:

Troubleshooting Virtual
Networking

© 2020 VMware, Inc.

Importance
Networks are used to access or control nearly every component in the vSphere environment.
When a network problem occurs, you must quickly diagnose and resolve it to minimize
negative impacts.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-2

Learner Objectives
After completing this module, you should be able to meet the following objectives:
• Analyze and troubleshoot standard switch problems
• Analyze and troubleshoot virtual machine connectivity problems
• Analyze and troubleshoot management network problems
• Analyze and troubleshoot distributed switch problems

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-3

Networking Troubleshooting Overview
In vSphere, networking problems can occur with the following types of connectivity:
• Virtual switch connectivity:
– Standard switches
– Distributed switches
• Virtual machine network connectivity
• ESXi host management network connectivity
A vSphere administrator should know how to troubleshoot common networking problems.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-4

Standard Switches
If a virtual machine loses network connectivity, the cause of the problem might be found
anywhere from the VM's NIC to the ESXi host’s physical network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-5

Standard Switch Log Files
When troubleshooting network problems, log files can help you to identify the root cause.
These log files are [Link], [Link], [Link], and [Link].

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-6

Sample Log Entries: vmnic Status
The [Link] file reports on many events, including uplinks that go up or down and network
redundancy that is lost or restored.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-7

Standard Switch IOChain Framework
A standard switch uses the ESXi network
IOChain framework. This framework is a group
of functions that provides connectivity
between the ports and the virtual switch.
Understanding the core constructs of the
vSphere networking layers can be useful for
troubleshooting connectivity issues.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-8

Network Troubleshooting Commands
You can run commands to test network connectivity, verify packet routing, and verify DNS
records.

Command Type Description

vmkping ESXi Verifies network connectivity between a VMkernel port
and a destination IP address.
vmkping -I vmk1 [Link]
traceroute ESXi\Linux Verifies the connection route that a network packet
takes to its destination. The route includes the IP
addresses of all the forwarding entities.
traceroute [Link]
nslookup ESXi\Linux Queries DNS records for specified systems.
nslookup [Link]
nc ESXi\Linux Short for netcat. Verifies connectivity to a TCP port on
a remote system.
nc -z [Link] 3260

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4-9

vSphere Commands: esxcli network
You can manage IP configurations, physical NICs, ports, virtual switches, and more with the
esxcli network command.
Example esxcli network commands:
• Bring vmnic2 online:
esxcli network nic up -n vmnic2
• Display the ARP and network discovery cache:
esxcli network ip neighbor list
• Display information about your distributed switches:
esxcli network vswitch dvs vmware list
• Configure the IP address for the vmk0 interface:
esxcli network ip interface ipv4 set -i vmk0 -ipv4 [Link]

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 10

vSphere Commands: pktcap-uw (1)
The pktcap-uw command is an enhanced packet capture and analysis tool that you can use
instead of the legacy tcpdump-uw command.
You run the pktcap-uw command to troubleshoot dropped VM packets or disconnected uplinks.
For example, you can run the command to help determine whether packets are dropped on the
ESXi host or the external network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 11

vSphere Commands: pktcap-uw (2)
By using the pktcap-uw command, you can capture frames at the virtual port, virtual switch,
and uplink levels, for example:
• Capture packets at a VM’s virtual switch port, where the port ID is 100663317:
pktcap-uw --switchport 100663317
• Capture packets on vmk0:
pktcap-uw --vmk vmk0
• Capture packets on vmnic6:
pktcap-uw --uplink vmnic6

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 12

vSphere Commands: pktcap-uw (3)
You can monitor traffic at many points in the
host network stack by running the pktcap-uw
command:
• Capture packets sent to the virtual switch
from the virtual NIC:
pktcap-uw --switchport 100663317 --
capture VnicTx
• Pass a list of packets from a virtual switch
port to the UplinkSnd point:
pktcap-uw --uplink vmnic0 --capture
PortOutput
• Capture packets sent to the uplink:
pktcap-uw --uplink vmnic0 --capture
UplinkSnd

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 13

Network Problem: ESXi Host Connectivity Issues
As an initial check from vSphere ESXi Shell, you ping a system that is known to be up and
accessible by the ESXi host.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 14

Identifying Possible Causes
You identify possible causes of the ESXi host network connectivity problem. If you know that
your hardware is functioning correctly, you take a top-down approach to troubleshooting,
starting with the ESXi host configuration.

Component ↓ Possible Cause

ESXi Host The ESXi host network configuration is incorrect.

The VLAN ID of the port group is incorrect.
The speed and duplex of the network links are not consistent.
The network link is down.
NIC teaming is not configured correctly.

Hardware The network adapter or server hardware is not supported.

(CPU, Memory, The physical hardware is faulty or misconfigured.
Network, Network performance is slow.
Storage)

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 15

Possible Cause: vSphere Standard Switch Configuration
To determine the cause of the ESXi host's connectivity problems, you verify that your ESXi host
network is configured correctly:
• Check vSphere standard switches, vmnics, port groups, and VMkernel ports:
esxcli network vswitch standard list.
• Check the VLAN IDs of port groups:
esxcli network vswitch standard portgroup list.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 16

Possible Cause: Distributed Virtual Switch Configuration
You check another possible cause of the connectivity problems of the ESXi host by verifying
that distributed virtual switches are configured correctly:
• Check vSphere distributed virtual switches, vmnics, port groups, and VMkernel ports:
esxcli network vswitch dvs vmware list.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 17

Possible Cause: ESXi Host Network Configuration
You verify that your ESXi host network is configured correctly:
• Check speed, duplex, network uplink, and NIC status (up or down):
esxcli network nic list.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 18

Resolving ESXi Network Misconfiguration
You can adjust the settings in your ESXi network configuration if they are not configured
correctly:
• List VMkernel network interfaces:
esxcli network ip interface list
• Enable and configure MTU:
esxcli network ip interface set –i <interfacename> -e true -mtu <mtu_size>
• Get IP address assigned to VMkernel:
esxcli network ip interface ipv4 get –i <interfacename>
• Set the IP address assigned to VMkernel:
esxcli network ip interface ipv4 set –i <interfacename> –ipv4 static
<address>
• Configure a VLAN:
esxcli network vswitch standard portgroup set –p <pg_name> -v <vlan_ID>

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 19

Possible Cause: NIC Teaming Misconfiguration
You verify that NIC teaming is configured correctly.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 20

Possible Cause: Unsupported or Faulty Hardware
You check whether the following ESXi network hardware issues exist:
• The network adapter or server hardware is not supported: Verify that the network hardware
is listed in the VMware Compatibility Guide.
• The physical hardware is faulty or misconfigured: Run esxcli network nic list.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 21

Possible Cause: Slow Network Performance
In the vSphere Client, you can display advanced traffic load information by selecting Monitor
> Performance > Advanced > Network.
You can isolate the traffic load by vmnic interface.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 22

Review of Virtual Machine Connectivity
If your VM loses network connectivity, the cause of the problem might be in the physical layer,
the virtual layer, or the guest operating system itself.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 23

Network Problem: VMs with No Network Connectivity
As an initial check, you ping the VM from another system.
If the ping command fails, you ping other VMs on the same network to determine the scope of
the problem.
You ping the name first and then the FQDN and the IP address. The result of the ping can help
you narrow the scope of the problem.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 24

Identifying Possible Causes
You can use the top-down approach to troubleshoot the problem. You start from the guest
operating system and work your way down to the ESXi host.

Component ↓ Possible Causes

Application or Guest OS The IP settings are misconfigured.

The firewall in the guest OS is blocking traffic.

Virtual Machine The port group name does not exist.

The virtual network adapter is not connected.

ESXi Host Underlying issues with the ESXi network connectivity exist.
Storage or resource contention occurs on the ESXi host.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 25

Possible Cause: IP Settings and Firewall Problems
Misconfigured IP settings and problems with firewalls might cause the VM network connectivity
problem.
You verify that the guest operating system network configuration (TCP/IP setting) is correct.
The firewall in the guest operating system might be blocking traffic. You should ensure that the
firewall does not block required ports.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 26

Possible Cause: Port Group Misconfiguration
The port group name that the VM uses is incorrect:
• View the distributed switch port group IDs on the ESXi host:
esxcli network vswitch dvs vmware list
• Correlate the port group ID with the name of the port group:
dcli network list
• Verify that the network adapter is connected to the correct port group.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 27

Possible Cause: ESXi Network Connectivity Problems
Storage or resource contention on the ESXi host can cause network connectivity issues:
• Verify that the VM has no underlying issues with storage and that it is not in resource
contention.
• Look for problems in the ESXi host network configuration, for example, port group ID, speed,
duplex settings, physical network link, and NIC teaming configuration.
• To eliminate a NIC failure or a physical configuration issue, connect the VM to a virtual
switch that uses NIC teaming.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 28

Network Problem: An ESXi Host Frequently Disconnects from vCenter Server
The ESXi host disconnects within 30–90 seconds, after it is added to the vCenter Server
inventory.
The problem is that dropped, blocked, or lost heartbeat packets are occurring between vCenter
Server and the ESXi host.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 29

Heartbeat Communication Between vCenter Server and ESXi
The ESXi host sends a heartbeat to vCenter Server to signal that the host is accessible by the
management network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 30

Identifying Possible Causes
You can use the top-down approach to troubleshoot the problem. You start from vCenter Server
and work your way down to the physical hardware.

Component ↓ Possible Cause

vCenter Server UDP port 902 is blocked on vCenter Server.
ESXi Host Ports are blocked by the firewall.
Hardware The network between the ESXi host and vCenter Server is
(CPU, Memory, congested.
Network, and Storage)

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 31

Possible Cause: Port Blocked by Firewall
If the firewall is enabled and UDP port 902 is blocked, you view the ports blocked by the
vCenter Server Appliance firewall.
To resolve this problem, you adjust the firewall settings on the vCenter Server Appliance virtual
machine:
• If ports are not configured, disable the firewall.
• If the firewall is configured to affect ports, ensure that the firewall is not blocking
UDP port 902.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 32

iptables Firewall
In vCenter Server Appliance, the iptables
firewall can be used in the following ways:
• Examine firewall rules from the command
line or an SSH session:
iptables –L --line-numbers
• Remove firewall rules.
• Modify firewall rules.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 33

Possible Cause: vCenter Server Not Using Port 902
By default, the vpxa agent on the ESXi host
sends heartbeats to vCenter Server (vpxd)
through UDP port 902.
A problem might occur if the host is configured
to send heartbeats over a port other than 902.
To determine the port that is used for sending
heartbeats, you use the
/etc/vmware/vpxa/[Link] file.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 34

Checking Heartbeat Port Number and Creating New Firewall Rule
If you prefer to use a nondefault port for heartbeats, the ESXi firewall must not block that port.
You edit the /etc/vmware/firewall/[Link] file.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 35

Changing the Heartbeat Port Number
To change the heartbeat port number, you can edit the Advanced vCenter Server Settings.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 37

Resolving Network Congestion
To resolve network congestion, you can apply short-term and long-term solutions:
• Short-term solution:
– To keep the ESXi host connected, increase the timeout limit in vCenter Server.
• Long-term solution:
– Resolve the underlying network congestion problems.
– If using distributed switches, use Network I/O Control to reprioritize traffic and increase
the number of shares for management traffic.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 38

Network Problem: vCenter Server Cannot Manage ESXi Host
When vCenter Server cannot manage an ESXi host, the ESXi host’s management network
might be misconfigured or manipulated from the command line.
For example, you can bring a physical network card up or down by running these esxcli
commands:
esxcli network nic up –n vmnic0
esxcli network nic down –n vmnic0
esxcli network nic list

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 39

Preventing Loss of Management Network Connectivity
A vSphere network rollback prevents an
accidental misconfiguration of management
networking and loss of connectivity.
For example, if you try to change the IP
address of your management VMkernel
interface, the vSphere Client returns an error
message.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 40

About Host Networking Rollback
If a host loses network connectivity as a result of a change in the network configuration, a
network configuration rollback is triggered on the host.
Several events can trigger a host networking rollback:
• Updating DNS and routing settings
• Updating the speed or duplex of a physical NIC
• Changing the IP settings of a management VMkernel network adapter
• Updating teaming and failover policies to a port group that contains the management
VMkernel network adapter
If a network disconnects for any of these reasons, the task fails, and the host reverts to the last
valid configuration.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 41

Management Network Connectivity
ESXi communicates with vCenter Server over
the management network.
ESXi loses the connection to vCenter Server if
the management network goes down:
• The IP address of the VMkernel port was
modified.
• The VLAN configuration was changed.
• The physical NIC failed.
To prevent a lost connection between an ESXi
host and vCenter Server, you can build
redundancy into the management network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 42

Recovering Lost Management Networks: Standard Switch
If the management network is on a standard switch, and you lose management network
connectivity, you can resolve the problem by using the Configure Management Network option
in the DCUI.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 43

Network Restore Options in the DCUI
To restore the network through the DCUI:
1. Select Network Restore Options.
2. Perform a full network restore.
You repair the management network on a misconfigured standard or distributed switch.
The Restore Network Settings option deletes all the current network settings except for the
management network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 44

Review of Distributed Switch Network Connectivity
The cause of a network connectivity problem might be found in the VMs, the vCenter Server
system, or the ESXi hosts that have NICs assigned to the distributed switch and the physical
network.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 45

Distributed Switch IOChain Framework
A distributed switch uses the
ESXi network IOChain
framework, which is a group
of functions that provides the
connectivity between the
ports and the virtual switch.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 46

Distributed Switch Log Files
Because a distributed switch is a vCenter Server component, you view VM, vCenter Server,
and ESXi host log files to troubleshoot networking problems.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 47

About Distributed Switch Rollback
The distributed switch rollback is triggered when invalid updates are made to distributed
switch-related objects.
Examples of events that might trigger a distributed switch rollback:
• Changing the MTU of a distributed switch
• Changing the following settings in the distributed port group of the management VMkernel
network adapter:
– NIC teaming and failover
– VLAN
– Traffic shaping
If an invalid configuration occurs, one or more hosts might be out of synchronization with the
distributed switch.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 48

Recovering from Distributed Switch Misconfiguration
Before changing a distributed switch configuration, you back up the configuration.
If your distributed switch loses network connectivity because of a misconfiguration, you can
restore it from your latest backup.
The vSphere Client provides options for backing up and restoring the distributed switch
configuration:
• Export: Backs up your distributed switch configuration
• Restore: Resets the configuration of a distributed switch from an exported configuration file
• Import: Creates a distributed switch from an exported configuration file

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 49

Migrating Networks from Standard Switches to a Distributed Switch
When you implement a distributed switch, you migrate your networks from the standard
switches to the distributed switch without downtime to ESXi hosts or VMs. You prioritize your
networks in terms of importance and determine the order in which to migrate your networks.

Network Type Description

Storage 1. Move this network first if you have a common subnet and infrastructure for
the management and storage networks.

Production 2. Move this network next because if VMs fail to communicate with each other
or the outside world, applications running inside the VMs might experience a
downtime.
Management 3. If downtime occurs on this network, connectivity is lost on ESXi hosts, and
you cannot manage the ESXi hosts from the vSphere Client.

vMotion 4. Migrate this network last because it is used only for VM migrations. Verify
that DRS is disabled and that no live migrations are occurring when you
migrate this network to the distributed switch.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 50

vSphere Commands: net-dvs
The net-dvs command displays information about your distributed switch configuration:
• Configuration information comes from a binary file called /etc/vmware/[Link].
• The ESXi host maintains and updates the [Link] file at 5-minute intervals.
• net-dvs -h lists the command's syntax and options.
In addition to displaying information, you can also run net-dvs to modify your distributed
switch configuration.
However, because net-dvs is an unsupported command, you run it at your own risk.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 51

Labs
Lab: Monitoring NIC Teaming During Failover
Lab: Monitoring and Recovering Switches
Lab: Applying the Troubleshooting Methodology
Lab: Troubleshooting Network Problems

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 52

Lab 5: Monitoring NIC Teaming During Failover
Monitor NIC teaming behavior when one of the links in the team goes down:
1. Verify the Distributed Switch Configuration
2. Verify Network Operation on the ESXi Host
3. Monitor the ESXi Host When the Active Link Goes Down
4. Monitor the ESXi Host When the Standby Link Goes Down
5. Reconfigure the Port Group pg-SA-Production-01

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 53

Lab 6: Monitoring and Recovering Distributed Switches
Use command-line tools to monitor distributed switches and recover from a distributed switch
failure:
1. Display Distributed Switch Information
2. Disable the Network Rollback Option
3. Recover from a Distributed Switch Failure
4. Enable the Network Rollback Option
5. Migrate the Management Network

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 54

Lab 7: Applying the Troubleshooting Methodology
Follow the troubleshooting methodology to solve a networking problem:
1. Run a Break Script
2. Narrow the Scope of the Problem to a VM
3. Narrow the Scope of the Problem to the ESXi Host
4. Resolve the Problem
5. Verify the Solution

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 55

Lab 8: Troubleshooting Network Problems
Identify, diagnose, and resolve virtual networking problems:
1. Run a Break Script
2. Verify That the System Is Not Functioning Correctly
3. Troubleshoot and Resolve the Problem
4. Verify the Solution

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 56

Review of Learner Objectives
After completing this module, you should be able to meet the following objectives:
• Analyze and troubleshoot standard switch problems
• Analyze and troubleshoot virtual machine connectivity problems
• Analyze and troubleshoot management network problems
• Analyze and troubleshoot distributed switch problems

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 57

Key Points
• Virtual network connectivity problems might occur with standard switches, distributed
switches, virtual machines, or management networks.
• A virtual machine connectivity problem might occur in the physical layer, the virtual layer,
or the guest operating system.
• The ping command is useful when troubleshooting ESXi host and VM connectivity issues.
• When an ESXi host frequently disconnects from vCenter Server, heartbeat packets are lost
between vCenter Server and the ESXi host.
• vSphere network rollback prevents an accidental misconfiguration of management
networking and loss of connectivity.
• You should back up the distributed switch configuration before changing this configuration.
• You can use the restore or the import function to reset the distributed switch configuration.
Questions?

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 4 - 58