Troubleshooting Cisco Nexus Switches and NX-OS

Cover
Title Page
Copyright Page
About the Authors
Dedications
Acknowledgments
Contents
Foreword
Introduction
Part I: Introduction to Troubleshooting Nexus Switches
Chapter 1 Introduction to Nexus Operating System (NX-OS)
Nexus Platforms Overview
Nexus 2000 Series
Nexus 3000 Series
Nexus 5000 Series
Nexus 6000 Series
Nexus 7000 Series
Nexus 9000 Series
NX-OS Architecture
The Kernel
System Manager (sysmgr)
Messages and Transactional Services
Persistent Storage Services
Feature Manager
NX-OS Line Card Microcode
File Systems
Flash File System
Onboard Failure Logging
Logflash
Understanding NX-OS Software Releases and Packaging
Software Maintenance Upgrades
Licensing
NX-OS High-Availability Infrastructure
Supervisor Redundancy
ISSU
NX-OS Virtualization Features
Virtual Device Contexts
Virtual Routing and Forwarding
Virtual Port Channel
Management and Operations Capabilities
NX-OS Advanced CLI
Technical Support Files
Accounting Log
Feature Event-History
Debug Options: Log File and Filters
Configuration Checkpoint and Rollback
Consistency Checkers
Feature Scheduler, EEM, and Python
Bash Shell
Summary
References
Chapter 2 NX-OS Troubleshooting Tools
Packet Capture: Network Sniffer
Encapsulated Remote SPAN
SPAN on Latency and Drop
SPAN-on-Latency
SPAN-on-Drop
Nexus Platform Tools
Ethanalyzer
Packet Tracer
NetFlow
NetFlow Configuration
Enable NetFlow Feature
Define a Flow Record
Define a Flow Exporter
Define and Apply the Flow Monitor
NetFlow Sampling
sFlow
Network Time Protocol
Embedded Event Manager
Logging
Debug Logfiles
Accounting Log
Event-History
Summary
References
Chapter 3 Troubleshooting Nexus Platform Issues
Troubleshooting Hardware Issues
Generic Online Diagnostic Tests
Bootup Diagnostics
Runtime Diagnostics
GOLD Test and EEM Support
Nexus Device Health Checks
Hardware and Process Crashes
Packet Loss
Interface Errors and Drops
Platform-Specific Drops
Nexus Fabric Extenders
Virtual Device Context
VDC Resource Template
Configuring VDC
VDC Initialization
Out-of-Band and In-Band Management
VDC Management
Line Card Interop Limitations
Troubleshooting NX-OS System Components
Message and Transaction Services
Netstack and Packet Manager
Netstack TCPUDP Component
ARP and Adjacency Manager
Unicast Forwarding Components
Unicast Routing Information Base
UFDM and IPFIB
EthPM and Port-Client
HWRL, CoPP, and System QoS
MTU Settings
FEX Jumbo MTU Settings
Troubleshooting MTU Issues
Summary
References
Part II: Troubleshooting Layer 2 Forwarding
Chapter 4 Nexus Switching
Network Layer 2 Communication Overview
Virtual LANs
VLAN Creation
Access Ports
Trunk Ports
Native VLANs
Allowed VLANs
Private VLANS
Isolated Private VLANs
Community Private VLANs
Using a Promiscuous PVLAN Port on Switched Virtual Interface
Trunking PVLANs Between Switches
Spanning Tree Protocol Fundamentals
IEEE 802.1D Spanning Tree Protocol
Rapid Spanning Tree Protocol
Spanning-Tree Path Cost
Root Bridge Election
Locating Root Ports
Locating Blocked Switch Ports
Verification of VLANS on Trunk Links
Spanning Tree Protocol Tuning
Multiple Spanning-Tree Protocol (MST)
MST Configuration
MST Verification
MST Tuning
Detecting and Remediating Forwarding Loops
MAC Address Notifications
BPDU Guard
BPDU Filter
Problems with Unidirectional Links
Spanning Tree Protocol Loop Guard
Unidirectional Link Detection
Bridge Assurance
Summary
References
Chapter 5 Port-Channels, Virtual Port-Channels, and FabricPath
Port-Channels
Basic Port-Channel Configuration
Verifying Port-Channel Status
Verifying LACP Packets
Advanced LACP Configuration Options
Minimum Number of Port-Channel Member Interfaces
Maximum Number of Port-Channel Member Interfaces
LACP System Priority
LACP Interface Priority
LACP Fast
Graceful Convergence
Suspend Individual
Port-Channel Member Interface Consistency
Troubleshooting LACP Interface Establishment
Troubleshooting Traffic Load-Balancing
Virtual Port-Channel
vPC Fundamentals
vPC Domain
vPC Peer-Keepalive
vPC Peer Link
vPC Member Links
vPC Operational Behavior
vPC Configuration
vPC Verification
Verifying the vPC Domain Status
Verifying the Peer-Keepalive
vPC Consistency-Checker
Advanced vPC Features
vPC Orphan Ports
vPC Autorecovery
vPC Peer-Gateway
vPC ARP Synchronization
Backup Layer 3 Routing
Layer 3 Routing over vPC
FabricPath
FabricPath Terminologies and Components
FabricPath Packet Flow
FabricPath Configuration
FabricPath Verification and Troubleshooting
FabricPath Devices
Emulated Switch and vPC+
vPC+ Configuration
vPC+ Verification and Troubleshooting
Summary
References
Part III: Troubleshooting Layer 3 Routing
Chapter 6 Troubleshooting IP and IPv6 Services
IP SLA
ICMP Echo Probe
UDP Echo Probe
UDP Jitter Probe
TCP Connect Probe
Object Tracking
Object Tracking for the Interface
Object Tracking for Route State
Object Tracking for Track-List State
Using Track Objects with Static Routes
IPv4 Services
DHCP Relay
DHCP Snooping
Dynamic ARP Inspection
ARP ACLs
IP Source Guard
Unicast RPF
IPv6 Services
Neighbor Discovery
IPv6 Address Assignment
DHCPv6 Relay Agent
DHCPv6 Relay LDRA
IPv6 First-Hop Security
RA Guard
IPv6 Snooping
DHCPv6 Guard
First-Hop Redundancy Protocol
HSRP
HSRPv6
VRRP
GLBP
Summary
Chapter 7 Troubleshooting Enhanced Interior Gateway Routing Protocol (EIGRP)
EIGRP Fundamentals
Topology Table
Path Metric Calculation
EIGRP Communication
Baseline EIGRP Configuration
Troubleshooting EIGRP Neighbor Adjacency
Verification of Active Interfaces
Passive Interface
Verification of EIGRP Packets
Connectivity Must Exist Using the Primary Subnet
EIGRP ASN Mismatch
Mismatch K Values
Problems with Hello and Hold Timers
EIGRP Authentication Issues
Interface-Based EIGRP Authentication
Global EIGRP Authentication
Troubleshooting Path Selection and Missing Routes
Load Balancing
Stub
Maximum-Hops
Distribute List
Offset Lists
Interface-Based Settings
Redistribution
Classic Metrics vs. Wide Metrics
Problems with Convergence
Active Query
Stuck in Active
Summary
References
Chapter 8 Troubleshooting Open Shortest Path First (OSPF)
OSPF Fundamentals
Inter-Router Communication
OSPF Hello Packets
Neighbor States
Designated Routers
Areas
Link State Advertisements
Troubleshooting OSPF Neighbor Adjacency
Baseline OSPF Configuration
OSPF Neighbor Verification
Confirmation of OSPF Interfaces
Passive Interface
Verification of OSPF Packets
Connectivity Must Exist Using the Primary Subnet
MTU Requirements
Unique Router-ID
Interface Area Numbers Must Match
OSPF Stub (Area Flags) Settings Must Match
DR Requirements
Timers
Authentication
Troubleshooting Missing Routes
Discontiguous Network
Duplicate Router ID
Filtering Routes
Redistribution
OSPF Forwarding Address
Troubleshooting OSPF Path Selection
Intra-Area Routes
Inter-Area Routes
External Route Selection
E1 and N1 External Routes
E2 and N2 External Routes
Problems with Intermixed RFC 1583 and RFC 2328 Devices
Interface Link Costs
Summary
References
Chapter 9 Troubleshooting Intermediate System-Intermediate System (IS-IS)
IS-IS Fundamentals
Areas
NET Addressing
Inter-Router Communication
IS Protocol Header
TLVs
IS PDU Addressing
IS-IS Hello (IIH) Packets
Link-State Packets
LSP ID
Attribute Fields
LSP Packet and TLVs
Designated Intermediate System
Path Selection
Troubleshooting IS-IS Neighbor Adjacency
Baseline IS-IS Configuration
IS-IS Neighbor Verification
Confirmation of IS-IS Interfaces
Passive Interface
Verification of IS-IS Packets
Connectivity Must Exist Using the Primary Subnet
MTU Requirements
Unique System-ID
Area Must Match Between L1 Adjacencies
Checking IS-IS Adjacency Capabilities
DIS Requirements
IIH Authentication
Troubleshooting Missing Routes
Duplicate System ID
Interface Link Costs
Mismatch of Metric Modes
L1 to L2 Route Propagations
Suboptimal Routing
Redistribution
Summary
References
Chapter 10 Troubleshooting Nexus Route-Maps
Conditional Matching
Access Control Lists
ACLs and ACL Manager Component
Interior Gateway Protocol (IGP) Network Selection
BGP Network Selection
Prefix Matching and Prefix-Lists
Prefix Matching
Prefix Lists
Route-Maps
Conditional Matching
Multiple Conditional Match Conditions
Complex Matching
Optional Actions
Incomplete Configuration of Routing Policies
Diagnosing Route Policy Manger
Policy-Based Routing
Summary
References
Chapter 11 Troubleshooting BGP
BGP Fundamentals
Address Families
Path Attributes
Loop Prevention
BGP Sessions
BGP Identifier
BGP Messages
OPEN
UPDATE
NOTIFICATION
KEEPALIVE
BGP Neighbor States
Idle
Connect
Active
OpenSent
OpenConfirm
Established
BGP Configuration and Verification
Troubleshooting BGP Peering Issues
Troubleshooting BGP Peering Down Issues
Verifying Configuration
Verifying Reachability and Packet Loss
Verifying ACLs and Firewalls in the Path
Verifying TCP Sessions
OPEN Message Errors
BGP Debugs
Demystifying BGP Notifications
Troubleshooting IPv6 Peers
BGP Peer Flapping Issues
Bad BGP Update
Hold Timer Expired
BGP Keepalive Generation
MTU Mismatch Issues
BGP Route Processing and Route Propagation
BGP Route Advertisement
Network Statement
Redistribution
Route Aggregation
Default-Information Originate
BGP Best Path Calculation
BGP Multipath
EBGP and IBGP Multipath
BGP Update Generation Process
BGP Convergence
Scaling BGP
Tuning BGP Memory
Prefixes
Paths
Attributes
Scaling BGP Configuration
Soft Reconfiguration Inbound Versus Route Refresh
Scaling BGP with Route-Reflectors
Loop Prevention in Route Reflectors
Maximum Prefixes
BGP Max AS
BGP Route Filtering and Route Policies
Prefix-List-Based Filtering
Filter-Lists
BGP Route-Maps
Regular Expressions (RegEx)
_ Underscore
^ Caret
$ Dollar Sign
[ ] Brackets
- Hyphen
[^] Caret in Brackets
( ) Parentheses and | Pipe
. Period
+ Plus Sign
? Question Mark
* Asterisk
AS-Path Access List
BGP Communities
Looking Glass and Route Servers
Logs Collection
Summary
Further Reading
References
Part IV: Troubleshooting High Availability
Chapter 12 High Availability
Bidirectional Forwarding Detection
Asynchronous Mode
Asynchronous Mode with Echo Function
Configuring and Verifying BFD Sessions
Nexus High Availability
Stateful Switchover
ISSU
Graceful Insertion and Removal
Custom Maintenance Profile
Summary
References
Part V: Multicast Network Traffic
Chapter 13 Troubleshooting Multicast
Multicast Fundamentals
Multicast Terminology
Layer 2 Multicast Addresses
Layer 3 Multicast Addresses
NX-OS Multicast Architecture
Replication
Protecting the Central Processing Unit
NX-OS Multicast Implementation
Static Joins
Clearing an MROUTE Entry
Multicast Boundary and Filtering
Event-Histories and Show Techs
IGMP
IGMPv2
IGMPv3
IGMP Snooping
IGMP Verification
PIM Multicast
PIM Protocol State and Trees
PIM Message Types
PIM Hello Message
PIM Register Message
PIM Register-Stop Message
PIM Join-Prune Message
PIM Bootstrap Message
PIM Assert Message
PIM Candidate RP Advertisement Message
PIM DF Election Message
PIM Interface and Neighbor Verification
PIM Any Source Multicast
PIM ASM Configuration
PIM ASM Verification
PIM ASM Event-History and MROUTE State Verification
PIM ASM Platform Verification
PIM Bidirectional
BiDIR Configuration
BiDIR Verification
PIM RP Configuration
Static RP Configuration
Auto-RP Configuration and Verification
BSR Configuration and Verification
Anycast-RP Configuration and Verification
Anycast RP with MSDP
PIM Anycast RP
PIM Source Specific Multicast
SSM Configuration
SSM Verification
Multicast and Virtual Port-Channel
vPC-Connected Source
vPC-Connected Receiver
vPC Considerations for Multicast Traffic
Duplicate Multicast Packets
Reserved VLAN
Ethanalyzer Examples
Summary
References
Part VI: Troubleshooting Nexus Tunneling
Chapter 14 Troubleshooting Overlay Transport Virtualization (OTV)
OTV Fundamentals
Flood Control and Broadcast Optimization
Supported OTV Platforms
OTV Terminology
Deploying OTV
OTV Deployment Models
OTV Site VLAN
OTV Configuration
Understanding and Verifying the OTV Control Plane
OTV Multicast Mode
OTV IS-IS Adjacency Verification
OTV IS-IS Topology Table
OTV IS-IS Authentication
Adjacency Server Mode
OTV Control Plane Policing (CoPP)
Understanding and Verifying the OTV Data Plane
OTV ARP Resolution and ARP-ND-Cache
Broadcasts
Unknown Unicast Frames
OTV Unicast Traffic with a Multicast Enabled Transport
OTV Multicast Traffic with a Multicast Enabled Transport
OTV Multicast Traffic with a Unicast Transport (Adjacency Server Mode)
Advanced OTV Features
First Hop Routing Protocol Localization
Multihoming
Ingress Routing Optimization
VLAN Translation
OTV Tunnel Depolarization
OTV Fast Failure Detection
Summary
References
Part VII: Network Programmability
Chapter 15 Programmability and Automation
Introduction to Automation and Programmability
Introduction to Open NX-OS
Shells and Scripting
Bash Shell
Guest Shell
Python
NX-SDK
NX-API
Summary
References
Index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y