Layer 3 Switch Troubleshooting

Posted on April 2, 2010 by Judson Bishop

The big topics of this chapter include HSRP, VRRP, GLBP, TCAM and the port error table. I broke out my notes from the BCMSN and I’ll summarize each topic from them, then move into the troubleshooting of each protocol.

Hot Standby Router Protocol (HSRP)
Cisco proprietary protocol to allow multiple routers to appear as a single gateway IP address. EAch router is assigned to a common HSRP group one active router and one standby router, the others, up to 16 group members, just listen. There is no preemption without configuration.

  • Election of one active router — Based on priority that is assigned when the router is configured for the group. Default priority is 100, range is from 0 – 255, highest priority for election is 255. Highest IP address on HSRP interface breaks ties.

  • Election of one standby router — Second highest priority. Listens to ehloo messages, by default every 3 seconds with hold time of 10 seconds.

  • Virtual router — IP and MAC address pair that end devices use as a gateway. The active router processes all packets and frames to this address, the virtual router processes no physical frames.

    • Troubleshoot
    sh standby …
    sh standby delay

    sh standby

    DSW1#sh standby 
Vlan10 - Group 10
  State is Active
  Virtual IP address is 10.2.1.254
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.704 secs
  Preemption enabled
  Active router is local
  Standby router is 10.2.1.2, priority 100 (expires in 10.480 sec)
  Priority 200 (configured 200)
  Group name is "hsrp-Vl10-10" (default)

    sh standby br

    DSW1#sh standby br
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Vl10        10   200 P Active  local           10.2.1.2        10.2.1.254

    sh standby delay

    DSW1#sh standby delay 
Interface          Minimum Reload 
Vlan10             1       5

    Virtual Router Redundancy Protocol (VRRP)
    VRRP is on open protocol that provides redundancy for the real IP address of a router or the virtual address of a group. By default all VRRP routers are configured to preempt the current master if their priority is greater.

  • Election of active router — If a real IP address is used, the master is the router with that IP. If a virtual one is used the master becomes the one with the highest priority, all others are in backup state. VRRP group numbers are from 0 to 255, router priority is from 1 to 254 with 254 being the highest and default priority is 100.

    • Troubleshoot
    sh vrrp
    sh vrrp brief

    sh vrrp

    R7#sh vrrp
FastEthernet0/0 - Group 1  
"Just playing"
  State is Master  
  Virtual IP address is 10.2.1.253
  Virtual MAC address is 0000.5e00.0101
  Advertisement interval is 1.000 sec
  Preemption enabled
  Priority is 200 
  Master Router is 10.2.1.247 (local), priority is 200 
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.218 sec

    sh vrrp brie

    R8#sh vrrp br
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Fa0/0              1   150 3414       Y  Backup  10.2.1.247      10.2.1.253

    Gateway Load Balancing Protocol (GLBP)
    Cisco proprietary protocol that is more robust than other redundancy protocols. When a client sends an ARP request looking for the router address, GLBP send back a reply with the virtual MAC supported by one of the routers in the GLBP group.

  • GLBP Active Virtual Gateway (GLBP-AVG) — Group members elect the AVG which replies to all host ARP requests. It has either the highest priority value or the highest IP address in the group is the tie breaker. Priority ranges from 0 to 255 with 255 being highest, 100 is the default. AVG assigns the virtual router address and assigns the secondary roles. Up to four virtual MAC address can be used in any group.
  • GLPB Active Virtual Forwarder (GLBP-AVF) — Each gateway assumes responsibility for forwarding packets that are sent to the virtual MAC address assigned to that gateway. GLBP load balances across the AVFs in round robin by default. AVFs are expected to send hellos to every other GLBP peer. If the hold time on the AVG expires without receiving a hello packet from an AVF the virtual MAC is assigned to another router.

    • Troubleshoot:
    sh glbp [brief]

    sh glbp

    R8(config-if)#do sh glbp
FastEthernet0/0 - Group 1
  State is Active
    2 state changes, last state change 00:03:58
  Virtual IP address is 10.2.1.252
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.776 secs
  Redirect time 600 sec, forwarder time-out 14400 sec
  Preemption enabled, min delay 0 sec
  Active is local
  Standby is 10.2.1.247, priority 150 (expires in 7.744 sec)
  Priority 200 (configured)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0004.dd69.fd01 (10.2.1.248) local
    00d0.bbef.64c1 (10.2.1.247)
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 00:03:48
    MAC address is 0007.b400.0101 (default)
    Owner ID is 0004.dd69.fd01
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.0102 (learnt)
    Owner ID is 00d0.bbef.64c1
    Redirection enabled, 597.584 sec remaining (maximum 600 sec)
    Time to live: 14397.584 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 10.2.1.247 (primary), weighting 100 (expires in 7.584 sec)

    Ternary Content Addressable Memory (TCAM):
    TCAM works with CEF to make extremely fast forwarding decisions. TCAM entries are composed of Value, Mask and Result combinations.

  • Values — Always 134 bits consisting of source and destination addresses along with relevant protocol infromation.
  • Masks — Also 134 bits set to match value bits, or not match value bits that do not matter. This is so that a switch can do a bitwise and with access lists or access control entities (ACE).
  • Results — Numerical values that represent what action to take after the TCAM lookup.

    • When the TCAM cannot process the packet it is punted to the CPU for processing. Here are some reasons why a packet is punted:

  • Protocols that send multicast or broadcast packets.
  • Administrative access to a switch such as telnet or ssh.
  • Packets that use features not supported in hardware.
  • A switch’s TCAM has reached capacity. TCAM might reach capacity with large routing tables or a large number of ACLs.

    Troubleshoot TCAM:
    3550:
    show tcam
    3560:
    show platform tcam
    show platform ip unicast counts

    3550#sh tcam inacl 1 statistics 

    3550#sh tcam inacl 1 statistics 
Ingress ACL TCAM#1: Number of active labels: 3
Ingress ACL TCAM#1: Number of masks   allocated:   14, available:  194
Ingress ACL TCAM#1: Number of entries allocated:   17, available: 1647

    3560#sh platform tcam utilization 

    3560#sh platform tcam utilization 

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                        544/4352         25/76    
 IPv4 IGMP groups + multicast routes:          144/1152          6/26    
 IPv4 unicast directly-connected routes:       544/4352         25/76    
 IPv4 unicast indirectly-connected routes:     176/1408         15/69    
 IPv6 Multicast groups:                        544/4352         25/76    
 IPv6 unicast directly-connected routes:       544/4352         25/76    
 IPv6 unicast indirectly-connected routes:     262/2096         11/42    
 IPv4 policy based routing aces:               256/256           2/2     
 IPv4 qos aces:                                768/768         324/324   
 IPv4 security aces:                           512/512          27/27    
 IPv6 policy based routing aces:                 0/0             0/0     
 IPv6 qos aces:                                  0/0             0/0     
 IPv6 security aces:                           204/510           5/5     

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

    Port Error Table:

    Error

    		Description

    		Cause
    Xmit-Err

    		Transmit buffer overlflow

    		Check for speed mismatch.
    Rcv-Err

    		Receive buffer overflow

    		Check for duplex mismatch.
    UnderSize

    		Has valid checksum but less than 64 bytes.

    		Host is sending invalid frames.
    Single-Col

    		A single collision occurred before the port transmitted the frame.

    		High bandwidth utilization or duplex mismatch.
    Multi-Col

    		Multiple collisions occur before the port transmitted the frame.

    		High bandwidth utilization or duplex mismatch.
    Late-Col

    		Collision detected further into frame being forwarded.

    		Cable too long or duplex mismatch.
    Excess-Col

    		Frame has experienced sixteen successive collisions, resulting in the frame being dropped.

    		High bandwidth, duplex mismatch or too many devices on a segment.
    Carri-Sen

    		Normal operation on a half-duplex link.

    		This is not a problem.
    Runts

    		Frames smaller than 64 bytes with a bad CRC.

    		Duplex mismatch or layer 1 problem.
    Giants

    		Frame size greater than 1518 bytes with a bad FCS.

    		Bad host NIC.

    Source: TSHOOT p.125

    • This entry was posted in Routing. Bookmark the permalink.

    Leave a comment