‘Classical CASE’ 分类存档

Troubleshooting IPoE Session that pending at “ACK-DPM-WAIT”


The article will talk about what is “ACK-DPM-WAIT”, and how to troubleshooting the similar scenario. Due to limitation info that couldn’t narrow down, in my CASE, so will update the article if the issue happen again and find RCA.

Problem Description

Version: 5.1.3 + induvial SMU
Platform: 9010 + Mod80 + A9K-MPA-4X10GE
BNG: IPOE, DHCP Proxy, 28k session

My customer found part of BNG session was failure. Trigger is due to customer power supply have issue that cause the asr9k re-power. After 9k reload, found dhcpd and arp have so many alarms, dhcpd was recovery after tried restart process multi times, but arp continue have SPIO alarm even if tried restart process, customer had enabled arp local disable on the BNG port.

The issue sessions got address correct from DHCP, but session would be deleted after 15min. After checked on asr9k, we found issue session pending on ACK_DPM_WAIT status. And the issue was auto recovery at approx.19:00-19:30. And at that timeslot, arp alarm disappear too.

How to decode TCP, UDP and RAW for IOS-XR

1. SPAN抓包
2. debug


RP/0/RP1/CPU0:CRS2(config)#udp directory /tmp/udp
RP/0/RP1/CPU0:CRS2(config)#ipv4 access-list hsrp-packet
RP/0/RP1/CPU0:CRS2(config-ipv4-acl)#20 permit udp any eq 1985 any eq 1985
RP/0/RP1/CPU0:CRS2(config-ipv4-acl)#30 deny ipv4 any any
RP/0/RP1/CPU0:CRS2(config)#ipv6 access-list v6-filter
RP/0/RP1/CPU0:CRS2(config-ipv6-acl)#10 deny ipv6 any any
RP/0/RP1/CPU0:CRS2#debug udp packet v4-access-list hsrp-packet v6-access-list v6-filter hex control-block location x/x/cpu0

You can check the capture by follow patch:
# cd /tmp/udp
#more xxxx

Multi Hierarchical CEF / Load Share


         |   |
         |                    |
    +----+----+          +----+----+
    | |          | |
    | RouterA |          | RouterB |
    +-\----\--+          +-/---/---+
       \    \             /   /
        \\   \           /   /
          \   \         /  //
           \   \F2/0   /  /
            \\  \     /  /
         F1/0 \  \ F3/0 / F4/0
               | CoreA |

在早期版本,不支持Multi hierarchical CEF,仅仅支持一层递归后的转发。这样产生了很多限制,例如今天提到的双PE结构。在特定版本后(包括IOS和IOX),CEF的行为有了改变,并且支持多层CEF。不过CEF的行为也要看平台,因为GSR上任何版本都不支持这种多层CEF。

TS for 6748 output drop

When you found have output queue drop for CEF720 LC, you can check follow step:
1. which port have issue, whether at same ASIC.
2. check whether have other error or have qos on issue port.
3. whether hw queue is full.

Follow is TS example:

Problem description:

Output queue have drop
1. After checked by follow command:
– show tech
– show inter switching x/x
– show int x/x counter de
– show inter x/x summary



1w0d: OSPF: DR/BDR election on TenGigabitEthernet6/4
1w0d: OSPF: Elect BDR
1w0d: OSPF: Elect DR
1w0d:        DR: (Id)   BDR: none
1w0d: OSPF: Rcv DBD from on TenGigabitEthernet6/4 seq 0x2011 opt 0x2 flag 0x2 len 1472  mtu 0 state EXSTART
Options: 0x02 (E)
0... .... = DN: DN-bit is NOT set
.0.. .... = O: O-bit is NOT set
..0. .... = DC: Demand circuits are NOT supported
...0 .... = L: The packet does NOT contain LLS data block
.... 0... = NP: Nssa is NOT supported
.... .0.. = MC: NOT multicast capable
.... ..1. = E: ExternalRoutingCapability
.... ...0 = MT
1w0d: OSPF: NBR Negotiation Done. We are the MASTER
1w0d: OSPF: Send DBD to on TenGigabitEthernet6/4 seq 0x2012 opt 0x52 flag 0x3 len 1452