Troubleshooting fabric FIA tail drop on ASR9k

Introduction

In TZ database, have more good documents that is troubleshooting fabric guide on ASR9k, but no analysis process that show how to troubleshooting fabric issue on real scenario/CASE. “Very lucky” ?  I matched a hot CASE that due to fabric issue cause online fail. So i summaried totally analysis process that will help CSE to narrow down similar issue.

Problem Description

  • Platform: 9922 + 4 36x10G + 2 8x100G
  • Version: 5.3.2 + SMU 

My customer online a new 9922 to replace old devices. After online, found their business have traffics drop. Base on online information, I found NP no more drop, and business traffics very less. (max 5g under 100g port, all bundle port when online ts)

But fia have lots of drop and amount packets in VOQ:

RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/10/cpu0
Wed May 11 13:03:04.841 Beijing
RP/0/RP0/CPU0:xxx#
PP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/11/cpu0 | i drop
Wed May 11 13:03:27.550 Beijing
                                Ingress drop:                        0
                                 Egress drop:                     1918
                                 Total drop:                      1918
                                Ingress drop:                    43820
                                 Egress drop:                    13121
                                 Total drop:                     56941
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0

RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/10/cpu0
Wed May 11 13:24:44.920 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
111       0              2              19        LC2_4_4
180       0              2              24326     LC11_1_1
192       0              2              648       LC11_1_1
276       0              2              12555     LC10_1_1

RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/2/cpu0
Wed May 11 13:26:37.671 Beijing

 ********** FIA-4 **********
Category: q_stats_a-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
180       0              2              38448     LC11_1_1
181       0              2              39954     LC11_1_1
182       0              2              51617     LC11_1_1
184       0              2              834       LC11_1_1
185       0              2              14837     LC11_1_1
186       0              2              6866      LC11_1_1
189       0              2              25281     LC11_1_1
190       0              2              54585     LC11_1_1
192       0              2              524       LC11_1_1
193       0              2              171       LC11_1_1
194       0              2              339       LC11_1_1
196       0              2              84        LC11_1_1
197       0              2              252       LC11_1_1
276       0              2              63093     LC10_1_1
277       0              2              29402     LC10_1_1
278       0              2              86107     LC10_1_1
279       0              2              101730    LC10_1_1
280       0              2              46332     LC10_1_1
281       0              2              8185      LC10_1_1
282       0              2              20360     LC10_1_1
283       0              2              23855     LC10_1_1
284       0              2              20146     LC10_1_1
285       0              2              15464     LC10_1_1
286       0              2              35014     LC10_1_1
287       0              2              8995      LC10_1_1

 ********** FIA-4 **********
Category: q_stats_b-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP  

All bundle ports use follow QOS prifile, I checked QOS drop for some bundles, no found any drop.

policy-map Diff-Sev
 class EXP-Control
  priority level 1 
  police rate percent 5 
  ! 
 ! 
 class EXP-Gold
  priority level 2 
  police rate percent 75 
  ! 
 ! 
 class EXP-Silver
  priority level 3 
  queue-limit 10 ms 
 ! 
 class EXP-Copper
  bandwidth remaining percent 50 
 ! 
 class class-default
  bandwidth remaining percent 50 
 ! 
 end-policy-map
! 

Troubleshooting

1. Clear counter at May 11 13:02

RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/0/cpu0
Wed May 11 13:02:54.243 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/10/cpu0
Wed May 11 13:03:04.841 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/11/cpu0
Wed May 11 13:03:14.907 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/1/cpu0                
Wed May 11 13:03:47.835 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/2/cpu0                
Wed May 11 13:04:02.986 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/3/cpu0
Wed May 11 13:04:38.460 Beijing

2. After checked fia drop, have follow found

  • lc02 FIA4 egress drop increased
  • lc10 FIA0 egress drop increased
  • lc10 FIA1 ingress/egress drop increased
  • lc11 FIA0 egress drop increased
  • lc11 FIA1 ingress/egress drop increased
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/0/cpu0 | i drop 
Wed May 11 13:04:56.381 Beijing
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/1/cpu0 | i drop
Wed May 11 13:05:06.780 Beijing
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/2/cpu0 | i drop
Wed May 11 13:05:14.797 Beijing
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                    31602
                                 Total drop:                     31602
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0

RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/3/cpu0 | i drop
Wed May 11 13:05:44.327 Beijing
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/10/cpu0 | i drop
Wed May 11 13:05:54.383 Beijing
                                Ingress drop:                        0
                                 Egress drop:                    32651
                                 Total drop:                     32651
                                Ingress drop:                   501560
                                 Egress drop:                   144065
                                 Total drop:                    645625
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/11/cpu0 | i drop
Wed May 11 13:06:03.356 Beijing
                                Ingress drop:                        0
                                 Egress drop:                    27203
                                 Total drop:                     27203
                                Ingress drop:                   771349
                                 Egress drop:                   170823
                                 Total drop:                    942172
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0
                                Ingress drop:                        0
                                 Egress drop:                        0
                                 Total drop:                         0

3. Checked what packets drop by “show drops”

  • There is a questions, lc2 no FIA4 in “show drops” … But lc2’s egress drop mostly same as follow.
  • lc10 fia 0 egress drop that relation wth “Egress Uc dq pkt-len-crc/RO-seq/len error drop
  • lc10 fia 1 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop“, other drops are ingress drop
  • lc11 fia 0 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop
  • lc11 FIA1 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop“, other drops are ingress drop 
RP/0/RP0/CPU0:xxx#sh drops     
Wed May 11 13:13:41.304 Beijing

                Node: 0/0/CPU0:
----------------------------------------------------------------

No NP 0 Drops
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

NP 4 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH                                 99              
----------------------------------------------------------------

NP 5 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH                                 176             
----------------------------------------------------------------

No FIA 0 Drops
----------------------------------------------------------------

No FIA 1 Drops
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

                Node: 0/1/CPU0:
----------------------------------------------------------------

No NP 0 Drops
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

NP 4 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH                                 277             
----------------------------------------------------------------

No NP 5 Drops
----------------------------------------------------------------

No Bridge 0 Drops
----------------------------------------------------------------

No Bridge 1 Drops
----------------------------------------------------------------

No FIA 0 Drops
----------------------------------------------------------------

No FIA 1 Drops
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

                Node: 0/2/CPU0:
----------------------------------------------------------------

No NP 0 Drops
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

No NP 4 Drops
----------------------------------------------------------------

No NP 5 Drops
----------------------------------------------------------------

No Bridge 0 Drops
----------------------------------------------------------------

No Bridge 1 Drops
----------------------------------------------------------------

No FIA 0 Drops
----------------------------------------------------------------

No FIA 1 Drops
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

                Node: 0/3/CPU0:
----------------------------------------------------------------

No NP 0 Drops
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

No NP 4 Drops
----------------------------------------------------------------

NP 5 Drops:
----------------------------------------------------------------
PARSE_DROP_IPV4_DISABLED                                     341             
----------------------------------------------------------------

No Bridge 0 Drops
----------------------------------------------------------------

No Bridge 1 Drops
----------------------------------------------------------------

No FIA 0 Drops
----------------------------------------------------------------

No FIA 1 Drops
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

                Node: 0/10/CPU0:
----------------------------------------------------------------

No NP 0 Drops
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

No Bridge 0 Drops
----------------------------------------------------------------

No Bridge 1 Drops
----------------------------------------------------------------

FIA 0 Drops:
----------------------------------------------------------------
Total drop:                                                  129021          
Egress drop:                                                 129021          
Egress Uc dq pkt-len-crc/RO-seq/len error drp                129104          
----------------------------------------------------------------

FIA 1 Drops:
----------------------------------------------------------------
Total drop:                                                  2734078         
Egress drop:                                                 567642          
Ingress drop:                                                2166436         
Ingress Tail drp                                             2166436         
Egress Uc dq pkt-len-crc/RO-seq/len error drp                567642          
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

                Node: 0/11/CPU0:
----------------------------------------------------------------

NP 0 Drops:
----------------------------------------------------------------
RSV_EGR_LAG_NOT_LOCAL_DROP_CNT                               3               
----------------------------------------------------------------

No NP 1 Drops
----------------------------------------------------------------

No NP 2 Drops
----------------------------------------------------------------

No NP 3 Drops
----------------------------------------------------------------

No Bridge 0 Drops
----------------------------------------------------------------

No Bridge 1 Drops
----------------------------------------------------------------

FIA 0 Drops:
----------------------------------------------------------------
Total drop:                                                  101719          
Egress drop:                                                 101719          
Egress Uc dq pkt-len-crc/RO-seq/len error drp                101778          
----------------------------------------------------------------

FIA 1 Drops:
----------------------------------------------------------------
Total drop:                                                  3604840         
Egress drop:                                                 648284          
Ingress drop:                                                2956556         
Ingress Tail drp                                             2956556         
Egress Uc dq pkt-len-crc/RO-seq/len error drp                648284          
----------------------------------------------------------------

No FIA 2 Drops
----------------------------------------------------------------

No FIA 3 Drops
----------------------------------------------------------------

4. Active ports, traffics pattern and topology when issue happened

  • More traffics will from bundle 101-4 to bundle 100 (bundle 102 have about 5.3G traffic), base on my port traffics summary, that should have 7.5G traffics.
  • Have some traffics, E.g: from TJ via SH01 to SH02 (issue 9k should SH01)

Base on follow information, both LC10 and LC11 are ingress drop on fia that similar have BP from 10G -> 100G, but 10G port only have 2M output, so that not make sense, continue checking:

5. Base on VOQ info on LC0/1/2, have lots of packets were handle in VOQ

And destination is LC10 and LC11, so that looks like Tomahawk LC have some issue.

Attention: When Tomahawk interoperability with typhoon on same chassis, mandatory 12 VQIs per 100G port to be compatible with typhoon 10G, single VQI used for traffic between two Tomahawk 40G/100G ports.

RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/0/cpu0 
Wed May 11 13:25:49.268 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-0 **********
Category: q_stats_b-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_a-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_b-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_a-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_b-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_a-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_b-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-4 **********
Category: q_stats_a-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
169       0              2              1652      LC11_0_0
170       0              2              133       LC11_0_0
171       0              2              484       LC11_0_0
174       0              2              731       LC11_0_0
176       0              2              266       LC11_0_0
178       0              2              627       LC11_0_0
179       0              2              649       LC11_0_0
180       0              2              10092     LC11_1_1
181       0              2              55274     LC11_1_1
182       0              2              26083     LC11_1_1
183       0              2              19448     LC11_1_1
184       0              2              8816      LC11_1_1
185       0              2              20263     LC11_1_1
186       0              2              50083     LC11_1_1
187       0              2              17152     LC11_1_1
188       0              2              14478     LC11_1_1
189       0              2              6659      LC11_1_1
190       0              2              25953     LC11_1_1
191       0              2              7155      LC11_1_1
192       0              2              1538      LC11_1_1
193       0              2              1185      LC11_1_1
194       0              2              1633      LC11_1_1
195       0              2              742       LC11_1_1
196       0              2              1113      LC11_1_1
197       0              2              1156      LC11_1_1
198       0              2              826       LC11_1_1
199       0              2              3742      LC11_1_1
200       0              2              496       LC11_1_1
201       0              2              1091      LC11_1_1
202       0              2              252       LC11_1_1
203       0              2              168       LC11_1_1
>>> you can found above info that 24 VOQ for LC11_1_1
>>> that due to NP1 have 2 active 100 port, each 100 port 12 VOQ, so 24 VOQ
276       0              2              12826     LC10_1_1
277       0              2              5937      LC10_1_1
278       0              2              5479      LC10_1_1
279       0              2              7374      LC10_1_1
280       0              2              2965      LC10_1_1
281       0              2              1537      LC10_1_1
282       0              2              3696      LC10_1_1
283       0              2              4167      LC10_1_1
284       0              2              6677      LC10_1_1

 ********** FIA-4 **********
Category: q_stats_b-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
285       0              2              2413      LC10_1_1
286       0              2              1801      LC10_1_1
287       0              2              1250      LC10_1_1

 ********** FIA-5 **********
Category: q_stats_a-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-5 **********
Category: q_stats_b-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
RP/0/RP0/CPU0:xxx# 
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/1/cpu0
Wed May 11 13:26:14.476 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-0 **********
Category: q_stats_b-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_a-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_b-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_a-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_b-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_a-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_b-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-4 **********
Category: q_stats_a-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
180       0              2              18352     LC11_1_1
181       0              2              25880     LC11_1_1
182       0              2              19200     LC11_1_1
183       0              2              36550     LC11_1_1
184       0              2              4620      LC11_1_1
185       0              2              6612      LC11_1_1
186       0              2              7165      LC11_1_1
187       0              2              4407      LC11_1_1
188       0              2              2767      LC11_1_1
189       0              2              8390      LC11_1_1
190       0              2              3588      LC11_1_1
191       0              2              5385      LC11_1_1
276       0              2              13964     LC10_1_1
277       0              2              3350      LC10_1_1
278       0              2              2862      LC10_1_1
279       0              2              8041      LC10_1_1
280       0              2              1146      LC10_1_1
281       0              2              3302      LC10_1_1
282       0              2              1141      LC10_1_1
283       0              2              192       LC10_1_1
284       0              2              5342      LC10_1_1
285       0              2              761       LC10_1_1
286       0              2              1843      LC10_1_1
287       0              2              1079      LC10_1_1

 ********** FIA-4 **********
Category: q_stats_b-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-5 **********
Category: q_stats_a-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-5 **********
Category: q_stats_b-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
RP/0/RP0/CPU0:xxx#      
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/2/cpu0
Wed May 11 13:26:37.671 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-0 **********
Category: q_stats_b-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_a-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_b-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_a-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_b-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_a-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_b-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-4 **********
Category: q_stats_a-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
180       0              2              38448     LC11_1_1
181       0              2              39954     LC11_1_1
182       0              2              51617     LC11_1_1
184       0              2              834       LC11_1_1
185       0              2              14837     LC11_1_1
186       0              2              6866      LC11_1_1
189       0              2              25281     LC11_1_1
190       0              2              54585     LC11_1_1
192       0              2              524       LC11_1_1
193       0              2              171       LC11_1_1
194       0              2              339       LC11_1_1
196       0              2              84        LC11_1_1
197       0              2              252       LC11_1_1
276       0              2              63093     LC10_1_1
277       0              2              29402     LC10_1_1
278       0              2              86107     LC10_1_1
279       0              2              101730    LC10_1_1
280       0              2              46332     LC10_1_1
281       0              2              8185      LC10_1_1
282       0              2              20360     LC10_1_1
283       0              2              23855     LC10_1_1
284       0              2              20146     LC10_1_1
285       0              2              15464     LC10_1_1
286       0              2              35014     LC10_1_1
287       0              2              8995      LC10_1_1

 ********** FIA-4 **********
Category: q_stats_b-4
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-5 **********
Category: q_stats_a-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-5 **********
Category: q_stats_b-5
Voq       ddr            pri            pktcnt    Slot_FIA_NP

6. Follow VOQ info on LC10, lots of packets via LC10 FIA1, NP1 was handle in VOQ

That VOQ map to hundredgi 0/10/0/3:

RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/10/cpu0
>>> this command can check maping between physical port and VOI            
Wed May 11 13:34:19.667 Beijing
Platform-manager VQI Assignment Information 
      Interface Name      | ifh Value | VQI | NP# 
--------------------------------------------------
      HundredGigE0_10_0_2 | 0x180001c0 | 276 |   1 
      HundredGigE0_10_0_3 | 0x18000200 | 288 |   1 

RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/10/cpu0
Wed May 11 13:34:55.667 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
276       0              2              17525     LC10_1_1  
<<< traffics from 0/10/0/2 to 0/10/0/3 in same NP 

 ********** FIA-0 **********
Category: q_stats_b-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_a-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
111       0              2              18        LC2_4_4
>>> base on LC10 VOQ information, less packets from LC10 to LC2
264       0              2              17        LC10_0_0

 ********** FIA-1 **********
Category: q_stats_b-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_a-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-2 **********
Category: q_stats_b-2
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_a-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-3 **********
Category: q_stats_b-3
Voq       ddr            pri            pktcnt    Slot_FIA_NP  

7. Follow VOQ info on LC11, lots of packets that via LC10/11 FIA1 was handle in VOQ

RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/10/cpu0            
Wed May 11 13:34:19.667 Beijing
Platform-manager VQI Assignment Information 
      Interface Name      | ifh Value | VQI | NP# 
--------------------------------------------------
      HundredGigE0_10_0_2 | 0x180001c0 | 276 |   1 
      HundredGigE0_10_0_3 | 0x18000200 | 288 |   1 

RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/11/cpu0
Wed May 11 13:34:42.501 Beijing
Platform-manager VQI Assignment Information 
      Interface Name      | ifh Value | VQI | NP# 
--------------------------------------------------
     HundredGigE0_11_0_2 | 0x1a000180 | 180 |   1 
     HundredGigE0_11_0_3 | 0x1a000280 | 192 |   1 

RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/11/cpu0
Wed May 11 13:35:09.550 Beijing

 ********** FIA-0 **********
Category: q_stats_a-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
276       0              2              14089     LC10_1_1
>>> traffics from lc11 to lc10

 ********** FIA-0 **********
Category: q_stats_b-0
Voq       ddr            pri            pktcnt    Slot_FIA_NP    

 ********** FIA-1 **********
Category: q_stats_a-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP    
111       0              2              51        LC2_4_4
168       0              2              73        LC11_0_0
180       0              2              15499     LC11_1_1
>>> local traffics from lc11 0/11/0/2 to 0/11/0/3 
276       0              2              4377      LC10_1_1

 ********** FIA-1 **********
Category: q_stats_b-1
Voq       ddr            pri            pktcnt    Slot_FIA_NP

8. We found xbar fabric down that cause the issue happened

+++ sh contr fabric crossbar link-status inst 0 location 0/2/CPU0 [13:47:43.871 Beijing Wed May 11 2016] ++++

PORT    Remote Slot  Remote Inst    Logical ID  Status
======================================================
00      0/2/CPU0            02             1        Up
01      0/2/CPU0            01             1        Up
02      0/2/CPU0            01             0        Up
03      0/2/CPU0            00             0        Up
04      0/2/CPU0            00             1        Up
05      0/2/CPU0            03             1        Up
06      0/2/CPU0            05             1        Up
07      0/FC3/SP            01             0        Down
08      0/2/CPU0            03             0        Up
09      0/FC3/SP            00             0        Up
10      0/2/CPU0            05             0        Up
11      0/FC4/SP            01             0        Up
12      0/FC4/SP            00             0        Up
14      0/FC2/SP            00             0        Up
15      0/FC2/SP            01             0        Down
16      0/FC1/SP            00             0        Up
17      0/FC1/SP            01             0        Down
20      0/FC0/SP            00             0        Up
22      0/FC0/SP            01             0        Down
23      0/2/CPU0            04             1        Up
24      0/2/CPU0            02             0        Up
25      0/2/CPU0            04             0        Up

9. Why fabric down will cause traffics drop? And how to explnation phenomonan?

When remote FIA rebuild the packets, found packets issue, then drop and report CRC, so there were amount CRC on FIA, that will continue consume credit, and due to packets drop, the credit will not return to RSP, so egress VOQ resource exhausted and you can found amount packets inqueue that to egress NP, so ingress FIA have tail drop. Btw, when fabric link down that only report one times in log each LC bootup, or you can check that by “show pfm location all“.

10. Which component have issue, LC2, slot2 or FC ?

Base on test, we found the issue only relation with slot2, and down link changed each reset LC2. And checked FC that should no issue. so RMA chassis and EFA.

11. Base on FA result, found FC slots bend pins on BP, and multi FC

本文出自 Frank's Blog

版权声明:


本文链接:Troubleshooting fabric FIA tail drop on ASR9k
版权声明:本文为原创文章,仅代表个人观点,版权归 Frank Zhao 所有,转载时请注明本文出处及文章链接
你可以留言,或者trackback 从你的网站

留言哦

blonde teen swallows load.xxx videos