Troubleshooting fabric FIA tail drop on ASR9k
17 6 月, 2016,由 frank撰写
Introduction
In TZ database, have more good documents that is troubleshooting fabric guide on ASR9k, but no analysis process that show how to troubleshooting fabric issue on real scenario/CASE. “Very lucky” ? I matched a hot CASE that due to fabric issue cause online fail. So i summaried totally analysis process that will help CSE to narrow down similar issue.
Problem Description
- Platform: 9922 + 4 36x10G + 2 8x100G
- Version: 5.3.2 + SMU
My customer online a new 9922 to replace old devices. After online, found their business have traffics drop. Base on online information, I found NP no more drop, and business traffics very less. (max 5g under 100g port, all bundle port when online ts)
But fia have lots of drop and amount packets in VOQ:
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/10/cpu0
Wed May 11 13:03:04.841 Beijing
RP/0/RP0/CPU0:xxx#
PP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/11/cpu0 | i drop
Wed May 11 13:03:27.550 Beijing
Ingress drop: 0
Egress drop: 1918
Total drop: 1918
Ingress drop: 43820
Egress drop: 13121
Total drop: 56941
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/10/cpu0
Wed May 11 13:24:44.920 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
111 0 2 19 LC2_4_4
180 0 2 24326 LC11_1_1
192 0 2 648 LC11_1_1
276 0 2 12555 LC10_1_1
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/2/cpu0
Wed May 11 13:26:37.671 Beijing
********** FIA-4 **********
Category: q_stats_a-4
Voq ddr pri pktcnt Slot_FIA_NP
180 0 2 38448 LC11_1_1
181 0 2 39954 LC11_1_1
182 0 2 51617 LC11_1_1
184 0 2 834 LC11_1_1
185 0 2 14837 LC11_1_1
186 0 2 6866 LC11_1_1
189 0 2 25281 LC11_1_1
190 0 2 54585 LC11_1_1
192 0 2 524 LC11_1_1
193 0 2 171 LC11_1_1
194 0 2 339 LC11_1_1
196 0 2 84 LC11_1_1
197 0 2 252 LC11_1_1
276 0 2 63093 LC10_1_1
277 0 2 29402 LC10_1_1
278 0 2 86107 LC10_1_1
279 0 2 101730 LC10_1_1
280 0 2 46332 LC10_1_1
281 0 2 8185 LC10_1_1
282 0 2 20360 LC10_1_1
283 0 2 23855 LC10_1_1
284 0 2 20146 LC10_1_1
285 0 2 15464 LC10_1_1
286 0 2 35014 LC10_1_1
287 0 2 8995 LC10_1_1
********** FIA-4 **********
Category: q_stats_b-4
Voq ddr pri pktcnt Slot_FIA_NP All bundle ports use follow QOS prifile, I checked QOS drop for some bundles, no found any drop.
policy-map Diff-Sev
class EXP-Control
priority level 1
police rate percent 5
!
!
class EXP-Gold
priority level 2
police rate percent 75
!
!
class EXP-Silver
priority level 3
queue-limit 10 ms
!
class EXP-Copper
bandwidth remaining percent 50
!
class class-default
bandwidth remaining percent 50
!
end-policy-map
! Troubleshooting
1. Clear counter at May 11 13:02
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/0/cpu0
Wed May 11 13:02:54.243 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/10/cpu0
Wed May 11 13:03:04.841 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/11/cpu0
Wed May 11 13:03:14.907 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/1/cpu0
Wed May 11 13:03:47.835 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/2/cpu0
Wed May 11 13:04:02.986 Beijing
RP/0/RP0/CPU0:xxx#clear controller fabric fia location 0/3/cpu0
Wed May 11 13:04:38.460 Beijing2. After checked fia drop, have follow found
- lc02 FIA4 egress drop increased
- lc10 FIA0 egress drop increased
- lc10 FIA1 ingress/egress drop increased
- lc11 FIA0 egress drop increased
- lc11 FIA1 ingress/egress drop increased
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/0/cpu0 | i drop
Wed May 11 13:04:56.381 Beijing
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/1/cpu0 | i drop
Wed May 11 13:05:06.780 Beijing
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/2/cpu0 | i drop
Wed May 11 13:05:14.797 Beijing
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 31602
Total drop: 31602
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/3/cpu0 | i drop
Wed May 11 13:05:44.327 Beijing
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/10/cpu0 | i drop
Wed May 11 13:05:54.383 Beijing
Ingress drop: 0
Egress drop: 32651
Total drop: 32651
Ingress drop: 501560
Egress drop: 144065
Total drop: 645625
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia stats location 0/11/cpu0 | i drop
Wed May 11 13:06:03.356 Beijing
Ingress drop: 0
Egress drop: 27203
Total drop: 27203
Ingress drop: 771349
Egress drop: 170823
Total drop: 942172
Ingress drop: 0
Egress drop: 0
Total drop: 0
Ingress drop: 0
Egress drop: 0
Total drop: 03. Checked what packets drop by “show drops”
- There is a questions, lc2 no FIA4 in “show drops” … But lc2’s egress drop mostly same as follow.
- lc10 fia 0 egress drop that relation wth “Egress Uc dq pkt-len-crc/RO-seq/len error drop“
- lc10 fia 1 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop“, other drops are ingress drop
- lc11 fia 0 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop“
- lc11 FIA1 egress drop that relation with “Egress Uc dq pkt-len-crc/RO-seq/len error drop“, other drops are ingress drop
RP/0/RP0/CPU0:xxx#sh drops
Wed May 11 13:13:41.304 Beijing
Node: 0/0/CPU0:
----------------------------------------------------------------
No NP 0 Drops
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
NP 4 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH 99
----------------------------------------------------------------
NP 5 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH 176
----------------------------------------------------------------
No FIA 0 Drops
----------------------------------------------------------------
No FIA 1 Drops
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------
Node: 0/1/CPU0:
----------------------------------------------------------------
No NP 0 Drops
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
NP 4 Drops:
----------------------------------------------------------------
RSV_DROP_MPLS_TXADJ_NO_MATCH 277
----------------------------------------------------------------
No NP 5 Drops
----------------------------------------------------------------
No Bridge 0 Drops
----------------------------------------------------------------
No Bridge 1 Drops
----------------------------------------------------------------
No FIA 0 Drops
----------------------------------------------------------------
No FIA 1 Drops
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------
Node: 0/2/CPU0:
----------------------------------------------------------------
No NP 0 Drops
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
No NP 4 Drops
----------------------------------------------------------------
No NP 5 Drops
----------------------------------------------------------------
No Bridge 0 Drops
----------------------------------------------------------------
No Bridge 1 Drops
----------------------------------------------------------------
No FIA 0 Drops
----------------------------------------------------------------
No FIA 1 Drops
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------
Node: 0/3/CPU0:
----------------------------------------------------------------
No NP 0 Drops
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
No NP 4 Drops
----------------------------------------------------------------
NP 5 Drops:
----------------------------------------------------------------
PARSE_DROP_IPV4_DISABLED 341
----------------------------------------------------------------
No Bridge 0 Drops
----------------------------------------------------------------
No Bridge 1 Drops
----------------------------------------------------------------
No FIA 0 Drops
----------------------------------------------------------------
No FIA 1 Drops
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------
Node: 0/10/CPU0:
----------------------------------------------------------------
No NP 0 Drops
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
No Bridge 0 Drops
----------------------------------------------------------------
No Bridge 1 Drops
----------------------------------------------------------------
FIA 0 Drops:
----------------------------------------------------------------
Total drop: 129021
Egress drop: 129021
Egress Uc dq pkt-len-crc/RO-seq/len error drp 129104
----------------------------------------------------------------
FIA 1 Drops:
----------------------------------------------------------------
Total drop: 2734078
Egress drop: 567642
Ingress drop: 2166436
Ingress Tail drp 2166436
Egress Uc dq pkt-len-crc/RO-seq/len error drp 567642
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------
Node: 0/11/CPU0:
----------------------------------------------------------------
NP 0 Drops:
----------------------------------------------------------------
RSV_EGR_LAG_NOT_LOCAL_DROP_CNT 3
----------------------------------------------------------------
No NP 1 Drops
----------------------------------------------------------------
No NP 2 Drops
----------------------------------------------------------------
No NP 3 Drops
----------------------------------------------------------------
No Bridge 0 Drops
----------------------------------------------------------------
No Bridge 1 Drops
----------------------------------------------------------------
FIA 0 Drops:
----------------------------------------------------------------
Total drop: 101719
Egress drop: 101719
Egress Uc dq pkt-len-crc/RO-seq/len error drp 101778
----------------------------------------------------------------
FIA 1 Drops:
----------------------------------------------------------------
Total drop: 3604840
Egress drop: 648284
Ingress drop: 2956556
Ingress Tail drp 2956556
Egress Uc dq pkt-len-crc/RO-seq/len error drp 648284
----------------------------------------------------------------
No FIA 2 Drops
----------------------------------------------------------------
No FIA 3 Drops
----------------------------------------------------------------4. Active ports, traffics pattern and topology when issue happened
- More traffics will from bundle 101-4 to bundle 100 (bundle 102 have about 5.3G traffic), base on my port traffics summary, that should have 7.5G traffics.
- Have some traffics, E.g: from TJ via SH01 to SH02 (issue 9k should SH01)
Base on follow information, both LC10 and LC11 are ingress drop on fia that similar have BP from 10G -> 100G, but 10G port only have 2M output, so that not make sense, continue checking:
5. Base on VOQ info on LC0/1/2, have lots of packets were handle in VOQ
And destination is LC10 and LC11, so that looks like Tomahawk LC have some issue.
Attention: When Tomahawk interoperability with typhoon on same chassis, mandatory 12 VQIs per 100G port to be compatible with typhoon 10G, single VQI used for traffic between two Tomahawk 40G/100G ports.
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/0/cpu0
Wed May 11 13:25:49.268 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_b-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_a-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_b-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_a-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_b-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-4 **********
Category: q_stats_a-4
Voq ddr pri pktcnt Slot_FIA_NP
169 0 2 1652 LC11_0_0
170 0 2 133 LC11_0_0
171 0 2 484 LC11_0_0
174 0 2 731 LC11_0_0
176 0 2 266 LC11_0_0
178 0 2 627 LC11_0_0
179 0 2 649 LC11_0_0
180 0 2 10092 LC11_1_1
181 0 2 55274 LC11_1_1
182 0 2 26083 LC11_1_1
183 0 2 19448 LC11_1_1
184 0 2 8816 LC11_1_1
185 0 2 20263 LC11_1_1
186 0 2 50083 LC11_1_1
187 0 2 17152 LC11_1_1
188 0 2 14478 LC11_1_1
189 0 2 6659 LC11_1_1
190 0 2 25953 LC11_1_1
191 0 2 7155 LC11_1_1
192 0 2 1538 LC11_1_1
193 0 2 1185 LC11_1_1
194 0 2 1633 LC11_1_1
195 0 2 742 LC11_1_1
196 0 2 1113 LC11_1_1
197 0 2 1156 LC11_1_1
198 0 2 826 LC11_1_1
199 0 2 3742 LC11_1_1
200 0 2 496 LC11_1_1
201 0 2 1091 LC11_1_1
202 0 2 252 LC11_1_1
203 0 2 168 LC11_1_1
>>> you can found above info that 24 VOQ for LC11_1_1
>>> that due to NP1 have 2 active 100 port, each 100 port 12 VOQ, so 24 VOQ
276 0 2 12826 LC10_1_1
277 0 2 5937 LC10_1_1
278 0 2 5479 LC10_1_1
279 0 2 7374 LC10_1_1
280 0 2 2965 LC10_1_1
281 0 2 1537 LC10_1_1
282 0 2 3696 LC10_1_1
283 0 2 4167 LC10_1_1
284 0 2 6677 LC10_1_1
********** FIA-4 **********
Category: q_stats_b-4
Voq ddr pri pktcnt Slot_FIA_NP
285 0 2 2413 LC10_1_1
286 0 2 1801 LC10_1_1
287 0 2 1250 LC10_1_1
********** FIA-5 **********
Category: q_stats_a-5
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-5 **********
Category: q_stats_b-5
Voq ddr pri pktcnt Slot_FIA_NP
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/1/cpu0
Wed May 11 13:26:14.476 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_b-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_a-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_b-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_a-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_b-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-4 **********
Category: q_stats_a-4
Voq ddr pri pktcnt Slot_FIA_NP
180 0 2 18352 LC11_1_1
181 0 2 25880 LC11_1_1
182 0 2 19200 LC11_1_1
183 0 2 36550 LC11_1_1
184 0 2 4620 LC11_1_1
185 0 2 6612 LC11_1_1
186 0 2 7165 LC11_1_1
187 0 2 4407 LC11_1_1
188 0 2 2767 LC11_1_1
189 0 2 8390 LC11_1_1
190 0 2 3588 LC11_1_1
191 0 2 5385 LC11_1_1
276 0 2 13964 LC10_1_1
277 0 2 3350 LC10_1_1
278 0 2 2862 LC10_1_1
279 0 2 8041 LC10_1_1
280 0 2 1146 LC10_1_1
281 0 2 3302 LC10_1_1
282 0 2 1141 LC10_1_1
283 0 2 192 LC10_1_1
284 0 2 5342 LC10_1_1
285 0 2 761 LC10_1_1
286 0 2 1843 LC10_1_1
287 0 2 1079 LC10_1_1
********** FIA-4 **********
Category: q_stats_b-4
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-5 **********
Category: q_stats_a-5
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-5 **********
Category: q_stats_b-5
Voq ddr pri pktcnt Slot_FIA_NP
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/2/cpu0
Wed May 11 13:26:37.671 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_b-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_a-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_b-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_a-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_b-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-4 **********
Category: q_stats_a-4
Voq ddr pri pktcnt Slot_FIA_NP
180 0 2 38448 LC11_1_1
181 0 2 39954 LC11_1_1
182 0 2 51617 LC11_1_1
184 0 2 834 LC11_1_1
185 0 2 14837 LC11_1_1
186 0 2 6866 LC11_1_1
189 0 2 25281 LC11_1_1
190 0 2 54585 LC11_1_1
192 0 2 524 LC11_1_1
193 0 2 171 LC11_1_1
194 0 2 339 LC11_1_1
196 0 2 84 LC11_1_1
197 0 2 252 LC11_1_1
276 0 2 63093 LC10_1_1
277 0 2 29402 LC10_1_1
278 0 2 86107 LC10_1_1
279 0 2 101730 LC10_1_1
280 0 2 46332 LC10_1_1
281 0 2 8185 LC10_1_1
282 0 2 20360 LC10_1_1
283 0 2 23855 LC10_1_1
284 0 2 20146 LC10_1_1
285 0 2 15464 LC10_1_1
286 0 2 35014 LC10_1_1
287 0 2 8995 LC10_1_1
********** FIA-4 **********
Category: q_stats_b-4
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-5 **********
Category: q_stats_a-5
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-5 **********
Category: q_stats_b-5
Voq ddr pri pktcnt Slot_FIA_NP6. Follow VOQ info on LC10, lots of packets via LC10 FIA1, NP1 was handle in VOQ
That VOQ map to hundredgi 0/10/0/3:
RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/10/cpu0
>>> this command can check maping between physical port and VOI
Wed May 11 13:34:19.667 Beijing
Platform-manager VQI Assignment Information
Interface Name | ifh Value | VQI | NP#
--------------------------------------------------
HundredGigE0_10_0_2 | 0x180001c0 | 276 | 1
HundredGigE0_10_0_3 | 0x18000200 | 288 | 1
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/10/cpu0
Wed May 11 13:34:55.667 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
276 0 2 17525 LC10_1_1
<<< traffics from 0/10/0/2 to 0/10/0/3 in same NP
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri pktcnt Slot_FIA_NP
111 0 2 18 LC2_4_4
>>> base on LC10 VOQ information, less packets from LC10 to LC2
264 0 2 17 LC10_0_0
********** FIA-1 **********
Category: q_stats_b-1
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_a-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-2 **********
Category: q_stats_b-2
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_a-3
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-3 **********
Category: q_stats_b-3
Voq ddr pri pktcnt Slot_FIA_NP 7. Follow VOQ info on LC11, lots of packets that via LC10/11 FIA1 was handle in VOQ
RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/10/cpu0
Wed May 11 13:34:19.667 Beijing
Platform-manager VQI Assignment Information
Interface Name | ifh Value | VQI | NP#
--------------------------------------------------
HundredGigE0_10_0_2 | 0x180001c0 | 276 | 1
HundredGigE0_10_0_3 | 0x18000200 | 288 | 1
RP/0/RP0/CPU0:xxx#show controllers pm vqi location 0/11/cpu0
Wed May 11 13:34:42.501 Beijing
Platform-manager VQI Assignment Information
Interface Name | ifh Value | VQI | NP#
--------------------------------------------------
HundredGigE0_11_0_2 | 0x1a000180 | 180 | 1
HundredGigE0_11_0_3 | 0x1a000280 | 192 | 1
RP/0/RP0/CPU0:xxx#show controllers fabric fia q-depth location 0/11/cpu0
Wed May 11 13:35:09.550 Beijing
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP
276 0 2 14089 LC10_1_1
>>> traffics from lc11 to lc10
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri pktcnt Slot_FIA_NP
111 0 2 51 LC2_4_4
168 0 2 73 LC11_0_0
180 0 2 15499 LC11_1_1
>>> local traffics from lc11 0/11/0/2 to 0/11/0/3
276 0 2 4377 LC10_1_1
********** FIA-1 **********
Category: q_stats_b-1
Voq ddr pri pktcnt Slot_FIA_NP8. We found xbar fabric down that cause the issue happened
+++ sh contr fabric crossbar link-status inst 0 location 0/2/CPU0 [13:47:43.871 Beijing Wed May 11 2016] ++++
PORT Remote Slot Remote Inst Logical ID Status
======================================================
00 0/2/CPU0 02 1 Up
01 0/2/CPU0 01 1 Up
02 0/2/CPU0 01 0 Up
03 0/2/CPU0 00 0 Up
04 0/2/CPU0 00 1 Up
05 0/2/CPU0 03 1 Up
06 0/2/CPU0 05 1 Up
07 0/FC3/SP 01 0 Down
08 0/2/CPU0 03 0 Up
09 0/FC3/SP 00 0 Up
10 0/2/CPU0 05 0 Up
11 0/FC4/SP 01 0 Up
12 0/FC4/SP 00 0 Up
14 0/FC2/SP 00 0 Up
15 0/FC2/SP 01 0 Down
16 0/FC1/SP 00 0 Up
17 0/FC1/SP 01 0 Down
20 0/FC0/SP 00 0 Up
22 0/FC0/SP 01 0 Down
23 0/2/CPU0 04 1 Up
24 0/2/CPU0 02 0 Up
25 0/2/CPU0 04 0 Up9. Why fabric down will cause traffics drop? And how to explnation phenomonan?
When remote FIA rebuild the packets, found packets issue, then drop and report CRC, so there were amount CRC on FIA, that will continue consume credit, and due to packets drop, the credit will not return to RSP, so egress VOQ resource exhausted and you can found amount packets inqueue that to egress NP, so ingress FIA have tail drop. Btw, when fabric link down that only report one times in log each LC bootup, or you can check that by “show pfm location all“.
10. Which component have issue, LC2, slot2 or FC ?
Base on test, we found the issue only relation with slot2, and down link changed each reset LC2. And checked FC that should no issue. so RMA chassis and EFA.11. Base on FA result, found FC slots bend pins on BP, and multi FC…
版权声明：
本文链接：Troubleshooting fabric FIA tail drop on ASR9k
版权声明：本文为原创文章，仅代表个人观点，版权归 Frank Zhao 所有，转载时请注明本文出处及文章链接