Archive

标签为 ‘Troubleshooting’的文章

Troubleshooting “%FABRIC-INGRESSQ-6-LINK_DOWN” on CRS

Introduction

Customer found 0/4/cpu0 have many ingressq asic error, after checked, that should match a know DDTS: CSCuu86430. The issue maybe was triggered when CRS-3 MSCs(140G) interactive with a CRS-X(400G) fabric. After trigger the issue, will found CRS-X’s fabric link of s1rx flapping. Have reload SMU under 514.

For this article, will show how to troubleshooting the fabric link flapping.

Troubleshooting

1. Customer found follow alarm:

LC/0/4/CPU0:Nov  8 00:30:26.752 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 
LC/0/1/CPU0:Nov  8 00:37:59.734 : fabricq_mgr[178]: %FABRIC-FABRICQ-3-PCL_PKT : Minor error in PCL of fabricq asic 0. PCL UC Lost Packet: CAOPCI: 0x18 (0/4, UC, LO):Lost Packet count= 1 
LC/0/4/CPU0:Nov  8 00:37:59.734 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 
LC/0/4/CPU0:Nov  8 10:27:27.265 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 
LC/0/4/CPU0:Nov  8 11:06:08.181 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 
LC/0/1/CPU0:Nov  8 11:08:46.132 : fabricq_mgr[178]: %FABRIC-FABRICQ-3-PCL_PKT : Minor error in PCL of fabricq asic 0. PCL UC Partial Packet: CAOPCI: 0x18 (0/4, UC, LO) 
LC/0/4/CPU0:Nov  8 11:18:34.733 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 
LC/0/4/CPU0:Nov  8 11:28:44.350 : ingressq[235]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 26 of Asic Instance 0 has been administratively shut down. 

完整阅读

Troubleshooting fabric FIA tail drop on ASR9k

Introduction

In TZ database, have more good documents that is troubleshooting fabric guide on ASR9k, but no analysis process that show how to troubleshooting fabric issue on real scenario/CASE. “Very lucky” ?  I matched a hot CASE that due to fabric issue cause online fail. So i summaried totally analysis process that will help CSE to narrow down similar issue.

Problem Description

  • Platform: 9922 + 4 36x10G + 2 8x100G
  • Version: 5.3.2 + SMU 

My customer online a new 9922 to replace old devices. After online, found their business have traffics drop. Base on online information, I found NP no more drop, and business traffics very less. (max 5g under 100g port, all bundle port when online ts)

完整阅读
blonde teen swallows load.xxx videos