关于Cat6k/7600 High CPU TS的相关命令
High Cpu Capture Packets No.1 by Span Way
A SPAN for the RP or SP port in Cisco IOS Software is available in Cisco IOS Software Release 12.1(19)E and later.
Example:
switch# monitor session 1 source interface switch# monitor session 1 destination interface
Now go to SP console:, we can do this on following way :
switch#remote log sw switch-sp# switch-sp#test monitor add 1 rp-inband rx test monitor add|del session# rp_inband|sp_inband [rx|tx|both]
Catalyst 6500/6000 Switch High CPU Utilization
http://www.cisco.com/en/US/partner/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml#utilities
This will allow us to see if the traffic being send to the CPU is valid and if so, if there are any possible actions which can be taken against this. Please make sure that the sniff traces is taken at the same intervals when we are having the output of the “show process cpu”.
High Cpu Capture Packets No.2 by Buffer
#Config t
#service internal
#exit
#sh platform capture buffer asic pinnacle slot 5 port 4 direction out priority lo
//notice:slot 5 is active
#sh platform capture buffer status
#sh platform capture buffer collect for 20
......
wait 20 seconds
......
#sh platform capture buffer status
#sh vlan internal usage
#sh platform capture buffer data filter
#sh platform capture buffer data sample 1
#sh platform capture buffer data sample 2
#sh platform capture buffer data sample 3
#sh platform capture buffer data sample 4
#sh platform capture buffer data sample 5
#config t
#no service internal
1. port 4 is RP, port 3 is SP.
2. direction out – to cpu, direction in – back from cpu
3. priority: lo, hi, very, cisco separate traffic to three queue at out direction.
BPDU in very high queue. Sometime, due to self-loop, will lead to SP will receive amount BPDU and high cpu, then you may capture packets by follow command:
“sh platform capture buffer asic pinnacle slot 5 port 3 direction out priority very”
Btw, if the LC have lost packets and asic is pinnacle, you can use top command and confirm problem too.
注意:由于这种方式是检查packet buffer中存留的信息,所以如果流量很快,那么这里面的信息很容易被refresh。
High Cpu Capture Packets No.3 by Netdr
#debug netdr capture rx interface g4/1
#no debug netdr capture
#show netdr cap
when you captureed by Netdr, can decord by follow way:
Wireshark使用技巧之三: 利用text2pcap转换Hex文本到pcap
High Cpu Capture Packets No.4 by ELAM
ELAM是在65/76上的一个internal工具,可以用来检测不同部位(为检测所预留的)连通性的工具,他不同于前面的几种方法,它一次只能抓到一个包,由于它是65/76中一种抓包方法,所以就一起放在这篇文章里了,对于cpu高,可能用处不是非常大。
TestMacNotification is a health monitoring test. If it fails, to debug it, the ELAM capturing is the best tool. The datapath in this test includes supervisor inband port, supervisor rewrite engine, supervisor earl, fabric, linecard rewrite engine, linecard earl. 4 ELAM captures can usually let the root-caused exposed. In the following example, suppose supervisor slot is 5 and linecard slot is 1:
The TestMacNotification test verifies that the data and control path between DFC modules and supervisor engines is working properly. This test also ensures Layer 2 MAC address consistency across Layer 2 MAC address tables. The test runs every six seconds. Ten consecutive failures causes the module to reset during bootup or runtime (default). After three consecutive resets, the module powers down.
A. ELAM capture in supervisor for diagnostic packet:
可以抓到SP EARL’s RBUS的包,包的目的index如果是LC,那么它将通过fabric到达LC
sh platform capture elam asic superman slot 5 instance 0
sh platform capture elam trigger dbus if VLAN=0x3EE
sh platform capture elam start|stat|data
B. ELAM capture in linecard for diagnostic packet:
抓到的包是由LC Rewrite Engine ASIC产生的
sh platform capture elam asic superman slot 1 instance 0/1
sh platform capture elam trigger dbus if VLAN=0x3EE
sh platform capture elam start|stat|data
C. ELAM capture in linecard for macn packet:
可以判断是否生成MN包,这个包的目的地址应该是SP的inband端口
sh platform capture elam asic superman slot 1 instance 0/1
sh platform capture elam trigger dbus if VLAN=0x3EE MN=1
sh platform capture elam start|stat|data
D. ELAM capture in supervisor for macn packet:
可以判断SP是否收到MN包,收到后,SP根据MN的内容更改L2的entry
sh platform capture elam asic superman slot 5 instance 0
sh platform capture elam trigger dbus if VLAN=0x3EE MN=1
sh platform capture elam start|stat|data
High Cpu Capture Packets No.5 by EEM and Netdr
EEM是一个比较实用的工具,可以自己设置脚本,并让其在某种情况下自行执行脚本,在之前的文章里我介绍过《SLA与EEM的联动》。在这里,用于抓取Punt到CPU的包。EEM脚本默认会运行20s,可以用maxrun来增加这个参数
注意:为了使netdr能抓到故障时的信息并且保存到存储地点(信息存储也需要一定的时间),可以用action x.x wait 30来增加等待时间,但这时一定要加大整个EEM Run的时间“maxrun”并大于wait的时间,否则信息会抓不下来。另外下面的OID一定要反应active RP的,建议用此脚本前在LAB做下测试。
event manager applet cpu_stats event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val 90 exit-op lt exit-val 80 poll-interval 2 maxrun 50 action 1.1 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %" action 1.2 cli command "enable" action 1.3 cli command "debug netdr capture rx" action 1.4 wait 25 action 1.5 cli command "show clock | append sup-bootdisk:cpu_stats" action 1.6 cli command "show log | append sup-bootdisk:cpu_stats" action 1.7 cli command "show proc cpu sort | append sup-bootdisk:cpu_stats" action 1.8 cli command "remote command switch show proc cpu sort | append sup-bootdisk:cpu_stats" action 1.9 cli command "show platform hardware capacity | append sup-bootdisk:cpu_stats" action 2.0 cli command "show diagnostic result module all detail | append sup-bootdisk:cpu_stats" action 2.1 cli command "show diagnostic events | append sup-bootdisk:cpu_stats" action 2.2 cli command "show spanning-tree | append sup-bootdisk:cpu_stats" action 2.3 cli command "show spanning-tree summary | append sup-bootdisk:cpu_stats" action 2.4 cli command "show spanning-tree detail | append sup-bootdisk:cpu_stats" action 2.5 cli command "show int | append sup-bootdisk:cpu_stats" action 2.6 cli command "no debug netdr capture" action 2.7 cli command "show netdr captured-packets | append sup-bootdisk:cpu_stats" action 2.8 cli command "debug netdr clear-capture" action 2.9 cli command "config ter" action 3.1 cli command "no event manager applet cpu_stats"