How to capture packets that dropped by NP/CPU?

之前讨论过在XR上,当我们遇到与本设备交互的TCP/UDP和RAW有问题时,可以用下面方法抓下来,然后分析《How to decode TCP, UDP and RAW for IOS-XR》。在76/65上,可以用Netdr,ELAM,CPU span,PB capture,那在咱们的ASR9k上是否有类似好用的工具?答案当然是肯定的,在咱们ASR9k上有两个方法可以抓punt到CPU的包:

在4.3.1以后,XR支持NP Monitor

详细的可以看下面文章: https://supportforums.cisco.com/docs/DOC-29010
这里要注意的是:Note that a captured packet will be DROPPED!
所以一定要注意在选择monitor的counter时,要选择真正的drop counter,而不要把正常的counter给monitor了,那样数据转发就会受影响了。暂时没有测试环境,没法贴出详细测试步骤,文档写的很清楚,详细看上面的文档。

在ASR9k上通过SPP Monitor抓包

下面是一个抓包实例:

RP/0/RSP0/CPU0:ASR9K.18#run attach 0/6/cpu0
Sat Oct 19 01:15:36.985 UTC

attach: Starting session 1 to node 0/6/cpu0

# spp_ui
spp-ui> node counter
copp_sampler
     Disable messages received:               1
-------------------------------
port3/classify
                     forwarded:           59294
Punted to default punt process:            2264
-------------------------------
port3/rx
           whole pkts received:          768891
      begin fragments received:             500
      total fragments received:            1248
        end fragments received:             500
           RX bailout/longjmps:          765405
incomplete chopper ctx after b:              40
-------------------------------
port2/tx
           packets transmitted:            1136
-------------------------------
port3/tx
           packets transmitted:            1139
-------------------------------
client/inject
               inject to port2:            1136
               inject to port3:            1139
-------------------------------
client/punt
              punted to client:          769391
-------------------------------
spp-ui>
spp-ui> trace start 100
spp-ui> trace stop
spp-ui> trace print
Packet serial 2
client/punt:
  length 128 phys_int_index 1 next_ctx 0xdeadbeef

下面拿出一个包来详细说明:
用SPP抓包最好的方法就是先抓一些包下来,然后看看offset,根据具体的需求,用过滤的方法注意过滤,然后最终抓到想要的包。因为不同的平台,不同的包,offset是不一样的,所以要现抓样本分析。

Packet serial 4
client/punt:
length 128 phys_int_index 1 next_ctx 0xdeadbeef
00: 00 70 72 00 00 03 00 65 7a 00 00 03 05 09 ff ff
10: 08 00 00 00 05 02 0f 00 00 00 00 00 02 00 10 00
20: 00 c0 00 00 c7 de 00 00 09 01 00 1f 00 03 0e 01
30: 00 00 00 00 3c 00 00 00 00 00 00 00 00 00 00 00
40: 00 07 02 ff 00 26 98 29 4c e8 01 25 68 98 cf a6
50: 08 00 45 00 00 2e 00 00 00 00 40 06 64 b5 03 03
60: 03 03 08 08 08 08 01 4d 03 e7 00 00 00 00 00 00
70: 00 00 50 00 00 00 8e 8c 00 00 00 01 02 03 04 05
80: 09 8b 66 5d 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 01 00 00 20 43 00 00 00 d3 5c 02 df 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

下面是常用过滤方法和offset的使用方法:

 spp-ui> trace filter set 94 2 0x0303
 !---src address, 94=0x50+dec14=dec(5*16+0)+dec14
 spp-ui> trace filter set 98 2 0x0808
 !---dst address, 有时候设置长度2或4都有问题,只能是1
 spp-ui> trace filter set 91 1 0x06
 !---TCP flag
 spp-ui> trace filter show
 !---查看定义的过滤表项
 spp-ui> trace filter clear 94
 !---清掉之前设置的offset过滤
 spp-ui> trace ascii save
 !---自动保存到 /tmp/spp_packet_trace_ascii,用“q”退出spp_ui,用more命令查看也可以

注意:在用第二种方法时,尽量控制在100个包以内,防止影响正常业务。

2021-0611更新:eXR SPP以及Tcpdump抓包

在64bit的eXR中,SPP抓包有些变化,不再是直接run进去执行spp,而是需要ssh进去,具体方法如下:

RP/0/RSP0/CPU0:CR5#admin show vm
Thu Jun 10 04:04:22.962 UTC

Location: 0/0
Id                Status        IP Address       HB Sent/Recv
-------------------------------------------------------------
sysadmin          running       192.0.8.1        NA/NA
default-sdr       running       192.0.8.3        179906/179906

Location: 0/RSP0
Id                Status        IP Address       HB Sent/Recv
-------------------------------------------------------------
sysadmin          running       192.0.0.1        NA/NA
default-sdr       running       192.0.0.4        3599690/3599690

RP/0/RSP0/CPU0:CR5#run
Thu Jun 10 13:31:18.180 UTC
ssssh[xr-vm_node0_RSP0_CPU0:~]$ssh 192.0.8.3
Last login: Thu Jun 10 03:54:13 2021 from 192.0.0.4
[xr-vm_node0_0_CPU0:~]$
[xr-vm_node0_0_CPU0:~]$
[xr-vm_node0_0_CPU0:~]$spp_ui
spp-ui> trace start 100
spp-ui> trace stop
spp-ui> trace print

但在我的测试里,这种方法可能抓不到traceroute的报文,具体原因不是很清楚,所以换了另一种抓包方式,这也是转成eXR带来的好处,直接用tcpdump抓包,如下:

[xr-vm_node0_0_CPU0:~]$ifconfig
eth-bfd   Link encap:Ethernet  HWaddr be:17:a1:12:a1:f0  
          inet6 addr: fe80::bc17:a1ff:fe12:a1f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:428 (428.0 B)  TX bytes:1296 (1.2 KiB)

eth-bfd.1282 Link encap:Ethernet  HWaddr 56:4b:47:50:4e:42  
          inet6 addr: fe80::544b:47ff:fe50:4e42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:232 (232.0 B)  TX bytes:648 (648.0 B)

eth-punt  Link encap:Ethernet  HWaddr ae:7d:f5:1a:cb:47  
          inet6 addr: fe80::ac7d:f5ff:fe1a:cb47/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:45553297 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4179755 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8728798955 (8.1 GiB)  TX bytes:693317196 (661.1 MiB)

eth-punt.1282 Link encap:Ethernet  HWaddr 56:4b:47:50:4e:53  
          inet6 addr: fe80::544b:47ff:fe50:4e53/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:45553293 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4179747 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8091052565 (7.5 GiB)  TX bytes:693316548 (661.1 MiB)

eth-vf1   Link encap:Ethernet  HWaddr 9a:8d:94:f9:48:86  
          inet6 addr: fe80::988d:94ff:fef9:4886/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:21065229 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16242307 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:18676667249 (17.3 GiB)  TX bytes:7018115092 (6.5 GiB)

eth-vf1.3073 Link encap:Ethernet  HWaddr 56:49:4b:00:02:03  
          inet addr:192.0.8.3  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::988d:94ff:fef9:4886/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:13698951 errors:0 dropped:1466854 overruns:0 frame:0
          TX packets:11465784 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:982255165 (936.7 MiB)  TX bytes:816854781 (779.0 MiB)

eth-vf1.3074 Link encap:Ethernet  HWaddr 56:4b:47:00:02:20  
          inet addr:172.0.2.32  Bcast:172.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::988d:94ff:fef9:4886/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9700  Metric:1
          RX packets:7366278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4776373 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:17399498878 (16.2 GiB)  TX bytes:6201250291 (5.7 GiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2320665896 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2320665896 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:364309539143 (339.2 GiB)  TX bytes:364309539143 (339.2 GiB)

[xr-vm_node0_0_CPU0:~]$tcpdump -i eth-punt.1282 -w traceroute.pcap
tcpdump: listening on eth-punt.1282, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
9807 packets captured
9812 packets received by filter
0 packets dropped by kernel
[xr-vm_node0_0_CPU0:~]$scp traceroute.pcap 192.0.0.4:/harddisk:
Warning: Permanently added '192.0.0.4' (ECDSA) to the list of known hosts.
traceroute.pcap                                                                                       100% 1988KB   1.9MB/s   00:00 
[xr-vm_node0_0_CPU0:~]$
[xr-vm_node0_0_CPU0:~]$ssh 192.0.0.4
Last login: Thu Jun 10 03:58:17 2021 from 192.0.0.4
[xr-vm_node0_RSP0_CPU0:~]$cd /harddisk:
[xr-vm_node0_RSP0_CPU0:/harddisk:]$ls
traceroute.pcap                                                  

存下来的pcap文件没法直接看,因为这里仍然带着Internal Header,其实思科应该出一个小工具,直接去掉Internal header,这样可以让Wireshark直接读取;虽然没有,我们也可以通过其他方式查看,用Wireshark打开后,另存为“K12 text file”,用编辑器打开,然后直接搜索DST/SRC 的Mac地址,然后删掉Internal Header:

+---------+---------------+----------+
03:14:50,816,351   ETHER
|0   |56|4b|47|50|4e|53|00|65|7a|00|00|01|01|d8|00|00|05|00|0f|00|00|00|00|00|00|00|00|00|56|4b|20|02|10|0b|00|00|04|00|04|40|09|01|05|56|00|03|0e|03|00|00|00|00|46|00|00|00|00|01|e8|00|00|00|00|00|00|00|cc|03|dc|77|4c|f1|ef|68|00|2e|c7|ba|55|1b|88|47|03|e8|71|01|45|b8|00|34|b2|eb|00|00|01|11|f2|0c|0a|00|00|03|0a|00|00|07|b2|e6|82|9e|00|20|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
+---------+---------------+----------+

用这个网址直接decode即可:https://hpd.gasmi.net

2023-0321更新:XRv9k Capture packets

之前在老东家时就像把xrv9k如何抓包的方法整理出来,但一直没做。。。最近用xrv9k做实验时,发现丢包,所以想看看进来的报文,因此顺便简单整理下,方便以后翻阅;

方法很简单,主要就是下面几个命令,具体可以参考这篇文章:https://community.cisco.com/t5/service-providers-knowledge-base/xrv9000-packet-path-troubleshooting-note/ta-p/3163691

debug controllers dpa packet-trace [line|inject] <# to trace>
show controllers dpa packet-trace 
clear controllers dpa packet-trace
本文出自 Frank's Blog

版权声明:


本文链接:How to capture packets that dropped by NP/CPU?
版权声明:本文为原创文章,仅代表个人观点,版权归 Frank Zhao 所有,转载时请注明本文出处及文章链接
你可以留言,或者trackback 从你的网站

留言哦

blonde teen swallows load.xxx videos