Netflow V9
31 7 月, 2013,由 frank撰写
最近研究了一下Netflow，发现这东西还是很不错的。建议如果流量异常有问题或负载分担问题，直接让客户配上monitor，不用配置exporter，直接看cache里的内容就可以了。Netflow里面有些迷惑的地方，这回整个测试了下，发现这东西还是比较有意思的。
2013-10-14: ASR9K NetFlow White Paper
https://supportforums.cisco.com/docs/DOC-36434
1. Template:
在v9的版本中，引进了template这个概念，这个东西感觉像个索引，告诉网管netflow的架构，默认export时间为1800s，也就是30mi，在抓netflow报文时，如果没有抓下template，那么你会发现通过wireshark无法解开cflow的报文，你也就没发分析了。另外template分两种，一种是正常的数据flow，另一种是option的flow，这就带来了强大的扩展。在下面的信息里，我把template的timeout改成了10s，cache timeout没改，所以你发现短时间内没有抓到任何flow流量：
RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 08:02:27.314 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (29117) Flows exported: 0 (0 bytes) >>> 刚开始为空 Flows dropped: 0 (0 bytes) Templates exported: 7 (644 bytes) >>> 数据template已经exported 7个，对应Flows exported Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) >>> 刚开始为空 Option data dropped: 0 (0 bytes) Option templates exported: 14 (392 bytes) >>> Option template已经exported 14个，对应Option data exported Option templates dropped: 0 (0 bytes) Packets exported: 21 (1036 bytes) Packets dropped: 0 (0 bytes) Total export over last interval of: 1 hour: 18 pkts 888 bytes 0 flows 1 minute: 18 pkts 888 bytes 0 flows 1 second: 0 pkts 0 bytes 0 flows RP/0/RSP0/CPU0:ios#sh flow exporter-map test Wed Jul 24 11:28:59.354 UTC Flow Exporter Map : test ------------------------------------------------- Id : 1 DestinationIpAddr : 12.1.1.1 SourceIfName : Loopback0 SourceIpAddr : 123.123.123.123 DSCP : 0 TransportProtocol : UDP TransportDestPort : 9995 Export Version: 9 Common Template Timeout : 10 seconds Options Template Timeout : 10 seconds Data Template Timeout : 10 seconds Interface-Table Export Timeout : 0 seconds Sampler-Table Export Timeout : 0 seconds RP/0/RSP0/CPU0:ios#show flow monitor-map test-monitor Wed Jul 24 11:31:12.103 UTC Flow Monitor Map : test-monitor ------------------------------------------------- Id: 1 RecordMapName: ipv4-raw ExportMapName: test CacheAgingMode: Normal CacheMaxEntries: 1000000 CacheActiveTout: 1800 seconds CacheInactiveTout: 15 seconds CacheUpdateTout: N/A
详细请看附件 “flow-test-1.pcap”
2. Export timeout：
在Netflow中，有多个export的时间点。
当流是active的时候，到达ActiveTimeout，就会把现有cache中这些active的数据组成UDP并发送出去，默认30mi；
当流是inactive的时候（就是在设备上没有这个流的任何数据包了），到达InactiveTimeout，就会把这些inactive的数据包组成UDP发送出去，默认15s，这个很快，所以如果遇到间接性的攻击流时，很难去cache中发现问题flow。
>>>>>>>>> 测试Inactive Timeout <<<<<<<<<
RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0 Wed Jul 24 11:36:53.723 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 0 High Watermark: 950000 Flows added: 8 Flows not added: 0 Ager Polls: 12919 - Active timeout 7 - Inactive timeout 1 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 8 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 8 Matching entries: 0 >>>现在没有流量 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:37:08.309 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (29117) Flows exported: 8 (456 bytes) >>>做测试时没有清掉，所以是8 Flows dropped: 0 (0 bytes) Templates exported: 1295 (119140 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 2588 (72464 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 3891 (192572 bytes) Packets dropped: 0 (0 bytes)
**************************开始打流量*****************************
RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:37:11.493 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (29117) Flows exported: 8 (456 bytes) >>> 没有变化 Flows dropped: 0 (0 bytes) Templates exported: 1296 (119232 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 2588 (72464 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 3891 (192572 bytes) Packets dropped: 0 (0 bytes) RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0 Wed Jul 24 11:37:15.053 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 1 High Watermark: 950000 Flows added: 9 Flows not added: 0 Ager Polls: 12940 - Active timeout 7 - Inactive timeout 1 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 8 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 8 IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort IPV4Prot IPV4TOS InputInterface ForwardStatus ByteCount PacketCount Dir 8.8.8.8 10.144.254.1 63 63 udp 0 Gi0/2/0/10 Fwd 21870 135 Ing Matching entries: 1 <<< 在cache里已经能看到了
**************************停止打流量*****************************
RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0 Wed Jul 24 11:37:22.749 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 1 High Watermark: 950000 Flows added: 9 Flows not added: 0 Ager Polls: 12948 - Active timeout 7 - Inactive timeout 1 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 8 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 8 IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort IPV4Prot IPV4TOS InputInterface ForwardStatus ByteCount PacketCount Dir 8.8.8.8 10.144.254.1 63 63 udp 0 Gi0/2/0/10 Fwd 31914 197 Ing Matching entries: 1 <<< 还存在 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:37:26.684 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (29117) Flows exported: 8 (456 bytes) >>>仍然没变 Flows dropped: 0 (0 bytes) Templates exported: 1297 (119324 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 2592 (72576 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 3897 (192868 bytes) Packets dropped: 0 (0 bytes) RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0 Wed Jul 24 11:37:32.666 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 1 High Watermark: 950000 Flows added: 9 Flows not added: 0 Ager Polls: 12957 - Active timeout 7 - Inactive timeout 1 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 8 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 8 IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort IPV4Prot IPV4TOS InputInterface ForwardStatus ByteCount PacketCount Dir 8.8.8.8 10.144.254.1 63 63 udp 0 Gi0/2/0/10 Fwd 34020 210 Ing Matching entries: 1 RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0 Wed Jul 24 11:37:35.756 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 0 High Watermark: 950000 Flows added: 9 Flows not added: 0 Ager Polls: 12961 - Active timeout 7 - Inactive timeout 2 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 9 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 9 Matching entries: 0 >>> 被clear了，去看下export信息 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:37:38.797 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (29117) Flows exported: 9 (513 bytes) >>> 增加到9了 Flows dropped: 0 (0 bytes) Templates exported: 1299 (119508 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 2594 (72632 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 3902 (193229 bytes) Packets dropped: 0 (0 bytes)
>>>>>>>>> 测试Active Timeout <<<<<<<<<
RP/0/RSP0/CPU0:ios#config ter Wed Jul 24 11:45:34.523 UTC RP/0/RSP0/CPU0:ios(config)#flow monitor-map test-monitor RP/0/RSP0/CPU0:ios(config-fmm)#cache ? entries Specify the number of entries in the flow cache permanent Disable removal of entries from flow cache timeout Specify the flow cache timeouts RP/0/RSP0/CPU0:ios(config-fmm)#cache timeout ? active Specify the active flow timeout inactive Specify the inactive flow timeout update Specify the update timeout RP/0/RSP0/CPU0:ios(config-fmm)#cache timeout active 30 RP/0/RSP0/CPU0:ios(config-fmm)#commit Wed Jul 24 11:47:20.443 UTC RP/0/RSP0/CPU0:ios(config-fmm)#end RP/0/RSP0/CPU0:ios# RP/0/RSP0/CPU0:ios#sh run int g0/2/0/10 Wed Jul 24 11:47:35.958 UTC interface GigabitEthernet0/2/0/10 ipv4 address 33.33.33.1 255.255.255.0 negotiation auto load-interval 30 flow ipv4 monitor test-monitor sampler 1-1000 ingress ! RP/0/RSP0/CPU0:ios#config ter Wed Jul 24 11:47:37.483 UTC RP/0/RSP0/CPU0:ios(config)#int g0/2/0/10 RP/0/RSP0/CPU0:ios(config-if)#no flow ipv4 monitor test-monitor sampler 1-1000 ingress RP/0/RSP0/CPU0:ios(config-if)#commit Wed Jul 24 11:47:50.972 UTC RP/0/RSP0/CPU0:ios(config-if)#do sh flow monitor-map test-monitor Wed Jul 24 11:48:29.900 UTC Flow Monitor Map : test-monitor ------------------------------------------------- Id: 1 RecordMapName: ipv4-raw ExportMapName: test CacheAgingMode: Normal CacheMaxEntries: 1000000 CacheActiveTout: 30 seconds >>>已经改了 CacheInactiveTout: 15 seconds CacheUpdateTout: N/A RP/0/RSP0/CPU0:ios(config-if)#flow ipv4 monitor test-monitor sampler 1-1000 ingress RP/0/RSP0/CPU0:ios(config-if)#commit Wed Jul 24 11:48:47.966 UTC >>>开始计时 RP/0/RSP0/CPU0:ios(config-if)#end RP/0/RSP0/CPU0:ios# RP/0/RSP0/CPU0:ios# RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:49:22.123 UTC >>> 大概过了35s Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (30838) Flows exported: 1 (57 bytes) >>>已经被export出去了 Flows dropped: 0 (0 bytes) Templates exported: 5 (460 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 8 (224 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 14 (805 bytes) Packets dropped: 0 (0 bytes) Total export over last interval of: 1 hour: 0 pkts 0 bytes 0 flows 1 minute: 14 pkts 748 bytes 1 flows 1 second: 2 pkts 156 bytes 1 flows RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 11:49:53.291 UTC >>> 大概又过了30s Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (30838) Flows exported: 2 (114 bytes) >>>已经被export出去了 Flows dropped: 0 (0 bytes) Templates exported: 8 (736 bytes) Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) Option data dropped: 0 (0 bytes) Option templates exported: 14 (392 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 24 (1370 bytes) Packets dropped: 0 (0 bytes)
>> 再次抓包，并在端口上初始化flow配置，确认template和正常的cflow信息 <<
抓包信息请看: “flow-test-2.pcap”
RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0 Wed Jul 24 12:10:10.049 UTC Flow Exporter: test Flow Exporter memory usage: 3280868 Used by flow monitors: test-monitor Status: Normal Transport UDP Destination 12.1.1.1 (9995) VRF default Source 123.123.123.123 (381) Flows exported: 3 (171 bytes) >>> 跟抓包信息对应 Flows dropped: 0 (0 bytes) Templates exported: 15 (1380 bytes) >>> 跟抓包信息对应 Templates dropped: 0 (0 bytes) Option data exported: 0 (0 bytes) >>> 可以看到只有option的templates，没有option的data Option data dropped: 0 (0 bytes) Option templates exported: 26 (728 bytes) Option templates dropped: 0 (0 bytes) Packets exported: 43 (2379 bytes) Packets dropped: 0 (0 bytes) Total export over last interval of: 1 hour: 41 pkts 2152 bytes 3 flows 1 minute: 20 pkts 1044 bytes 1 flows 1 second: 0 pkts 0 bytes 0 flows P/0/RSP0/CPU0:ios#show flow monitor-map test-monitor Wed Jul 24 12:14:55.437 UTC Flow Monitor Map : test-monitor ------------------------------------------------- Id: 1 RecordMapName: ipv4-raw ExportMapName: test CacheAgingMode: Normal CacheMaxEntries: 1000000 CacheActiveTout: 30 seconds CacheInactiveTout: 15 seconds CacheUpdateTout: N/A RP/0/RSP0/CPU0:ios#show flow exporter-map test Wed Jul 24 12:15:07.023 UTC Flow Exporter Map : test ------------------------------------------------- Id : 1 DestinationIpAddr : 12.1.1.1 SourceIfName : Loopback0 SourceIpAddr : 123.123.123.123 DSCP : 0 TransportProtocol : UDP TransportDestPort : 9995 Export Version: 9 Common Template Timeout : 10 seconds Options Template Timeout : 10 seconds Data Template Timeout : 10 seconds Interface-Table Export Timeout : 0 seconds Sampler-Table Export Timeout : 0 seconds
3. Cache中的flow
当在端口配上“flow ipv4 monitor xxx sampler xxx ingress”时，就会激活netflow，进来的数据包会首先进到flow cache中，这个cache可以更改，最大1000000个条目。这里面的每个条目在被export出去之前都是累加的，例如在下面的信息中，我用了sample 1/1000。
RP/0/RSP0/CPU0:ios#sh flow monitor test-monitor cache location 0/2/cpu0 Thu Jul 25 16:43:29.250 UTC Cache summary for Flow Monitor test-monitor: Cache size: 1000000 Current entries: 1 High Watermark: 950000 Flows added: 7 Flows not added: 0 Ager Polls: 1776 - Active timeout 5 - Inactive timeout 1 - TCP FIN flag 0 - Watermark aged 0 - Emergency aged 0 - Counter wrap aged 0 - Total 6 Periodic export: - Counter wrap 0 - TCP FIN flag 0 Flows exported 6 IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort BGPDstOrigAS BGPSrcOrigAS BGPNextHopV4 IPV4DstPrfxLen IPV4SrcPrfxLen IPV4Prot IPV4TOS InputInterface OutputInterface L4TCPFlags ForwardStatus FirstSwitched LastSwitched ByteCount PacketCount Dir SamplerID 8.8.8.8 10.144.254.1 63 63 0 0 0.0.0.0 24 0 udp 0 Gi0/2/0/10 Gi0/2/0/0 0 Fwd 00 00:27:24:562 00 00:31:09:544 1034954 22499 Ing 1 Matching entries: 1
在这里可以看到，22499是累计值，就像我上面说的，他会一直累积到Timeout并Export，另外这也是sample后的数据包个数，所以此时真实的数据包大概应该有22499000，但你可能会发现这个数值跟端口的 packets input计数不一样，即使同时clear端口和flow，也不一样，而且差很多。端口数据和netflow的数据应该是通过不同方式获得的，所以可能会有些延时差异。
2018-1-12: Netflow USe CASE in TS
1. 如果点到点ping不同，完全丢包，中间又是MPLS环境，我们如何确认报文到了哪里了？这时可以通过netflow来检测MPLS报文中的内容，如TOS，SRC/DST Address等
2. 如果有攻击流量，可以也可以通过netflow找出来，可以看我最新更新的博文：asr9k netflow and qos order in the inbound direction
4. 如何通过netflow信息绘图
例如下面的，ActiveTimout是300s，实际是317s，所以此流的平均数据包为33358000/317s = 105230 PPS，这个数也大致等于端口的100k PPS。
对于字节数，可以通过 (1534468 * 8)/300s = 40919147bits/sec，大致等于端口统计的48Mbits/sec。
对于这个平均值要打上什么时间，可以是flow的start时间，也可以是flow的end时间，只要统一就ok。
这样一个点就绘出来了，在lab我只打了1条流，如果多个流存在，要把所有流加起来求平均值，这样就可以计算出端口总共的带宽。
另外默认netflow不计算2层包头，所以如果打64byte的小包，而且流量很大，那么统计出来的结果会差很多，因为每个包用于计算的大小只有64-14=50byte。
RP/0/RSP0/CPU0:ios#sh int g0/2/0/10 | i input Thu Jul 25 16:31:20.059 UTC output flow control is off, input flow control is off Last input 00:00:00, output 00:17:08 30 second input rate 48000000 bits/sec, 100000 packets/sec 105600753 packets input, 6336051711 bytes, 0 total input drops 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
flow monitor-map test record ipv4 cache entries 1000000 ! sampler-map 1-1000 random 1 out-of 1000 Int x/x/x/x flow ipv4 monitor test sampler 1-1000 ingressshow flow monitor test cache format record match ipv4 protocol eq tcp location x/x/cpu0
抓包附件：
flow-test-1/2-check-blog
你好 我也正在研究netflow 但是手边没有支持的路由器 你是用的虚拟机吗？我该怎么办？您的那个数据包的附件在哪啊？没找到下载的地方 我想看看 能否发到我的邮箱[email protected] 谢谢
最近也在研究netflow 但是支持v9的模拟器好像没有呢 不知到博主怎么弄的。您的附件怎么下载啊！能否发到邮箱[email protected] 谢谢
请问能把那两个cap包发到我的邮箱吗？[email protected]
已经给你发了，我用的是真实设备做的。
请问能把那两个cap包发到我的邮箱吗？我最近也在看v9这个,但是没有实际的包,[email protected]
Frank,您好。你能把附件cap包给我发一下吗？邮箱地址：[email protected].
谢谢！
Hi, All
新的Wordpress支持上传附件，我已经更新了此文章，你们可以直接下载文章后附上的zip包，里面包含两个抓包文件
Dear Frank,
非常谢谢！
你好 请问为什么那个zip包里的东西我用wireshark打开是UDP报文呢
wireshark默认是UDP，你需要自己手动把他decode下，方式是选择”分析” -> “decode as” -> 选择UDP的端口号 -> current选择“CFLOW”就可以了
楼主你好：
我最近在研究netflow，但是网上没有资料比较少，不知道楼主能不能私底下叫流一下。
谢谢
我netflow研究的也不深，皮毛而已，如果有问题，可以发到这里一起讨论下
[…] 然后又发现抓下来的包，no template found，又根据这两篇解决：Wireshark needs templates to decipher Cisco NetFlow v9 和 Netflow V9 […]