Telemetry Receiver by UDP+KV-GPB

9 8 月, 2020,由 frank撰写

In the article, only show as follow (because easy to show how to work):

Protocol: UDP; Port: 5234

Encoding: KV-GPB

For more detailed information, you can reference my colleague’s GitHub, which includes more examples, e.g GRPC.

GRPC Update Info:

If use GRPC not TCP/UDP, that will have standard function, you will not need to write “DECODE_FN_MAP…”, function will auto generate, direct to use. Simple list steps:

Get GRPC protobuf (e.g “mdt_grpc_dialout”) from cisco / bigmuddy-network-telemetry-proto
Install/use compile tools, suggest install python virtual env, you can reference 在MacOS中部署Python虚拟开发环境

(telemetry-protocol) [root@telemetry telemetry-protocol]# python -V
Python 3.6.8
(telemetry-protocol) [root@telemetry telemetry-protocol]# easy_install pip
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install protobuf
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio_tools
(telemetry-protocol) [root@telemetry telemetry-protocol]# python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. cisco_grpc_dialout.proto
(telemetry-protocol) [root@telemetry telemetry-protocol]# ls -l|grep cisco
-rw-r--r--.  1 root    root       2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py   #generated
-rw-r--r--.  1 root    root       3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py   #generated
-rw-r--r--.  1 root    root        485 Aug 10 16:15 cisco_grpc_dialout.proto   #proto file

Follow all file if you need grpc protocol:

[root@telemetry grpc-dailout]# ls -l
total 36
-rw-r--r--. 1 root root  2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #Contains the server Stub class and the client Stub class, as well as the service RPC interface to be implemented.
-rw-r--r--. 1 root root  3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #message serialization classes
-rw-r--r--. 1 root root   484 Aug 10 15:36 mdt_grpc_dialout.proto 
-rw-r--r--. 1 root root  3722 Aug 10 16:11 service_grpc_dialout_no_tls.py #service python
-rw-r--r--. 1 root root 19220 Aug 10 15:58 telemetry_pb2.py  #decode gpb-kv messages, generate as below

grpc-dailout.zip

Install Protobuf of python version

For detailed info, can reference google official doc: Protocol Buffer Basics: Python and Download Protocol Buffers.

[root@telemetry telemetry-protocol]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.12.4/protobuf-python-3.12.4.zip
[root@telemetry telemetry-protocol]# unzip protobuf-python-3.12.4.zip 
[root@telemetry telemetry-protocol]# cd protobuf-3.12.4/
[root@telemetry protobuf-3.12.4]# ./configure 
[root@telemetry protobuf-3.12.4]# make
[root@telemetry protobuf-3.12.4]# make check
......
PASS: protobuf-test
PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
============================================================================
Testsuite summary for Protocol Buffers 3.12.4
============================================================================
# TOTAL: 6
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
[root@telemetry protobuf-3.12.4]# make install
[root@telemetry protobuf-3.12.4]# cd ./python/
[root@telemetry python]# python3 setup.py build
[root@telemetry python]# python3 setup.py test
[root@telemetry python]# python3 setup.py install
[root@telemetry python]# protoc --version
libprotoc 3.12.4

Compile Proto File

In the example, only show KV-GPB, so only require 1 proto file (telemetry.proto) for all sensor-path, if you use GPB for encoding, each sensor-path requires 1 proto file. The XR Proto file download from the official link: cisco / bigmuddy-network-telemetry-proto

[root@telemetry telemetry-protocol]# protoc -I=./ --python_out=./ ./telemetry.proto
[root@telemetry telemetry-protocol]# ls -l
total 6340
drwxr-xr-x. 13 root    root       4096 Aug  9 15:58 protobuf-3.12.4
-rw-r--r--.  1 root    root    6100223 Jul 29 07:58 protobuf-python-3.12.4.zip
-rw-r--r--.  1 root    root       7493 Aug  9 15:34 telemetry_server_udp_gpb-kv.py
-rw-r--r--.  1 root    root      19220 Aug  9 16:28 telemetry_pb2.py   <<<
-rw-r--r--.  1 root    root       8187 Aug  9 16:27 telemetry.proto
-rw-r--r--.  1 tcpdump tcpdump  344548 Aug  9 15:15 telemetry-udp.pcap

Telemetry Config On XR

How to get sensor path, please reference my last blog: How To Get Telemetry sensor path for show cmd on IOS XR?

telemetry model-driven
 destination-group nms
  address-family ipv4 10.75.37.91 port 5432
   encoding self-describing-gpb
   protocol udp
  !
 !
 sensor-group interface
  sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface[interface-name='GigabitEthernet0/0/0/1']
 !
 subscription nms
  sensor-group-id interface strict-timer
  sensor-group-id interface sample-interval 20000
  destination-id nms
  source-interface GigabitEthernet0/0/0/2
 !
!

Telemetry UDP Server by Python

Come From: AlexFengCisco / Telemetry_Receiver

[root@telemetry telemetry-protocol]# more telemetry_server_udp_gpb-kv.py
'''
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |          MSG TYPE             |           ENCODING_TYPE       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         MSG_VERSION           |           FLAGS               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                           MSG_LENGTH                          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     ~                                                               ~
     ~                      PAYLOAD (MSG_LENGTH bytes)               ~
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     MSG TYPE (2 bytes)  = 1 (for MDT)
     ENCODING_TYPE (2 bytes) = 1 (GPB), 2 (JSON)
     MSG_VERSION (2 bytes) = 1
     FLAGS (2 bytes) = 0
     MSG_LENGTH (4 bytes)
'''


import socket, struct
import json
from google.protobuf.descriptor import FieldDescriptor
import time
import pprint
import telemetry_pb2
from google.protobuf.json_format import MessageToJson

DECODE_FN_MAP = {
    FieldDescriptor.TYPE_DOUBLE: float,
    FieldDescriptor.TYPE_FLOAT: float,
    FieldDescriptor.TYPE_INT32: int,
    FieldDescriptor.TYPE_INT64: int, #long
    FieldDescriptor.TYPE_UINT32: int,
    FieldDescriptor.TYPE_UINT64: int,#long
    FieldDescriptor.TYPE_SINT32: int,
    FieldDescriptor.TYPE_SINT64: int,#long
    FieldDescriptor.TYPE_FIXED32: int,
    FieldDescriptor.TYPE_FIXED64: int,#long
    FieldDescriptor.TYPE_SFIXED32: int,
    FieldDescriptor.TYPE_SFIXED64: int,#long
    FieldDescriptor.TYPE_BOOL: bool,
    FieldDescriptor.TYPE_STRING: str,
    FieldDescriptor.TYPE_BYTES: bytes,#lambda b: bytes_to_string(b),
    FieldDescriptor.TYPE_ENUM: int,
}

def field_type_to_fn(msg, field):
    if field.type == FieldDescriptor.TYPE_MESSAGE:
        # For embedded messages recursively call this function. If it is
        # a repeated field return a list
        result = lambda msg: proto_to_dict(msg)
    elif field.type in DECODE_FN_MAP:
        result = DECODE_FN_MAP[field.type]
    else:
        raise TypeError("Field %s.%s has unrecognised type id %d" % (
                         msg.__class__.__name__, field.name, field.type))
    return result


def proto_to_dict(msg):
    result_dict = {}
    extensions = {}
    for field, value in msg.ListFields():
        conversion_fn = field_type_to_fn(msg, field)

        # Skip extensions
        if not field.is_extension:
            # Repeated fields result in an array, otherwise just call the
            # conversion function to store the value
            if field.label == FieldDescriptor.LABEL_REPEATED:
                result_dict[field.name] = [conversion_fn(v) for v in value]
            else:
                result_dict[field.name] = conversion_fn(value)
    return result_dict


# Bind Socket UDP port 5432 as Telemetry recevice server
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 5432))

count = 0

start_time = time.time()


while True:
    count += 1
    buf, addr = sock.recvfrom(65535)
    Telemetry_content = telemetry_pb2.Telemetry()

    print(buf.hex())
    print(buf)
    print("Message Length {}".format(len(buf)))
    #print(len(str(buf)))
    #Handle Telemetry UDP GPB and GPB-kv  from IOX
    if buf[0:1] == b'\x00': ##check the binary daa , no official document
        print("Telemetry GPB  message from IOX")
        Telemetry_content.ParseFromString(buf[12:])
        print('Node :'+Telemetry_content.node_id_str)
        print('IP Address (source port) :' + str(addr))
        print('Encodig Path :' + Telemetry_content.encoding_path)
        content_json_dict = proto_to_dict(Telemetry_content.data_gpb)  # Old proto_to_dict from google public code
        print(MessageToJson(Telemetry_content)) # new MessageToJson from google public code , will replace proto_to_dict
        print("*"*20)
        print(content_json_dict)


        if len(str(Telemetry_content.data_gpbkv)) > 2: # Handle GPB kv , in case of unstable A9KV , sometimes sent empty content message
            print('GPB kv format')
            Fields_list = Telemetry_content.data_gpbkv[0].fields[1].fields
            #print(Fields_list)

            json_dict = proto_to_dict(Telemetry_content.data_gpbkv[0])
            #pprint.pprint(json_dict)
            print(json_dict)

            for field in Fields_list:

                #print(field.fields)
                if field.string_value:
                    print(field.name + ':' + field.string_value)
                if field.uint32_value:
                    print(field.name + ':' + str(field.uint32_value))


    print("="*200)

Run the script:

[root@telemetry telemetry-protocol]# python3 telemetry_server_udp_gpb-kv.py
0001000100010000000000700a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b508d8e
b'\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00p\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bP\x8d\x84\xf0\x98\xbd.h\x8d\x84\xf0\x98\xbd.'
Message Length 124 <<<
Telemetry GPB  message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
  "nodeIdStr": "iosxrv9000-1",
  "subscriptionIdStr": "nms",
  "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
  "collectionId": "3466",
  "msgTimestamp": "1596974694925",
  "collectionEndTime": "1596974694925"
}
********************
{}
========================================================================================================================================================================================================
000100010001000000000a300a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b48fe8d
b"\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\n0\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bH\xfe\x83\xf0\x98\xbd.P\xfe\x83\xf0\x98\xbd.Z\xbd\x13\x08\x8c\x84"
Message Length 2620 <<<
Telemetry GPB  message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
  "nodeIdStr": "iosxrv9000-1",
  "subscriptionIdStr": "nms",
  "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
  "collectionId": "3466",
  "collectionStartTime": "1596974694910",
  "msgTimestamp": "1596974694910",
  "dataGpbkv": [
    {
      "timestamp": "1596974694924",
      "fields": [
        {
          "name": "keys",
          "fields": [
            {
              "name": "interface-name",
              "stringValue": "GigabitEthernet0/0/0/1"
            }
          ]
        },
        {
          "name": "content",
          "fields": [
            {
              "name": "interface-handle",
              "stringValue": "GigabitEthernet0/0/0/1"
            },
......
********************
{}
GPB kv format
{'timestamp': 1596974694924, 'fields': [{'name': 'keys', 'fields': [{'name': 'interface-name', 'string_value': 'GigabitEthernet0/0/0/1'}]}, {'name': 'content', 'fields': [{'name': 'interface-handle', 'string_value': 'GigabitE}
interface-handle:GigabitEthernet0/0/0/1
interface-type:IFT_GETHERNET
hardware-type-string:GigabitEthernet
state:im-state-up
line-state:im-state-up
encapsulation:ether
encapsulation-type-string:ARPA
mtu:1514
is-l2-transport-enabled:false
state-transition-count:1
is-dampening-enabled:false
speed:1000000
duplexity:im-attr-duplex-unknown
media-type:im-attr-media-unknown
link-type:im-attr-link-type-force
in-flow-control:im-attr-flow-control-off
out-flow-control:im-attr-flow-control-off
bandwidth:1000000
max-bandwidth:1000000
is-l2-looped:false
loopback-configuration:no-loopback
description:to server-1
fast-shutdown:false
if-index:7
is-intf-logical:false
========================================================================================================================================================================================================
^CTraceback (most recent call last):
  File "telemetry_server_udp_gpb-kv.py", line 90, in 
    buf, addr = sock.recvfrom(65535)
KeyboardInterrupt

Capture packets and checking

[root@telemetry telemetry-protocol]# tcpdump -i ens192 -w telemetry-udp-new.pcap
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
^C380 packets captured
381 packets received by filter
0 packets dropped by kernel

telemetry-udp-new1.pcap

本文出自 Frank's Blog

本文链接：Telemetry Receiver by UDP+KV-GPB
版权声明：本文为原创文章，仅代表个人观点，版权归 Frank Zhao 所有，转载时请注明本文出处及文章链接

Linux, Programmer, 学习笔记, 网络技术 Linux,Python,Telemetry

你可以留言,或者trackback 从你的网站

DX6 on Linux by uSID«

»EVPN Summary

留言哦

返回顶端

『无题』

I persist, therefore I am!!!

Memo

Perseverance sometimes equals genius!
There are only two creatures in the world who can surmount the pyramid:

—–the eagle and the snail.
Useful Link

1. IANA - AFI
2. IANA - SAFI
3. IANA - BGP Ex Comm
4. RPM pbone.net
5. Linux Packages Search
Useful Toots

Linux/http server:

python -m SimpleHTTPServer <port>

python -m http.server <port>

PS: only one thread, if require more threads, can replace by “caddy”.

Win10/http server:

http://www.rejetto.com/hfs/

Win10/md5:

certutil -hashfile .\xxxx MD5
分类
- CCDE (16)
- CCIE SEC (4)
- CCIE SP (55)
- CCIE VOICE (3)
- Classical CASE (26)
- Linux (76)
- Mobile Phone (4)
- Programmer (44)
- Solution (3)
- Uncategorized (6)
- Windows常用技术 (10)
- 似水流年 (17)
- 学习笔记 (191)
- 常用资料 (45)
- 网络技术 (39)
近期文章
其他操作

Telemetry Receiver by UDP+KV-GPB

Install Protobuf of python version

Compile Proto File

Telemetry Config On XR

Telemetry UDP Server by Python

Capture packets and checking

留言哦

『无题』

Memo

Useful Link

Useful Toots

分类

近期文章

其他操作