Telemetry Receiver by UDP+KV-GPB

In the article, only show as follow (because easy to show how to work):

Protocol: UDP; Port: 5234

Encoding: KV-GPB

For more detailed information, you can reference my colleague’s GitHub, which includes more examples, e.g GRPC.

AlexFengCisco / Telemetry_Receiver

GRPC Update Info:

If use GRPC not TCP/UDP, that will have standard function, you will not need to write “DECODE_FN_MAP…”, function will auto generate, direct to use. Simple list steps:

  1. Get GRPC protobuf (e.g “mdt_grpc_dialout”) from cisco / bigmuddy-network-telemetry-proto
  2. Install/use compile tools, suggest install python virtual env, you can reference 在MacOS中部署Python虚拟开发环境
(telemetry-protocol) [root@telemetry telemetry-protocol]# python -V
Python 3.6.8
(telemetry-protocol) [root@telemetry telemetry-protocol]# easy_install pip
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install protobuf
(telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio_tools
(telemetry-protocol) [root@telemetry telemetry-protocol]# python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. cisco_grpc_dialout.proto
(telemetry-protocol) [root@telemetry telemetry-protocol]# ls -l|grep cisco
-rw-r--r--.  1 root    root       2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py   #generated
-rw-r--r--.  1 root    root       3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py   #generated
-rw-r--r--.  1 root    root        485 Aug 10 16:15 cisco_grpc_dialout.proto   #proto file  
Follow all file if you need grpc protocol:
[root@telemetry grpc-dailout]# ls -l
total 36
-rw-r--r--. 1 root root  2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #Contains the server Stub class and the client Stub class, as well as the service RPC interface to be implemented.
-rw-r--r--. 1 root root  3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #message serialization classes
-rw-r--r--. 1 root root   484 Aug 10 15:36 mdt_grpc_dialout.proto 
-rw-r--r--. 1 root root  3722 Aug 10 16:11 service_grpc_dialout_no_tls.py #service python
-rw-r--r--. 1 root root 19220 Aug 10 15:58 telemetry_pb2.py  #decode gpb-kv messages, generate as below
grpc-dailout.zip

Install Protobuf of python version

For detailed info, can reference google official doc: Protocol Buffer Basics: Python and Download Protocol Buffers.

[root@telemetry telemetry-protocol]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.12.4/protobuf-python-3.12.4.zip
[root@telemetry telemetry-protocol]# unzip protobuf-python-3.12.4.zip 
[root@telemetry telemetry-protocol]# cd protobuf-3.12.4/
[root@telemetry protobuf-3.12.4]# ./configure 
[root@telemetry protobuf-3.12.4]# make
[root@telemetry protobuf-3.12.4]# make check
......
PASS: protobuf-test
PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
============================================================================
Testsuite summary for Protocol Buffers 3.12.4
============================================================================
# TOTAL: 6
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
[root@telemetry protobuf-3.12.4]# make install
[root@telemetry protobuf-3.12.4]# cd ./python/
[root@telemetry python]# python3 setup.py build
[root@telemetry python]# python3 setup.py test
[root@telemetry python]# python3 setup.py install
[root@telemetry python]# protoc --version
libprotoc 3.12.4

Compile Proto File

In the example, only show KV-GPB, so only require 1 proto file (telemetry.proto) for all sensor-path, if you use GPB for encoding, each sensor-path requires 1 proto file. The XR Proto file download from the official link: cisco / bigmuddy-network-telemetry-proto

[root@telemetry telemetry-protocol]# protoc -I=./ --python_out=./ ./telemetry.proto
[root@telemetry telemetry-protocol]# ls -l
total 6340
drwxr-xr-x. 13 root    root       4096 Aug  9 15:58 protobuf-3.12.4
-rw-r--r--.  1 root    root    6100223 Jul 29 07:58 protobuf-python-3.12.4.zip
-rw-r--r--.  1 root    root       7493 Aug  9 15:34 telemetry_server_udp_gpb-kv.py
-rw-r--r--.  1 root    root      19220 Aug  9 16:28 telemetry_pb2.py   <<<
-rw-r--r--.  1 root    root       8187 Aug  9 16:27 telemetry.proto
-rw-r--r--.  1 tcpdump tcpdump  344548 Aug  9 15:15 telemetry-udp.pcap

Telemetry Config On XR

How to get sensor path, please reference my last blog: How To Get Telemetry sensor path for show cmd on IOS XR?

telemetry model-driven
 destination-group nms
  address-family ipv4 10.75.37.91 port 5432
   encoding self-describing-gpb
   protocol udp
  !
 !
 sensor-group interface
  sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface[interface-name='GigabitEthernet0/0/0/1']
 !
 subscription nms
  sensor-group-id interface strict-timer
  sensor-group-id interface sample-interval 20000
  destination-id nms
  source-interface GigabitEthernet0/0/0/2
 !
!

Telemetry UDP Server by Python

Come From: AlexFengCisco / Telemetry_Receiver

[root@telemetry telemetry-protocol]# more telemetry_server_udp_gpb-kv.py
'''
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |          MSG TYPE             |           ENCODING_TYPE       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         MSG_VERSION           |           FLAGS               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                           MSG_LENGTH                          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     ~                                                               ~
     ~                      PAYLOAD (MSG_LENGTH bytes)               ~
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     MSG TYPE (2 bytes)  = 1 (for MDT)
     ENCODING_TYPE (2 bytes) = 1 (GPB), 2 (JSON)
     MSG_VERSION (2 bytes) = 1
     FLAGS (2 bytes) = 0
     MSG_LENGTH (4 bytes)
'''


import socket, struct
import json
from google.protobuf.descriptor import FieldDescriptor
import time
import pprint
import telemetry_pb2
from google.protobuf.json_format import MessageToJson

DECODE_FN_MAP = {
    FieldDescriptor.TYPE_DOUBLE: float,
    FieldDescriptor.TYPE_FLOAT: float,
    FieldDescriptor.TYPE_INT32: int,
    FieldDescriptor.TYPE_INT64: int, #long
    FieldDescriptor.TYPE_UINT32: int,
    FieldDescriptor.TYPE_UINT64: int,#long
    FieldDescriptor.TYPE_SINT32: int,
    FieldDescriptor.TYPE_SINT64: int,#long
    FieldDescriptor.TYPE_FIXED32: int,
    FieldDescriptor.TYPE_FIXED64: int,#long
    FieldDescriptor.TYPE_SFIXED32: int,
    FieldDescriptor.TYPE_SFIXED64: int,#long
    FieldDescriptor.TYPE_BOOL: bool,
    FieldDescriptor.TYPE_STRING: str,
    FieldDescriptor.TYPE_BYTES: bytes,#lambda b: bytes_to_string(b),
    FieldDescriptor.TYPE_ENUM: int,
}

def field_type_to_fn(msg, field):
    if field.type == FieldDescriptor.TYPE_MESSAGE:
        # For embedded messages recursively call this function. If it is
        # a repeated field return a list
        result = lambda msg: proto_to_dict(msg)
    elif field.type in DECODE_FN_MAP:
        result = DECODE_FN_MAP[field.type]
    else:
        raise TypeError("Field %s.%s has unrecognised type id %d" % (
                         msg.__class__.__name__, field.name, field.type))
    return result


def proto_to_dict(msg):
    result_dict = {}
    extensions = {}
    for field, value in msg.ListFields():
        conversion_fn = field_type_to_fn(msg, field)

        # Skip extensions
        if not field.is_extension:
            # Repeated fields result in an array, otherwise just call the
            # conversion function to store the value
            if field.label == FieldDescriptor.LABEL_REPEATED:
                result_dict[field.name] = [conversion_fn(v) for v in value]
            else:
                result_dict[field.name] = conversion_fn(value)
    return result_dict


# Bind Socket UDP port 5432 as Telemetry recevice server
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 5432))

count = 0

start_time = time.time()


while True:
    count += 1
    buf, addr = sock.recvfrom(65535)
    Telemetry_content = telemetry_pb2.Telemetry()

    print(buf.hex())
    print(buf)
    print("Message Length {}".format(len(buf)))
    #print(len(str(buf)))
    #Handle Telemetry UDP GPB and GPB-kv  from IOX
    if buf[0:1] == b'\x00': ##check the binary daa , no official document
        print("Telemetry GPB  message from IOX")
        Telemetry_content.ParseFromString(buf[12:])
        print('Node :'+Telemetry_content.node_id_str)
        print('IP Address (source port) :' + str(addr))
        print('Encodig Path :' + Telemetry_content.encoding_path)
        content_json_dict = proto_to_dict(Telemetry_content.data_gpb)  # Old proto_to_dict from google public code
        print(MessageToJson(Telemetry_content)) # new MessageToJson from google public code , will replace proto_to_dict
        print("*"*20)
        print(content_json_dict)


        if len(str(Telemetry_content.data_gpbkv)) > 2: # Handle GPB kv , in case of unstable A9KV , sometimes sent empty content message
            print('GPB kv format')
            Fields_list = Telemetry_content.data_gpbkv[0].fields[1].fields
            #print(Fields_list)

            json_dict = proto_to_dict(Telemetry_content.data_gpbkv[0])
            #pprint.pprint(json_dict)
            print(json_dict)

            for field in Fields_list:

                #print(field.fields)
                if field.string_value:
                    print(field.name + ':' + field.string_value)
                if field.uint32_value:
                    print(field.name + ':' + str(field.uint32_value))


    print("="*200)

Run the script:

[root@telemetry telemetry-protocol]# python3 telemetry_server_udp_gpb-kv.py
0001000100010000000000700a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b508d8e
b'\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00p\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bP\x8d\x84\xf0\x98\xbd.h\x8d\x84\xf0\x98\xbd.'
Message Length 124 <<<
Telemetry GPB  message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
  "nodeIdStr": "iosxrv9000-1",
  "subscriptionIdStr": "nms",
  "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
  "collectionId": "3466",
  "msgTimestamp": "1596974694925",
  "collectionEndTime": "1596974694925"
}
********************
{}
========================================================================================================================================================================================================
000100010001000000000a300a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b48fe8d
b"\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\n0\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bH\xfe\x83\xf0\x98\xbd.P\xfe\x83\xf0\x98\xbd.Z\xbd\x13\x08\x8c\x84"
Message Length 2620 <<<
Telemetry GPB  message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
  "nodeIdStr": "iosxrv9000-1",
  "subscriptionIdStr": "nms",
  "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
  "collectionId": "3466",
  "collectionStartTime": "1596974694910",
  "msgTimestamp": "1596974694910",
  "dataGpbkv": [
    {
      "timestamp": "1596974694924",
      "fields": [
        {
          "name": "keys",
          "fields": [
            {
              "name": "interface-name",
              "stringValue": "GigabitEthernet0/0/0/1"
            }
          ]
        },
        {
          "name": "content",
          "fields": [
            {
              "name": "interface-handle",
              "stringValue": "GigabitEthernet0/0/0/1"
            },
......
********************
{}
GPB kv format
{'timestamp': 1596974694924, 'fields': [{'name': 'keys', 'fields': [{'name': 'interface-name', 'string_value': 'GigabitEthernet0/0/0/1'}]}, {'name': 'content', 'fields': [{'name': 'interface-handle', 'string_value': 'GigabitE}
interface-handle:GigabitEthernet0/0/0/1
interface-type:IFT_GETHERNET
hardware-type-string:GigabitEthernet
state:im-state-up
line-state:im-state-up
encapsulation:ether
encapsulation-type-string:ARPA
mtu:1514
is-l2-transport-enabled:false
state-transition-count:1
is-dampening-enabled:false
speed:1000000
duplexity:im-attr-duplex-unknown
media-type:im-attr-media-unknown
link-type:im-attr-link-type-force
in-flow-control:im-attr-flow-control-off
out-flow-control:im-attr-flow-control-off
bandwidth:1000000
max-bandwidth:1000000
is-l2-looped:false
loopback-configuration:no-loopback
description:to server-1
fast-shutdown:false
if-index:7
is-intf-logical:false
========================================================================================================================================================================================================
^CTraceback (most recent call last):
  File "telemetry_server_udp_gpb-kv.py", line 90, in 
    buf, addr = sock.recvfrom(65535)
KeyboardInterrupt

Capture packets and checking

[root@telemetry telemetry-protocol]# tcpdump -i ens192 -w telemetry-udp-new.pcap
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
^C380 packets captured
381 packets received by filter
0 packets dropped by kernel

telemetry-udp-new1.pcap

1+
你可以留言,或者trackback 从你的网站

留言哦

blonde teen swallows load.xxx videos