Telemetry Receiver by UDP+KV-GPB
8月 9th, 2020,由 frank撰写
In the article, only show as follow (because easy to show how to work):
Protocol: UDP; Port: 5234
Encoding: KV-GPB
For more detailed information, you can reference my colleague’s GitHub, which includes more examples, e.g GRPC.
AlexFengCisco / Telemetry_Receiver
GRPC Update Info:
If use GRPC not TCP/UDP, that will have standard function, you will not need to write “DECODE_FN_MAP…”, function will auto generate, direct to use. Simple list steps:
- Get GRPC protobuf (e.g “mdt_grpc_dialout”) from cisco / bigmuddy-network-telemetry-proto
- Install/use compile tools, suggest install python virtual env, you can reference 在MacOS中部署Python虚拟开发环境
(telemetry-protocol) [[email protected] telemetry-protocol]# python -V Python 3.6.8 (telemetry-protocol) [[email protected] telemetry-protocol]# easy_install pip (telemetry-protocol) [[email protected] telemetry-protocol]# pip install grpcio (telemetry-protocol) [[email protected] telemetry-protocol]# pip install protobuf (telemetry-protocol) [[email protected] telemetry-protocol]# pip install grpcio_tools (telemetry-protocol) [[email protected] telemetry-protocol]# python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. cisco_grpc_dialout.proto (telemetry-protocol) [[email protected] telemetry-protocol]# ls -l|grep cisco -rw-r--r--. 1 root root 2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #generated -rw-r--r--. 1 root root 3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #generated -rw-r--r--. 1 root root 485 Aug 10 16:15 cisco_grpc_dialout.proto #proto fileFollow all file if you need grpc protocol:[[email protected] grpc-dailout]# ls -l total 36 -rw-r--r--. 1 root root 2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #Contains the server Stub class and the client Stub class, as well as the service RPC interface to be implemented. -rw-r--r--. 1 root root 3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #message serialization classes -rw-r--r--. 1 root root 484 Aug 10 15:36 mdt_grpc_dialout.proto -rw-r--r--. 1 root root 3722 Aug 10 16:11 service_grpc_dialout_no_tls.py #service python -rw-r--r--. 1 root root 19220 Aug 10 15:58 telemetry_pb2.py #decode gpb-kv messages, generate as belowgrpc-dailout.zip
Install Protobuf of python version
For detailed info, can reference google official doc: Protocol Buffer Basics: Python and Download Protocol Buffers.
[[email protected] telemetry-protocol]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.12.4/protobuf-python-3.12.4.zip [[email protected] telemetry-protocol]# unzip protobuf-python-3.12.4.zip [[email protected] telemetry-protocol]# cd protobuf-3.12.4/ [[email protected] protobuf-3.12.4]# ./configure [[email protected] protobuf-3.12.4]# make [[email protected] protobuf-3.12.4]# make check ...... PASS: protobuf-test PASS: protobuf-lazy-descriptor-test PASS: protobuf-lite-test PASS: google/protobuf/compiler/zip_output_unittest.sh PASS: protobuf-lite-arena-test PASS: no-warning-test ============================================================================ Testsuite summary for Protocol Buffers 3.12.4 ============================================================================ # TOTAL: 6 # PASS: 6 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ [[email protected] protobuf-3.12.4]# make install [[email protected] protobuf-3.12.4]# cd ./python/ [[email protected] python]# python3 setup.py build [[email protected] python]# python3 setup.py test [[email protected] python]# python3 setup.py install [[email protected] python]# protoc --version libprotoc 3.12.4
Compile Proto File
In the example, only show KV-GPB, so only require 1 proto file (telemetry.proto) for all sensor-path, if you use GPB for encoding, each sensor-path requires 1 proto file. The XR Proto file download from the official link: cisco / bigmuddy-network-telemetry-proto
[[email protected] telemetry-protocol]# protoc -I=./ --python_out=./ ./telemetry.proto [[email protected] telemetry-protocol]# ls -l total 6340 drwxr-xr-x. 13 root root 4096 Aug 9 15:58 protobuf-3.12.4 -rw-r--r--. 1 root root 6100223 Jul 29 07:58 protobuf-python-3.12.4.zip -rw-r--r--. 1 root root 7493 Aug 9 15:34 telemetry_server_udp_gpb-kv.py -rw-r--r--. 1 root root 19220 Aug 9 16:28 telemetry_pb2.py <<< -rw-r--r--. 1 root root 8187 Aug 9 16:27 telemetry.proto -rw-r--r--. 1 tcpdump tcpdump 344548 Aug 9 15:15 telemetry-udp.pcap
Telemetry Config On XR
How to get sensor path, please reference my last blog: How To Get Telemetry sensor path for show cmd on IOS XR?
telemetry model-driven destination-group nms address-family ipv4 10.75.37.91 port 5432 encoding self-describing-gpb protocol udp ! ! sensor-group interface sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface[interface-name='GigabitEthernet0/0/0/1'] ! subscription nms sensor-group-id interface strict-timer sensor-group-id interface sample-interval 20000 destination-id nms source-interface GigabitEthernet0/0/0/2 ! !
Telemetry UDP Server by Python
Come From: AlexFengCisco / Telemetry_Receiver
[[email protected] telemetry-protocol]# more telemetry_server_udp_gpb-kv.py ''' +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MSG TYPE | ENCODING_TYPE | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MSG_VERSION | FLAGS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MSG_LENGTH | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ ~ ~ PAYLOAD (MSG_LENGTH bytes) ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ MSG TYPE (2 bytes) = 1 (for MDT) ENCODING_TYPE (2 bytes) = 1 (GPB), 2 (JSON) MSG_VERSION (2 bytes) = 1 FLAGS (2 bytes) = 0 MSG_LENGTH (4 bytes) ''' import socket, struct import json from google.protobuf.descriptor import FieldDescriptor import time import pprint import telemetry_pb2 from google.protobuf.json_format import MessageToJson DECODE_FN_MAP = { FieldDescriptor.TYPE_DOUBLE: float, FieldDescriptor.TYPE_FLOAT: float, FieldDescriptor.TYPE_INT32: int, FieldDescriptor.TYPE_INT64: int, #long FieldDescriptor.TYPE_UINT32: int, FieldDescriptor.TYPE_UINT64: int,#long FieldDescriptor.TYPE_SINT32: int, FieldDescriptor.TYPE_SINT64: int,#long FieldDescriptor.TYPE_FIXED32: int, FieldDescriptor.TYPE_FIXED64: int,#long FieldDescriptor.TYPE_SFIXED32: int, FieldDescriptor.TYPE_SFIXED64: int,#long FieldDescriptor.TYPE_BOOL: bool, FieldDescriptor.TYPE_STRING: str, FieldDescriptor.TYPE_BYTES: bytes,#lambda b: bytes_to_string(b), FieldDescriptor.TYPE_ENUM: int, } def field_type_to_fn(msg, field): if field.type == FieldDescriptor.TYPE_MESSAGE: # For embedded messages recursively call this function. If it is # a repeated field return a list result = lambda msg: proto_to_dict(msg) elif field.type in DECODE_FN_MAP: result = DECODE_FN_MAP[field.type] else: raise TypeError("Field %s.%s has unrecognised type id %d" % ( msg.__class__.__name__, field.name, field.type)) return result def proto_to_dict(msg): result_dict = {} extensions = {} for field, value in msg.ListFields(): conversion_fn = field_type_to_fn(msg, field) # Skip extensions if not field.is_extension: # Repeated fields result in an array, otherwise just call the # conversion function to store the value if field.label == FieldDescriptor.LABEL_REPEATED: result_dict[field.name] = [conversion_fn(v) for v in value] else: result_dict[field.name] = conversion_fn(value) return result_dict # Bind Socket UDP port 5432 as Telemetry recevice server sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.bind(('0.0.0.0', 5432)) count = 0 start_time = time.time() while True: count += 1 buf, addr = sock.recvfrom(65535) Telemetry_content = telemetry_pb2.Telemetry() print(buf.hex()) print(buf) print("Message Length {}".format(len(buf))) #print(len(str(buf))) #Handle Telemetry UDP GPB and GPB-kv from IOX if buf[0:1] == b'\x00': ##check the binary daa , no official document print("Telemetry GPB message from IOX") Telemetry_content.ParseFromString(buf[12:]) print('Node :'+Telemetry_content.node_id_str) print('IP Address (source port) :' + str(addr)) print('Encodig Path :' + Telemetry_content.encoding_path) content_json_dict = proto_to_dict(Telemetry_content.data_gpb) # Old proto_to_dict from google public code print(MessageToJson(Telemetry_content)) # new MessageToJson from google public code , will replace proto_to_dict print("*"*20) print(content_json_dict) if len(str(Telemetry_content.data_gpbkv)) > 2: # Handle GPB kv , in case of unstable A9KV , sometimes sent empty content message print('GPB kv format') Fields_list = Telemetry_content.data_gpbkv[0].fields[1].fields #print(Fields_list) json_dict = proto_to_dict(Telemetry_content.data_gpbkv[0]) #pprint.pprint(json_dict) print(json_dict) for field in Fields_list: #print(field.fields) if field.string_value: print(field.name + ':' + field.string_value) if field.uint32_value: print(field.name + ':' + str(field.uint32_value)) print("="*200)
Run the script:
[[email protected] telemetry-protocol]# python3 telemetry_server_udp_gpb-kv.py 0001000100010000000000700a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b508d8e b'\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00p\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bP\x8d\x84\xf0\x98\xbd.h\x8d\x84\xf0\x98\xbd.' Message Length 124 <<< Telemetry GPB message from IOX Node :iosxrv9000-1 IP Address (source port) :('10.75.37.85', 45761) Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface { "nodeIdStr": "iosxrv9000-1", "subscriptionIdStr": "nms", "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface", "collectionId": "3466", "msgTimestamp": "1596974694925", "collectionEndTime": "1596974694925" } ******************** {} ======================================================================================================================================================================================================== 000100010001000000000a300a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b48fe8d b"\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\n0\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bH\xfe\x83\xf0\x98\xbd.P\xfe\x83\xf0\x98\xbd.Z\xbd\x13\x08\x8c\x84" Message Length 2620 <<< Telemetry GPB message from IOX Node :iosxrv9000-1 IP Address (source port) :('10.75.37.85', 45761) Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface { "nodeIdStr": "iosxrv9000-1", "subscriptionIdStr": "nms", "encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface", "collectionId": "3466", "collectionStartTime": "1596974694910", "msgTimestamp": "1596974694910", "dataGpbkv": [ { "timestamp": "1596974694924", "fields": [ { "name": "keys", "fields": [ { "name": "interface-name", "stringValue": "GigabitEthernet0/0/0/1" } ] }, { "name": "content", "fields": [ { "name": "interface-handle", "stringValue": "GigabitEthernet0/0/0/1" }, ...... ******************** {} GPB kv format {'timestamp': 1596974694924, 'fields': [{'name': 'keys', 'fields': [{'name': 'interface-name', 'string_value': 'GigabitEthernet0/0/0/1'}]}, {'name': 'content', 'fields': [{'name': 'interface-handle', 'string_value': 'GigabitE} interface-handle:GigabitEthernet0/0/0/1 interface-type:IFT_GETHERNET hardware-type-string:GigabitEthernet state:im-state-up line-state:im-state-up encapsulation:ether encapsulation-type-string:ARPA mtu:1514 is-l2-transport-enabled:false state-transition-count:1 is-dampening-enabled:false speed:1000000 duplexity:im-attr-duplex-unknown media-type:im-attr-media-unknown link-type:im-attr-link-type-force in-flow-control:im-attr-flow-control-off out-flow-control:im-attr-flow-control-off bandwidth:1000000 max-bandwidth:1000000 is-l2-looped:false loopback-configuration:no-loopback description:to server-1 fast-shutdown:false if-index:7 is-intf-logical:false ======================================================================================================================================================================================================== ^CTraceback (most recent call last): File "telemetry_server_udp_gpb-kv.py", line 90, in buf, addr = sock.recvfrom(65535) KeyboardInterrupt
Capture packets and checking
[[email protected] telemetry-protocol]# tcpdump -i ens192 -w telemetry-udp-new.pcap tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes ^C380 packets captured 381 packets received by filter 0 packets dropped by kernel
