Telemetry Receiver by UDP+KV-GPB
9 8 月, 2020,由 frank撰写
In the article, only show as follow (because easy to show how to work):
Protocol: UDP; Port: 5234
Encoding: KV-GPB
For more detailed information, you can reference my colleague’s GitHub, which includes more examples, e.g GRPC.
AlexFengCisco / Telemetry_Receiver
GRPC Update Info:
If use GRPC not TCP/UDP, that will have standard function, you will not need to write “DECODE_FN_MAP…”, function will auto generate, direct to use. Simple list steps:
- Get GRPC protobuf (e.g “mdt_grpc_dialout”) from cisco / bigmuddy-network-telemetry-proto
- Install/use compile tools, suggest install python virtual env, you can reference 在MacOS中部署Python虚拟开发环境
(telemetry-protocol) [root@telemetry telemetry-protocol]# python -V Python 3.6.8 (telemetry-protocol) [root@telemetry telemetry-protocol]# easy_install pip (telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio (telemetry-protocol) [root@telemetry telemetry-protocol]# pip install protobuf (telemetry-protocol) [root@telemetry telemetry-protocol]# pip install grpcio_tools (telemetry-protocol) [root@telemetry telemetry-protocol]# python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. cisco_grpc_dialout.proto (telemetry-protocol) [root@telemetry telemetry-protocol]# ls -l|grep cisco -rw-r--r--. 1 root root 2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #generated -rw-r--r--. 1 root root 3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #generated -rw-r--r--. 1 root root 485 Aug 10 16:15 cisco_grpc_dialout.proto #proto fileFollow all file if you need grpc protocol:[root@telemetry grpc-dailout]# ls -l total 36 -rw-r--r--. 1 root root 2695 Aug 10 16:16 cisco_grpc_dialout_pb2_grpc.py #Contains the server Stub class and the client Stub class, as well as the service RPC interface to be implemented. -rw-r--r--. 1 root root 3805 Aug 10 16:16 cisco_grpc_dialout_pb2.py #message serialization classes -rw-r--r--. 1 root root 484 Aug 10 15:36 mdt_grpc_dialout.proto -rw-r--r--. 1 root root 3722 Aug 10 16:11 service_grpc_dialout_no_tls.py #service python -rw-r--r--. 1 root root 19220 Aug 10 15:58 telemetry_pb2.py #decode gpb-kv messages, generate as belowgrpc-dailout.zip
Install Protobuf of python version
For detailed info, can reference google official doc: Protocol Buffer Basics: Python and Download Protocol Buffers.
[root@telemetry telemetry-protocol]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.12.4/protobuf-python-3.12.4.zip [root@telemetry telemetry-protocol]# unzip protobuf-python-3.12.4.zip [root@telemetry telemetry-protocol]# cd protobuf-3.12.4/ [root@telemetry protobuf-3.12.4]# ./configure [root@telemetry protobuf-3.12.4]# make [root@telemetry protobuf-3.12.4]# make check ...... PASS: protobuf-test PASS: protobuf-lazy-descriptor-test PASS: protobuf-lite-test PASS: google/protobuf/compiler/zip_output_unittest.sh PASS: protobuf-lite-arena-test PASS: no-warning-test ============================================================================ Testsuite summary for Protocol Buffers 3.12.4 ============================================================================ # TOTAL: 6 # PASS: 6 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ [root@telemetry protobuf-3.12.4]# make install [root@telemetry protobuf-3.12.4]# cd ./python/ [root@telemetry python]# python3 setup.py build [root@telemetry python]# python3 setup.py test [root@telemetry python]# python3 setup.py install [root@telemetry python]# protoc --version libprotoc 3.12.4
Compile Proto File
In the example, only show KV-GPB, so only require 1 proto file (telemetry.proto) for all sensor-path, if you use GPB for encoding, each sensor-path requires 1 proto file. The XR Proto file download from the official link: cisco / bigmuddy-network-telemetry-proto
[root@telemetry telemetry-protocol]# protoc -I=./ --python_out=./ ./telemetry.proto [root@telemetry telemetry-protocol]# ls -l total 6340 drwxr-xr-x. 13 root root 4096 Aug 9 15:58 protobuf-3.12.4 -rw-r--r--. 1 root root 6100223 Jul 29 07:58 protobuf-python-3.12.4.zip -rw-r--r--. 1 root root 7493 Aug 9 15:34 telemetry_server_udp_gpb-kv.py -rw-r--r--. 1 root root 19220 Aug 9 16:28 telemetry_pb2.py <<< -rw-r--r--. 1 root root 8187 Aug 9 16:27 telemetry.proto -rw-r--r--. 1 tcpdump tcpdump 344548 Aug 9 15:15 telemetry-udp.pcap
Telemetry Config On XR
How to get sensor path, please reference my last blog: How To Get Telemetry sensor path for show cmd on IOS XR?
telemetry model-driven destination-group nms address-family ipv4 10.75.37.91 port 5432 encoding self-describing-gpb protocol udp ! ! sensor-group interface sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface[interface-name='GigabitEthernet0/0/0/1'] ! subscription nms sensor-group-id interface strict-timer sensor-group-id interface sample-interval 20000 destination-id nms source-interface GigabitEthernet0/0/0/2 ! !
Telemetry UDP Server by Python
Come From: AlexFengCisco / Telemetry_Receiver
[root@telemetry telemetry-protocol]# more telemetry_server_udp_gpb-kv.py
'''
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MSG TYPE | ENCODING_TYPE |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MSG_VERSION | FLAGS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MSG_LENGTH |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ ~
~ PAYLOAD (MSG_LENGTH bytes) ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
MSG TYPE (2 bytes) = 1 (for MDT)
ENCODING_TYPE (2 bytes) = 1 (GPB), 2 (JSON)
MSG_VERSION (2 bytes) = 1
FLAGS (2 bytes) = 0
MSG_LENGTH (4 bytes)
'''
import socket, struct
import json
from google.protobuf.descriptor import FieldDescriptor
import time
import pprint
import telemetry_pb2
from google.protobuf.json_format import MessageToJson
DECODE_FN_MAP = {
FieldDescriptor.TYPE_DOUBLE: float,
FieldDescriptor.TYPE_FLOAT: float,
FieldDescriptor.TYPE_INT32: int,
FieldDescriptor.TYPE_INT64: int, #long
FieldDescriptor.TYPE_UINT32: int,
FieldDescriptor.TYPE_UINT64: int,#long
FieldDescriptor.TYPE_SINT32: int,
FieldDescriptor.TYPE_SINT64: int,#long
FieldDescriptor.TYPE_FIXED32: int,
FieldDescriptor.TYPE_FIXED64: int,#long
FieldDescriptor.TYPE_SFIXED32: int,
FieldDescriptor.TYPE_SFIXED64: int,#long
FieldDescriptor.TYPE_BOOL: bool,
FieldDescriptor.TYPE_STRING: str,
FieldDescriptor.TYPE_BYTES: bytes,#lambda b: bytes_to_string(b),
FieldDescriptor.TYPE_ENUM: int,
}
def field_type_to_fn(msg, field):
if field.type == FieldDescriptor.TYPE_MESSAGE:
# For embedded messages recursively call this function. If it is
# a repeated field return a list
result = lambda msg: proto_to_dict(msg)
elif field.type in DECODE_FN_MAP:
result = DECODE_FN_MAP[field.type]
else:
raise TypeError("Field %s.%s has unrecognised type id %d" % (
msg.__class__.__name__, field.name, field.type))
return result
def proto_to_dict(msg):
result_dict = {}
extensions = {}
for field, value in msg.ListFields():
conversion_fn = field_type_to_fn(msg, field)
# Skip extensions
if not field.is_extension:
# Repeated fields result in an array, otherwise just call the
# conversion function to store the value
if field.label == FieldDescriptor.LABEL_REPEATED:
result_dict[field.name] = [conversion_fn(v) for v in value]
else:
result_dict[field.name] = conversion_fn(value)
return result_dict
# Bind Socket UDP port 5432 as Telemetry recevice server
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 5432))
count = 0
start_time = time.time()
while True:
count += 1
buf, addr = sock.recvfrom(65535)
Telemetry_content = telemetry_pb2.Telemetry()
print(buf.hex())
print(buf)
print("Message Length {}".format(len(buf)))
#print(len(str(buf)))
#Handle Telemetry UDP GPB and GPB-kv from IOX
if buf[0:1] == b'\x00': ##check the binary daa , no official document
print("Telemetry GPB message from IOX")
Telemetry_content.ParseFromString(buf[12:])
print('Node :'+Telemetry_content.node_id_str)
print('IP Address (source port) :' + str(addr))
print('Encodig Path :' + Telemetry_content.encoding_path)
content_json_dict = proto_to_dict(Telemetry_content.data_gpb) # Old proto_to_dict from google public code
print(MessageToJson(Telemetry_content)) # new MessageToJson from google public code , will replace proto_to_dict
print("*"*20)
print(content_json_dict)
if len(str(Telemetry_content.data_gpbkv)) > 2: # Handle GPB kv , in case of unstable A9KV , sometimes sent empty content message
print('GPB kv format')
Fields_list = Telemetry_content.data_gpbkv[0].fields[1].fields
#print(Fields_list)
json_dict = proto_to_dict(Telemetry_content.data_gpbkv[0])
#pprint.pprint(json_dict)
print(json_dict)
for field in Fields_list:
#print(field.fields)
if field.string_value:
print(field.name + ':' + field.string_value)
if field.uint32_value:
print(field.name + ':' + str(field.uint32_value))
print("="*200)
Run the script:
[root@telemetry telemetry-protocol]# python3 telemetry_server_udp_gpb-kv.py
0001000100010000000000700a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b508d8e
b'\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00p\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bP\x8d\x84\xf0\x98\xbd.h\x8d\x84\xf0\x98\xbd.'
Message Length 124 <<<
Telemetry GPB message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
"nodeIdStr": "iosxrv9000-1",
"subscriptionIdStr": "nms",
"encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
"collectionId": "3466",
"msgTimestamp": "1596974694925",
"collectionEndTime": "1596974694925"
}
********************
{}
========================================================================================================================================================================================================
000100010001000000000a300a0c696f73787276393030302d311a036e6d73323e436973636f2d494f532d58522d7066692d696d2d636d642d6f7065723a696e74657266616365732f696e746572666163652d78722f696e746572666163653a0a323031392d31322d3033408a1b48fe8d
b"\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\n0\n\x0ciosxrv9000-1\x1a\x03nms2>Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface:\n2019-12-03@\x8a\x1bH\xfe\x83\xf0\x98\xbd.P\xfe\x83\xf0\x98\xbd.Z\xbd\x13\x08\x8c\x84"
Message Length 2620 <<<
Telemetry GPB message from IOX
Node :iosxrv9000-1
IP Address (source port) :('10.75.37.85', 45761)
Encodig Path :Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
{
"nodeIdStr": "iosxrv9000-1",
"subscriptionIdStr": "nms",
"encodingPath": "Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface",
"collectionId": "3466",
"collectionStartTime": "1596974694910",
"msgTimestamp": "1596974694910",
"dataGpbkv": [
{
"timestamp": "1596974694924",
"fields": [
{
"name": "keys",
"fields": [
{
"name": "interface-name",
"stringValue": "GigabitEthernet0/0/0/1"
}
]
},
{
"name": "content",
"fields": [
{
"name": "interface-handle",
"stringValue": "GigabitEthernet0/0/0/1"
},
......
********************
{}
GPB kv format
{'timestamp': 1596974694924, 'fields': [{'name': 'keys', 'fields': [{'name': 'interface-name', 'string_value': 'GigabitEthernet0/0/0/1'}]}, {'name': 'content', 'fields': [{'name': 'interface-handle', 'string_value': 'GigabitE}
interface-handle:GigabitEthernet0/0/0/1
interface-type:IFT_GETHERNET
hardware-type-string:GigabitEthernet
state:im-state-up
line-state:im-state-up
encapsulation:ether
encapsulation-type-string:ARPA
mtu:1514
is-l2-transport-enabled:false
state-transition-count:1
is-dampening-enabled:false
speed:1000000
duplexity:im-attr-duplex-unknown
media-type:im-attr-media-unknown
link-type:im-attr-link-type-force
in-flow-control:im-attr-flow-control-off
out-flow-control:im-attr-flow-control-off
bandwidth:1000000
max-bandwidth:1000000
is-l2-looped:false
loopback-configuration:no-loopback
description:to server-1
fast-shutdown:false
if-index:7
is-intf-logical:false
========================================================================================================================================================================================================
^CTraceback (most recent call last):
File "telemetry_server_udp_gpb-kv.py", line 90, in
buf, addr = sock.recvfrom(65535)
KeyboardInterrupt
Capture packets and checking
[root@telemetry telemetry-protocol]# tcpdump -i ens192 -w telemetry-udp-new.pcap tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes ^C380 packets captured 381 packets received by filter 0 packets dropped by kernel
