How To Deploy STC VM by QEMU?
Introduction
Virtual STC inclulde 3 components:
- License Server VM
- Application: Spirent TestCenter
- Virtual TestCenter: STC port VM
Different platoform have diff version, e.g Virtual TestCenter have ESXi platform or QEMU platform, different platform have diff License, if use incorrect platform or License, will have alarm, e.g:
So I summarized hwo to deploy STC VM by QEMU in the article.
Install CentOS 7 and config basic config
Config network adapter after install system
[[email protected] ~]# more /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=18aeb3df-a3ce-4550-96d3-3bbe236cdb01
DEVICE=ens192
ONBOOT=yes
IPADDR0=10.75.58.101
PREFIX0=24
GATEWAY0=10.75.58.1
DNS1=x.x.x.x
DOMAIN=cisco.comChange hostname and added local DNS
[[email protected] ~]# hostnamectl status
Static hostname: localhost.localdomain
Icon name: computer-vm
Chassis: vm
Machine ID: 7722efff25664298a497dec2458b7288
Boot ID: 17cda692aefc4a0a9ff7fa53498cd8d4
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1127.el7.x86_64
Architecture: x86-64
[[email protected] ~]#
[[email protected] ~]# hostnamectl set-hostname frank
[[email protected] ~]#
[[email protected] ~]# hostnamectl status --static
frank
[[email protected] ~]# more /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.75.58.101 frank frank.cisco.com
[[email protected] ~]#
[[email protected] ~]# ping frank
PING frank (10.75.58.101) 56(84) bytes of data.
64 bytes from frank (10.75.58.101): icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from frank (10.75.58.101): icmp_seq=2 ttl=64 time=0.024 ms
^C
--- frank ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.024/0.046/0.069/0.023 ms
[[email protected] ~]#
[[email protected] ~]# more /etc/resolv.conf
# Generated by NetworkManager
search cisco.com
nameserver x.x.x.xDeploy Virtual TestCenter
There are four ways，due to image is QEMU, so suggest to use first or second:
- Server is Linux, direct to use KVM/QEMU + PCI Passthrough
- Server is Linux, direct to use KVM/QEMU + Bridge
- Server is ESXI, require nested virtualization, ESXI do PCI Passthrough first, then Linux VM do PCI Passthrough second… <— my testbed
- Server is ESXI, direct to use KVM/QEMU + Bridge on Linux VM
After install the STCv by install guide (can be found at the end of the article), do “installer/setup_setup_mgmt_bridge.py –n ens192 –b admin“, centos’s ens192 and vnet0 will link to bridge “admin”, as follow:
[[email protected] ~]# brctl show
bridge name bridge id STP enabled interfaces
admin 8000.000c290edc77 no ens192
vnet0
Nested Virtualization + Passthrough – Failed
Conclusion is not working in my testbed… Maybe have workaround, but that take us lots of time… so I just summarize the procedure that can be reviewed in future.
Enable Virtualization at CentOS7 VM
Enable PCI Passthrough on ESXI
Assign the PCI card in CentOS7 VM
ESXI report error (not capture the alarm) and cannot assign the passthrough card after did the step, so rollback ESXI passthrough…
Just Enable Passthrough on STCv – Failed
Enable IOMMU
Although we enable IOMMU at ESXI page, but linux not auto enable, check by “dmesg | grep -i -e DMAR -e IOMMU”, change grub.cfg to enable:
[[email protected] ~]# more /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"Then write to grub2.cfg by follow cmd:
[[email protected] ~]# grub2-mkconfig -o /etc/grub2.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1127.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1127.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-7722efff25664298a497dec2458b7288
Found initrd image: /boot/initramfs-0-rescue-7722efff25664298a497dec2458b7288.img
done
[[email protected] ~]# more /etc/grub2.cfg |grep iommu
linux16 /vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on
linux16 /vmlinuz-0-rescue-7722efff25664298a497dec2458b7288 root=/dev/mapper/centos-root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=onLaungh with passthrough
Enable passthrough and laungh STCv by follow cmd (bus info can be found by “lshw -c network” or “lspci“)
./launchvm.py --driver dpdk --port-type hostdev --test-dev 0000:13:00.0,0000:1b:00.0 --ports 2 --cpu 5But STCv hang and couldn’t boot successful, don’t know why, need check by Spirent, so stop at here.
Just use Bridge on STCv – Workable
Set up vP1 and vP2
[[email protected] /]# more /etc/sysconfig/network-scripts/ifcfg-ens224
DEVICE=ens224
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE=virbr1
[[email protected] /]# more /etc/sysconfig/network-scripts/ifcfg-ens256
DEVICE=ens256
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE=virbr2Set up virbr1 and virbr2
[[email protected] network-scripts]# more ifcfg-virbr1
DEVICE="virbr1"
BOOTPROTO="static"
ONBOOT="yes"
TYPE="Bridge"
NM_CONTROLLED="no"
[[email protected] network-scripts]# more ifcfg-virbr2
DEVICE="virbr2"
BOOTPROTO="static"
ONBOOT="yes"
TYPE="Bridge"
NM_CONTROLLED="no"Bound with STCv in Launcher script
10. Test Bridge virbr1, virbr2Check by STC Application
Config license server at “Tools -> License Management -> Virtual License -> add”; Then connect to chassis and checking license; will success if no other issue, as follow:
Reference
Thanks JiaMing’s help at “Nested Virtualization” and “Passthrough” part. Follow is official install guide: