ESXI tips
17 11 月, 2021,由 frank撰写
修改WEB访问的端口
默认https的端口是443，某些情况下，443有可能被禁了，所以想用其他端口，此时可以直接在下面文件中修改端口，重启即可：
[root@localhost:~] more /etc/vmware/rhttpproxy/config.xml | grep 443
<httpsPort>443</httpsPort>通过命令行启动虚机
重启ESXI主机后，如果发现重要的虚拟机忘记放入开启自启动了，此时还无法通过web页面访问ESXI，但SSH可以访问，此时就需要通过命令行来启动虚拟机，方法如下：
[root@localhost:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
10 xrv9k-731-1 [xxx] xrv9k-731-1/xrv9k-731-1.vmx otherGuest64 vmx-09
12 xrv9k-731-2 [xxx] xrv9k-731-2/xrv9k-731-1.vmx otherGuest64 vmx-09
14 VMware vCenter Server [xxx] VMware vCenter Server/VMware vCenter Server.vmx other3xLinux64Guest vmx-10 VMware vCenter Server Appliance
2 centos7.9-beihe [xxx] centos7.9/centos7.9.vmx centos7_64Guest vmx-18
3 centos7.9-qiufan [xxx] centos7.9-xxxxxx/centos7.9.vmx centos7_64Guest vmx-18
4 centos7.9-songjian [xxx] centos7.9-xxxxxxxx/centos7.9.vmx centos7_64Guest vmx-18
6 win10 [xxx] win10/win10.vmx windows9_64Guest vmx-18
8 EVE-C-112 [xxx] EVE-C-112/EVE-C-112.vmx ubuntu64Guest vmx-11
9 centos7-yangsuit [xxx] centos7-yangsuit/centos7-yangsuit.vmx centos7_64Guest vmx-18
[root@localhost:~] vim-cmd vmsvc/power.on 6
Powering on VM:远程更改ESXI的管理IP
机房变更，需要更改ESXI的管理IP地址，但又不方便去现场连显示器和键盘，是否可以远程更改IP地址？答案是可以的，只要你能ssh到ESXI的shell里：
[root@localhost:~] TERM=xterm
[root@localhost:~] dcui
如果你发现输入f2没有反应，那么在MacOS里的键盘设置中直接勾选下面项目即可：
离线升级7.0.0 to 7.0.3
先下载升级包：VMware-ESXi-7.0U3-18644231-depot.zip
然后通过Web上传到存储空间中，通过SSH进入，执行下面命令（注意-d后面要写绝对路径，否则会报错，即是你在image目录下，也要输入绝对路径）：
[root@localhost:~] esxcli software sources profile list -d /vmfs/volumes/datastore1/image/VMware-ESXi-7.0U3-18644231-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -----------------
ESXi-7.0U3-18644231-standard VMware, Inc. PartnerSupported 2021-10-05T00:00:00 2021-10-05T00:00:00
ESXi-7.0U3-18644231-no-tools VMware, Inc. PartnerSupported 2021-10-05T00:00:00 2021-09-17T16:22:28
[root@localhost:~] 然后用下面命令升级，注意/tmp目录下的空间十分有限，如果遇到下面告警，就近到tmp目录下，删掉里面的所有内容：
Error unlocking file /tmp/_tmp_stagebootbank.lck: [Errno 28] No space left on device然后执行升级，重启后完成升级：
[root@localhost:~] esxcli software profile update -d /vmfs/volumes/datastore1/image/VMware-ESXi-7.0U3-18644231-depot.zip -p ESXi-7.0U3-18644231-standard
Update Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_atlantic_1.0.3.0-8vmw.703.0.0.18644231......2022-01-16 更新：安装VIB固件更新
有时候需要安装第三方固件更新一些工具包，如Mellanox的工具包，但安装时会有下面报错：
[root@localhost:~] esxcli software vib install -v /tmp/mft-4.18.0.107-10EM-700.0.0.15843807.x86_64.vib
[InstallationError]
('MEL_bootbank_mft_4.18.0.107-0', 'Could not find a trusted signer: certificate has expired')
cause = ('MEL_bootbank_mft_4.18.0.107-0', 'Could not find a trusted signer: certificate has expired')
vibs = ['MEL_bootbank_mft_4.18.0.107-0']
Please refer to the log file for more details.这时可以通过下面方法不去确认证书：
[root@localhost:~] esxcli software acceptance set --level=CommunitySupported
Host acceptance level changed to 'CommunitySupported'.
[root@localhost:~] esxcli software vib install -v /tmp/mft-4.18.0.107-10EM-700.0.0.15843807.x86_64.vib --no-sig-check
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: MEL_bootbank_mft_4.18.0.107-0
VIBs Removed:
VIBs Skipped:
[root@localhost:~] esxcli software acceptance set --level=PartnerSupported
Host acceptance level changed to 'PartnerSupported'.更改vCenter7的IP地址
注意：这部分是更改IP的操作，但更改IP后，发现无法访问，所以删了重装（重装过程看下一个章节）；重装后发现遗漏了一个重要步骤，那就是需要更改下/etc/hosts中的域名。。。
直接通过Console接口进去后，更改IP总是报下面错误：
经过查找，需要通过Alt + F1（在MacOS，Option+F1）进入shell，然后执行下面命令通过命令行修改IPv4地址、网关和DNS：
/opt/vmware/share/vami/vami_config_net
无DNS部署vCenter7.0.3
首先完成第一阶段的安装，配置如下，FQDN和DNS都设置成自己：
直接登录vCenter，需要通过Alt + F1（在MacOS，Option+F1）进入shell，修改/etc/hosts，加入下面一行
10.114.252.142 localhost最后通过“https://10.114.252.142:5480”登录vCenter并完成第二阶段的安装，配置如下：
这部分参考网站：无DNS解析环境下部署VCenter(VCSA)7.0
2022-01-17 更新：在ESXI开启SR-IOV
最近在测试虚拟测试仪的性能，需要使用SR-IOV，PCI Passthrough跟SR-IOV有什么区别呢？下面是从网上趴下来的图片，看后会很清晰，简单理解就是一个是独享，一个是共享；
网卡用的是Mellanox CX-5单口100g，需要单独开启SR-IOV，详细可以看这里，下面是简单总结：
安装Mellanox工具包
下载mft并安装，具体参考本文中的：“2022-01-16 更新：安装VIB固件更新”；
查看网卡的默认状态
[root@localhost:~] /opt/mellanox/bin/mst status
MST devices:
------------
mt4119_pciconf0
mt4119_pciconf1
mt4125_pciconf2
mt4119_pciconf3
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf1 q
Device #1:
----------
Device type: ConnectX5
Name: MCX515A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE single-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: mt4119_pciconf1
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7]
PF_BAR2_SIZE 0
NON_PREFETCHABLE_PF_BAR False(0)
VF_VPD_ENABLE False(0)
STRICT_VF_MSIX_NUM False(0)
VF_NODNIC_ENABLE False(0)
NUM_PF_MSIX_VALID True(1)
NUM_OF_VFS 0
NUM_OF_PF 1
PF_BAR2_ENABLE False(0)
SRIOV_EN True(1)
PF_LOG_BAR_SIZE 5
VF_LOG_BAR_SIZE 1
NUM_PF_MSIX 63
NUM_VF_MSIX 19
INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0)
PCIE_CREDIT_TOKEN_TIMEOUT 0
PARTIAL_RESET_EN False(0)
SW_RECOVERY_ON_ERRORS False(0)
RESET_WITH_HOST_ON_ERRORS False(0)
ADVANCED_POWER_SETTINGS False(0)
CQE_COMPRESSION BALANCED(0)
IP_OVER_VXLAN_EN False(0)
MKEY_BY_NAME False(0)
ESWITCH_IPV4_TTL_MODIFY_ENABLE False(0)
PRIO_TAG_REQUIRED_EN False(0)
UCTX_EN True(1)
PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
LRO_LOG_TIMEOUT0 6
LRO_LOG_TIMEOUT1 7
LRO_LOG_TIMEOUT2 8
LRO_LOG_TIMEOUT3 13
LOG_DCR_HASH_TABLE_SIZE 11
DCR_LIFO_SIZE 16384
ROCE_CC_PRIO_MASK_P1 255
CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1)
CLAMP_TGT_RATE_P1 False(0)
RPG_TIME_RESET_P1 300
RPG_BYTE_RESET_P1 32767
RPG_THRESHOLD_P1 1
RPG_MAX_RATE_P1 0
RPG_AI_RATE_P1 5
RPG_HAI_RATE_P1 50
RPG_GD_P1 11
RPG_MIN_DEC_FAC_P1 50
RPG_MIN_RATE_P1 1
RATE_TO_SET_ON_FIRST_CNP_P1 0
DCE_TCP_G_P1 1019
DCE_TCP_RTT_P1 1
RATE_REDUCE_MONITOR_PERIOD_P1 4
INITIAL_ALPHA_VALUE_P1 1023
MIN_TIME_BETWEEN_CNPS_P1 0
CNP_802P_PRIO_P1 6
CNP_DSCP_P1 48
LLDP_NB_DCBX_P1 False(0)
LLDP_NB_RX_MODE_P1 OFF(0)
LLDP_NB_TX_MODE_P1 OFF(0)
DCBX_IEEE_P1 True(1)
DCBX_CEE_P1 True(1)
DCBX_WILLING_P1 True(1)
KEEP_ETH_LINK_UP_P1 True(1)
KEEP_IB_LINK_UP_P1 False(0)
KEEP_LINK_UP_ON_BOOT_P1 False(0)
KEEP_LINK_UP_ON_STANDBY_P1 False(0)
NUM_OF_VL_P1 _4_VLs(3)
NUM_OF_TC_P1 _8_TCs(0)
NUM_OF_PFC_P1 8
DUP_MAC_ACTION_P1 LAST_CFG(0)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
IB_ROUTING_MODE_P1 LID(1)
PCI_WR_ORDERING per_mkey(0)
MULTI_PORT_VHCA_EN False(0)
PORT_OWNER True(1)
ALLOW_RD_COUNTERS True(1)
RENEG_ON_CHANGE True(1)
TRACER_ENABLE True(1)
IP_VER IPv4(0)
BOOT_UNDI_NETWORK_WAIT 0
UEFI_HII_EN False(0)
BOOT_DBG_LOG False(0)
UEFI_LOGS DISABLED(0)
BOOT_VLAN 1
LEGACY_BOOT_PROTOCOL PXE(1)
BOOT_LACP_DIS True(1)
BOOT_VLAN_EN False(0)
BOOT_PKEY 0
ATS_ENABLED False(0)
DYNAMIC_VF_MSIX_TABLE False(0)
EXP_ROM_UEFI_x86_ENABLE False(0)
EXP_ROM_PXE_ENABLE True(1)
ADVANCED_PCI_SETTINGS False(0)
SAFE_MODE_THRESHOLD 10
SAFE_MODE_ENABLE True(1) [root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf3 q
Device #1:
----------
Device type: ConnectX5
Name: MCX515A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE single-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: mt4119_pciconf3
Configurations: Next Boot
......
NUM_OF_VFS 0
NUM_OF_PF 1 更改网卡驱动并激活SR-IOV
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf1 s SRIOV_EN=1 NUM_OF_VFS=4
Device #1:
----------
Device type: ConnectX5
Name: MCX515A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE single-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: mt4119_pciconf1
Configurations: Next Boot New
SRIOV_EN True(1) True(1)
NUM_OF_VFS 0 4
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
[root@localhost:~]
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf3 s SRIOV_EN=1 NUM_OF_VFS=4
Device #1:
----------
Device type: ConnectX5
Name: MCX515A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE single-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: mt4119_pciconf3
Configurations: Next Boot New
SRIOV_EN True(1) True(1)
NUM_OF_VFS 0 4
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.重启ESXI，可以看到已经改了，但可以看到此时没有VF：
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf1 q |grep NUM_OF_VFS
NUM_OF_VFS 4
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4119_pciconf3 q |grep NUM_OF_VFS
NUM_OF_VFS
[root@localhost:~] lspci -d | grep -E "0000:23|0000:81"
0000:23:00.0 Ethernet controller: Mellanox Technologies ConnectX-5 EN network interface card 100GbE single-port (MCX515A-CCAT) [vmnic2]
0000:81:00.0 Ethernet controller: Mellanox Technologies ConnectX-5 EN network interface card 100GbE single-port (MCX515A-CCAT) [vmnic6]打开SR-IOV，可以通过Mellanox工具打开，这时也可以直接在ESXI打开，如下：
[root@localhost:~] lspci -d | grep -E "0000:23|0000:81"
0000:23:00.0 Ethernet controller: Mellanox Technologies ConnectX-5 EN network interface card 100GbE single-port (MCX515A-CCAT) [vmnic2]
0000:23:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.35.0_VF_0]
0000:23:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.35.0_VF_1]
0000:23:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.35.0_VF_2]
0000:23:00.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.35.0_VF_3]
0000:81:00.0 Ethernet controller: Mellanox Technologies ConnectX-5 EN network interface card 100GbE single-port (MCX515A-CCAT) [vmnic6]
0000:81:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.129.0_VF_0]
0000:81:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.129.0_VF_1]
0000:81:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.129.0_VF_2]
0000:81:00.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [PF_0.129.0_VF_3]
注意，如果是1块网卡的2个口，那么用前面的Address 就无法区分了，而ESXI的Web中只能看Address，所以只能进Shell通过lspci来区分，如下所示：
[root@localhost:~] lspci |grep Mellanox
0000:98:00.0 Ethernet controller: Mellanox Technologies ConnectX-6 Dx EN NIC; OCP3.0; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623436AN-CDA) [vmnic0]
0000:98:00.1 Ethernet controller: Mellanox Technologies ConnectX-6 Dx EN NIC; OCP3.0; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623436AN-CDA) [vmnic1]
0000:98:00.2 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_0]
0000:98:00.3 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_1]
0000:98:00.4 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_2]
0000:98:00.5 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_3]
0000:98:00.6 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_4]
0000:98:00.7 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_5]
0000:98:01.0 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_6]
0000:98:01.1 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.0_VF_7]
0000:98:01.2 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_0]
0000:98:01.3 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_1]
0000:98:01.4 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_2]
0000:98:01.5 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_3]
0000:98:01.6 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_4]
0000:98:01.7 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_5]
0000:98:02.0 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_6]
0000:98:02.1 Ethernet controller: Mellanox Technologies ConnectX Family nmlx5Gen Virtual Function [PF_0.152.1_VF_7]2022-01-21 更新：Disable Flow control on ESXI Server
ESXI默认是开启Flow Control的，用下面命令查看并关闭Flow Control，详细可以看官方文档：
[root@localhost:~] esxcli network nic pauseParams list
NIC Pause Params Supported Pause RX Pause TX Auto Negotiation Auto Negotiation Resolution Avail RX Auto Negotiation Resolution TX Auto Negotiation Resolution
------ ---------------------- -------- -------- ---------------- --------------------------------- ------------------------------ ------------------------------
vmnic0 true true true false false false false
vmnic1 true true true false false false false
vmnic2 true true true false false false false
vmnic4 true true true false false false false
vmnic5 true true true false false false false
vmnic6 true true true false false false false
vmnic8 true false false true false false false
vmnic9 true true true true false false false
[root@localhost:~]
[root@localhost:~] esxcli network nic pauseParams set
Error: Missing required parameter -n|--nic-name
Usage: esxcli network nic pauseParams set [cmd options]
Description:
set Set pause parameters for a NIC
Cmd options:
-a|--auto=<bool> Enable/disable auto negotiation.
-n|--nic-name=<str> Name of NIC whose pause parameters should be set. (required)
-r|--rx=<bool> Enable/disable pause RX flow control.
-t|--tx=<bool> Enable/disable pause TX flow control.
[root@localhost:~]
[root@localhost:~] esxcli network nic pauseParams set -n vmnic4 -r f
[root@localhost:~] esxcli network nic pauseParams set -n vmnic5 -r f
[root@localhost:~] esxcli network nic pauseParams set -n vmnic4 -t f
[root@localhost:~] esxcli network nic pauseParams set -n vmnic5 -t f
[root@localhost:~]
[root@localhost:~] esxcli network nic pauseParams list
NIC Pause Params Supported Pause RX Pause TX Auto Negotiation Auto Negotiation Resolution Avail RX Auto Negotiation Resolution TX Auto Negotiation Resolution
------ ---------------------- -------- -------- ---------------- --------------------------------- ------------------------------ ------------------------------
vmnic0 true true true false false false false
vmnic1 true true true false false false false
vmnic2 true true true false false false false
vmnic4 true false false false false false false
vmnic5 true false false false false false false
vmnic6 true true true false false false false
vmnic8 true false false true false false false
vmnic9 true true true true false false false
2023-06-27 更新：命令行确认CPU和Memory信息
需要确认一台ESXI Server的CPU和内存的相关信息，可是找不到iL0口的用户名和密码了，所以只能ssh到这台ESXI上，直接用命令行来查看相关信息，如下所示8颗4G内存，2颗E5504 4线程Intel CPU，符合web页面显示的信息：
[root@localhost:~] smbiosDump|grep "Size: "
Max. Size: 32 kB
Current Size: 32 kB
Max. Size: 256 kB
Current Size: 256 kB
Max. Size: 4096 kB
Current Size: 4096 kB
Max. Size: 32 kB
Current Size: 32 kB
Max. Size: 256 kB
Current Size: 256 kB
Max. Size: 4096 kB
Current Size: 4096 kB
Max. Size: 288 GB
Size: 4 GB
Size: 4 GB
Size: 4 GB
Size: No Memory Installed
Size: No Memory Installed
Size: No Memory Installed
Size: No Memory Installed
Size: No Memory Installed
Size: 4 GB
Size: 4 GB
Size: 4 GB
Size: No Memory Installed
Size: No Memory Installed
Size: No Memory Installed
Size: 4 GB
Size: 4 GB
Size: 0x00018
ROM Size: 4096 kB[root@localhost:~] vim-cmd hostsvc/hosthardware | grep E5504 -A 5 -B 5
(vim.host.CpuPackage) {
index = 0,
vendor = "intel",
hz = 2000070759,
busHz = 133338039,
description = "Intel(R) Xeon(R) CPU E5504 @ 2.00GHz",
threadId = (short) [
0,
1,
2,
3
--
(vim.host.CpuPackage) {
index = 1,
vendor = "intel",
hz = 2000070781,
busHz = 133338018,
description = "Intel(R) Xeon(R) CPU E5504 @ 2.00GHz",
threadId = (short) [
4,
5,
6,
7