Security Recommendation and Hardening on Virtual Environments - Chapter One

 Today I want to start a series of discussions about some security tips and prospective best practices. I will start with some simple steps on ESXi configuration as the main asset between vCenter objects. At each post there is 4 or 5 security recommendation titles about VMware security hardening and are absolutely consistent with VMware published checklists:

1. ‌Built-in firewall rules:

As one of the first steps for ESXi hardening you can start from a limitation of permitted connections "To / From" the host and restricting unused transmit or blocking suspected traffics. So you may need to revision firewall rules and control what is permitted and what is not? or are their usage permanent or temporary for a specific time duration? Check your list again and for example, if you always want to have permanently SSH access to your hosts, limit allowed IP addresses to only your management system IP address.

2. Using SNMP version 3:

Because of the security nature of SNMP protocol on version3 in comparison with older versions 1 & 2 (based on the support of encryption, authentication & hashing algorithms), it's strongly recommended to use SNMPv3. Old versions are using only a community string for SNMP communication that is clear-text data and certainly is a security breach. So for monitoring ESXi hosts, it's better to configure only SNMPv3 settings by "esxcli system snmp set --v3targets ... (I will explain how to do it in another post)

3. VIB Verification:

VIB or vSphere Installation Bundle is a package file (like a ZIP) contains some installation files related to the ESXi. As the Kyle said there are 3 main parts of VIBs: Archive (Payload), XML (Descriptor) and a Signature file for trust level verification and you can configure it to each of 4 below-mentioned acceptance level depends on your system management policies:

I.   Partner: VIB creating and testing will be done by a partner and there is no VMware verification.

II.  VMware Certified: All processes will be done by VMware itself.

III. VMware Accepted: Testing will be done by partners but result verification relies on VMware. 

IV. Community: All processes executed outside of the VMware partner program and are not supported.

It's a good suggestion to don't trust to all community VIB packages

4. NTP configuration:

Time, Time and Time ... This is so important to remember to set it before doing every other configuration on your hosts. It's recommended to set at least one NTP server outside of your virtual infrastructure (like a router) for all of the hosts. ( I described it before on this post how to do it by CLI)

5. Versions of TLS:

It's always a real problem, Which version of TLS we should use on our managed hosts? and what version must be disabled? It's strongly recommended to use only TLS 1.2 but somehow maybe some of the associated management products to the ESXi host can only communicate with an older version. So before disable versions of 1.0 or 1.1, check this matter out.

I think it will be my last post in 2018, so I hope to have a great and perfect year in 2019. 

Happy New Year guys :)

Welcome to the CloudCred

So at last i decide to take steps to VMware vExpert seriously :)

Thanks to CloudCred for helping improving our knowledge about virtualization & cloud computing,

and sharing our experiences in this massive area. Today is a new day, for learning, understanding, reviewing and teaching to us and other good people of the world who are eager to learn about every corner of unlimited virtual universe.

Clustering & VM Monitoring

Shall we enable VM (and APP) Monitoring in the cluster settings? It's a big challenge for all virtual admins and countermeasures for answering to this big question, is a real matter.
VMware Tools is an important part of VM setup and for some reason it's necessary to install it on every guest OS (Windows, Linux and etc), because of it's role for time synchronization, better device detection and sending heartbeats. When you decide to enable VM monitoring in the cluster level, it will check heartbeat recieving from each VM. If there is no heartbeats or any network/storage traffics from VM (after default 2 min by clustering advanced options "das.iostatsInterval") VM will be restart but for confronting fault VM restarting, it will do it ONLY 3 times for each hour.
So we should ask ourselves in what situations it can be happened? and why VMware Tools shouldn't work? Generally Blue Screen states or any system faults can cause it. so it's better to reset the VM. As the VMware said, there is 3 states (Low/ Medium/ High) options that will configure 4 below custom settings:
1. Failure interval: It's about heartbeat recieving interval, so after this time HA will restart the VM.
2. Minimum Uptime: HA waiting time before VM startup and it's actions. (any heartbeat, storage, network traffics)
3. Maximum per-VM resets: I explained it before ;)
4. Maximum resets time window: Settings 3 will be executed on this time duration. (per hour)

Thanks to GS Khalsa for his related post on VMware.

ESXi 6.7 & WinSrv 2012R2

Whenever we don't care about servers updating, it can became a big problem. When you update your hosts, naturally you can see a new tip on your console like this: "vmware tools running out of date".  If you update to VMware ESXi 6.7 and there is  many virtual machines running on Windows Server 2012 R2  (he-he, yea exactly as i mentioned before without any regular update procedures or sadly WSUS server to do that on your environment) you will encounter with a great challenge. Because there is a series of prerequisites for installation of VMware Tools (VM version14) on this type of windows server:

VMware Tools ==> DotNet Framework 4.6 (or higher versions) ==> Update KB2919355

But the problem arises when you want to install this update manually (remember it's included some another update files and you need consider their order) it will warn "is not applicable with your system". So to fix that you need to install two hotfixes: KB2939087 and KB2975061 then try again for KB2919355. After successfully installation you can continue your setup procedure and at last install VMware Tools on your VMs.

 

VMSA-2018-0027

VMware announced  a security breach VMSA-2018-0027 and released related patch files. It's about a security issue on these VMware products: ESXi, Workstation and Fusion that contain uninitialized stack memory usage in the virtual machine network adapter type: VMXNET3 and on some situation allow a guest OS to execute code on the ESXi host. Remember other types of Adapter (VMXNET2, E1000, E1000E) are not affected.

It's severity is critical for all of the products and below i mentioned related patch for ESXi versions:

ESXi600-201811401-BG

ESXi650-201811301-BG

ESXi670-201811401-BG

For installation of patch files, one the best ways is using VMware Update Manager (VUM) or if there is not, run ESXCLI:

esxcli software vib update -d {path of imported file on ESXi folders}/{PatchFileName.zip} 

 

What is the VMKernel Core Dump - Part I

Generally coredump will be generated whenever the OS kernel sends certain signals to specified process, specially when the process send an access request to the out of address memory space. Often system will be crashed in this situation and generated errors give us related information about hardware faults or application bugs.

Sometimes you may encountered a ESXi host has been crashed, it will try to write diagnostics information on a file that has been name "VMkernel Core Dump". This file contains information about halt experience of host named purple screen state and has a high degree of importance, because in this situation, you don't have access to your system data and logs. So it's necessary to gather and analyze coredump files from all of ESXi host into one or more repositories.

There are two mechanisms for collection of coredump files: DiskDump to saving on specified permitted disk and NetDump to send coredump information by the network. If ESXi can't save coredump information on it's disk, there may be an issue with storage devices or it's connection to the host (Failed Array Controller, RAID Problem, broken physical path to storage, FC/SCSI connectivity problem, SAN switch failure and so on). So you should configure at least one alternative target to save coredump information.

But before that let's check about what is the netdump exactly?

netdump is a protocol for sending coredump information from a failed ESXi to the dump collector service that has these characteristics:

1. Listen on UDP port 6500.

2. Support only IPv4

3. Clear-text network traffic

4. No Authentication /Authorization

To retrieve current configuration for coredump saving location:

# esxcli system coredump partition get

# esxcli system coredump network get  (it can be used by check option too)

If the service is not enabled:

# esxcli system coredump network set --enable true

# esxcli system coredump partition set --enable true --smart

To set new configuration for coredump:

# esxcli system coredump partition set --partition="mpx.vmhba2:C0:T0:L0"

# esxcli system coredump network set --interface-name vmk0 --server-ipv4 10.10.10.10 --server-port 6500

To find-out which storage devices we have on the host:
# esxcli storage core path list

For the older version of VMware ESXi:

# esxcfg-dumppart --list

# esxcfg-dumppart --get-active

# esxcfg-dumppart --smart-activate

  

Network Dump Collector is a built-in service within vcenter server that provides a way of host coredump information gathering.But remember that NetDump does not work if aggregation protocols

such as LACP or Etherchannel has been configured for the vmkernel traffic.VMware recommends for segregation of VMkernel networking for NetDump by VLAN or physical LAN separation to prevent traffic interception. (In ESXi 5.0 VLAN tagging configured at the vSwitch level are ignored during network core dump transmission.)

Also the name structure and format of recieved coredump file is something like this: yyyy-mm-dd-hh_mm-N.zdump .

Maximum default size of zdump file is 2GB and older dump files automatically will be deleted. (The Dump Collector service has a non-configurable 60-second timeout and if no information is received in this period, the partial file will be deleted.)

Thanks to VMware for more information about it:  

VMWARE KB103251

Just after do your job by CLI remember to do /sbin/auto-backup.sh for saving configuration changes on your hosts:)

volume disk mounting problem with ubuntu 18

Today i want to share a good experience with you all that can be useful especially if your desktop is a linux OS. Last month i upgraded my OS and after that every time when i plugged my USB hard disk to my PC, system (ubuntu) didn't detect the Mass storage and unfortunately couldn't mount it, So many ways i tried to fix but this problem and nothing succeeded. After executing 'lsusb' command you can see your device is detected on USB port, but not as a mounted volume. Then before plugging the disk again, type this command to see what was happening on my system:

tail -F /var/log/syslog

Reported result is something like below:

"
 kernel: [  605.363832] usb 2-3: new SuperSpeed USB device number 3 using xhci_hcd
 usbguard-daemon[1015]: message repeated 3 times: [ IPC connection denied: uid=1000 gid=1000 pid=4988]
 usbguard-daemon[1015]: uid=0 pid=963 device.rule='block id 1058:1078 via-port "2-3" with-interface 08:06:50' type='Device.Insert' result='SUCCESS' device.system_name='/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3'
 usbguard-daemon[1015]: uid=0 pid=963 result='SUCCESS' device.system_name='/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3' target.new='block' type='Policy.Device.Update' device.rule='block id 1058:1078 target.old='block'
 usbguard-daemon[1015]: Ignoring unknown UEvent action: sysfs_devpath=/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3 action=bind
 kernel: [  605.384367] usb 2-3: New USB device found, idVendor=1058, idProduct=1078
 kernel: [  605.384374] usb 2-3: New USB device strings: Mfr=2, Product=3, SerialNumber=1
 kernel: [  605.384379] usb 2-3: Product:
 kernel: [  605.384384] usb 2-3: Manufacturer:
 kernel: [  605.384388] usb 2-3: SerialNumber:
 kernel: [  605.384755] usb 2-3: Device is not authorized for usage
 mtp-probe: checking bus 2, device 3: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3"
 mtp-probe: bus: 2, device: 3 was not an MTP device
 usbauth[6138]: Config file not found or empty.
 fwupd[5094]: failed to add USB device: 1058:1078 is not supported: USB error on device 1058:1078 : Entity not found [-5]
 upowerd[1346]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb2/2-3
 usbguard-daemon[1015]: IPC connection denied: uid=1000 gid=1000 pid=4988
" 
sudo -i (prompt to root access)
cd /sys/bus/usb/devices/2-3

First i had thought it was happened because of 

usbguard APP on my ubuntu, but after changing 

it's default decision to allow and nothing happened, try to investigated logs on line 11: "Device is not authorized for usage". So i tried to find-out what was the problem:

There is many files to show each attributes of my disk and one file named 'authorized'.

'cat authorized' reported back '0' and i changed it to '1' by 'vi' editor or 'leafpad' App  and after save the file, the disk volume mounted successfully at last.

There is an important point you should always consider, if your system can't mount volume, your VMs on VMware Workstation and other virtualization Type II host can't, even your system detect device type and you choose different guest OS like windows for your VM.