VMFS Recovery before the last night of 2021 :)

We had a situation that one ESXi host had been crashed because of multiple power outages, and unfortunately one of its virtual machine couldn't be power-on via any action. I tried to start the VM through the GUI and CLI, and after attaching the VMDK files to a new VM, the operation failed and didn't let me bring it back to the circle of service, because the VMFS datastore was truly corrupted.

Reinstalling the ESXi host and, copying the virtual machine's file to another datastore are two options that weren't possible to do. why? There was no stable backup file and everything was in dastastore1, so any type of ESXi installation may be led to data loss and, it was a risky way. Also, copying the virtual disk files was not possible in that state, since all VMDK files were corrupted. Then we decided to attach the whole array to a Linux OS like an Ubuntu system and as you may know, this popular recovery method: Mounting the VMFS partition inside the Ubuntu:

# vmfs-fuse /dev/sdb1 /mnt/myvol/    (For example in my case sdb1 means disk 2 partition 1)

However, we face another error that mentioned VMFS version 6 is not supported. So, consider if you want to recover a VMFS6 datastore, you should install vmfs6-tools, not the vmfs-tools that is used for the older versions like VMFS5. Also, you can get it through apt command or git from GitHub link:

# apt-get install vmfs6-tools

 

Now we could run the vmfs6-fuse without any problem and mount the datastore inside the mentioned directory. Although I don't want to investigate details of file/directory-related applications in this post, but as another important point is you should know you can mount them only inside the /mnt/ directory. so to move them again to another directory, you may have issue with apps like nautilus to work for security considerations.

Finally, we achieved to mount the datastore, and migrate the vmdk files to another storage, then attach them to another installed OS in a new VM, and power it up. Although inside the Guest OS, we forced the reset the NTFS permissions and take the ownership of the required files to grant access. It was absolutely a long way to go for us, but we did it successfully

As an easy-to-do recommendation keep your backup files in at least two safe places, then you are less likely to encounter such situations. I hope you guys have a safe year in 2022 with more protection and truly fewer issues.

 

 

Log4Shell (Log4j) or Log 4 Hell ?!

 

From four days ago we did whatever we could do to make secure many infrastructures against the recent RCE (remote code execution) zero-day vulnerability: Apache log4j. Especially we focused on virtualization environments and critical components like vCenter Servers, vRealize Suite, Horizon View Connection Server's Apache Tomcat, and Unified Access Gateway (UAG). I truly confess it's a bad situation, very very bad. Personally, I believe that no exploit was worse than this in 2021, because there are many solutions from a variety of vendors that use Apache as the web App (and subsequently log4j as the Apache's logging component): From Cisco, VMware, Dell to many Industrial Applications of OT Networks.

The log4j is a Java-based logging framework of Apache web services, but its default JNDI feature (Java Naming & Directory Interface) is the primary point of this zero-day. The severity rate of this vulnerability is critical (10/10) that was publicly announced on Dec 9th, 2021 by the developer team and led them to release of the CVE-2021-44228.

All versions below 2.14.1 are vulnerable by default. However, in the last day Apache Software Foundation (ASF) announced the second vulnerability CVE-2021-45046 that version 2.15.0 can be affected too. So, the suggested workarounds are not enough and cannot address this catastrophic exploit lonely. SANS said: "The new version 2.16.0 has been made available to completely fix the issue (so far at least)"

As ASF explained about the Apache log4j 2 update: "In version 2.16.0 the message lookups feature has been completely removed. Lookups in configuration still work. Furthermore, Log4j now disables access to JNDI by default. JNDI lookups in configuration now need to be enabled explicitly. Also, Log4j now limits the protocols by default to only Java, LDAP, and LDAPS and limits the LDAP protocols to only accessing Java primitive objects. Hosts other than the localhost need to be explicitly allowed."

This vulnerability lets the attacker execute RCEs on target without authentication, which potentially is a high risk that you should never ignore. However, VMware announced VMSA-2021-0028 which listed all the exploitable products and also their current workarounds and possible fixes. Of course, there are many other products that are under investigation to find further weaknesses against the possibility of this vulnerability. So, I strongly recommend executing any immediate remediation action, because the situation is still @#$%ed up and one of the best choices is the following FAQ of this "In the Wild" exploit.

At last, you should be aware that in most products you cannot handle directly this issue and need to review the vendor's provided solution to fix it. While these actions are not enough to protect against the attacker's activities, but it was mentioned on many FAQs that adding this line: "-Dlog4j2.formatMsgNoLookups=true" has no require to revert back to the latest setting. So, it seems the patching will roll the products forward and is accommodated with the provided workarounds. Then I think it's not just a temporary solution.

VMware UAG deployment and thumbprint detection issue in front of the firewall

 In isolated network structures that you are forced to secure incoming connections from outside the network, it’s absolutely necessary to deploy the VMware Unified Access Gateway (UAG) for securing the VDI environment include of a limited type of access and also hardening the internal Horizon Connection Servers (CS). In most scenarios, the UAG server(s) will be deployed in the DMZ boundary (subnet / VLAN) in the standalone form or load-balanced structure, so if there is a firewall between these network boundaries, probably restrict the communications between them. (CS & UAG servers)
 UAG as the heir of Security Server and Access Point is based on Photos OS VMware’s built-in Linux distribution OS and can act as the Edge service for VMware Workspace ONE and Horizon suite. It also supports the Java KeyStore (JKS) certificate format that it’s recommended you learn how to work with Linux built-in SSL tools like OpenSSL and KeyTool, Because in the UAG deployment, in the first step of configuration, you should set the Horizon Connection Server’s URL address and thumbprint, then the UAG should validate the CS certificate. There are two methods for this operation:

1. Set the SHA1 thumbprint manually for the deployed CS, no different it's a self-signed certificate, or even generated by an internal CA service. 
2. Add the CA certification path (root & subordinate) for an external valid certificate if it doesn't exist to validate the announced certificate by the CS.
   

 

 In some experiences, I saw the configured firewall policy includes SSL inspection may prevent directly secured channel establishment, even if the firewall trusts the corresponding SSL certificate. It means the firewall changes the SSL connection between the CS and UAG, in the way that the firewall acts as the middle of communication and secures two sides of the channel: UAG to FW and, CS to FW and also, their reverse-path in the firewall rules.

 I mentioned before, we can import the trusted certificate used by Horizon CS (that has VDM as its friendly name too) to the UAG Java KeyStore but still, you need to consider the firewall intervention between SSL communication, especially if you used hardware appliance devices, like Sophos, Juniper, Fortinet, PaloAlto. You can convert the certificate (openssl) and import its root CA to the Java KeyStore (keytool) and also, explore inside the cacerts file (UAG Java KeyStore) in the following path:

# vi /lib/jvm/jre/lib/security/cacerts 

Because if you want to import the SSL anyway, you will still encounter some other issues in the road of certificate validation in UAG. Thus I will explain them later in another post. When I understood my firewall is like an intermediate between the CS and UAG SSL communication, therefore although I set the SHA1 thumbprint, UAG couldn't connect to the CS successfully. So as the next step, I tried the curl command like this:

# curl -v https://horizon-cs.company.com:443

And after reviewing the generated result with more precision, I got the firewall accepts the certificate but change the issuer’s information and replaces it with the firewall’s default issued certificate. Now I was ensuring that's the primary issue. So, I ran the following command to find the X509 format of the responding certificate’s thumbprint:

# echo | openssl s_client -connect horizon-cs.company.com:443 |& openssl x509 -fingerprint -noout

Then I find out if I replace the Horizon certificate thumbprint with the required result, then the secured connection will be established successfully.

 

Congratulation Now the UAG is connected to the CS.