We had a situation that one ESXi host had been crashed because of multiple power outages, and unfortunately one of its virtual machine couldn't be power-on via any action. I tried to start the VM through the GUI and CLI, and after attaching the VMDK files to a new VM, the operation failed and didn't let me bring it back to the circle of service, because the VMFS datastore was truly corrupted.
Reinstalling the ESXi host and, copying the virtual machine's file to another datastore are two options that weren't possible to do. why? There was no stable backup file and everything was in dastastore1, so any type of ESXi installation may be led to data loss and, it was a risky way. Also, copying the virtual disk files was not possible in that state, since all VMDK files were corrupted. Then we decided to attach the whole array to a Linux OS like an Ubuntu system and as you may know, this popular recovery method: Mounting the VMFS partition inside the Ubuntu:
# vmfs-fuse /dev/sdb1 /mnt/myvol/ (For example in my case sdb1 means disk 2 partition 1)
However, we face another error that mentioned VMFS version 6 is not supported. So, consider if you want to recover a VMFS6 datastore, you should install vmfs6-tools, not the vmfs-tools that is used for the older versions like VMFS5. Also, you can get it through apt command or git from GitHub link:
# apt-get install vmfs6-tools
Now we could run the vmfs6-fuse without any problem and mount the datastore inside the mentioned directory. Although I don't want to investigate details of file/directory-related applications in this post, but as another important point is you should know you can mount them only inside the /mnt/ directory. so to move them again to another directory, you may have issue with apps like nautilus to work for security considerations.
Finally, we achieved to mount the datastore, and migrate the vmdk files to another storage, then attach them to another installed OS in a new VM, and power it up. Although inside the Guest OS, we forced the reset the NTFS permissions and take the ownership of the required files to grant access. It was absolutely a long way to go for us, but we did it successfully
As an easy-to-do recommendation keep your backup files in at least two safe places, then you are less likely to encounter such situations. I hope you guys have a safe year in 2022 with more protection and truly fewer issues.
No comments:
Post a Comment