vSAN Cluster Stuck Due to DDH
I upgraded my lab vSAN cluster to 7.0 U1, and then got all the disk format versions updated without issue. Then, the vSAN health service flagged a warning saying that the vSAN objects need to have a re-layout process done. I started this process, but after some time, the process hung at 72% and the cluster is now reporting 2 different disks on 2 of my hosts are failed and it is attempting to evacuate the data off of them via the Dying Disk Handling process.
This is erroneous as DDH is going solely by the response time here. Since I'm using consumer grade SSDs with no dedicated controller, I assume the drives slowed down due to the load from the object rebalance, but now nothing can proceed because it thinks the disks need to be removed, but doing so would render the entire vSAN datastore unreachable (4 hosts with FTT=1 and RAID 5 erasure coding). I tried unmounting the one disk group it said was fully evacuated, but that process timed out.
My VMs are still running, but everything seems hung on the vSAN side. Is there a way I can cancel the DDH evacuate process and tell vSAN that these SSDs are still good?