I messed up.
TL;DR: I didn't have backups.
I think I already know my answer, but I figured I would ask in case there is something that I am missing that could save my tail.
On Monday I noticed that a drive had failed on my HP dl380 G8 and I ordered a new one. Yesterday morning I went to get into one of the services hosted on that machine, and I couldn't get in. I went look at the box and noticed a second drive failed.
This host is running proxmox 7.2, and zfs was not used.
I forced the server to shutdown and rebooted it. When it came backup there was a prompt asking if I wanted to ignore the drive failures and enable the drives, I pressed yes. The server booted into grub rescue with an error about "arch-independant ELF magic."
Not sure what to do, I booted into a live cd to see what was still there. I had drive errors when trying to use `fdisk -l` and ran `fsck /dev/sda` to fix the errors. I was able to load the drives, and could navigate the file structure.
It was at about this time in my googling that I discovered that this could have been caused by a grub update, and I attempted to perform the grub recovery steps from the proxmox forum. This never worked, and I would get an error when getting to the chroot step, saying "unable to run /bin/bash, permission denied".
Seeing as I was able to mount the LVM's I used qemu-convert to pull the raw images off the server and place them on external storage, but that brings up the next problem. Any VM that was running is corrupted. VM's that I had turned off, I can access by mounting the vhd file in a live cd.
So, my question is, am I fucked?