Weirdest stutter/freeze issue when interacting with things after updating drivers

I'm at a loss. I know there are a lot of stuttering/fps problems with the game but I can't say I've seen anything similar to what I'm experiencing right now so I figured I'd just make a post in case somebody has experienced the same.

I have a new rig with 9800x3d and 9070 XT so my specs are fine. I played for hours yesterday with 100+ fps outside until eventually I got a CTD. I saw there were new AMD drivers (25.4.1) so I figured I'd install them in case it was better optimized. I sure as hell am regretting that now.

I launch the game, and after a few seconds casting spells causes major FPS drops. I look around and see that it might be shaders that need recompiling, so I force the game to do that but little luck. My FPS is bouncing from 5 to 80 and everything is just laggy hell. I try disabling AMD anti-lag which almost fixes the general FPS drops but whenever I try to open inventory, menu, crates, etc., my game just visually locks up. However, if I alt+tab/unfocus the game and go right back, the interface action goes through just fine. What the hell is going on? Anybody seen anything similar?

I haven't tried downgrading back to previous drivers yet, but I guess that is the next step. Either way it's certainly one of the weirdest performance bugs I've ever encountered. My rig is proven to be more than powerful enough, so certainly something in the software has gone real cooked.