VPN site to site traffic loss

I have been working through this issue for the last 16 days and can't make any progress. A customer has a site to site VPN configured using Ubiquiti's UDM Pro routers, one at each site, and fiber from their ISP on both sides. Here is the issue and some observations: Every few hours, Ping, RDP, SMB all stop, on all devices from Site B to Site A. The wan doesn't go down at Site A or Site B, users at both sites still have Internet during this event The tunnel from what we see using ssh (ipsec statusall) into the udmp's, doesn't go down. It stays down for 10 minutes During the event all devices simultaneously lose connection, in both locations (no ping/rdp/smb from site A to site B and no ping/rdp/smb from Site B to Site A) There are multiple internal DNS servers at site A and ping/traffic times out even when using server/device IP addresses and not DNS

I have talked to Ubiquiti and sent logs, they don't see the tunnel go down in the logs. I have checked for updates on all ubiquiti devices. I have power cycled all devices I have confirmed that there isn't any IP conflict between any of the subnets at either site. The Fiber gateway at Site B has been power cycled The remote user VPN at Site A (not site to site) is not having this issue Servers have all received restarts since this problem started. Creating another site to site using "site C", Site C to Site A has intermittent issues but not as consistent as Site A to Site B. Site C to Site B experiences no packet loss.

Any thoughts or direction would be extremely helpful. I've been doing this a long time and I am at a loss. Everything I have found on the internet doesn't line up with all the factors. I feel like it's an ISP issue at Site A but I am not even sure what to yell at the ISP for, because the internet doesn't go down. Thanks in advanced.