Can't crossplay

I'm trying to get some crossplay going between Xbonks Series S and my Steam Deck and it's just not happening. I try to join the Xbox from my Steam Deck and it seems to recognize the "server" - the world name shows up, but I get fail to connect. A password was set when the Xbox server, but it doesn't prompt me to enter a password, it just says failed to connect.

Trying to join the Steam Deck from the Xbonks I get a slightly different flavor of not working. We don't even see the server name come up, it just straight up says failed to join.

This is my 11 year old son playing on the Xbonks and my grown ass adult self playing on Steam Deck. If anyone knows the secret to being a terrible parent and fixing the Xbonks settings to allow my son to play Valheim multiplayer, I that might be the key here.