Passport?

I legally changed my name and gender marker on my state driver's license, but I never updated my passport. It's still good until 2028. I can't tell if it's better to have the original in case I need to go back in the closet temporarily, or have everything matching. My license gender marker is X, which I will be changing to M asap. Anyone have thoughts on this?