RedditPP Mascot

RedditPP

r/sysadmin • … • u/Lazy_Side_6830

Moving iLO credentials into a vault with offline caching

The September cumulative updates broke RDS on our session hosts, 2019 and 2022 both. It runs fine for a few hours after the patch, then the first round of logoffs deadlocks the whole session stack and a hard reset is the only way back, and of course Microsoft like the true (con)sistently helpful company they are only initially alerted M365 subscribers after fixing it.

Getting to the boxes was my problem, no RDP means iLO, and I know where iLO lives and I know the credentials exist. They were in a runbook page on the internal wiki. Which is hosted on a VM. On a host I was trying to reach. I was a dog chasing its own tail for a good week. Ended up driving in at 11pm to plug a crash cart into the host directly like it's 2009.

Anyway, after all was said and done, we made a long overdue decision to stop keeping anything out-of-band in a system that needs the network up. We are currently moving it all into self-hosted Passwork which sits on separate infrastructure from the estate it holds credentials for, and turning offline caching on for those records so they sit readable on my laptop even with no connection at all. Records will get marked in passwork for offline and cached encrypted on the device, with the cache wiping itself if the device doesn't sync inside a window I set as admin.

The aforementioned shift solves an issue but opens others up:

1. How do you stop cached credentials going stale? If someone rotates an iLO password and my laptop hasn't synced since, I'm holding a dead credential at exactly the moment I need a live one.

2. How do you stop people copying them out? Nothing stops someone opening a cached credential and pasting it into a notes app for convenience which puts me back where I started with the wiki.

3. What about phones? A couple of the guys who'd need this offshore would be caching to phones I can't remote wipe the way I can a laptop.

4. Who gets it? Do I give offline access to everyone who might need it at 2am or keep it to a couple of us and accept that coverage depends on who's around?

Thank you in advance for any advice given.

— 0