RedditPP Mascot

RedditPP

r/sysadmin • … • u/WhenSharksCollide

That escalated quickly...

Pretty sure this is the first thread I've started so sorry for formatting and etc. I generally just lurk and try to borrow knowledge from this sub and others like it. If this is the wrong place to post this, let me know I guess.

​

TL;DR at the end.

​

Long Version:

Previously I was working as a software support technician, although I have had prior minor IT roles. Recently I applied for a assistant network engineering position I was not sure I qualified for, saw some red flags, and underestimated them. I now am the only local person working at a small ISP that appears to have any networking experience. I know I am being paid less than a position of this type is worth (\~$40k yearly, I think this type of position is worth at least $60k in my area, but I haven't done too much research in that area.) I direct report to the COO, and I've been told repeatedly that "you will be taught everything you need to know." but have yet to find anyone with the necessary knowledge to maintain most of the systems that are in daily use by our customers. The field tech have the fiber and installs handled, but it's become apparent that on the backend the company is lacking in many ways.

Password policy? Nope. Bus factor? Probably less than 1. Backups? "Yeah, I think they backed that up last month." Test servers? Nadda. Co-location of critical services? The COO *just* (as in, within the last month) decided that it might be a good idea to co-locate *anything*.

There is no CTO, no SEC staff, no full time support staff, no local engineer (I guess aside from myself now...) and seemingly little to no plan or concern that the whole business could be blown out of the water by one or two small mistakes.

I'm currently stuck in the "fight or flight" mode. I could learn alot and make a name for myself here, or I could run it into the ground. I've got a solid six months of cash backed up for my cheap lifestyle, and I know they are not paying me what the position would be worth to someone with experience, which means if I dive in and learn everything I will out-grow my salary at a rapid pace, and then be inclined to leave them in a similar position to how I arrived (But hopefully more stable if I have anything to say about it.)

I'm looking for advice or anyone who has had to work their way through a similar situation.

TL;DR I guess I am a network engineer/system admin/SEC officer now. Everything is held together with moldy tape and hairspray. Advice much appreciated.

Edit/Update: The dawn of day 3, the biggest worstest.

No time to describe in full right now, but I am pulling the eject lever. COO is asking me to use a tool I have almost no training on or time with to find a problem he isn't sure exists, moments after I and the resident biologist found some concerning shit on a different part of the network. It's not worth it, I'll take the unplanned vacation just as soon as I've had ten minutes to lay out this mess to the COO and make it apparent he is in over his head.

UPDATE #2: So, I handed in my keys and left immediately following a meeting with the COO. In addition to the problems I have outlined above, the massive responsibilities that were forked on me, and the lack of training or experience present in most of the staff on this continent, early this morning I managed to discover that a support ticket I was investigating yesterday is very possibly a full blown security event involving a local municipality. Not only is that concerning on it's own, but the lack of fucks given by the COO when I brought that issue up in our meeting, plus the fact that I found the ticket on accident yesterday and it is over a week old with no prior response...that shit shook my bones. So, for CYA reasons I forced an impromtu meeting on the COO, abused single party consent on recorded conversations, and laid out the top level issues that I could see. Most of the next half hour was the COO saying verbatim the same things he said during our meeting yesterday and downplaying pretty much anything I would consider an immediate concern. After seeing his reaction to what I had discovered, or lack of one, I knew it was time to get out. Dropped my key on his desk, handed some scribbled notes to the biologist that is currently keeping the cellular systems alive on his own, and got the hell out. Three days of wasted time, and I am now jobless for the first time in years, but one of the things I do not want to do in my career is ride an ISP into a grave covered over with CVE's and the tears of my forefathers.

If anyone is in NY and needs a JUNIOR systems/network engineer, or an L2 tech...and you are not running a flaming shitshow, DM me. I learn quick and I've touched a little bit of everything by now.

— 0