The TCP packet of death

This is another story from back when I was one of the admins for a medium-sized students' dormitory (~500 connected clients).

Physical layout consisted of 4 houses connected by fiber with every house sporting several Netgear managed switches, thus providing 100 Mbit connections.

So, one evening I was sitting in my room and reading when someone knocked on my door: "Internet is down!"

First thing to check: Local issue? Tried connecting to the router remotely - no dice. Not only is the "Internet down" but the whole network. Went over to the server room and logged in locally and discovered that the router was alive and kicking, also the uplink was fine but the connection to our network was borked. Around that time a fellow admin showed up who forgot more about networking than I'll ever learn.

However, he too failed to spot an immediate source of this error. Uplink - fine. No errors in the logs. Blinkenlights on the switches in the server room are indeed blinking and showing not only connected ports but also active ones! We tried several things over the course of the next hour but did not find anything. TCPdump didn't show any incoming traffic on our network facing card.

Until, by pure chance (or maybe desperation?), we rebooted the switches. Because the network suddenly came to life again... and ground to a halt mere seconds later.

Okay, time for another reset, this time with TCPdump to watch what's happening. And that's when we saw it: A TCP packet which consisted entirely of zeros (yes, even the leading bytes which should include stuff like source and target port) and just one single 0x0F somewhere. And it flooded the network for the short time it worked.

We then worked out what had obviously happened: Some computer in our network broadcasted those malformed packets which in turn caused our switches to freeze in such a state that they _appeared_ to still be working (I kid you not, if you didn't watch them closely you'd notice nothing - it was just that the pattern of blinking and steady lights didn't change over time as it usually would in a live network).

The solution was to pull out all the cables and connect them one by one until the switch failed again. That way we discovered the culprit - who wasn't home at the time (of course). He later told us that he came home to discover that his PC (which he had left on to complete a slow and huge download) had bluescreened which obviously didn't agree with his NIC which then decided to retaliate against the cruel world in the only way it could.