Funny network problems

Yeah, Jenkins (the CI tool, previously Hudson) took down a whole building at a previous job.

All the workers were multicasting to the controller and vice-versa, and the switches amplified to everyone.

Interesting morning.

A friend of mine took down the entire university network. On the day that the first-year students were registering (which traditionally is one week before the rest).

He was using this OS that was developed by the CS department at Stellenbosch, called Gneiss (programmed in Oberon, of course), so everything was really low-level (unlike with say Linux, that would tell non-root users to bugger off). And he was synchronising multiple nodes using subnet broadcast. Now subnet-broadcasting is not supposed to get out of the subnet… but this time it did, and the other routers helped and repeated the broadcast all the way to the medical campus (which is like 50km away). A massive DDOS attack. The techs figured out where it is coming from and literally unplugged the fibre connection to the CS department and everything calmed down…

It’s like the one time a website that was hosted on my server was compromised on Christmas day and pulled down the hole network of the DC Centre. The tech also just unplugged the server.

No amount of pleading would get them to connect the network for just 5 mins. so I could just quickly fix it by disabling that user’s account and it being Christmas time everyone I know in Jo’burg was down at the coast.
It took me to get in the car and drive the family back to Mosselbay on boxing day, get a flight to Jo’burg on the first flight out, rent a car, drive to the DC, fix the problem in 5min. and then waiting util the end of the day to fly back home.
That was the last phisical server I ever did. From there on I only do Virtual Servers (that was long before Azure and AWS).

1 Like

Pffft, so low level…

Have you ever been part of a 15000 person corporate, where someone accidentally mails the entire company as a CC?
And then at least 500 people are stupid enough to reply-all asking to be removed from the thread…

Human-powered DDOS :see_no_evil:

I reckon we should be more precise with the term ‘virtual’. I am not keen on the term ‘the cloud’ either since we’ll produce the perception that there are server farms in the clouds…

I always say: the cloud is just someone else’s computer.

Well, I suppose that reduces the concept a bit TOO much. In theory it does have advantages, cloud providers do have this thing where the data is in more than one place (depending on service level) and in theory the server could move locations easily, unlike a real physical server.

For myself, I think the biggest advantage to virtual is simply the speed at which you can set something up. You can (for example) have ansible scripts (or playbooks) that provision a network of servers, with a public and private network, load balancing, databases with redundancy, literally with the click of a button. Well, probably more like typing a command for us terminal-warriors, but you get the point. You can scale up the number of servers on your platform when the traffic goes up, and scale them down (to spend less money) when the traffic is down. Server falls over? Quickly build another one. Trash the old one… no point in wasting the time fixing it.

In the old days, I had to order several physical servers from a data center (Hetzner back in the day), wait 2-3 days for them to be provisioned, then do our own deployment stuff. Server fell over? Data is on that one server. But even that was better than the days of collocation, when I physically had to get in my car and drive to Great Westerford (and get security clearance) to get at them…

It is much easier and faster to get into your Virtual Server than the physical one for sure!

Or client racking and stacking new appliance on the network, plugging in the I/O and switching on, total utter chaos. The appliance default setings were creating a loop on the network. Took em a few hours to find that issue…

On the Virtual stuff, my day job is working for one of the VM OEMs, Solutions architecting.