The eagle eyed among you may have noticed that we just recently came back from around a 37 hour downtime. This was not planned by us (otherwise we would have told you in advance!), but was due to a hardware failure on our server and our provider being unable to replace it in a timely fashion.

 

On Tuesday evening at around 21:40 UTC, one of the routers at the datacenter was hit with a large DDoS. This attack wasn't aimed at us at all, but rather at the router itself (motives unknown), and along with a few other clients, caused our IPs to stop routing correctly. This was the initial cause of the downtime.

 

Within 5-10 minutes, we were back up again after the router had recovered from the attack and then our primary networking controller on the server physically failed. We have redundancies built in to combat situations like this, and what simply should have happened is that the connection fell back to our secondary networking controller, and the server continued as normal until a hardware replacement could be sourced for our primary controller. What actually happened was that the network connection fell back to our secondary controller, and then within 15 minutes, our secondary controller failed as well, leaving us completely without networking, and in what was, given the chance of these failures, an unprecedented situation.

 

This was diagnosed and our ISP informed us the server would have to undergo a full rebuild as the networking is contained on the motherboard. They completed this pretty quickly (it was diagnosed and rebuilt within 16 hours), but they then informed us that the server room we are located in was full and they were having trouble finding a spot for a new box. This is what took all of the further time, while we were waiting for them to free up space. Eventually they found us a spot and re-provisioned the server, and here we are back again.

 

So, apologies for the downtime, but it was a little out of our control!

Is there a Tip.it Twitter or something we will be able to get information on next time in case this happens in the future? :)

Yes there is but on this occasion no notice was posted to it. @runetips

This period without Tip.It was difficult, not gonna lie, I rely on these forums more than I care to admit whilst doing my uni work.

^I never knew how much I loved these forums until they went down. I felt empty on the inside.

Glad we are back up and running but I have to say it was rather disappointing that with FB and Twitter setup there was virtually no message given out in the interim as to wtf was going on. Admittedly FB got a not eventually but it was near 24 hours in to the issue.

I had to divert all of my forum time to the HLF :/

i warned you not to reveal the wu-tang secret

Sounds like you were as prepaired as possable...

Sometimes sh** just happens.

I missed the site too and am glad its up and running.

 

Passes some glue to reattach the hair Im betting some of you lost while waiting =)

Is there a Tip.it Twitter or something we will be able to get information on next time in case this happens in the future?

Glad to see you guys/ladies back up and running once again. :thumbsup:

 

I hope all of you can catch whoever caused the DDoS and put them in jail for a very long time. :angry:

 

~D. V. "I missed these forums and their community!" Devnull

Good to have it back. No apology needed for something that was obviously not your fault. Slight apology needed for not letting us know anything through social media.

Glad it's back. It's only when the forums are down that I realize how often I check them :D

Good to have it back. No apology needed for something that was obviously not your fault. Slight apology needed for not letting us know anything through social media.

 

We did post something on Twitter:

My apoligies then it must've gone up after I checked - in that case you're all wonderful :)

Got on TIF this morning, thinking, "Man I haven't been on here for 2 days. It's gonna take a while to catch up on all the posts I haven't yet rea-- oh."

Whoa, what are the chances of both networking controllers to physically fail within such a short time?

Got on TIF this morning, thinking, "Man I haven't been on here for 2 days. It's gonna take a while to catch up on all the posts I haven't yet rea-- oh."

I read this and wanted to burst out laughing. I know exactly the feeling. Any chance you'd also double-take and go "Wait, what?" :huh:

 

 

Whoa, what are the chances of both networking controllers to physically fail within such a short time?

I think Peter would agree with me saying "Ridiculously Remote"... :geek:

 

 

~D. V. "Just my mental $0.25, curiosity included..." Devnull

Whoa, what are the chances of both networking controllers to physically fail within such a short time?

I think Peter would agree with me saying "Ridiculously Remote"... :geek:

 

This is where our provider's sales people start trying to sell us backups for our backups, right? :lol: