Well,

Apparently someone decied to go into the cabinet where the servers are and then kick the power on the server. Then they decided to rewire their server and somehow plugged out network cable back into the wrong port.

Sorry for the downtime, we'll be looking for a new host now.

 

thought my life was over for a second there when i couldn't get on...

 

Sabotage, I tell you...

 

I thought maybe you guys had changed DNS or something. Yeh, that's bad. I had a server colo'd here in Virginia a few years back. Some very dumb datacenter employee was tasked with reloading the OS on a customer's box. Said customer just happened to have their box next to mine, long story short, tech reloaded OS on wrong server, mine.

They took 24 production websites off line and offered me not even a month of free service. Their CEO told me to go F myself basically. That guy is still on my list...

 

I thought Vic was back to work for my350z

 

LOL that is a common problem with datacenter operations...

 

 

Yah,

At that time we had some major paging of memory for some reason, I'll be tracking this down a bit more as well.

Average: pgpgin/s pgpgout/s fault/s majflt/s
10:20:01 AM 0.00 397.11 2683.23 0.00
10:30:01 AM 1222.78 5234.36 2834.05 2.45

 

Shoot, I'm glad that my company has its own rows of locked racks... avoid something like this...

 

I think a lot of datacenter employees get fatigued from the constant monotony of the job and do stuff like that. Not really an excuse though, when you consider they are entrusted with caring for the lifeblood of a lot of companies.

 

Yeh but don't the datacenter employees have access through your locks. If not, you have no remote hands assistance. I've had servers in Texas and California before, I couldn't lock their people out unless I'm going to hop on a plane everytime there's a problem.

 

You said that another customer's box was [presumably] in your rack. And that they mixed up another customer's with yours...? All of our rows of racks have our own servers... so mimimal risk of mixing up our servers with other customers.

Also, our colo is only a few minutes away from our centralized [primary] operations.

 

i cant live without the Z forums....

 

Kyle was in range before he moved, we are looking to relocate closer to one of us for this exact reason.

 

still cleaning up the mess a bit the hard shutdown on the db server = wasn't the best

 

OMG.... I thought the world was about to end!

 

I thought you guys did not pay your electric bill . . .

 

Sort of reverse happed to my boss before I got hired on, he called the cage monkeys to hit the power on a router and after they said they had done it, he still could not attach to it, after about four rounds of "we just recycled the power again"/"I still can't get on" he drove to the datacenter and the cage monkey walked him over to his rack, but it was not his rack, his was one row over. This guy had been rebooting someone else's router.

Once I got on board, we moved all our stuff over to a different facility and have been trouble free for over four years. The real beauty for me is our new datacenter is less than a mile from my house

Chris

 

 

you would not believe the guy that I had on the phone last night helping me.. oh the pain that was.. You would think they would hire someone with more brains than that..