Outage: 7/30
#1
Lost in Transit
Thread Starter
Join Date: Oct 2004
Location: Houston, TX
Posts: 10,003,830
Likes: 0
Received 0 Likes
on
0 Posts
Outage: 7/30
Well,
Apparently someone decied to go into the cabinet where the servers are and then kick the power on the server. Then they decided to rewire their server and somehow plugged out network cable back into the wrong port.
Sorry for the downtime, we'll be looking for a new host now.
Apparently someone decied to go into the cabinet where the servers are and then kick the power on the server. Then they decided to rewire their server and somehow plugged out network cable back into the wrong port.
Sorry for the downtime, we'll be looking for a new host now.
#4
Registered User
Join Date: May 2006
Location: Virginia, USA
Posts: 1,211
Likes: 0
Received 0 Likes
on
0 Posts
I thought maybe you guys had changed DNS or something. Yeh, that's bad. I had a server colo'd here in Virginia a few years back. Some very dumb datacenter employee was tasked with reloading the OS on a customer's box. Said customer just happened to have their box next to mine, long story short, tech reloaded OS on wrong server, mine.
They took 24 production websites off line and offered me not even a month of free service. Their CEO told me to go F myself basically. That guy is still on my list...
They took 24 production websites off line and offered me not even a month of free service. Their CEO told me to go F myself basically. That guy is still on my list...
#6
New Member
iTrader: (8)
Join Date: Jul 2005
Location: SOCAL
Posts: 3,022
Likes: 0
Received 0 Likes
on
0 Posts
Originally Posted by silvrhand
Well,
Apparently someone decied to go into the cabinet where the servers are and then kick the power on the server. Then they decided to rewire their server and somehow plugged out network cable back into the wrong port.
Sorry for the downtime, we'll be looking for a new host now.
Apparently someone decied to go into the cabinet where the servers are and then kick the power on the server. Then they decided to rewire their server and somehow plugged out network cable back into the wrong port.
Sorry for the downtime, we'll be looking for a new host now.
Trending Topics
#8
Lost in Transit
Thread Starter
Join Date: Oct 2004
Location: Houston, TX
Posts: 10,003,830
Likes: 0
Received 0 Likes
on
0 Posts
Yah,
At that time we had some major paging of memory for some reason, I'll be tracking this down a bit more as well.
Average: pgpgin/s pgpgout/s fault/s majflt/s
10:20:01 AM 0.00 397.11 2683.23 0.00
10:30:01 AM 1222.78 5234.36 2834.05 2.45
At that time we had some major paging of memory for some reason, I'll be tracking this down a bit more as well.
Average: pgpgin/s pgpgout/s fault/s majflt/s
10:20:01 AM 0.00 397.11 2683.23 0.00
10:30:01 AM 1222.78 5234.36 2834.05 2.45
#9
350Z-holic
iTrader: (2)
Join Date: Jun 2001
Location: teh interwebnets
Posts: 17,685
Likes: 0
Received 0 Likes
on
0 Posts
Originally Posted by C4talyst
I thought maybe you guys had changed DNS or something. Yeh, that's bad. I had a server colo'd here in Virginia a few years back. Some very dumb datacenter employee was tasked with reloading the OS on a customer's box. Said customer just happened to have their box next to mine, long story short, tech reloaded OS on wrong server, mine.
They took 24 production websites off line and offered me not even a month of free service. Their CEO told me to go F myself basically. That guy is still on my list...
They took 24 production websites off line and offered me not even a month of free service. Their CEO told me to go F myself basically. That guy is still on my list...
#10
Registered User
Join Date: May 2006
Location: Virginia, USA
Posts: 1,211
Likes: 0
Received 0 Likes
on
0 Posts
Originally Posted by ASTIG_Z
LOL that is a common problem with datacenter operations...
#11
Registered User
Join Date: May 2006
Location: Virginia, USA
Posts: 1,211
Likes: 0
Received 0 Likes
on
0 Posts
Originally Posted by yobri
Shoot, I'm glad that my company has its own rows of locked racks... avoid something like this...
#12
350Z-holic
iTrader: (2)
Join Date: Jun 2001
Location: teh interwebnets
Posts: 17,685
Likes: 0
Received 0 Likes
on
0 Posts
Originally Posted by C4talyst
Yeh but don't the datacenter employees have access through your locks. If not, you have no remote hands assistance. I've had servers in Texas and California before, I couldn't lock their people out unless I'm going to hop on a plane everytime there's a problem.
Also, our colo is only a few minutes away from our centralized [primary] operations.
#14
Lost in Transit
Thread Starter
Join Date: Oct 2004
Location: Houston, TX
Posts: 10,003,830
Likes: 0
Received 0 Likes
on
0 Posts
Kyle was in range before he moved, we are looking to relocate closer to one of us for this exact reason.
Originally Posted by yobri
You said that another customer's box was [presumably] in your rack. And that they mixed up another customer's with yours...? All of our rows of racks have our own servers... so mimimal risk of mixing up our servers with other customers.
Also, our colo is only a few minutes away from our centralized [primary] operations.
Also, our colo is only a few minutes away from our centralized [primary] operations.
#18
Mad Scientist
iTrader: (6)
Join Date: Jul 2004
Location: San Jose, CA
Posts: 2,795
Likes: 0
Received 0 Likes
on
0 Posts
Sort of reverse happed to my boss before I got hired on, he called the cage monkeys to hit the power on a router and after they said they had done it, he still could not attach to it, after about four rounds of "we just recycled the power again"/"I still can't get on" he drove to the datacenter and the cage monkey walked him over to his rack, but it was not his rack, his was one row over. This guy had been rebooting someone else's router.
Once I got on board, we moved all our stuff over to a different facility and have been trouble free for over four years. The real beauty for me is our new datacenter is less than a mile from my house
Chris
Once I got on board, we moved all our stuff over to a different facility and have been trouble free for over four years. The real beauty for me is our new datacenter is less than a mile from my house
Chris
#20
Lost in Transit
Thread Starter
Join Date: Oct 2004
Location: Houston, TX
Posts: 10,003,830
Likes: 0
Received 0 Likes
on
0 Posts
you would not believe the guy that I had on the phone last night helping me.. oh the pain that was.. You would think they would hire someone with more brains than that..