Why the Site Went Down!

remek

Founder
Excellent !
Extraordinaire !
Well Done !
Joined
Jan 30, 2006
Messages
4,310
Reaction score
111
Points
48
The past 24 hours have been the most nerve wrecking hours we've had here on the admin side of The PE Gym.

It all started with a simple idea: let's upgrade our server to increase our its speed, memory, and space nearly 10-fold. This would lead to faster page loads, and a better PE Gym experience for all of us here.

If only it all went as planned . . .

Our server provider has been a great company thus far (so great, in fact, we've stuck with them instead of hosting our own servers). They have overall been one of the best hosting services we've dealt with. Until today.

Midway through the server change process, the technicians of our hosting company accidentally deleted the entire site, including the forums, the articles, the guides, the product reviews, and everything else that encompasses the great community we got going here. The odds of this happening are rare to say the least.

They also accidentally deleted our failsafe: every single one of our hourly back-ups (we duplicate the site many times over each day in case things go wrong). Even worse, our most recent off-site back-ups were corrupted as well.

After several red bulls and man-hours of labor, some good news went our way: we we're able to recover the database and thus all the content - from forum posts to articles. We also were able to recover most of the files.

The site isn't 100 percent fixed, but we're on our way to getting everything back to normal over the next few days. For now, you should be able to view all the guides, articles, forum posts, and be able to post comments, reviews, on the forums, etc. Some of the pictures and the looks might be off, but the functionality should be there.

On the bright side, we've been rebuilding some of the site from the ground up anyways to help make the article-posting process easier (it's painful as hell, as calixto will tell you all about!) as well as get out many more community features. More on this soon.

The other bright side is we've now become as obsessed with backups and security as teenager girls are with Edward from Twilight. Over the next couple of weeks we'll be implementing more redundancy into the system than a class-A military server.

We'll also be working towards that original goal: a faster, more fluid PE Gym experience.

If you see anything out of whack over the next few days, please post about them in this thread.

Sorry again for the delay!
Aaron
 
Last edited:
  • Like
Reactions: UYN

Gyrta

Registered
Well Done !
Joined
Dec 1, 2009
Messages
1,509
Reaction score
50
Points
0
Finally back!
 

BigRooster

Senior Member, Member of the Month Apr 2018, PEG
Joined
Mar 14, 2010
Messages
3,727
Reaction score
127
Points
83
I was extremely concerned. Tried connecting a few times yesterday. Nothing was posted about being under repair, then on another attempt it showed maitnence being done. Then this morning I try to connect and it showed something about Apache Server, or something like that. It took three times to get connected. Well needless to say I am glad as are many others that you are up and running.
 

Pegasus

Administrator, PE Gym Editor, PEGym Hero; ,
Staff member
Excellent !
Joined
Jul 15, 2009
Messages
44,295
Reaction score
1,172
Points
133
Computers seem to go like this.
 

Unit

Retired ModeratorPEGym Hero
Joined
Aug 1, 2010
Messages
2,607
Reaction score
83
Points
0
Location
Western Rockies
Thanks for all your hard work and dead Red Bulls. I love this place. It seems like this kind of thing happens enough to computers that I was not that supprised that it went down for longer than planned. Had no idea how bad it was. Thanks again.:caked:
 

Dangler

Sr. Admin, ΦWizardΦ, White Tiger Award 2015
Staff member
Excellent !
Joined
Mar 20, 2010
Messages
41,668
Reaction score
556
Points
133
Location
North Texas
I found these very troubling, as you did:
"the technicians of our hosting company accidentally deleted the entire site"
"They also accidentally deleted our failsafe"

This reeks of no-oversight, lack of experience, not watching out for the customers best interests, not paying attention, and not following or having written procedures to follow.

I hope at one point remek, you sit down with these people and express your dissatisfaction with their lack of performance and incompetent sloppiness.
But hey, what the heck, its just a mistake, computers do strange things....
Thanks to you & your crew for bailing things out. I am glad for the updates & improvements, and appreciate all you do.
Dangler
 

diesel121

Registered
Well Done !
Joined
Mar 29, 2010
Messages
999
Reaction score
36
Points
0
Location
east coast
Thanks for your hard work fellas! My daily routine gets a bit thrown off when I can't visit the gym. Glad to be up and running again!


*diesel
 

wind chill

Registered
Well Done !
Joined
May 25, 2010
Messages
2,065
Reaction score
77
Points
0
Location
Europe
i thought they shut down this site permanently
good thing it didn't happen
 

JonPop

Retired Sr. Administrator , PEGym Hero
Joined
Feb 8, 2008
Messages
16,754
Reaction score
482
Points
0
Location
Reunited with his dear wife Holly.
I was a fish out of water. Thanks Control for keeping me in the loop.
Thanks Aaron for letting us know what happened.
When I was still receiving emails after Midnight (ET), I knew you guys
were still hard at work. Kudos for getting the PE Gym back up and running.
(jP
 

JonasKay

Registered
Well Done !
Joined
Aug 11, 2010
Messages
572
Reaction score
58
Points
0

MrBigDick

Retired Moderator, PEGym Hero
Joined
Jul 26, 2009
Messages
11,313
Reaction score
254
Points
0
Age
55
Location
Wherever I choose to be.....
Yes, thank you all for your hard work and dedication to this awesome site. I can imagine the thought of losing all of the hard work that has gone into building this home for us was quite stressful. Control and management rock! Good job guys.
 

Batwoman

Moderator Reserve, PEGym Hero
Joined
Dec 31, 2009
Messages
9,932
Reaction score
439
Points
0
Ditto to all of the above!
It was very strange to see the site just.... gone. Thinking about what it would be like if it were gone for good was sobering.... I am SO glad to see it back up again!
Thanks for your efforts and dedication.
 

not2big

Senior Super Moderator, PEGym Hero
Staff member
Well Done !
Joined
Jan 1, 2010
Messages
23,962
Reaction score
1,003
Points
133
Location
Jersey Shore
I think I lost an inch or so due to stress waiting for the forums to come back up.
 

remek

Founder
Excellent !
Extraordinaire !
Well Done !
Joined
Jan 30, 2006
Messages
4,310
Reaction score
111
Points
48
Thanks for the understanding, guys (and gal)! It makes all the effort worth it knowing we have a community behind us.

I found these very troubling, as you did:
"the technicians of our hosting company accidentally deleted the entire site"
"They also accidentally deleted our failsafe"

This reeks of no-oversight, lack of experience, not watching out for the customers best interests, not paying attention, and not following or having written procedures to follow.

I hope at one point remek, you sit down with these people and express your dissatisfaction with their lack of performance and incompetent sloppiness.
But hey, what the heck, its just a mistake, computers do strange things....
Thanks to you & your crew for bailing things out. I am glad for the updates & improvements, and appreciate all you do.
Dangler

Yea, mistakes do happen, but we will indeed be sitting down with them once we're done tweaking everything back to normal. They'll get an ear-full to say the least.

I was a fish out of water. Thanks Control for keeping me in the loop.
Thanks Aaron for letting us know what happened.
When I was still receiving emails after Midnight (ET), I knew you guys
were still hard at work. Kudos for getting the PE Gym back up and running.
(jP

Yea, it appears the hard work never ends around here. Control is the man. He hasn't slept except for cat naps since it happened two nights ago. The good news is we're almost to a complete workable site with everything important back to normal.

Not sure if you tried to email me JP, but as a side effect of this whole mess I lost my email accounts as well.
 

JonPop

Retired Sr. Administrator , PEGym Hero
Joined
Feb 8, 2008
Messages
16,754
Reaction score
482
Points
0
Location
Reunited with his dear wife Holly.
Yeah Aaron, I did send you several e mails, was wondering why I got no response.
Got your e mail fixed now? (jP

(You're welcome Miss Titty. I tried to keep everyone informed.)
 

remek

Founder
Excellent !
Extraordinaire !
Well Done !
Joined
Jan 30, 2006
Messages
4,310
Reaction score
111
Points
48
Yeah Aaron, I did send you several e mails, was wondering why I got no response.
Got your e mail fixed now? (jP

(You're welcome Miss Titty. I tried to keep everyone informed.)

Not quite. Focusing on the site first, then the emails.
 

GTO

Retired Moderator, PEGym Editor, PEGym Hero
Joined
May 21, 2010
Messages
5,449
Reaction score
163
Points
83
Location
Oh ... just around ...
I found these very troubling, as you did:
"the technicians of our hosting company accidentally deleted the entire site"
"They also accidentally deleted our failsafe"

This reeks of no-oversight, lack of experience, not watching out for the customers best interests, not paying attention, and not following or having written procedures to follow.

I hope at one point remek, you sit down with these people and express your dissatisfaction with their lack of performance and incompetent sloppiness.
But hey, what the heck, its just a mistake, computers do strange things....
Thanks to you & your crew for bailing things out. I am glad for the updates & improvements, and appreciate all you do.
Dangler

Gotta agree with Dangler on this one ... being in IT myself, those in the trade know this is something that should NEVER, NEVER happen.

But as we look on the bright side, the site is back and getting better. Thanks!!
 
  • Like
Reactions: Dangler