The Great TMN Crash of 2014
-
- Administrator
- Posts: 15705
- Joined: Sun Jan 18, 2009 5:47 am
- Status: We got a German here who wants to die for his country! Oblige him!
The Great TMN Crash of 2014
First, allow me to apologize to each and every one of you for what happened over the weekend. I view any downtime as unacceptable and I certain view what happened this weekend as a complete breakdown in how our site should run. For those that are interested I'm starting this thread to explain and discuss what happened. Here's a breakdown of what went down, starting on Friday night.
Friday:
At 7:45PM I was notified by a member that the site was down. After initial troubleshooting I contacted our hosting provider and learned they were having database server issues and hoped to have them fixed quickly. I wasn't given an estimate for when we would be back online but was assured there was nothing we could do. Given that our primary database and any backups are managed by our host moving the site to another server elsewhere wasn't an option.
Saturday:
At 11:30AM I contacted the host for an update. They said they were still looking at it and no completion time was available.
At 2:30PM I contacted the host again. This time I was told that our site wasn't affected by the issue and our database had too many connections because the software was poorly written. When I told them that the database code hadn't changed in 6 months they had no answer.
Between 3:00 and 7:30 I contacted the host numerous times with escalating levels of frustration. I learned they did unannounced maintenance on our server and it corrupted a number of databases, of which ours was one. They had tried to repair our database and it was too corrupted to be saved. I told them to restore from a backup with they tried only to find out that the backup was corrupted as well.
At 8:30PM I requested a supervisor to discuss the situation with. At this point we were faced with not having any database backups newer than the end of last year since all of the ones from this year were on their servers and were all apparently corrupted. After a heated discussion they allowed me to have the raw database files for our site. I began repairing each file manually. The corrupted files were the table that stored posts with 306,000 records and the private messages table with 26,000 records. Over the course of last night and today I have been able to successfully recover the tables.
At 3:45 the database had been repaired, checked and installed on a server that we control. The site was brought back up and is functioning.
The site is currently going to be a little sluggish based on the fact that the install was hurried in order to get the site running.
Moving forward we have been left with no option but to move our site from a VPS environment to a dedicated server of our own. That way we can control backups, site and database settings and all aspects of the server. Until we find a new home the site can run on its current setup without any issues.
There will be abbreviated downtime when we move the site again. That downtime will be 15-20 minutes and will be one of our normal downtimes that happen late at night.
If you see anything not functioning on the website let us know. So far everything looks good. Hopefully it will stay that way.
Again, on behalf of the leadership team I apologize for the downtime. Moving forward we will take steps to make sure it doesn't happen again.
Friday:
At 7:45PM I was notified by a member that the site was down. After initial troubleshooting I contacted our hosting provider and learned they were having database server issues and hoped to have them fixed quickly. I wasn't given an estimate for when we would be back online but was assured there was nothing we could do. Given that our primary database and any backups are managed by our host moving the site to another server elsewhere wasn't an option.
Saturday:
At 11:30AM I contacted the host for an update. They said they were still looking at it and no completion time was available.
At 2:30PM I contacted the host again. This time I was told that our site wasn't affected by the issue and our database had too many connections because the software was poorly written. When I told them that the database code hadn't changed in 6 months they had no answer.
Between 3:00 and 7:30 I contacted the host numerous times with escalating levels of frustration. I learned they did unannounced maintenance on our server and it corrupted a number of databases, of which ours was one. They had tried to repair our database and it was too corrupted to be saved. I told them to restore from a backup with they tried only to find out that the backup was corrupted as well.
At 8:30PM I requested a supervisor to discuss the situation with. At this point we were faced with not having any database backups newer than the end of last year since all of the ones from this year were on their servers and were all apparently corrupted. After a heated discussion they allowed me to have the raw database files for our site. I began repairing each file manually. The corrupted files were the table that stored posts with 306,000 records and the private messages table with 26,000 records. Over the course of last night and today I have been able to successfully recover the tables.
At 3:45 the database had been repaired, checked and installed on a server that we control. The site was brought back up and is functioning.
The site is currently going to be a little sluggish based on the fact that the install was hurried in order to get the site running.
Moving forward we have been left with no option but to move our site from a VPS environment to a dedicated server of our own. That way we can control backups, site and database settings and all aspects of the server. Until we find a new home the site can run on its current setup without any issues.
There will be abbreviated downtime when we move the site again. That downtime will be 15-20 minutes and will be one of our normal downtimes that happen late at night.
If you see anything not functioning on the website let us know. So far everything looks good. Hopefully it will stay that way.
Again, on behalf of the leadership team I apologize for the downtime. Moving forward we will take steps to make sure it doesn't happen again.
We will be cruel to Pitt. And through our cruelty they will know who we are. And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us. And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives. And Pitt will be sickened by us. And Pitt will talk about us. And Pitt will fear us.
Re: The Great TMN Crash of 2014
Great work. We appreciate that effort you guys put in here.
WARNING the above post does not reflect my views or the views of TMN in any way.
-
- Hall of Fame
- Posts: 32154
- Joined: Sat Jan 31, 2009 8:52 am
- Status: Tried to retire. Failed. Trying again.
- Location: Cheat Lake, WV
- Contact:
Re: The Great TMN Crash of 2014
Yeah, what Aaron said. Thanks, Kevin - that's a lot of hassle to go through for no compensation other than the satisfaction of maintaining the best Mountaineer fan site on the interwebz. I really appreciate all the work you and the rest of the leadership team do to keep this place open for us.Aaron wrote:Great work. We appreciate that effort you guys put in here.
The Old Grad formerly known as wvu1980ae.
- eerfrombirth
- Hall of Fame
- Posts: 20504
- Joined: Fri Jan 30, 2009 5:34 pm
- Location: Danville, WV
Re: The Great TMN Crash of 2014
Yeah thanks Kev, I figured it was bad for it to be down that long.
The seeds we sow today determine the kind of fruit we'll reap tomorrow.
- sc-wvu4ever
- MVP
- Posts: 7877
- Joined: Tue Jan 20, 2009 9:19 pm
- Location: South Charleston
- Contact:
Re: The Great TMN Crash of 2014
Kevin, thank you and all of the leadership for what you do. I really appreciate the site, my favorite place on the world wide interwebz
From Weirton to Welch and Martinsburg to Matawan the hills resound with sounds of Gold and Blue Football.
- WVUgirl2003
- MVP
- Posts: 7528
- Joined: Tue Jan 20, 2009 9:20 pm
- Status: I will go in this way and find my own way out.
- Location: Hilo, Hawaii
Re: The Great TMN Crash of 2014
I totally thought I just got kicked out!
THANKS!
![smile :)](./images/smilies/1.gif)
THANKS!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
- WVUgirl2003
- MVP
- Posts: 7528
- Joined: Tue Jan 20, 2009 9:20 pm
- Status: I will go in this way and find my own way out.
- Location: Hilo, Hawaii
Re: The Great TMN Crash of 2014
or that the new owner of TOS that must not be named hacked this site or something!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
- Ryan
- Administrator
- Posts: 55171
- Joined: Mon Jan 19, 2009 1:53 pm
- Location: Pittsburgh, PA formerly Mangtang
Re: The Great TMN Crash of 2014
I've often considered banning you for all of your offensive posts and behavior.WVUgirl2003 wrote:I totally thought I just got kicked out!![]()
THANKS!
The sexual harassment of Aphetto was really over the line, but we let it slide.
Re: The Great TMN Crash of 2014
I'm ashamed but i thought i got banned at first lol
WARNING the above post does not reflect my views or the views of TMN in any way.
- Tito
- Hall of Fame
- Posts: 28179
- Joined: Wed Jan 21, 2009 9:54 am
- Status: One post over the line sweet Jesus.
- Location: The Senior from Warwood with 4 seconds on his stopped clock.
Re: The Great TMN Crash of 2014
See how good a warning feels instead of an arrest? Pay it forward...Aaron wrote:I'm ashamed but i thought i got banned at first lol
![Image](http://www.mysmiley.net/imgs/smile/angel/angelic.gif)
___________________________________________________________________________________
DISCLAIMER-WARNING: Please be advised that the above post is for amusement only. Any emotional involvement is done so at your own risk. Thanks for your interest.![Image](https://i.imgur.com/Bvhb7bJ.gif)
___________________________________________________________________________________
True, Father Time has long and cruelly sodomized me with the splintered haft of his great scythe, but I shall out-live the God-damned lot of you!
DISCLAIMER-WARNING: Please be advised that the above post is for amusement only. Any emotional involvement is done so at your own risk. Thanks for your interest.
![Image](https://i.imgur.com/Bvhb7bJ.gif)
___________________________________________________________________________________
True, Father Time has long and cruelly sodomized me with the splintered haft of his great scythe, but I shall out-live the God-damned lot of you!
- WVUgirl2003
- MVP
- Posts: 7528
- Joined: Tue Jan 20, 2009 9:20 pm
- Status: I will go in this way and find my own way out.
- Location: Hilo, Hawaii
Re: The Great TMN Crash of 2014
He still doesn't come around very much anymore. I feel so ashamed.Ryan wrote:I've often considered banning you for all of your offensive posts and behavior.WVUgirl2003 wrote:I totally thought I just got kicked out!![]()
THANKS!
The sexual harassment of Aphetto was really over the line, but we let it slide.
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
-
- Administrator
- Posts: 15705
- Joined: Sun Jan 18, 2009 5:47 am
- Status: We got a German here who wants to die for his country! Oblige him!
Re: The Great TMN Crash of 2014
I appreciate the kind words. It truly is a pleasure keeping the site running for everyone. Given this week is an off week I'm planning on moving the site to a dedicated server sometime over the weekend. I'll keep everyone updated as to when that goes down. We'll also be making provisions for a hot backup server so if one goes down the site will automatically flip to the other. That will be down the road.
We will be cruel to Pitt. And through our cruelty they will know who we are. And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us. And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives. And Pitt will be sickened by us. And Pitt will talk about us. And Pitt will fear us.
-
- Junior Varsity
- Posts: 1037
- Joined: Tue Jan 12, 2010 9:16 am
- Status: Apathy is a bitch and the death of men.
- Location: Morgantown, WV (Originally a Wetzel County Boy)
Re: The Great TMN Crash of 2014
Sometimes these things happen with this new fangled technology thing and the inter webs.
Thanks for the time you put in to bring the site back up for us.
Thanks for the time you put in to bring the site back up for us.
- Ryan
- Administrator
- Posts: 55171
- Joined: Mon Jan 19, 2009 1:53 pm
- Location: Pittsburgh, PA formerly Mangtang
Re: The Great TMN Crash of 2014
He's a pansy. Most of us would have taken a little sexual harassment in stride.WVUgirl2003 wrote:He still doesn't come around very much anymore. I feel so ashamed.Ryan wrote:I've often considered banning you for all of your offensive posts and behavior.WVUgirl2003 wrote:I totally thought I just got kicked out!![]()
THANKS!
The sexual harassment of Aphetto was really over the line, but we let it slide.
-
- Junior Varsity
- Posts: 1037
- Joined: Tue Jan 12, 2010 9:16 am
- Status: Apathy is a bitch and the death of men.
- Location: Morgantown, WV (Originally a Wetzel County Boy)
Re: The Great TMN Crash of 2014
If i got sexually harassed it would be the most action ive had in months.