»You are not logged in :: [login] or [register] ::
You must be registered to post messages.
inspiration » A - General Discussion » 01 - Announcements » Outage: 9 Dec - 11 Dec 2005
Author Topic: Outage: 9 Dec - 11 Dec 2005
Ryan Dean
Lord of Elements
LVL: 772
posted 11 December 2005 08:04 AM      $post_id   Author's Homepage   Email Ryan Dean   Send New Private Message    Reply With Quote 
Howdy Everyone-
I'm afraid I have to report a serious case of downtime. The server hosting Dragonsworn (known as ace) apparently went offline sometime on Friday, 9 December according to reports. I was informed that the machine wasn't functioning correctly on Saturday, 10 December.

Working with remote hands, I was able to get the server operational again on 10 December, but noticed something that could have occurred one of two ways: a failing filesystem, or deliberate tampering. In order to insure the system was clean, a fresh 'world' build was performed and installed. Once the pristine libraries and binaries were in place, the system was rebooted.

The system never recovered from the reboot. Working with remote hands again, it was discovered that the system was hanging after mounting the root (/) filesystem, but before executing any of the normal startup scripts (/etc/rc, init, etc).

A new hard drive was purchased for the system, operating under the assumption that the drive was quite possibly failing. The drive was loaded locally with the latest version of the operating system (FreeBSD 6.0; was previously running 4.11), had all of the latest OS security patches and bugfixes applied, and installed all the software needed to run the site.

The new hard drive was installed in the server around 0600 on Sunday, 11 December 2005, and the core services were restored as of 0700 on Sunday, 11 December 2005. (The hour was a bunch of data moving from old drive to new and configuration restoration for the core services).

As of now, the server appears to be running just fine, but not all services on the system have been fully restored. All of web and DNS have been though, so there /shouldn't/ be any trouble accessing Dragonsworn.

As always, if you come across anything odd / unusual, or if something isn't working for you, feel free to shoot me an email. I can be reached at rtdean@dragonsworn.com or rtdean@cytherianage.net. Those email addresses are on different servers, so if you ping me at both, I should always get your message.
-Ryan T. Dean

PS- All times listed are in EST.

edited by Ryan Dean on December 11th, 2005

Director, Dragonsworn

Posts: 271 | ID: 29  |  Aligned: unaligned  |  House: Element