Thank you Ventures Online. You may have noticed wetwired was down for a whole day this week. Below is the reason.

We have completed our root cause analysis on the service outage that occurred on the multi-host servers Phoenix, Paperboy, Contra, and Eclipse between Sunday August 15th and Monday August 16th, 2004.

Late Sunday evening we detected a possible security event on one of the four servers. After an initial review of the situation, we determined that a common outside source had gained privileged access to Paperboy, Contra and Eclipse. We later determined that this same attacker had attempted to gain privileged access to Phoenix, which resulted in system instability requiring an OS reload on that server.

We immediately took the effected servers offline in order to preserve data integrity and to initiate an intrusion analysis to determine the point of entry and to identify any trojans that may have been installed. Unfortunately, the depth of the compromise was severe and all attempts to isolate the infected binaries and libraries were unsuccessful.

Identifying the point of entry is also proving to be complicated as there is no obvious source at this time. All of the servers involved were running OS versions patched by the Fedora Legacy project which makes a best effort to provide timely patches to security vulnerabilities, but makes no guarantees as to the timeliness of patch release. At this point it appears the attacker was able to exploit an un-patched service.

Consequently, once these factors were analyzed, we determined that it would be in everyone’s best interest to re-install the servers with new operating systems and migrate the data despite the outage that would be incurred.

This event comes at an unfortunate time for us as well. We are currently in the middle of two very large projects that will culminate with the release of a highly robust and flexible hosting environment and a new line of dedicated server hardware. Both will allow us to extend more functionality down to the end user and are scheduled to be online in early Q4.

That being said, we will be providing compensation in the form of a service credit on your account. Our SLA does not provide for server downtime, however, we feel that this is a way that we can show you we are committed to outstanding service and support. The amount of credit due to you will equal 50% of your normal monthly bill for hosting service.

We understand the inconvenience this has caused everyone. If you have comments or concerns, we encourage you to talk to our support staff so we can address them directly.

Regards,

Heather Bennett
Director of Customer Operations

Posted in General |

This entry was posted on Wednesday, August 18th, 2004 at 3:23 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.