Friends and Customers,
We are experiencing a stereotypical Friday the 13th which feels unending. As most of you know by now, our web hosting server went offline about 11 am yesterday. Our host server tech teams have encountered a “Perfect Storm” of rare (and I do mean rare, not simply uncommon) events, all of which add up to the devastating truth that we are still offline this evening.
If you are interested in the details, I’ve been posting updates here:
http://www.facebook.com/pages/Long-Beach-WA/beachdogcom/50438687881
Most of you want to know:
When will our sites be back online? The best estimate I have is later this evening, Saturday, Feb 14.
Is my email lost? No. Depending on the sender’s server configurations, your mail may be sending bounce messages back to the sender and it may not. In either event, the mail is sitting on a different server than your website. As soon as the web server goes back online, mail will start being delivered. It will take awhile for all to be delivered but none should be lost.
Have we lost any data? This is perhaps the worst news in the storm. Yes. The original server data was not recoverable so the host server techs went to restore from a backup. At that point it was discovered that the backup system had failed on/since December 2nd and gone unnoticed by them. As a result, the most current data available is from December 2nd.
What does that mean for me? As soon as the server is online and accessible by us, we’ll be conducting triage to determine where we are. We’ll focus on high-traffic sites first, followed by e-commerce and other business sites and finally the personal and family sites. We’ll be working as quickly as possible and will contact each of you with the status of your own site as soon as we have that information.
We’re estimating about 250 hours of work from our Long Beach office to get everyone back to February 13th status. With 5 of us able to make an impact on the project, we expect to have everyone back up to speed in just a number of days. We appreciate your patience.
Bad news for database-driven site owners (that means all of you using WordPress, bulletin boards and other dynamically-updated site features). There is no way to recover the data that was created online. Backups for that data were to be happening online and the company let us down big time. In the name of professionalism, that’s all I’m willing to say right now but rest assured that this is not the end of the story.
You have my grave and humble apology. While none of what has happened this weekend could have been foreseen and was certainly beyond my control, I also know what it feels like to have my business taken offline and my email non-existent. I share the pit in your stomach as you learn there is data loss and potentially more online commerce lost as we work to put Humpty Dumpty back together.
You have my absolute promise to stay with you every step of the way until each of you is solidly on the other side of this unfortunate series of events. And now, you have my personal email address. Please feel free to hit ‘reply’ to this message with any questions you may have or even to leave a phone number if you want a call back. Keith is returning most calls so that I can spend more time working on the recovery. If you really need or want to talk with me, just say so in your email.
Of course, we will be compensating you in some way for your inconvenience and I’ll figure out what that will be as soon as we get the fires put out.
Most sincerely,
Keleigh
Keleigh Schwartz, Alpha Dog
