LJ Owwwies? 0,o;

[Indigo]

*pokes lj* and here I thought I'd post an entry before going to bed... *pout*

*snatches a cookie and joins everyone else under a chair*

[Aevelyn_LL]

Current error message:

LiveJournal is currently down due to a massive power failure at our data center. We'll provide updates at /powerloss/ as they're available.

Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.

[Myrror]

*wince* I hope everything's ok.

This also holds up my drawing--for one exchange, I was waiting for the weekend because I'd have more time, and now I can't get to the description of the chosen one's character. I don't like missing that exchange.

Ah, well, if it doesn't come back up in time, perhaps the exchanges mod will extend the deadline because of people not being able to post their entries and such

[Aevelyn_LL]

It's wonderful timing that this came so soon after the announced sell out. I'm tempted to just start my own blog on my site. But I like the friend's page sooo much!

[AirRaiser]

Nother Update

Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.

[Lumi]

*looks hopeful* so maybe it'll be back this evening? *hopes so*

[Myrror]

Me too--that will still give me Sunday and Monday to do that drawing.

*scans a whorling*

[sera]

Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.

Update #6: We're up again, but only partially. Some database clusters are still reconstructing/syncing. See status.livejournal.com.

status.livejournal.com/ <-- Current updateyness

www.livejournal.com/misc/whereami.bml <-- Tells where you are

^^ My section is still down. . . ;.; I'm on Filemignon! ^^

[Lumi]

WOOOOOOOOOOOOHOOOOO!!!! It's BAAAACCCKKK!!!! XD!!!!!
*cling sung luffs her lj* UPDAteagt time!!!!

[storm unlogged]

*reads the whereami? thing and frowns* They moved my lj to the MadCow server??
Was on Chef. Huh. Weirdness abounds. Ah well at least I can fiddle around since Chef is on readonly for a while. Must be some of us got moved when they reloaded the backup. *shrugs*

[Aevelyn_LL]

They moved mine as well. I used to be on Porkchop, now I'm on Roastbeef.