AO3 temporarily down - and up again :)

[samvara]

ETA: Back up, down for approximately 1 hour and 15 mins and yes, suspicion was correct.

Botheration. Systems are looking into it and my suspicion is we'll find a runaway process hung and choked it to death.

Further updates when I have them :)

Comments

Drive-by encouragement!

[dizmo]

Look at it this way. It just wouldn't be a beta without some... betabumps.

Track down and smite that runaway process! Because wow, AO3's like fifteen kinds of awesome. :)

Re: Drive-by encouragement!

[samvara]

*grins* Thank you! I confess I'd much rather it happens now than over Christmas :)

Re: Drive-by encouragement!

[dizmo]

Yeah, imagine it going down just as Yuletide goes live. Aieeee.

Re: Drive-by encouragement!

[samvara]

*shudder*

We've got a HUGE amount of performance data from this weekend and a fabulous list of behind the scenes things we can optimise :)

Not to mention a lot of great feedback on small (and large!) things we could fix to make it better!

Re: Drive-by encouragement!

[dizmo]

Yaaaaaaaay crunchy numbers!!! \0/

Re: Drive-by encouragement!

[samvara]

aaand back up!

sidra is teaching me how to access and kill said process until we have automated the fix.

Re: Drive-by encouragement!

[dizmo]

*dives back in* Wheee!

*pets the server*

[elke_tanzer]

AO3 rocks my socks, and I've got this irrational worry: I'm hoping I didn't do anything inadvertent while trying to upload and edit that might've broken our shiny new toy, you know?

:-)

Re: *pets the server*

[samvara]

*hugs* It was NOT YOU.

We have this thing called 'passenger' which is a Ruby process. It is possible for it to get jammed and eat up the CPU at 100% which then stops everything from responding. We do not love it but we cannot upgrade it right now because we need it for something else. I have now been trained in how to un-jam it - we are investigating automating this un-jamming process.

That was my super technical explanation!

Re: *pets the server*

[elke_tanzer]

Awww, thank you! It's so good to know...

... that it wasn't me figuring out that parenthesis are not good characters to try to put in a tag wuzzle -- I got a webpage announcing some sort of freeform incompatibility error when I tried that, so I went back and took the parenthesis out and resubmitted all of my story's metadata and all seemed well then, but slow.

... and that it wasn't me clicking on the "none" pairing wuzzle on my Works that I'd put on a few of my stories before realizing it wasn't necessary -- the page took an infinite time to load so I ended up shutting my browser to recover.

:-)

(I'm hypothesizing that your manual unjamming process involves something like the Solaris commands top and/or ps -ef and then kill -9 unless you've got some sort of graphic user interface for server administration -- I spent too much time administering such systems in another life!)

Thank you for all of your work!!

Re: *pets the server*

[samvara]

*grins*

Give it a good workout and tell how it went - we are aiming for 'robust' software here :p

and yes, top and kill :) I now have PuTTY running on my innocent little machine. New skills for me!

Re: *pets the server*

[elke_tanzer]

*gigglefit*

I just caused the software to kick out some kind of other error message to the programmers related to a doppelganger fic's Characters and/or Pairings... and sent a message off to Support about it.

HOMG I can't believe I've just asked the questions I've just asked.

*sporfle*

*blush*

*continued gigglefit*

I adore the OTW and AO3!!!

Re: *pets the server*

[samvara]

*snork* I just saw your support request - congratulations on raising the doppelganger problem :)

Re: *pets the server*

[elke_tanzer]

*delighted laughter*

You would not believe how many doppelganger fics are in my WiP bin... they're all sort of hopping up and down a little, hoping that the AOOO will be an impetus for me to finish writing them!

Re: *pets the server*

[samvara]

I love me a good doppelganger fic!