Eric: Day two of the MT Blackout....
Two days without Narbonic, without No Stereotypes, without Digger, without Ascent (okay, Ascent wouldn't have updated), without Fans, without Rip and Teri, without Lil Mell, without American Elf, without Achewood's extended dance mix, without... without... without....
This isn't a knock on Joey Manley. Not at all. I suspect Mr. Manley's brain is close to exploding right about now, and I feel really badly. But it's also interesting from a Webcomics Anthropology standpoint. When there is a failure among the major providers, the ripple effect is tremendous.
We've known this for a while, of course. There was a time when Keenspace regularly dropped off the net, and it's not immune even today. Keenspot, if I remember correctly, had some significant problems at one point. And one of the publicly stated motivating factors of moving TalkAboutComics off of that selfsame server was because TAC is subject to occasional attacks by assholes, and everyone was sick of that bringing Modern Tales, Girlamatic, Graphic Smash, Serializer.Net, and all the affiliates down as well. Pretty much any situation where everyone lives on one server is asking for a day when everyone goes down at once, and then a big chunk of the webcomics community is just plain AWOL.
Some people, I'm certain, are demanding to know why Manley doesn't have each community on a different server. The answer to that's pretty simple, or so it seems to me (I've never spoken to Manley about this, mind): economics. You always have to take more money in than you lay out, or your business fails.
Do I think Manley needs more redundancy? I'm sure he feels he does, right about now. One thing about systems administration and IT, though -- it's always after something catastrophic happens that you see what you need to do to prevent that catastrophe. I suspect things will change after this. I'm certain Manley will do something both for his subscribers and for the artists he has under contract to help make up for what happened (extensions of subscriptions being the obvious for the former), and things will go on. It's one of those sad things that happens in IT, and by now we should all know it.
One point's been made, though. Right now, the only webcomic residing on a Manley server that's up and running... is Gossamer Commons. Okay, it's not a Manley strip (and so Manley's standards of quality don't necessarily apply -- there's a disadvantage to not having editorial oversight), but still. We're still there.
Posted by Eric Burns-White at March 31, 2005 11:13 AM
Comment from: Wednesday posted at March 31, 2005 11:29 AM
It's all because Alisin kissed Rikk.
Comment from: djcoffman posted at March 31, 2005 11:44 AM
It's a good day to read Yirmumah. :) http://yirmumah.net
Comment from: larksilver posted at March 31, 2005 12:03 PM
I honestly hadn't noticed that Modern Tales wasn't up. I've been reading Narbonic.com, and it, and Digger (which I'm not expecting to read again until Friday) are the only things I currently read on MT. Granted, if I decide to subscribe that will change, I'm sure.. but still.
Just saying, Dave and Mell and Helen are still performing their antics.. just not on their "primary" home.
Comment from: J.(Channing)Wells posted at March 31, 2005 12:30 PM
Yah, visit Narbonic.com. Shaenon's being a dear and posting 'em manual for us. Sadly, this is not an option for either "Li'l Mell" or "More Fun", which have no formal home of their own outside of MT. Ah well.
Comment from: alschroeder posted at March 31, 2005 12:48 PM
Magellen has updated, at its keenspace location. Ditto Killroy and Tina, at killroyandtina.com. And it may be a good time to check out the non-MT-family comics to review, anyway.
But I'll say one thing for Manley. This doesn't happen often to the MT servers. More than you can say for a lost of hosters, pay or not.----Al
Comment from: alschroeder posted at March 31, 2005 12:49 PM
I meant LOT of hosters. Sorry about that.---Al
Comment from: Brandon E. posted at March 31, 2005 2:05 PM
Yea, I was dissappointed when I went through my comics list last night after midnight and found a third of them were down because of this.
But it never even crossed my mind to blame people working with Modern Tales and affiliated. Shit happens.
I saw some people mentioning that they had heard it was done in malice, but without any sources to back them up. Was this a hacker's work or is this accidental typical server failure?
Comment from: Shaenon posted at March 31, 2005 2:10 PM
Yes, this is very upsetting. I'll rerun Li'l Mell and More Fun next week, so nobody misses those installments. Narbonic is running as usual on narbonic.com. And, when MT gets back online, I have a LOT of re-uploading to do. I was hella ahead of schedule on Narbonic.
Nearly every webcomic I read these days is on the MT sites. I'm seriously jonesing right now. Achewood alone is not enough!
Comment from: Christopher B. Wright posted at March 31, 2005 2:16 PM
I would assume it's a technical snafu before I'd assume it was done out of malice. Tech snafus (hardware failure, cpu failure, bad ram, etc) can be a LOT more disruptive than someone hacking and deleting... with a hack and delete you can at least restore backup files and move on... with hardware failure there's nothing to restore *to*, and you have to figure out what exactly has gone wrong before you can fix it.
Comment from: Reinder Dijkhuis posted at March 31, 2005 2:53 PM
Brandon: Lots of attempted DOSing was going on at the time. Whether or not this had any causal relation to the hard drive failure, I can't tell (and I don't think Joey knows either), but the constant pressure from malicious visitors wasn't helping the sites at all.
I'm trying to keep my readers informed through this blog entry. I'll post what I know when I learn it. Bottom line: we may have to wait a few more days, and MT family cartoonists may have to do some data entry. One or two will have to do a lot of data entry.
Comment from: Christopher B. Wright posted at March 31, 2005 3:10 PM
Well, never mind then. That just sucks.
Comment from: Tim Tylor posted at March 31, 2005 4:57 PM
The DOS stinks mightily. I can't help wondering who'd try to do that against MT - someone with some beehive in their bonnet about the site, or just a random vandal?
Comment from: Christopher B. Wright posted at March 31, 2005 5:08 PM
I'd hate to think that someone would be deliberately malicious to a friggin' webcomic publisher. Not that a bored script kiddie being randomly destructive is much better...
Comment from: Brandon E. posted at March 31, 2005 5:14 PM
It was done by a club of angry syndicated cartoon charecters led by a fat lasagna loving cat and a group of cute and "innocent" children who leave dashed lines in their wake.
Comment from: Phalanx posted at March 31, 2005 5:16 PM
First Keenspace and now MT. Talk about a bad March.
I'm just glad my comic updates on Sundays. If it doesn't clear up by then I'll have to crank up my Keenspace account, which is something I haven't done for while.
But someone out there must really hate MT comics. *grumbles*
Comment from: Tim Tylor posted at March 31, 2005 6:19 PM
BTW, Ursula's got a temporary link to the latest Digger in her forum.
Comment from: Phil Kahn posted at March 31, 2005 7:59 PM
Heh. At first I read that and thought, "MT? He can't be that sad about MegaTokyo."
Comment from: Shadowydreamer posted at March 31, 2005 8:05 PM
Glych made her #1 fan happy too.. She's got the latest No Stereotypes up in her forum as well. Here
Comment from: kirabug posted at March 31, 2005 9:28 PM
Are Real Life and Mac Hall also on those servers? I haven't gotten through to either of them in days... and I think they're on the same server/service, I just don't know which service it is.
Comment from: Brandon E. posted at March 31, 2005 10:44 PM
I think Real Life has been having its own problems lately with the site update that Greg has been trying to implement.
I didn't think Mac Hall was an issue because I typically have trouble connecting to them anyway (that is probably a problem on my end though).
Comment from: Tragic Lad posted at March 31, 2005 11:06 PM
The MT family accounts for much of my online comics reading habits. I feel like the local comics store has just hung a big 'closed due to renovations' sign in the window and I have to mozey on down to the drug store to see what, if anything, they have to read.
I truly feel for Joey Manley - He's already the hardest working guy in webcomics, and now after dueling the scriptkiddies over TAC, he's now got to deal with hard drive issues and restore all of the MT family sites. oy vey.
I don't know about the rest of you, but I'm all for buying Joey a good, stiff, drink. I get the feeling that by the time it's all back online he'll need it.
Comment from: Eric M. posted at April 1, 2005 12:02 AM
Joey has updates on his progress, artists have updates on how they're dealing with the blackout (delaying updates, mirroring elsewhere, etc.) over at:
Joey's latest: "Very close to finishing the temporary archival-only version of Girlamatic. The DNS has propagated, and weĖre waiting for ev1serversĖ router tables to update. They say thatĖll take another 5 hours. ...
After that, the other sites will be much easier Ō IĖve rewritten every php script FROM SCRATCH since the crash, to get Girlamatic online. The other sites will use the same code that the new Girlamatic site uses."
Comment from: RoboYuji posted at April 1, 2005 12:44 AM
If I can't get to Mac Hall, I can't get to Real Life, so I've always assumed they were on the same server.
But I'm TOTALLY jonesing for Fans right now . . . :(
Comment from: Eric Burns posted at April 1, 2005 12:50 AM
You and me both, Robo. It's like they knew there was a cliffhanger.
Comment from: RoboYuji posted at April 1, 2005 12:59 AM
And it wasn't even Friday's comic yet!
Comment from: Bo Lindbergh posted at April 1, 2005 1:29 AM
You just had to open your mouth, hadn't you?
Right now, the only webcomic residing on a Manley server that's up and running... is Gossamer Commons.
Murphy knows a challenge when he hears one.
Comment from: ZorbaTHut posted at April 1, 2005 5:21 PM
As I see it, the only big mistake Manley made was not having an offsite backup of the PHP scripts, since he's been talking about having to rewrite them. (I've got all my important code stored in no less than five separate places - three personal computers, one private server in Texas, and one colocation backup system for said private server.) And better comic backups would have been a good idea too.
But beyond that? Yeah, this happens sometimes. Pick up the pieces and move on, and maybe once it's affordable, buy a backup server. :P
Comment from: Tangent posted at April 1, 2005 7:18 PM
It makes me wish I knew some actual hackers who could search down and annihilate the computers of the DOSers who've been attacking MT. What type of scum tries to destroy a company like Modern Tales or Keenspot, who are giving quality entertainment on the internet on a daily basis?
I swear, sometimes I think the Internet should only be allowed for those who pass an intelligence and morality test. But I suppose that would eliminate most of the AOLers... ;)
(BTW, I am, of course, implying that everyone here is intelligent and moral enough to be on the Net, as we're not DOSers or Spammers. And AOL does have a number of intelligent reasonable users... it's just it has so many people that the percentage of jerks and idiots using it are significantly higher...)
Comment from: Ray Radlein posted at April 1, 2005 9:33 PM
I've been getting the damnedest DOS-like bombardment here at home the past few days (when my internets have been there at all, that is): Some Comcast IP in Pennsylvania keeps hitting me on a variety of ports that I've never heard of before. It'll hammer one port for a while, go away, and then come back a few minutes later and hammer at another port. SANS doesn't have any of the ports listed as being associated with any known malware; however, most of them are used by different product activation and verification schema (mostly for products I've never heard of). If I had to guess, I'd guess that someone has uncovered a previously-unknown and currently unpatched vulnerability on some DRM library which is used by a lot of different products, and is going around jiggling doorknobs to see how many computers running those products they can find.
Comment from: joeymanley posted at April 3, 2005 10:23 AM
I'll be posting a long postmortem later tonight/early tomorrow morning (putting some finishing touches on the new control panel).
The biggest mistake I made was not upgrading the scripts I wrote 3 years ago (with "PHP for Dummies" at my elbow) long ago. If you guys could see those scripts, you'd be shocked that the site didn't have a catastrophic failure earlier. I should be imprisoned for excessive torture of a poor MySQL server (the homepage of Modern Tales along made 32 separate connections to the database every time it was called). That's why I had to rewrite the scripts before relaunching -- because they left us vulnerable to DoS attacks (even accidental DoS attacks, like, you know, if 200 people had pointed Teleport Pro at the site at the same time).
So it is my own stupid fault, even if malice was involved on the part of the folks who brought us down (not sure if it was or not, yet -- I have spent my time trying to get the sites back up, not investigating -- will do that later). Note that I did/do have more than one server -- I just host a LOT of websites that people don't know I host. Yours wasn't the only comic site I host that didn't go down, Eric -- grin -- though it's a funny thought. Cat and Girl, Small Stories Online, Joda.us, etc., etc., all weathered the storm just as handily. It was just the oldest server in my server group, the only one still at rackspace -- the provider I've been trying for two years to migrate away from. Nothing like a crash to accelerate those plans!
Hope to have everything back up 100% to speed tomorrow. Actually, better than before, thanks to the quick recode (which takes advantage of much of the work I've been doing on WCN over the past year and a half).
Thanks everybody! Your support is appreciated!
Post a comment
Thanks for signing in, . Now you can comment. (sign out)(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)