Dryline Hosting

[Ryan McGinnis]

Yeah, Mike got Farked, torrented and a few other things all in one day. We're trying to get him a work around right now. He'll be back up.

Well, the FARKing was moderately hard, according to FARK's ticker maybe 20,000 pageviews in the course of a day, with maybe a meg or two of photos on each page. That's like 40 gigs, give or take 10 gigs, tops, right? Torrenting shouldn't matter at all... you do understand how peer to peer works, yes? It has nothing to do with http, operates on a different port, is not a server client relationship, doesn't involve your server at all, etc.

You may not be able to comment on customer issues, but out of curiousity... is pulling the plug SOP over at drylinehosting.com any time a site gets unusually popular? Not trying to ruffle your feathers but, candidly, that seems like a pretty shortsighted thing to do, PR wise. Now hundreds of thousands of people (including the chasers you cater to) get to see a giant advert about how great your hosting is (headlined by 30 point text saying the site is down) instead of the content they were looking for. Talk about irony.

This kind of thing is why I decided to use blogspot for image posting a long time ago. Google never yanks my cord outta the wall just because I'm popular. They just raise my pagerank. (And it's free.) I know that you do not run a billion dollar company or anything and can't give things away for free, but you do compete with billion dollar companies, and when they're perceived as giving better, more reliable service than you, I'm not sure what your market advantage is.

**edit** 12:43AM 01/03:

Just a heads up that, as David pointed out to me in PM, this post came off more than a little bit offensive to him. It was worded poorly, and I did not mean to infer that David's service sucked; it was, actually, a misfired attempt to ask him what the heck was up with pulling a webpage that was getting a lot of traffic. He does take pride in his service, and I suspect that had I worded it better so that he could see what I was getting at as opposed to my post's snarky undertone, he would not have been as upset. I apologize for putting the question to him in this manner -- I could have phrased it much better.

 

[fplowman]

Whaaaa???

Im a happy Dryline customer.. please splain Lucy...

Fred

 

[David Drummond]

Well, the FARKing was moderately hard, according to FARK's ticker maybe 20,000 pageviews in the course of a day, with maybe a meg or two of photos on each page. That's like 40 gigs, give or take 10 gigs, tops, right? Torrenting shouldn't matter at all... you do understand how peer to peer works, yes? It has nothing to do with http, operates on a different port, is not a server client relationship, doesn't involve your server at all, etc.

The farking or other such massive influx of traffic is not about pageviews or bandwidth usage (although it can certainly eat up a person's allocated bandwidth real quick). It's the http requests. There are only so many virtual doors to go through at any given time for EVERYONE on the server. At one point there were over 20k http requests per hour, just from Fark.com and that doesn't include the other traffic to this site (seemed the pictures managed to get direct linked from a number of other places besides fark.com) and the other traffic to the other sites on the server. Just to understand the http requests. Calling up one page can mount MANY http requests depending on what's on it. That's just for ONE viewer.

As I explained to you via PM, it's much like Walmart on black Friday when the doors open. Only so many people can squeeze through until the entrance gets clogged up. And yes, I have a full understanding of how peer to peer works, I mentioned that because I saw someone post it, not anything to do with anything else.

You may not be able to comment on customer issues, but out of curiousity... is pulling the plug SOP over at drylinehosting.com any time a site gets unusually popular? Not trying to ruffle your feathers but, candidly, that seems like a pretty shortsighted thing to do, PR wise. Now hundreds of thousands of people (including the chasers you cater to) get to see a giant advert about how great your hosting is (headlined by 30 point text saying the site is down) instead of the content they were looking for. Talk about irony.

No we do not pull a website for being popular. In fact this is the first time ever we have had to pull a website for monopolizing the server resources (in this case CPU time and memory allocation). We worked to try and accomodate it for several hours until it became apparent that it just wasn't going to happen. We then had a decision to make. Suspend the site or let it take down all the rest of the websites with it. I did what any other shared host would do and spared all the other customers the pain. As for shortsighted? I dunno....shut one site down or lose them all? What would you choose? The giant "advert" you are talking about is a standard suspension page. Any time a site gets suspended (which is almost always because someone didn't pay their bill on time) it automatically goes to that page. I can't change it from one site to the next. It's a limitation of cpanel. Ironic, maybe I don't know. I was just trying to keep the rest of the server up at that point.

This kind of thing is why I decided to use blogspot for image posting a long time ago. Google never yanks my cord outta the wall just because I'm popular. They just raise my pagerank. (And it's free.)

Well since you don't actually get any disk space to store stuff there you can't test that, but if you did I would tell you to start backing your huge hard drive up on there and see how long before they pull your site....errr...BLOG..yeah.

I know that you do not run a billion dollar company or anything and can't give things away for free, but you do compete with billion dollar companies, and when they're perceived as giving better, more reliable service than you, I'm not sure what your market advantage is.

Your right, I don't run a million dollar company. If I did I probably wouldn't be here right now personally answering this question (although I do tend to give a lot of things away free). I'd probably have some peon doing it if it got answered at all. We try to PERSONALLY take care of everyone's needs. Something a billion dollar company could care less about. For some things, no doubt they do have a better service. We aren't all things to everyone. Some sites outgrow standard shared hosting and they need to move on to more dedicated solutions. You might recall Tim had to do the same thing with ST here recently.

Market advantage? I'll let our other customers post their own experiences and that can stand for itself.

And yeah I did pull this thread initially. I acted on instinct rather than better judgement and after discussing it with another mod put it back.

 

[Dan Robinson]

High traffic sites are not the norm, and no 'normal' web server is built to handle one site suddenly getting insane amounts of traffic. It's like the bathrooms at the office, the gas pumps at a gas station, and the lanes on a highway. They are designed to handle normal traffic flow with a limited ability for higher capacity. When an unusual circumstance occurs where higher than normal traffic is presented, there is going to be a problem: traffic jam. It is no one's fault, just the nature of the situation.

A site that routinely gets high traffic (like every day or once a week) is usually best put on its own dedicated server with non-shared resources. For most web sites, a dedicated server is overkill (like building a gas station with 100 pumps to handle the one day a month with high traffic). It's a trade-off. You can build for the rare high traffic days at high expense, or you can build for the normal days at more affordable costs (with the tradeoff of capacity problems during the rare high traffic days). It's just economics.

A site that gets high amounts of traffic is going to have problems on shared servers - just a fact of life.

A tactic that works well for high traffic events is to thumbnail and link all photos on a page, or break the photos up into a series of pages. Having all of the photos full-sized on one page forces the server to attempt to load them all at once every time someone views the page.

 

[Tim Vasquez]

As a site administrator I can tell you that the Fark effect a.k.a. Slashdot effect can be a serious mess, and the problem can really scale out of control depending on what kind of content is on the page. The pipeline feeding any given server is not infinitely large, and a smart administrator will be thinking about all the other customers relying on the server. It was great when our Mr. T Forecasting School comic got Farked last year, but I was also monitoring server load & bandwidth consumption and was ready to zap the directory if things got bad.

I applaud Ryan's edit and hopefully some info has been learned here. The bottom line is if you expect a Slashdot effect, the only sure way to stay afloat is to get dedicated hosting and keep on top of things. Unfortunately that can cost some $$.

Tim

 

[David Drummond]

I did want to add that the standard suspended page that displays might have given the wrong impression that Mike hadn't paid his bill or something.

That was certainly not the case at all. It was suspended solely out of trying to keep all the other customers online. The limitations of the cpanel software only allow me one suspended page for everything. He has expressed to me that it might have appeared that way to some people and as such we will be making some changes to that page.

As I told Mike, I will be analyzing the traffic from today over the coming week, see in fact WHERE it was all coming from (I am sure it was not all from fark.com) and see if there is any place where we could make improvements or if in fact it WAS just too much for the server to handle. If there is room for improvement there, it will be made.

I congratulate Mike on hitting a level with his site where he could really make use of a dedicate server. Few personal sites ever hit that level. It's actually quite an accomplishment.

 

[Mike Hollingshead]

Well I'd say if I hit this every day I'd probably really need a dedicated server, but I don't sustain that stuff..far from it. And like Dan mentions, it's probably not worth it. It'd be nice if one could spread that out over the month and not get it all that day. Life would be simpler.

It would not surprise me if Fark did this on its own. It's been extremely rare to be on a host and have it survive during the fark(one must realize how huge they've become). Guessing most all news outlets check out the links frequently. I've only been on one server that utterly impressed me for being able to handle it. Prices went up and I was moved from that one. Like was mentioned from the start of this, it's not likely to affect most, and if it does, it certainly won't be most of the time. Since this is my job now, I have to be bugged when I can't stay running during a fark or high hitting deal. If it wasn't my "job" and it was like before, then I'd just say, ok cool, let me know when it is over and back up(oh how I sooooo miss those days sometimes).

I've been sustaining 2k to 3k visitors a day for a while now(with higher peaks in there), and have had no server issues while at dryline. I have a lot of images online now, so that's a healthy number of folks looking at a large selection of images. I may have had problems come up, but it was unavoidable things that I didn't understand, such as local ISP issues. That or problems my ftp created...which I can now say for 100% certainty is my program and not a server glitch(still does it at the new host). Questions were always promptly answered. I would not be nearly as nice to myself as David has been to me.

The best part here is that the current servers got a real good test and it can be looked at. I've seen David talk to a lot of different people about different things relating to hosting, and can't recall seeing him tripped up on anything.

One interesting thing to me is how long it was on and working. It was placed on fark late the previous night, and what time was it shut off? 11:30 a.m.? I'd almost say it was half handling it. Then again I guess the longer it is up the more locations will start to link to it(would love to see what happened with hits/visits).

I actually expected it to go down once I learned it was on fark. It's always been the norm everywhere I've been. In the end, if there is something that can be changed, I'm sure David will do it.

 

[Edward Ballou]

As pure interest, how are things handled financially during a peak popularity cycle? Do you just burn through your allotted bandwidth and then the site shuts off, or do you get an extra bill? Or perhaps is there some contact made between David & Mike to discuss matters? Not trying to intrude, but am curious on how things are handled.

Way to be a resource hog Mike Congrats - they really are some great pictures!

 

[Eric Whitehill]

There are ways to avoid being farked/slashdotted - several customers I have worked with in the past were linked to by slashdot, and you _can_ stay up - it's just a matter of how you configure some things and overall load.

One of my clients had a 2.2 Ghz machine with 1GB of ram and T-1 for upstream. The client went from 6,000 hits a day to over 36,000. The way the site was configured, the T-1 went full before the site actually went down. It's an interesting config, and doesn't work for everyone. If someone wants some pointers, let me know.......

(and no, Akamai was not the answer.)

 

[Mike Hollingshead]

As pure interest, how are things handled financially during a peak popularity cycle? Do you just burn through your allotted bandwidth and then the site shuts off, or do you get an extra bill? Or perhaps is there some contact made between David & Mike to discuss matters? Not trying to intrude, but am curious on how things are handled.

Way to be a resource hog Mike Congrats - they really are some great pictures!

The sites go down before bandwidth limits are reached. The biggest spurt I ever saw was after the November 2004 Auroras after a fark and follow up massive e-mail chain(one that actually was linking to my site.....not bill bob took these hurricane pictures). That was 820,000 visitors in 20 days. It came out to around 30,000,000 file hits and 400 gigs transfer. So, if godaddy says 2,000 gigs transfer, they'd pretty much have to also be saying they could handle very hard hitting, since it would be a requirement(cause it'd be 5 times harder than THAT......which was the worst I'd seen....and someone else will have to tell you what exactly kept that server up and running....I have no clue).

The funny thing is I think that was the only time I went past what I was given for bandwidth. The guy kept it running and said it would cost me, but then forgot to bill the reseller I was going through.

 

[Dan Robinson]

Hard bandwidth limits are usually better than the alternative. That is when the host quietly lets a site go over the limits and then charges overages (which are usually charges passed on from the host's connectivity provider). Overages are usually astronomical, and for a typical Fark/Slashdot could stick a site owner with thousands in overage fees. It does happen - JibJab is a good example. I think JibJab's overage fees were in the five figure range.

 

[David Drummond]

As pure interest, how are things handled financially during a peak popularity cycle? Do you just burn through your allotted bandwidth and then the site shuts off, or do you get an extra bill? Or perhaps is there some contact made between David & Mike to discuss matters? Not trying to intrude, but am curious on how things are handled.

Ed, the service is setup it automatically suspends when the bandwidth limit is reached. This is primarily to prevent any new signups from doing something naughty, like uploading warez etc. We recently upgraded the bandwidth ceilings on all packages and so far NO ONE has hit them.

As far as how we handle it. We have yet to charge for overages. It's been my unwritten policy that if someone just happens to go over a little on their bandwidth, we don't charge for it. If it's happening regularly, we usually just ask them to upgrade to the next plan. Again though, since our last upgrade on bandwidth on all the packages, I don't recall anyone that has hit their ceiling on bandwidth.

EDIT: I wanted top add that if someone who is a customer is expecting a peak in bandwidth usage, like their site getting featured somewhere, if they will let us know ahead of time if they can I can temporarily raise the bandwidth cap even higher so it doesn't automatically shut off during the peak when they need it most.

 

[David Drummond]

I got some preliminary numbers on the farking, and with Mike's permission I wanted to share so you guys can see some of the numbers we were talking about.

These numbers were JUST for extremeinstability.com and doesn't account for other traffic on the server.

Yesterday 780,591 http requests with 35796 full pages hit.

The majority of it came between 8 and 11 am. With 9 am hour being a real kicker with 105497 http requests in that hour alone. This is when the server completely crashed and I struggled with it to recover after that for some time.


Additionally, there were another 12942 direct requests for the images alone on top of the regular http requests. (Images directly linked elsewhere)


Then there was the residual traffic Umscheid got from the image of his someone direct linked there as well.

 

[Karen Politte]

I think that David Drummond has excelled at looking after every one of his sites during this awkward and inconsiderate "attack" or whatever you call it. He hosts my domain and has always been perpetually attentive and quick to respond to any issues I had that were within his control.

I for one am certainly not going to blast him for pulling the site in question in order to save the rest of the Dryline community. What would you have had him do - leave things as they were and have everybody's e-mail and websites down for days or weeks on end? Personally, I send and expect a lot of emails domestically - and with a sick family member - internationally......and I appreciate the way in which David considered everybody in this dilemma.

At the end of the day - this Fark thing is only a fad and it will pass. Give it 5 years and the site might not even be there anymore. But I know mine will be!

KL

 

[Dan Robinson]

Furthermore, the practical benefits of this type of publicity (Fark, etc) are (at least in my experience) not worth the costs. High web site traffic results in a lot of 'oohs and ahhs' and 'that's really cool' from people, but at the end of the day, what does that bring? Not much or anything at all, at least when it happens to me. Certainly doesn't help pay my mortgage, LOL. It's nice to get the exposure, yes, but IMO not worth any bandwidth overages or server headaches! Publicity like this can be more of a curse than a blessing sometimes.