here's what a black iPhone does in traffic
the modded Jesus phone:
click to go to the project
this year’s traffic:
...according to Google Analytics
the blue line flat on the bottom represents around 3,500 unique visitors a day.
click to enlarge
the vertical line represents visitors *a day*, so yes, last wednesday I’ve seen upwards of 60,000 unique visitors…
according to AWstats it’s 70,650 to be exact, on the 31st of October…
legend to go with the spikes: ;-)
1 DIY obsolete iBook logic board repair
2 iNoPhone
3 how to turn your iPod mini into a flash based iPod
4 put flash memory into (almost) ANY iPod!
5 Slurpr – the mother of all wardrive boxes
7 How to build a fully encrypted NAS on OpenBSD
This was one wave my webserver wasn’t able to digest fully. It was doing alright until Digg came along and from then on, I had to keep my fingers crossed… And ‘normally’ whenever a project of mine hits the frontpage, it automagically disappears from there once my server can’t handle the traffic anymore. I mean, people can’t see it anymore and people don’t digg it anymore… Makes sense, right? But this time, even though the project itself already was long time gone from the frontpage, it also was still there because it reached a Top 10 in all topics, a list which is on the frontpage and so the traffic kept on coming …and coming …and coming, hour after hour…
This does have me thinking though, I mean, the counter now counted only what it was able to handle, how high would that spike be if I got a server farm which would be able to handle such a wave? Next to that it somehow doesn’t feel right that Digg is able to also make sure other visitors from other sites aren’t being served well. I mean, I couldn’t care that much if a visitor from Digg isn’t served the page he/she’s looking for, they post mirrors in the thread and all that. But for every other visitor from another site it’s just all a bad user experience.
My still single webserver itself is doing great though, it’s now got an uptime of 519 days(!). There was only no way in the world it could handle this traffic and I doubt very much if I would’ve been able to with my second server next to it. I haven’t colocated that one yet, but maybe it’s also better cause now I know what kind of waves my next setup should be able to catch ;-)
My server’s been able to handle a slashdot twice this year, but this wave from Digg was about four times that of a slashdot…
Kind of odd though, needing to optimize a machine for around fourty times the traffic it won’t handle 99% of the time. And extremely odd if you think there’s only one site (Digg) responsible for those extreme spikes.
wanna see another spike? here’s one from the ever reliable(...) Alexa.
I guess I’m gonna redirect the traffic from Digg next time something like this happens, to Flickr for instance. That should be possible with Apache’s .htaccess. That’s exactly what happened in the thread this time, someone was pointing to one pic on Flickr and that seemed to please the crowd. It makes sense on a site where ‘articles’ reduced to single pictures (with [PIC] in the headline) are able to reach the frontpage time after time.
*^_^*permalink - 
add to del.icio.us
previous: project: paint it black: the iPhone
What’s the actual traffic like, in megabits per second?
Well, I still haven’t got anything installed which reads out exactly that. But… in a period of 8 and a half hours on the Digg-frontpage (it was longer there, but these are numbers I’ve got) I burned 30GB of traffic. In a period of 24 hours I burned 45GB… Now this was in a time my Apache-install was crapping along. I counted 600 visitors a minute for a long time, but they hardly got anything ;-) Anyway, calculated on this I’d say it was around 8-10 mbits.
Next time something like this happens, I’ll make sure to have static content and I’ll probably burn more… with 600 visitors a minute and roughly 1 megabyte each I should be able to top 80 megabits, not 8…
By my calculation you’re looking at 4mbits averaged over the 24 hours. So in practice that’s going to be about double(-ish) peaked. Pretty good. I’ll assume most of that is imagery, which is static content that is pretty easy to serve up (or pull that in from flickr ;). My boxen will handle several hundred mbits/sec doing that. (Though of course, in practice they mostly do php work.)
I assume your website also runs on php? If so, do you have an opcode cache running like eAccelerator? That /really/ helps keep the load down. The difference on one of my older webserver is that without it, apache uses 50% cpu time, and with it only 10%. Second, you’re running OpenBSD right? You should consider switching to Linux as it’s much faster at handling this kind of task. (This is some older info, but it shows the general point: http://bulk.fefe.de/scalability/)
Third: get something to load balance that, preferably, won’t fall over once you get some traffic incoming. I.e., something dedicated to the cause. This is something OpenBSD should be able to handle nicely. (As can linux, though I use a hardware box myself.)
That’s as far as web servers go, but there’s an equally important thing to fix first: your database needs its own box. I’m not sure how database-heavy your website is, but when I’m getting hit with traffic my database server is the first to notice. And back in the day when I didn’t have a box specifically for that, it held back apache’s performance.
I know you’re going to come up with something cool and digg-prone again so it’s good having a web setup that can take a hit ;) Oh and slashdot really isn’t what it used to be ;)
Doesn’t your colo hoster make graphs of traffic at the switch port?
(Yay google btw: q=45+gigabytes+over+24+hours+in+megabits+per+second)
Ah, I just read that the webserver already does run linux, so ignore that bit. (But gentoo? boo! ;)
Well, 1st, Gentoo is still an old relic somehow, although I do get a nice uptime on the box!
Second, especially since this last hit, I’m a little confused on how I can maximize the gear I’ve got, since it dawned on me that I need to be able to swallow bigger peaks than I would’ve imagined somehow ;-)
I haven’t decided yet on how to utilize the machinery at its best.
I’ve got:
my current webserver:
2× 2.4GHz Xeon 400FSB, 2GB mem, 2×80GB IDE softraid 1
(this one is the one now serving this page)
my ‘new’ webserver (not colocated yet, but first I need to make up my mind…)
2×2.4GHz Xeon 533FSB, 4GB mem, 4×74GB SCSI U320, hardware RAID10. (in short, double the memory and probably triple the Disk I/O)
Now the idea is to have them both colocated, so the new server comes alongside the old one and that should make a nice setup possible.
Well, maybe we should discuss this on Nadya’s party ;-)
Hey Markie, as allways: great stuff! ;)
RewriteRule .* http://flickr.com/ [R=302,L]
Try this rule (not tested). Make sure you use a 302 (move temporarely) so google won’t kick you out of it’s index.
so long, prego
just stop using apache, and use lighttpd with php-fcgi, you will be amased what it can serve