Yesterday, this blog had 3212 hits (HTTP requests), out of those, 666 were from Bots. That includes Search Engines, Feed Readers (excluding IE/FF), generic bots and calls from scripts libraries (Perl, Jakarta, Java, etc.).
That means 20% of the total traffic to this website was not from people, but from machines. But wait, it gets worse.
I estimate that there is another 10% of requests coming from Bots pretending to be browsers (using fake user-agent). How do I know that? Well, how is it possible that the same user clicks on 5 pages in 5 seconds? Maybe it is possible. But how do you explain that the request for those pages never come paired with a request for a single GIF, CSS or JS file?
So, about 30% of our resources of network are being used by Bots, not humans. The same thing happened on the Web 1.0 era. On the beginning of the year 2000 it was insane the number of bots requests coming to a website.
Subscribe to this Site
January, 2008