|
|
| Spider(?), faked user-agent |
1217996 |
35.21% |
|
| Unknown! |
730509 |
21.12% |
|
| BaiDu Spider (baidu.com) |
325768 |
9.42% |
|
| Yahoo! Slurp (yahoo.com) |
271912 |
7.86% |
|
| GoogleBot (google.com) |
215543 |
6.23% |
|
| BingBot (bing.com) |
200830 |
5.81% |
|
| MSNBot (msn.com) |
147465 |
4.26% |
|
| Unknown |
137956 |
3.99% |
|
| Sogou web spider (sogou.com) |
73052 |
2.11% |
|
| Yandex.ru robot |
22921 |
0.66% |
|
| MSNBot Media (msn.com) |
14449 |
0.42% |
|
| DotBot (dotnetdotcom.org) |
10786 |
0.31% |
|
| Spider-on-the-fly (discoverlife.org) |
10117 |
0.29% |
|
| ExaBot (exava.com) |
6269 |
0.18% |
|
| Synapse |
6117 |
0.18% |
|
| Facebookexternalhit (facebook.com) |
6077 |
0.18% |
|
| MJ12bot (majestic12.co.uk) |
5649 |
0.16% |
|
| Twiceler (cuill.com) |
4200 |
0.12% |
|
| Ask Jeeves/Teoma (ask.com) |
3559 |
0.1% |
|
| Stack Rambler (rambler.ru) |
3108 |
0.09% |
|
| Speedy Spider (entireweb.com) |
2905 |
0.08% |
|
| Yeti (naver.com) |
2897 |
0.08% |
|
| Yahoo! Slurp China (yahoo.com.cn) |
2426 |
0.07% |
|
| 80legs.com crawler |
2074 |
0.06% |
|
| IA Archiver (alexa.com) |
1955 |
0.06% |
|
| LWP Simple |
1870 |
0.05% |
|
| Java VM |
1781 |
0.05% |
|
| Charlotte (searchme.com) |
1572 |
0.05% |
|
| Fast Enterprise Crawler (fast.no) |
1209 |
0.03% |
|
| DiscoBot (discoveryengine.com) |
1017 |
0.03% |
|
| Jakarta Commons-HttpClient |
1000 |
0.03% |
|
| NV32ts |
894 |
0.03% |
|
| DoCoMo (docomo.ne.jp) |
882 |
0.03% |
|
| Libwww-perl |
788 |
0.02% |
|
| Sosospider (soso.com) |
778 |
0.02% |
|
| Voyager (kosmix.com) |
715 |
0.02% |
|
| Google Translate (google.com) |
675 |
0.02% |
|
| ZyBorg (wisenut.com) |
582 |
0.02% |
|
| Aotech.net E-mail collector |
515 |
0.01% |
|
| Findlinks (wortschatz.uni-leipzig.de) |
475 |
0.01% |
|
| ODP link checker (pmoz.info) |
465 |
0.01% |
|
| AhrefsBot (ahrefs.com) |
448 |
0.01% |
|
| Python-urllib fetcher (Naver Japan / Korea) |
441 |
0.01% |
|
| Ezooms Bot |
398 |
0.01% |
|
| VoilaBot (voila.fr) |
376 |
0.01% |
|
| Integrated Computer Solutions (ics.com) |
368 |
0.01% |
|
| BecomeBot (become.com) |
352 |
0.01% |
|
| BOTW Spider (botw.org) |
352 |
0.01% |
|
| MojeekBot (mojeek.com) |
344 |
0.01% |
|
| Envolk spider (envolk.com) |
339 |
0.01% |
|
| Indy Library (E-mail collector) |
329 |
0.01% |
|
| Blekkobot (blekko.com) |
325 |
0.01% |
|
| DoCoMo (goo.ne.jp) |
314 |
0.01% |
|
| Search17Bot (search17.com) |
278 |
0.01% |
|
| Goldfire Server |
264 |
0.01% |
|
| ScoutJet (scoutjet.com) |
253 |
0.01% |
|
| JyxoBot (jyxo.cz) |
246 |
0.01% |
|
| Kalooga Bot (kalooga.com) |
245 |
0.01% |
|
| Bender (benderthewebrobot.tumblr.com) |
244 |
0.01% |
|
| GoogleBot-Mobile (google.com) |
243 |
0.01% |
|
| LinkBot Pro (link checking) |
234 |
0.01% |
|
| Fast Web Crawler (fast.no) |
232 |
0.01% |
|
| WGet (downloading tool) |
231 |
0.01% |
|
| Heritrix (accelobot.com) |
219 |
0.01% |
|
| WordPress (wordpress.com) |
216 |
0.01% |
|
| LibraBot (microsoft.com) |
210 |
0.01% |
|
| Yahoo Seeker (yahoo.com) |
209 |
0.01% |
|
| YacyBot (yacy.net) |
204 |
0.01% |
|
| GigaBot (gigablast.com) |
192 |
0.01% |
|
| www.socialayer.com Agent |
183 |
0.01% |
|
| StumbleUpon (stumbleupon.com) |
180 |
0.01% |
|
| Xenu Link Sleuth (link cheching) |
177 |
0.01% |
|
| TencentTraveler |
175 |
0.01% |
|
| MLBot (metadatalabs.com) |
170 |
0% |
|
| GrubNG (grub.org) |
151 |
0% |
|
| RetrevoPageAnalyzer (retrevo.com) |
150 |
0% |
|
| Baidu Transcoder (baidu.com) |
138 |
0% |
|
| eduCrawler |
131 |
0% |
|
| YoudaoBot (youdao.com) |
127 |
0% |
|
| SnapBot (snap.com) |
119 |
0% |
|
| Netcraft Web Server Survey |
117 |
0% |
|
| Archive.org Bot |
114 |
0% |
|
| http_load 29jun2005 |
113 |
0% |
|
| SPBot (seoprofiler.com) |
112 |
0% |
|
| Lynx (file transferring tool) |
111 |
0% |
|
| Atrax Solutions atraxbot (atraxsolutions.com) |
106 |
0% |
|
| NaverBot (naver.com) |
104 |
0% |
|
| Win Http Services (microsoft.com) |
103 |
0% |
|
| Raymond Balmès/Biible (bible.info) |
99 |
0% |
|
| Curl (file transferring tool) |
97 |
0% |
|
| Grub-client (grub.org) |
91 |
0% |
|
| TurnitinBot (turnitin.com) |
90 |
0% |
|
| NiceBot |
88 |
0% |
|
| PSBot (picsearch.com) |
88 |
0% |
|
| IRLbot (irl.cs.tamu.edu) |
87 |
0% |
|
| Ichiro (help.goo.ne.jp) |
85 |
0% |
|
| Convera Crawler (authoritativeweb.com) |
83 |
0% |
|
| DataFountains/DMOZ (ucr.edu) |
82 |
0% |
|
| JikeSpider (shoulu.jike.com) |
82 |
0% |
|
| Pittsburgh Nutch Crawler |
81 |
0% |
|
| Total |
3459021 |
|