Depending on who you ask, some people don't like them and do "ban" or
use the
robots.txt to give instructions about their site to web robots.
There are "good" bots and "bad" bots, the "good" ones are usually obediant and respect any "instructions", the "bad" ones don't.
kedaha wrote:I spotted another one today: Majestic-12 [Bot]; I wonder how many there are.
This is a list, that is used in the phpBB software,
Code: Select all
Google [Bot]
Yahoo [Bot]
Bing [Bot]
AdsBot [Google]
Alexa [Bot]
Alta Vista [Bot]
Ask Jeeves [Bot]
Baidu [Spider]
Exabot [Bot]
FAST Enterprise
FAST WebCrawler
Francis [Bot]
Gigabot [Bot]
Google Adsense [Bot]
Google Desktop
Google Feedfetcher
Heise IT-Markt [Crawler]
Heritrix [Crawler]
IBM Research [Bot]
ICCrawler - ICjobs
MSN NewsBlogs
MSN [Bot]
MSNbot Media
Majestic-12 [Bot]
Metager [Bot]
NG-Search [Bot]
Nutch [Bot]
Nutch/CVS [Bot]
OmniExplorer [Bot]
Online link [Validator]
SEO Crawler
SEOSearch [Crawler]
Seekport [Bot]
Sensis [Crawler]
Seoma [Crawler]
Snappy [Bot]
Steeler [Crawler]
Synoo [Bot]
Telekom [Bot]
TurnitinBot [Bot]
Voyager [Bot]
W3 [Sitesearch]
W3C [Linkcheck]
W3C [Validator]
WiseNut [Bot]
Yacy [Bot]
Yahoo MMCrawler [Bot]
Yahoo Slurp [Bot]
YahooSeeker [Bot]
ichiro [Crawler]
psbot [Picsearch]
They can be edited, deleted , etc :
For example:
Code: Select all
Bot name Last visit Options Mark
Google [Bot] Mon Oct 31, 2016 4:58 pm Deactivate Edit Delete
I got this from a different forum, one that I administer, On this forum I am not
a admin, and don't know , nor have access to how it is set up.
Post by roseway » but it baffles me why they need to be manually registered.
I am sorry, maybe I worded previous post poorly, they are not "manually" registered,
it is more of a "default" list, and it they are not listed as "registered" users, it is a seperate group. " Spiders/Robots ",
Manage bots
“Bots”, “spiders” or “crawlers” are automated agents most commonly used by search engines to update their databases. Since they rarely make proper use of sessions they can distort visitor counts, increase load and sometimes fail to index sites correctly. Here you can define a special type of user to overcome these problems.
And that is where it gets to be a "manual" process, when the admins need to "define"
special types of users, etc. Decide which bots to allow, and where they are allowed
to go, etc,......
There are some really "bad bots", that even go so far as to try to "look like "
the google bot, (for example), I think these are what give the "good bots" such a
bad reputation,
by Bulkley » I say should because I'm never quite sure that's all they do.
I think in many cases, and this happened to me, I thought it was a "google bot",
because that was what it said it was, but it was trying to change it's "profile" as
if it was a normal user, so my imediate reaction was to "ban" and block the google
bot, but later when I was asking about it, on another forum, I learned that it was not
a real google bot, the real google bot does not try to do that sort of thing.
Adminstrating a server/website, forums, etc. is much more complicated then
many people realize, it is important that the owner keeps track of what the "bots"
and visisitors are doing, if the site owner can not, or does not want to , then they
need to have administrators that can be trusted, and know how to deal with those
"tasks",... ..
I don't really know that much, in fact I just barely "scratch" the surface, all though
it is something that interests me, and I am constantly trying to learn more,....