Page 1 of 2

Robots Spiders and Bots

Posted: Tue Nov 14, 2017 6:55 pm
by Dr Bunsen Honeydew
For some reason since the forum upgrade we have lost just about all of them. The only regular visitor now is Baidu from China. It has mystified Terry, does anyone here have knowledge of how we get them back.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 7:49 pm
by Lurcher300b
Have you looked on the manage bot page in the phpbb admin section checking they are activated.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 8:14 pm
by terrybooth
Yes. All bots are activated.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 8:19 pm
by terrybooth
94 guests online at the moment - looking at them, many appear to be bots. But they are not registered.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 8:58 pm
by Dr Bunsen Honeydew
Ah! that would explain it, how do we register them or are they nasty fishing bots.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 9:44 pm
by terrybooth
The one's we want are the search engine ones. I've created a robots.txt file to see if that makes a difference.

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 11:19 pm
by Lurcher300b
Does the bot page show the last time they visited?

Re: Robots Spiders and Bots

Posted: Tue Nov 14, 2017 11:39 pm
by Dr Bunsen Honeydew
Yes. i just copied a few but only the top three have been here and only two today.

Baidu [Spider] Tue Nov 14, 2017 11:33 pm Deactivate Edit Delete
Exabot [Bot] Tue Nov 14, 2017 3:22 am Deactivate Edit Delete
Alexa [Bot] Fri Nov 10, 2017 6:49 am Deactivate Edit Delete
AdsBot [Google] Never Deactivate Edit Delete
Alta Vista [Bot] Never Deactivate Edit Delete
Ask Jeeves [Bot] Never Deactivate Edit Delete
Bing [Bot] Never Deactivate Edit Delete
FAST Enterprise [Crawler] Never Deactivate Edit Delete
FAST WebCrawler [Crawler] Never Deactivate Edit Delete
Francis [Bot] Never Deactivate Edit Delete
Gigabot [Bot] Never Deactivate Edit Delete
Google Adsense [Bot] Never Deactivate Edit Delete
Google Desktop Never Deactivate Edit Delete
Google Feedfetcher Never Deactivate Edit Delete

There is about 100 more listed as never visited. Obviously the formatting is wrong but I am sure you can see what I mean.

Re: Robots Spiders and Bots

Posted: Wed Nov 15, 2017 1:16 am
by Lurcher300b
Try adding the website into googles search list.

https://www.google.com/webmasters/tools ... -url?pli=1

Re: Robots Spiders and Bots

Posted: Wed Nov 15, 2017 10:06 am
by Dr Bunsen Honeydew
Done, should everyone here do this at it seems to be an individual choice.

I have a thought, could it be they are all going to a Portal that is no longer there !!!!