Robots Spiders and Bots

Forum for admin topics, member introductions and general non-hifi chitchat.
Forum rules
1. No ad-hominem
2. No spamming or shilling
User avatar
Dr Bunsen Honeydew
Posts: 26394
Joined: Wed Jul 11, 2012 7:26 pm
Location: Muppet Labs
Great Britain

Robots Spiders and Bots

Unread post by Dr Bunsen Honeydew » Tue Nov 14, 2017 6:55 pm

For some reason since the forum upgrade we have lost just about all of them. The only regular visitor now is Baidu from China. It has mystified Terry, does anyone here have knowledge of how we get them back.

Lurcher300b
Posts: 407
Joined: Tue Jun 14, 2016 1:58 pm
Great Britain

Re: Robots Spiders and Bots

Unread post by Lurcher300b » Tue Nov 14, 2017 7:49 pm

Have you looked on the manage bot page in the phpbb admin section checking they are activated.

User avatar
terrybooth
Posts: 3693
Joined: Wed Jul 11, 2012 6:49 pm
Location: West Yorkshire
Great Britain

Re: Robots Spiders and Bots

Unread post by terrybooth » Tue Nov 14, 2017 8:14 pm

Yes. All bots are activated.
Pioneer PL71/DL103/ Phono2/HiFiPi/P90SA/TIS/CubixPro two-up

User avatar
terrybooth
Posts: 3693
Joined: Wed Jul 11, 2012 6:49 pm
Location: West Yorkshire
Great Britain

Re: Robots Spiders and Bots

Unread post by terrybooth » Tue Nov 14, 2017 8:19 pm

94 guests online at the moment - looking at them, many appear to be bots. But they are not registered.
Pioneer PL71/DL103/ Phono2/HiFiPi/P90SA/TIS/CubixPro two-up

User avatar
Dr Bunsen Honeydew
Posts: 26394
Joined: Wed Jul 11, 2012 7:26 pm
Location: Muppet Labs
Great Britain

Re: Robots Spiders and Bots

Unread post by Dr Bunsen Honeydew » Tue Nov 14, 2017 8:58 pm

Ah! that would explain it, how do we register them or are they nasty fishing bots.

User avatar
terrybooth
Posts: 3693
Joined: Wed Jul 11, 2012 6:49 pm
Location: West Yorkshire
Great Britain

Re: Robots Spiders and Bots

Unread post by terrybooth » Tue Nov 14, 2017 9:44 pm

The one's we want are the search engine ones. I've created a robots.txt file to see if that makes a difference.
Pioneer PL71/DL103/ Phono2/HiFiPi/P90SA/TIS/CubixPro two-up

Lurcher300b
Posts: 407
Joined: Tue Jun 14, 2016 1:58 pm
Great Britain

Re: Robots Spiders and Bots

Unread post by Lurcher300b » Tue Nov 14, 2017 11:19 pm

Does the bot page show the last time they visited?

User avatar
Dr Bunsen Honeydew
Posts: 26394
Joined: Wed Jul 11, 2012 7:26 pm
Location: Muppet Labs
Great Britain

Re: Robots Spiders and Bots

Unread post by Dr Bunsen Honeydew » Tue Nov 14, 2017 11:39 pm

Yes. i just copied a few but only the top three have been here and only two today.

Baidu [Spider] Tue Nov 14, 2017 11:33 pm Deactivate Edit Delete
Exabot [Bot] Tue Nov 14, 2017 3:22 am Deactivate Edit Delete
Alexa [Bot] Fri Nov 10, 2017 6:49 am Deactivate Edit Delete
AdsBot [Google] Never Deactivate Edit Delete
Alta Vista [Bot] Never Deactivate Edit Delete
Ask Jeeves [Bot] Never Deactivate Edit Delete
Bing [Bot] Never Deactivate Edit Delete
FAST Enterprise [Crawler] Never Deactivate Edit Delete
FAST WebCrawler [Crawler] Never Deactivate Edit Delete
Francis [Bot] Never Deactivate Edit Delete
Gigabot [Bot] Never Deactivate Edit Delete
Google Adsense [Bot] Never Deactivate Edit Delete
Google Desktop Never Deactivate Edit Delete
Google Feedfetcher Never Deactivate Edit Delete

There is about 100 more listed as never visited. Obviously the formatting is wrong but I am sure you can see what I mean.

Lurcher300b
Posts: 407
Joined: Tue Jun 14, 2016 1:58 pm
Great Britain

Re: Robots Spiders and Bots

Unread post by Lurcher300b » Wed Nov 15, 2017 1:16 am

Try adding the website into googles search list.

https://www.google.com/webmasters/tools ... -url?pli=1

User avatar
Dr Bunsen Honeydew
Posts: 26394
Joined: Wed Jul 11, 2012 7:26 pm
Location: Muppet Labs
Great Britain

Re: Robots Spiders and Bots

Unread post by Dr Bunsen Honeydew » Wed Nov 15, 2017 10:06 am

Done, should everyone here do this at it seems to be an individual choice.

I have a thought, could it be they are all going to a Portal that is no longer there !!!!

Post Reply