rexCrawler
rexCrawler is a regular expression web-crawler plugin. Able to parse websites, save data and show…
Plugin Tag: crawler
BotRx DeTx Plugin
DeTx uses a global database of attackers (Bots) and also includes Artificial Intelligence and data…
JaspreetChahal’s wordpress bot detector lite
This plugin detects a few bots (e.g. Google, Google ads, Alta vista etc) and sends…
Robots.txt Extender
Dynamic robots.txt for Multisite! Change parameters, or don't, for each site of your network without…
Page Modified
Display the results of your Page Modified crawls right within your WordPress Admin.
SEObot Monitor for Googlebot, Bingbot and search engine spiders
With SEObot Monitor for Googlebot you will be able to dump the server logs to…
Block Common Crawl via robots.txt
Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI…