If you want to control the robots or spiders that visit your site, this application is the solution #Robots txt #Generate text #Control spider #Robot #Spider #Generate
If you want to control the robots or spiders that visit your site, robots-txt generator is the solution.
It is very common that the webmaster of each site spends a lot of money paying for bandwidth and it is unfair that a significant portion of that bandwidth is consumed spiders. With Robots txt generator, you can stop or control the bots.
Some robots or spiders saturate your server making countless queries per minute?
With Robots txt generator, you can limit how many requests can make any robot or spider per hour or per minute. The program has a database of more than 300 bots, allowing you to search on the database the source and owner of any bot you have found in your logs.
You can even specify different restrictions for different bots. Per example, you can allow google bot, but disallow google-image-bot. You can specify a different restriction level for each one of the 300 bots in the database or can specify some restrition to the “general” bots, plus some specific for some specifi bots. Like giving a ban to some specific bot.
Robots txt generator 1.01
add to watchlist add to download basket send us an update REPORT- runs on:
-
Windows 2003
Windows Vista
Windows XP
Windows 2K - file size:
- 80 KB
- filename:
- Robots.txt-Generator.rar
- main category:
- Internet
- developer:
- visit homepage
Context Menu Manager
ShareX
Bitdefender Antivirus Free
calibre
7-Zip
Microsoft Teams
IrfanView
4k Video Downloader
Zoom Client
Windows Sandbox Launcher
- 4k Video Downloader
- Zoom Client
- Windows Sandbox Launcher
- Context Menu Manager
- ShareX
- Bitdefender Antivirus Free
- calibre
- 7-Zip
- Microsoft Teams
- IrfanView