Robots.txt is a visual editor tool for Robot Exclusion Files and a log manager package tool.
Robots.txt allows a user to quickly and easily create the robots.txt files required to instruct search engine spiders, which parts of a Web site are not to be indexed and made searchable by the general Web public and then to identify spiders, which do not keep to those instructions.
The program provides the user a way to log onto his FTP or local network server and then select the documents and directories which are not to be made searchable.
Here are some key features of "Robots.txt":
· Visually generate industry standard Robots.txt files
· Identify malicious and unwanted spiders and ban them from your site
· Direct search engine crawlers to the appropriate pages for multilingual sites
· Keep spiders out of sensitive and private areas of your Web site
· Upload the correctly formatted robots.txt file directly to your FTP server not switching from Robots.txt Editor
· Track spider visits
· Create spider visits reports in HTML, Microsoft Excel CSV and XML formats
Requirements:
· Internet Explorer 4.0 or higher
· WinSock 2.0
· Intel Pentium 166 processor or higher
· 64 MB RAM
· 14K internet connection
· 8MB free hard disk space (plus extra space for log files you will import from your server)
Limitations:
· you cannot add a new spider to the list;
· you cannot edit spider information;
· you cannot save the generated instruction file robots.txt;
· web server log import is restricted to one file;
· you cannot export log analysis results;
· report generation and their customization are unavailable.
Find us on Google+
