RoboGen is a program, for Windows 2000, XP, and Vista, to help you more easily manage the robots.txt files for your websites. Instead of the time consuming manual management process, described in the tutorial, you can either edit your robot exclusion files on your local computer or on a remote FTP site.
RoboGen is a visual editor for Robot Exclusion Files.
The user is allowed to quickly and easily create the ROBOTS.TXT files that are required to instruct web search engines which parts of a your web site are not to be indexed or made searchable by the public browsing the web.
RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable.
RoboGen maintains a database of over 180 search engine user-agents, which are selectable from a drop down menu.
Here are some key features of "RoboGen Standard Edition":
· Create Robot Exclusion Files by selecting documents and directories.
· Log into FTP servers and even upload robots.txt from RoboGen.
· Manage information for multiple servers.
· Database of over 180 know user-agents and 10 major search engines.
· Ability to edit the robot database and add additional user-agents.
· RoboGen recognizes common document roots when logging into FTP servers and suggests that they be set as the document root.
Requirements:
· 2 megabytes of free hard drive space
· 12 megabytes of RAM (16 or more recommended).
· Microsoft Internet Explorer 3.0 or newer (used for FTP access).
· Website with your own domain name or document root.
Limitations:
· 5 Trial Saves only
Find us on Google+
