Visual Web Spider

Visual Web Spider is a multi-threaded web crawler whose purpose is to help you collect and index webpages on the Internet. The tool is able to index pages that contain specific keywords and phrases.

You are welcomed by a clean feature lineup where you can define new projects with the aid of a wizard-like approach. You are offered step-by-step guidance throughout the entire configuration process. The projects may be saved to a file on your computer so you can edit them in the future.

You can create a new project by providing information about the starting URLs. You may add links from custom files (TXT, CSV), generate URLs or create search engine links.

What’s more, you are allowed to set up crawling rules and depth, specify the rules to index only relevant webpages (e.g. index all pages, index only one page of each website), select the type of information that you want to index (e.g. page keywords, page size, HTTP status code, page description, page domain, last modified time), as well as pick the project filename.

You are allowed to start, pause or stop each project, reset failed tasks, filter the information using custom conditions, sort data by ID, page URL, HTTP status code, page title, page text, and backward link, resolve page URLs, and delete duplicate files.

Visual Web Spider gives you the possibility to export data to Microsoft Access Database or MySQL database, or save the information to ASCII, CSV or HTML file format. You can specify the saving directory and export all results into a single item or a user-defined number of results per file.

Last but not least, you can export the link map to HTML file format, save broken or failed links to HTML, TXT or CSV, as well as pick the preferred encoding type. Tests have shown that the application carries out a task quickly while remaining light on system resources, to the overall performance of the computer is not hampered.

All things considered, Visual Web Spider provides a straightforward working environment and several powerful features for helping you automatically index or collect specific pages.