InterroBot

A web crawler can be a useful addition to your software collection, especially if you're a domain administrator. Besides being a way to boost your website's ranking in the search results hierarchy, such tools can also help one with troubleshooting or maintenance-related duties.

More specifically, users can get a more thorough idea as to how a domain operates behind the scenes. InterroBot is a modern, practical tool that can help users filter their website's content, see any assets or pages that failed to load, as well as look for HTTP headers, status codes, and much more.

The program is well-suited to handle crawling tasks in a surprisingly accessible way. Users can create a new project by inputting a domain URL, which the software should start crawling without delay. As it's crawling, users will be able to see the links the program goes through, while also observing the domain's pie chart.

The pie chart provides easy-to-understand information about the pages, assets, and external links of a website. Any errors will also be pointed out.

Through the Search interface, users will be able to inspect the links crawled by the app more thoroughly. With the use of the filtering functions on offer, users can handily choose to view only specific pages or assets, while also having the possibility to exclusively inspect any client or server errors. Pages that fail to load in due time also can be filtered in or out, likewise for missing media.

You may also pause or resume any previously stopped crawls through the software. The option to re-index already crawled pages is likewise available for users.

InterroBot is a functional piece of software that packs several web crawling features within a modern, accessible interface. The program works as expected, as we were able to crawl a number of domains without any problems.