DRKSpiderJava

Checking the consistency of a website can be done faster using crawlers, which are tools that scan each link and webpage of a site, with the purpose of indexing them.

DRKSpiderJava is such an utility that does more than indexing, as it allows you to find broken links and inspect a website’s structure. In order to properly work, the application requires Java installed and running on your computer.

The program can check all the links of a website with ease, by starting its scan from a root page. By adjusting the depth of a search, you can set the crawler to scan every link found on a certain page, up to a number of times. You can also limit the scan by certain external links, URL exclusion list, and the optional setting for obeying robots.txt site definition.

Once the scan is complete, you are displayed with a tree view of each page, representing the hierarchical distribution of each one inside the website. Doing so provides you with a clear view of every page of a website.

Once DRKSpiderJava finishes crawling a website, it provides you with a list of all the safe or broken links. This list can be exported to XML or CSV, then forwarded to a specialist for further analysis.

Each item in the list displays contextual information, such as link tag, URL, anchor text, nofollow status, link status code, and depth. One an alternate side, the application can help you optimize a website, by analyzing its SEO score and seeing what it lacks.

DRKSpiderJava has what it takes to help you check the consistency of a website and the links it contains, using intuitive and multi-threaded crawlers.