ItSucks

ItSucks is a java web spider (web crawler) with the ability to download (and resume) files. ItSucks is also highly customizable with regular expressions and download templates.

The application also provides a swing GUI and a console interface. All backend functionalities are also available in a separate library, they can be easily used for other projects.

Here are some key features of "ItSucks":

￭ Designed to support multiple protocols. At this time only http/https is implemented. ￭ Running multiple crawl jobs simultaniously. ￭ Number of working threads for every crawl job is changeable. (Not over the GUI available yet.) ￭ Search depth is configurable. ￭ Support for individual priority of an URL. ￭ Customizable processing chain to process the downloaded data. ￭ Save and load one or multiple Jobs. (serialize / deserialize) ￭ Fully programmed with Java 1.5 features (generics etc.). ￭ Allowed hostnames are configurable by regular expressions. ￭ Type of files to be downloaded are configurable by regular expressions. ￭ A base URL can be set to follow only URL's beginning with this prefix. ￭ A highly configurable advanced filter can hold multiple regular expressions. For every expression, actions can be defined when the regular expression matches the URL and actions to be executed when the expression does not match. ￭ Possible actions are: follow the URL (Accept), do not follow the URL (Reject), change the priority of the URL ￭ A filter can inject additional parameters into the job. ￭ Support to observe every event fired by the framework. ￭ Possibility to filter events by category and type. ￭ Keep track of multiple downloads and one time. ￭ Add and remove download jobs. ￭ A integrated regular expression tester to test/debug your regular expression directly. ￭ Load and save downloads as templates. ￭ Start your download templates on the console after creating it with the GUI.

Requirements:

￭ Java JRE 1.5 or better ￭ Min. 256 MB RAM (the more the better, java craves for memory)