14 DAY TRIAL // Web sites have quickly become one of the simplest, most efficient ways of getting information. They can be complex and dynamic, but are also simple to work with and fast to set up. Despite all these qualities, they do have several drawbacks. Unlike traditional programs that work with one document that is pretty much self-contained, a web site is often made up of many little bits and, because of that, there is the issue of saving it. Fortunately, most modern browsers are capable of saving a web archive, which is a self-contained snapshot of the web page. While this is great for single pages of a site, it is less than ideal for taking down an entire site, especially since all the pages are self contained and all the links between them get broken. Which is why programs that download an entire site have been made. Unfortunately, due to the fact that downloading a part of the web can turn out to be a much bigger task than you anticipated, these programs are usually very hard to work with and are full of options with labels that make no sense to the average user.
Some of them, however, are more user-friendly, like SiteCrawler.
What it does SiteCrawler is an application that can download entire sites locally, preserving their functionality and look. It has all the functionality that you can expect from such an application; however, it is well packaged into small chunks that are more easily digestible and more readily understood.
Working with it At a first glance, the process of snagging a web site locally is a simple one. However, since the web is indeed a very tangled web of pages, following a link may lead simply to the next article or to a totally different site, and each page can potentially led to any number of other pages, creating a vast tangle that can easily fill up all the hard drives in your part of the planet. To avoid such nasty occurrences, there are various options that can be used.
Setting up a download beyond pointing the program in the right direction, by giving it a web link, simply becomes a matter of setting up the appropriate restrictions so you don't download more than you have to. The way the program is set up, you should be able to download most pages without any additional problems, and you will never go overboard. This is because the program comes set up to download as little as possible. If it does come to a site that is not being entirely fetched, you can simply change the Scope setting to the next level.
If you crave a little more control, you can make use of the restrictions tab. Once again, the options here deal with limiting how much you download. Most of these are self explanatory, such as the maximum amount of time to spend on a session or file, the maximum number of files and the maximum size of a file or session. The only option that is not well documented is the 'Crawling Level' which determines how far down the site hierarchy the program should download. If you imagine the side as a pyramid, with the main page at the top, and below it the pages and images that are linked from it, and then below them the pages and images that are linked from them, you get the idea of levels, and of how far a website can stretch through links. The levels setting simply tells it how far down that pyramid the program should download. A setting of 0 or 1 simply downloads the file you link to, while 2 will download that and the next one below it. The amount of web site downloaded increases geometrically with each level, so exercise should be considered with this option.
Fine control While all the options mentioned above are very good for the vast majority of uses, sometimes you need to be very specific. Fortunately, this program can accommodate such needs.
Tucked away in the third tab, there are the filters. These are similar to the rules found in most applications that allow you to automate certain tasks. Here they allow you to automatically exclude or include certain content from downloading, regardless of the other settings that are present in this respect. They cannot however trump the restrictions that may be in place, they are only there to give fine control over the scope. They work similarly to all rules present out there, in that there is an action and a condition, and if the condition is met, the action takes place. The only real difference is that here, defining the rule starts with the action and then it is followed by the condition, which is simply a matter of ordering the two and has no impact on the functionality.
The Good
Very straightforward to use, well designed and documented. Makes the entire thing seem less complex and slowly eases you into the advanced options, should you need them.
The Bad
Sometimes while downloading a site, the program becomes unresponsive and does not accept commands. From an interactive point of view the program freezes, but it comes back when it finishes doing whatever it was doing in the background.
The Truth
There are many such programs out there, but few are as easy to use. This program doesn't swamp you with options from the get-go and, even when you do start getting into the complicated options, it does so in a gradual manner.
Here are some screenshots, click to enlarge:
