theArchivist 2.1x (b10) review

theArchivist is a web-crawler for downloading and archiving web content, rewriting absolute links as it goes.

License: Freeware OS: Mac OS X File size: 659K Developer: Brian Johnson Price: $0.00 Updated: 18 Jan 2006

0

theArchivist is a web-crawler for downloading and archiving web content, rewriting absolute links as it goes. Variable "stopping rules" make sure you don't try to archive the whole web!

Starting with a particular URL it retrieves the web page, scans it for links, and then attempts to retrieve all files linked to the page.
This behavior repeats for each file retrieved and continues until one of several stop criteria is reached.

If desired, the application will rewrite absolute URLs relative to the download hierarchy, producing a completely self-sufficient archive.

What's New:
Corrected parsing of javascript function references,
added php and asp to recognized "html" file extensions,
fixed bug that truncated crawls done without the "legal servers" restriction.