Hi all! I've just released a first cut of a web crawler framework I've been working on for a few months.
https://github.com/brianmadden/krawler . I've also opened a new channel
#krawler for discussions, etc. Bug reports, comments, questions all welcome. 🙂