Not sure if this is the place to announce it, but I've just released a first cut of a web crawler framework I've been working on for a few months.
https://github.com/brianmadden/krawler . I've also opened a new channel
#C3QPU9R1N for discussions, etc.