xcrawler
a light framework of crawler with gevent, requests, leveldb
Features
- multiple crawling task, use coroutine of gevent;
- a url pool to manage urls (downloaded or to download), use leveldb;
- a proxy pool to avoid blocked by the site while quering very frequently;
Installation
sudo python setup.py install
Usage
Sorry for no docs right now, just see the example or read the source code