• Search by APIs
• Search by Words
• Search Projects

• Java
• Python
• JavaScript
• TypeScript
• C++
• Scala
• Blog

Projects Related with crawler

• 1. webmagic
  under Apache License 2.0 license
• A scalable web crawler framework for Java.
• 2. WebCollector
  under GNU General Public License v3.0 license
• WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
• 3. gecco
  under MIT License license
• Easy to use lightweight web crawler（易用的轻量化网络爬虫）
• 4. SeimiCrawler
  under Apache License 2.0 license
• 一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
• 5. xxl-crawler
  under GNU General Public License v3.0 license
• A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
• 6. storm-crawler
  under Apache License 2.0 license
• A scalable, mature and versatile web crawler based on Apache Storm
• 7. sparkler
  under Apache License 2.0 license
• Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
• 8. CrawlerForReader
  under Apache License 2.0 license
• Android 本地网络小说爬虫，基于jsoup及xpath
• 9. ache
  under Apache License 2.0 license
• ACHE is a web crawler for domain-specific search.
• 10. ispider
  under Apache License 2.0 license
• A Distributed Crawler System Designed By Java.
• 11. CrawlerPack
  under Apache License 2.0 license
• Java 網路資料爬蟲包
• 12. crawler-commons
  under Apache License 2.0 license
• A set of reusable Java components that implement functionality common to any web crawler
• 13. elasticsearch-river-web
  under Apache License 2.0 license
• Web Crawler for Elasticsearch
• 14. JewelCrawler
  under GNU General Public License v3.0 license
• 豆瓣电影爬虫——a crawler which is able to crawl movie detail and short comments, save them to database mysql, also include Sentiment analysis based on comments
• 15. zongtui-webcrawler
  under GNU General Public License v2.0 license
• 基于hadoop思维的分布式网络爬虫。
• 16. DouyuCrawler
  under Apache License 2.0 license
• [Deprecated]一个Java程序，用于抓取斗鱼弹幕。
• 17. WebVideoBot
  under MIT License license
• Web crawler.
• 18. customer-review-crawler
  under The Unlicense license
• A crawler to collect reviews and product infomation on Amazon.com
• 19. java-Crawler
  under MIT License license
• :heavy_check_mark: Some web crawler code implemented in Java . 各类爬虫代码
• 20. AppCrawler
  under Apache License 2.0 license
• Automated GUI testing tool for Android Applications
• 21. gecco-htmlunit
  under MIT License license
• Gecco crawler downloader for htmlunit
• 22. gecco-redis
  under MIT License license
• Gecko crawler supports distributed by redis
• 23. supermonkey
  under Apache License 2.0 license
• A crawler for automated Android UI testing.
• 24. crawler-jsoup-maven
  under Apache License 2.0 license
• This is a crawler(reptile)
• 25. codes-scratch-crawler
  under Apache License 2.0 license
• 读书笔记《自己动手写网络爬虫》，自己敲的代码。主要记录了网络爬虫的基本实现，网页去重的算法，网页指纹算法，文本信息挖掘
• 26. Dodder
  under MIT License license
• A distributed DHT crawler that sniffs torrents from BitTorrent network
• 27. AIPa
  under Apache License 2.0 license
• A compact, flexible Java multi-threaded crawler framework (Ai Pa), built-in Jsoup, zero-cost hands-on.一款小巧、灵活的Java多线程爬虫框架（AiPa）内嵌Jsoup 零成本上手
• 28. crawler4j
  under Apache License 2.0 license
• Open Source Simple Web Crawler for Java. Simple Flexible And Lightweight
• 29. hot-crawler
  under MIT License license
• A springboot-based hot news crawler.
• 30. BUbiNG
  under Apache License 2.0 license
• The LAW next generation crawler.
• 31. PicCrawler
  under Apache License 2.0 license
• 使用RxJava2 和 Java 8的特性开发的图片爬虫
• 32. ankus_crawler
  under Apache License 2.0 license
• 33. vw-crawler
  under MIT License license
• :beetle:简单轻便的Java爬虫框架，只要会一点简单的正则表达式和简单的css选择器就能轻松的采集数据。
• 34. news-crawl
  under Apache License 2.0 license
• News crawling with Storm-crawler - stores content as WARC
• 35. webcrawler-verifier
  under MIT License license
• Java library providing functionality to verify that user-agents are who they claim to be.
• 36. DistributedCrawler
  under Apache License 2.0 license
• DistributeCrawler的Maven版
• 37. visual-spider
  under MIT License license
• 用JavaFX开发基于crawler4j的图形化的网络爬虫
• 38. flink-crawler
  under Apache License 2.0 license
• Continuous scalable web crawler built on top of Flink and crawler-commons
• 39. vscrawler
  under Apache License 2.0 license
• a crawler framework appropriate grab
• 40. tom-crawler
  under Apache License 2.0 license
• spring整合webmagic，mybatis，dungproxy