NetDiscovery

@Tony沈哲 on weibo License Codacy Badge

功能特点:

最新版本

模块名 最新版本
netdiscovery-core-core Download
netdiscovery-core-engine Download
netdiscovery-core-engine-monitor Download
netdiscovery-downloader-htmlunit Download
netdiscovery-downloader-httpclient Download
netdiscovery-downloader-okhttp Download
netdiscovery-downloader-selenium Download
netdiscovery-pipeline-couchbase Download
netdiscovery-pipeline-elasticsearch Download
netdiscovery-pipeline-mongo Download
netdiscovery-pipeline-redis Download
netdiscovery-queue-kafka Download
netdiscovery-queue-rabbitmq Download
netdiscovery-queue-redis Download
netdiscovery-queue-rocketmq Download
netdiscovery-kotlin-coroutines Download
netdiscovery-kotlin-dsl Download

NetDiscovery 是基于 Vert.x、RxJava 2 等框架实现的爬虫框架。目前仍在不断地完善中,期望能成为通用的爬虫框架。

对于 Java 工程,如果使用 gradle 构建,由于默认没有使用 jcenter(),需要在相应 module 的 build.gradle 中配置

repositories {
    mavenCentral()
    jcenter()
}

在 NetDiscovery 中,Spider 可以单独运行,Spider 也可以交给 SpiderEngine 来控制。

SpiderEngine 可以在运行之前注册到 Etcd/Zookeeper,然后由 monitor 对 SpiderEngine 进行监控。

Stargazers over time

Stargazers over time

详细功能查看wiki

下载

案例:

基于本人的开源项目

TODO List:

Contributors:

Contributing:

Pull requests are welcome

联系方式:

Wechat:fengzhizi715

Java与Android技术栈:每周更新推送原创技术文章,欢迎扫描下方的公众号二维码并关注,期待与您的共同成长和进步。

License

Copyright (C) 2017 - present, Tony Shen.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.