hot-crawler is a web crawler about hot information on excellent websites. It can help you quickly access valuable information on many websites. It is compatible with multiple devices, such as personal computers, mobile phones, and tablets. These websites are mainly composed of two languages, Chinese and English. As shown in the figure below.
To develop Hot-Crawler, you need to install the following softwares:
We strongly suggest using IntelliJ IDEA to develop this project.
Clone project source to your local computer
$ git clone https://github.com/tagnja/hot-crawler.gitActive develop configuration in src/main/resources/application.yml
spring:
profiles:
active: devStartup your local Redis Server before running the project.
Running the project with Spring Boot maven plugin.
cd hot-crawler
mvn spring-boot:runVisting the website by http://localhost:8080
For example of crawler development to view crawler development tutorial.
For using techniques of the project to view use techniques list.
For the complete crawled websites to view websites crawler list.
For more info about update log to view update log.
Contributing is welcome! Please check out the Contributing to hot-crawler guide.
hot-crawler is released under the MIT License.