Introduction

ES-Fastloader

The ES-Fastloader uses the fault tolerance and parallelism of Hadoop and builds individual ElasticSearch shards in multiple reducer nodes, then transfers shards to ElasticSearch cluster for serving. The loader will create a Hadoop job to read data from data files in HDFS, repartitions it on a per-node basis, and finally writes the generated indices to ES shards. In DiDi we have been using ES-Fastloader to create large-scale ElasticSearch indices from TB/PB level sequence files in Hive.

Features

Requirements

Developer guide

Contributing

Welcome to contribute by creating issues or sending pull requests. See Contributing Guide for guidelines.

Who is using ES-Fastloader?

滴滴出行

License

ES-Fastloader is licensed under the Apache License 2.0. See the LICENSE file.

Contact us

微信交流群