License Build Status

A new highly flexible, highly scalable document ingestion system.

See the web site and the documentation for more info

Getting Started

Please see the documentation in the wiki

Project Status

Current release version: 1.0-beta2.

Can be used with gradle configuration:

repositories {
  maven {
    url ''
  maven {
    url ''

dependencies {
  compile ('org.jesterj:ingest:1.0-beta2')

The extra repos are for a patched version of cassandra, and should go away in future releases (see The clojars repo is for is for a clojure based implementation of docopt, which will hopefully become unnecessary in future versions.

JDK versions

Presently only JDK 8 has been supported. JDK 9/10 will not be explicitly supported. Now that JDK 11 is out as an LTS version, support for it will commence. JDK 11 is supported in the master branch, but not yet released

Slack Chanel

If you want an invite to our slack channel just send mail to:

                     ______    _           __            _                  
   ____ ___  _______/ ____ \  (_)__  _____/ /____  _____(_)____  _________ _
  / __ `/ / / / ___/ / __ `/ / / _ \/ ___/ __/ _ \/ ___/ // __ \/ ___/ __ `/
 / /_/ / /_/ (__  ) / /_/ / / /  __(__  ) /_/  __/ /  / // /_/ / /  / /_/ / 
 \__, /\__,_/____/\ \__,_/_/ /\___/____/\__/\___/_/__/ (_)____/_/   \__, /  
/____/             \____/___/                     /___/            /____/   


In this release we have the following features

Release 0.1 is intended to be the smallest functional unit. Plans and steps will need to be assembled in code etc and only run locally, only single node supported. Documents indexed will have fields for mod-time, file name and file size.

Progress for 1.0

The Java config feature is experimental but working out better than expected. I wanted to use what I had built for a project, but the lack of externalized configuration was a blocker. It was a quick fix but it's turning out to be quite pleasant to work with. The down side is I'm not sure how it would carry forward to later stages of the project so it might still go away. Feedback welcome.

TODO for 1.0 final

Release 1.0 is intended to be the usable for single node systems, and therefore suitable for production use on small to medium sized projects.

TODO for 2.0

Release 2.0 is intended to be similar to 1.0 but with a very basic web control UI. At this point it should be possible to install the war file, start a node,

TODO for 3.0

Release 3.0 is intended to be the first release to spread work across nodes.

What is FTI?

FTI stands for Fault Tolerant Indexing. For our purposes this means that once a scanner is pointed at a document source, it is guaranteed to eventually do one of the following things with every qualifying document:

It will do this no matter how many nodes fail, or how many times Solr is rebooted