Rainbow is a tool that helps improve the I/O performance of wide tables stored in columnar formats on HDFS.




More information in our project main page.

HDFS Column Store

In many industrial and academic data analytical applications, huge amount of structured data is stored as wide two-dimension tables in columnar file formats on HDFS. Popular columnar file formats, including RC File, ORC, Parquet and Carbondata, are widely supported in data analytical systems over HDFS such as Hive, Spark, Presto and Impala.

These file formats partition the data into row groups and place data inside a row group in columnar manner. A row group is stored in an HDFS block. This is an efficient way for that it introduces most advantages of column store into Hadoop ecosystem without affecting parallelism, scalability and fault-tolerance.

Wide Tables

With these file formats, tables on HDFS are becoming very wide, from a few hundred columns to tens of thousands. Wide Table has some important advantages:

Although everything looks good having the wide tables stored as columnar formats on HDFS, the I/O efficiency and query performance are far from optimal.

In an experimental example, given a 400GB, 1187-column table stored as Parquet in a single node HDFS. The read bandwidth of HDFS is 100MB/s. A query took 907 seconds to read 8 columns (0.3% data, i.e. 1.2GB) from the table. While ideally, it should take only 12 seconds.

Such a huge gap between ideal and reality is caused by disk seeks. The columns read by a query may not be continuous on disk so that seek cost becomes the major part of I/O cost.

We wrote a Paper (SIGMOD'17) on narrowing the gap and we have Rainbow to implement the solutions in the paper.

Note: Rainbow does not contain any source code used in the original implementations described in section 5 and 6 of the paper. Copyrights and patents of the original implementations are not included in Rainbow.

How to Use it?

Rainbow is just a command line tool. It calculates the optimized data layout and generates Hive SQL statements. The generated CREATE TABLE and LOAD DATA statements can be used in Hive to apply the optimized data layout.

Rainbow does not need to be installed. It does not change anything except your CREATE TABLE statements.

Follow the steps in Rainbow CLI to use the command line interface of Rainbow.

Or follow the steps in Rainbow Web to use the Web user interface of Rainbow.

Contact us

For feedback and questions, feel free to email us:

Welcome to contribute and submit pull requests :)

Project Structure