Another HDFS Federation solution from bytedance.


Compared with ViewFS and WebHDFS:

Multiple language supportYesNoYes
Unified APIs providedYesYesNo
Mount table stored inZookeeperConfigurationConfiguration
* Client-side configurations are not usually easy to change
Client-side libraryNoHeavyLight
Call latency introducedMediumNo latencyHigh
Centralized cache supportYesYesNo


Benchmark result with NameNode Benchmark 0.4:

Direct TPSNNProxy TPS (3 proxy instances)
create_write @ 16maps263,294247,799
create_write @ 128maps3,008,6042,563,465
rename @ 128maps474,671471,240

(numOfFiles 512 each map, NameNode CPU: Xeon E5-2630 v2 * 2, network RTT 0.1ms)


IMPORTANT: NNProxy works only with patched NameNode. See /hadoop-patches in git repository. These patches add proxy support to hadoop-common library. Each of them must be applied before compiling.

Build with maven:

$ cd nnproxy/hadoop-nnproxy
$ mvn package -DskipTests

Built jar are shaded with necessary libraries, which can run with hadoop jar.


Organize working directory as follows:


Then run bin/nnproxy with configuration arguments to start proxy server. Note that before starting proxy server, a mount table must be deployed first.

For example:

# Setup zk environment, for simplicity, not necessary
export NNPROXY_MOUNT_TABLE_ZKPATH="/hadoop/hdfs/mounts"

export NNPROXY_OPTS="-Ddfs.nnproxy.mount-table.zk.quorum=$NNPROXY_ZK_QUORUM -Ddfs.nnproxy.mount-table.zk.path=$NNPROXY_MOUNT_TABLE_ZKPATH"

# Deploy mount table, example above mounts / to hdfs://toutiaonn0 and /data 
# to hdfs://toutiaonn1. In which case, accessing /data/recommend would be 
# transparently redirected to hdfs://toutiaonn1/data/recommend.
# In this example, toutiaonn0 and toutiaonn1 are fs names configured with 
# auto failover HA. Using ip:port directly (non-HA setup) are also fesible.
echo 'hdfs://toutiaonn0 /' > mounts
echo 'hdfs://toutiaonn1 /data' >> mounts
cat mounts | bin/nnproxy load $NNPROXY_OPTS

# Run proxy (remember these args)
bin/nnproxy proxy $NNPROXY_OPTS

Look into class com.bytedance.hadoop.hdfs.ProxyConfig for more configurations available.

Proxy server metrics are also available with hadoop-metrics2.

Mount table management

Once proxy server is started, it watches mount table node on ZooKeeper for further changes.

Run bin/nnproxy dump $NNPROXY_OPTS to dump current mount table to stdout.

Run bin/nnproxy load $NNPROXY_OPTS to load mount table from stdin.

One can also alter data stored in mount table ZK node directly (or possibly build APIs or UIs). All these changes take effect immediately.

Mount table are delimited text with 3 columns: targetFS, path and attributes, in that order. For example:

hdfs://toutiaonn0   /        none
hdfs://toutiaonn1   /data    none

Each row advises proxy to redirect path and its children to specified targetFS. attributes are currently reserved and is of no use.

Future work


Copyright 2016 Bytedance Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.