Another HDFS Federation solution from bytedance.
Compared with ViewFS and WebHDFS:
|Multiple language support||Yes||No||Yes|
|Unified APIs provided||Yes||Yes||No|
|Mount table stored in||Zookeeper||Configuration||Configuration|
|* Client-side configurations are not usually easy to change|
|Call latency introduced||Medium||No latency||High|
|Centralized cache support||Yes||Yes||No|
Benchmark result with NameNode Benchmark 0.4:
|Direct TPS||NNProxy TPS (3 proxy instances)|
|create_write @ 16maps||263,294||247,799|
|create_write @ 128maps||3,008,604||2,563,465|
|rename @ 128maps||474,671||471,240|
(numOfFiles 512 each map, NameNode CPU: Xeon E5-2630 v2 * 2, network RTT 0.1ms)
IMPORTANT: NNProxy works only with patched NameNode. See
/hadoop-patches in git repository. These patches add proxy support to
hadoop-common library. Each of them must be applied before compiling.
Build with maven:
$ cd nnproxy/hadoop-nnproxy $ mvn package -DskipTests
Built jar are shaded with necessary libraries, which can run with
Organize working directory as follows:
bin/ bin/nnproxy lib/ lib/hadoop-nnproxy-0.1.0.jar
bin/nnproxy with configuration arguments to start proxy server.
Note that before starting proxy server, a mount table must be deployed first.
# Setup zk environment, for simplicity, not necessary export NNPROXY_ZK_QUORUM="127.0.0.1:2181" export NNPROXY_MOUNT_TABLE_ZKPATH="/hadoop/hdfs/mounts" export NNPROXY_OPTS="-Ddfs.nnproxy.mount-table.zk.quorum=$NNPROXY_ZK_QUORUM -Ddfs.nnproxy.mount-table.zk.path=$NNPROXY_MOUNT_TABLE_ZKPATH" # Deploy mount table, example above mounts / to hdfs://toutiaonn0 and /data # to hdfs://toutiaonn1. In which case, accessing /data/recommend would be # transparently redirected to hdfs://toutiaonn1/data/recommend. # In this example, toutiaonn0 and toutiaonn1 are fs names configured with # auto failover HA. Using ip:port directly (non-HA setup) are also fesible. echo 'hdfs://toutiaonn0 /' > mounts echo 'hdfs://toutiaonn1 /data' >> mounts cat mounts | bin/nnproxy load $NNPROXY_OPTS # Run proxy (remember these args) bin/nnproxy proxy $NNPROXY_OPTS
Look into class
com.bytedance.hadoop.hdfs.ProxyConfig for more configurations available.
Proxy server metrics are also available with hadoop-metrics2.
Once proxy server is started, it watches mount table node on ZooKeeper for further changes.
bin/nnproxy dump $NNPROXY_OPTS to dump current mount table to stdout.
bin/nnproxy load $NNPROXY_OPTS to load mount table from stdin.
One can also alter data stored in mount table ZK node directly (or possibly build APIs or UIs). All these changes take effect immediately.
Mount table are delimited text with 3 columns:
attributes, in that order.
hdfs://toutiaonn0 / none hdfs://toutiaonn1 /data none
Each row advises proxy to redirect
path and its children to specified
attributes are currently reserved and is of no use.
readonlyattribute in mounts
Copyright 2016 Bytedance Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.