This guide is a collection of distributed training examples (that can act as boilerplate code) and a tutorial of basic distributed TensorFlow. Many of the examples focus on implementing well-known distributed training schemes, such as those available in dist-keras which were discussed in the author's blog post.
Almost all the examples can be run on a single machine with a CPU, and all the examples only use data-parallelism (i.e., between-graph replication).
The motivation for this guide stems from the current state of distributed deep learning. Deep learning papers typical demonstrate successful new architectures on some benchmark, but rarely show how these models can be trained with 1000x the data which is usually the requirement in industy. Furthermore, most successful distributed cases use state-of-the-art hardware to bruteforce massive effective minibatches in a synchronous fashion across high-bandwidth networks; there has been little research showing the potential of asynchronous training (which is why there are a lot of those examples in this guide). Finally, the lack of documenation for distributed TF was the largest motivator for this project. TF is a great tool that prides itself on its scalability, but unfortunately there are few examples that show how to make your model scale with data size.
The aim of this guide is to aid all interested in distributed deep learning, from beginners to researchers.
See the Basics-Tutorial folder for notebooks demonstrating core concepts used in distributed TensorFlow. The rest of the examples assume understanding of the basics tutorial.
Servers.ipynb -- basics of TensorFlow serversParameter-Server.ipynb -- everything about parameter serversLocal-then-Global-Variables.ipynb -- creates a graph locally then make global copies of the variables Useful for graphs that do local updates before pushing global updates (e.g., DOWNPOUR, ADAG, etc.)Multiple-Workers -- contains three notebooks: one parameter server notebook and two worker notebooks The exercise shows how global variables are communicated via the parameter server and how local updates can be made by explicitly placing ops on local devicesThe complete list of examples is below. The first example, Non-Distributed-Setup, shows the basic learning problem we want to solve distributively; this example should be familiar to all since it doesn't use any distributed code. The second example, Distributed-Setup, shows the same problem being solved with distributed code (i.e., with one parameter server and one worker). The remaining examples are a mix of synchronous and non-synchronous training schemes.
Non-Distributed-SetupDistributed-SetupHogWild (Asychronous SGD)DOWNPOURDOWNPOUR-Easy1AGN (Accumulated Gradient Normalization)Synchronous-SGDSynchronous-SGD-different-learning-ratesSAGN (Synchronous Accumulated Gradients Normalization)Multiple-GPUs-Single-MachineDynamic SGD TODOAsynchronous Elastic Averaging SGD (AEASGD) TODOAsynchronous Elastic Averaging Momentum SGD (AEAMSGD) TODO1This is the same as the DOWNPOUR example except that is uses SGD on the workers instead of Adagrad.
All the training examples (except the non-distributed example) live in a folder. To run them, move to the example directory and run the bash script.
cd <example_name>/
bash run.shIn order to completely stop the example, you'll need to kill the python processes associated with it. If you want to stopped training early, then there will be python processes for each of the workers in addition to the parameter server processes. Unfortunately, the parameter server processes continue to run even after the workers are finished--these will always need to be killed manually. To kill all python processes, run pkill.
sudo pkill python