This Repository is Reinforcement Learning Agent FrameWork

This repository is designed to provide an easy demo reinforcement learning framework for those studying deep reinforcement learning.

This framework is based on a tensorflow. And the basic model is implemented in example_model directory. If you want to use your own model, please refer provided model in example_model directory

We provide a tutorial to train the agent for the environment, and tutorials by action and input shape are provided as follows.

Environment

Continuous Action MLP - bipedalwalker, pendulum
Discrete Action MLP - LunarLander
Discrete Action CNN - Breakout

Algorithms

Continuous Action MLP - DDPG, TD3, PPO, PPO2
Discrete Action MLP - Vanilla PG, A2C, PPO, DQN, QRDQN, IQN
Discrete Action CNN - Vanilla PG, A2C, PPO, DQN, QRDQN, IQN

Our tutorial is being done in the gym environment provided by openai and you need to install the openai gym and box2d to run the tutorial code.

Installation

from git repository

https://github.com/RLOpensource/tensorflow_RL
pip install .

cpu version

pip install tensorflow-rl[tf-cpu]

gpu version

pip install tensorflow-rl[tf-gpu]

If you install this repository by only

pip install tensorflow-rl

tensorflow is not installed

Requirements

tensorflow
box2d
gym
numpy
tensorboardX

Implemented

[x] Vanilla Policy Gradient
[x] Advantage Actor Critic
[x] Proximal Policy Optimization
[x] Deep Deterministic Policy Gradient
[x] Value based Reinforcement Learning
[ ] Soft Actor Critic
[ ] LSTM train Algorithm

Demonstration

1. Continuous Action BipedalWalker

Script : bipedalwalker_td3.py, bipedalwalker_ddpg.py, bipedalwalker_ppo.py, bipedalwalker_ppo2.py
Environment : BipedalWalker-v2
Orange : td3, Blue: ddpg, SkyBlue: ppo, Pink: ppo2
Episode : 600
Image : td3

BipedalWalker

2. Continuous Action Pendulum

Script : pendulum_td3.py, pendulum_ddpg.py
Environment : Pendulum-v0
Orange : ddpg, Blue: td3
Episode : 300
Image : td3

Pendulum

3. Discrete Action CNN Breakout

Script : breakout_rollout_a2c.py, breakout_rollout_ppo.py, breakout_rollout_vpg.py
Environment : BreakoutDeterministic-v4 with Multi-processing
Blue : ppo, Orange : a2c, Red : vpg
Episode : 600
Image : PPO

Breakout

4. Discrete Action MLP LunarLander

Script : lunarLander_rollout_a2c.py, lunarLander_rollout_ppo.py, lunarLander_rollout_vpg.py
Environment : LunarLander-v2 with Multi-processing
Blue : ppo, Orange : a2c, Red : vpg
Episode : 350
Image : PPO

LunarLander

5. Value Based Reinforcement Learning with CNN

Script : breakout_value_dqn.py, breakout_value_qrdqn.py, breakout_value_iqn.py
Environment : BreakoutDeterministic-v4 with Multi-processing
Green : IQN, Blue : QRDQN, Pink : DQN
Episode : 280
Image : IQN

Breakout

6. Value Based Reinforcement Learning with MLP

Script : lunarLander_value_dqn.py, lunarLander_value_qrdqn.py, lunarLander_value_iqn.py
Environment : LunarLander-v2 with Multi-processing
Orange : IQN, Blue : QRDQN, Red : DQN
Episode : 250
Image : IQN