Reinforcement Learning in Python

Implementing Reinforcement Learning (RL) Algorithms for global path planning in tasks of mobile robot navigation. Comparison analysis of Q-learning and Sarsa algorithms fo the environment with cliff, mouse and cheese.
DOI

Related works:

Description

RL Algorithms implemented in Python for the task of global path planning for mobile robot. Such system is said to have feedback. The agent acts on the environment, and the environment acts on the agent. At each step the agent:

The environment:

Goal is to learn how to take actions in order to maximize the reward. The objective function is as following:

Q[s, a] = Q[s, a] + λ (r + γ max(Q[s, a]) – Q[s, a]),

where,
Q[s, a] - value of the objective function on the next step,
Q[s, a] - value of the objective function on the current position,
max(Q
[s, a]) – Q[s, a]) - choosing maximum value from the possible next steps,
s – current position of the agent,
a – current action,
λ – learning rate,
r – reward that is got in the current position,
γ – gamma (reward decay, discount factor),
s - next chosen position according to the next chosen action,
a
- next chosen action.

The major component of the RL method is the table of weights - Q-table of the system state. Matrix Q is a set of all possible states of the system and the system response weights to different actions. During trying to go through the given environment, mobile robot learns how to avoid obstacles and find the path to the destination point. As a result, the Q-table is built. Looking at the values of the table it is possible to see the decision for the next action made by agent (mobile robot).


Experimental results with different Environments sre shown and described below.
Code is supported with a lot of comments. It will guide you step by step through entire idea of implementation.

Each example consists of three files:

Content

Codes (it'll send you to appropriate folder):


Experimental results (figures and tables on this page):


RL Q-Learning Environment-1. Experimental results

Environment-1 with mobile robot, goal and obstacles

RL_Q-Learning_E-1 RL_Q-Learning_E-1


Q-learning algorithm resulted chart for the environment-1

Represents number of episodes via number of steps and number of episodes via cost for each episode

RL_Q-Learning_C-1


Final Q-table with values from the final shortest route for environment-1

RL_Q-Learning_T-1
Looking at the values of the table we can see the decision for the next action made by agent (mobile robot). The sequence of final actions to reach the goal after the Q-table is filled with knowledge is the following: down-right-down-down-down-right-down-right-down-right-down-down-right-right-up-up.
During the experiment with Q-learning algorithm the found shortest route to reach the goal for the environment-1 consist of 16 steps and the found longest rout to reach the goal consists of 185 steps.


RL Q-Learning Environment-2. Experimental results

Bigger environment-2 with more obstacles

RL_Q-Learning_E-2


Q-learning algorithm resulted chart for the environment-2

Represents number of episodes via number of steps and number of episodes via cost for each episode

RL_Q-Learning_C-2


Final Q-table with values from the final shortest route for environment-1

RL_Q-Learning_T-2


RL Q-Learning Environment-3. Experimental results

Super complex environment-3 with a lot of obstacles

RL_Q-Learning_E-3


Comparison analysis of Q-Learning and Sarsa algorithms

RQ-learning_via_Sarsa


MIT License

Copyright (c) 2018 Valentyn N Sichkar

github.com/sichkar-valentyn

Reference to:

Valentyn N Sichkar. Reinforcement Learning Algorithms for global path planning // GitHub platform. DOI: 10.5281/zenodo.1317898