Echo-AI

Development to resume from June

Python package containing all mathematical backend algorithms used in Machine Learning. The full documentation for Echo is provided here.

About the Project
- Activation Functions
Repository Structure
Setup Instructions

About

Echo-AI Package is created to provide an implementation of the most promising mathematical algorithms, which are missing in the most popular deep learning libraries, such as PyTorch, Keras and TensorFlow.

Activation Functions

The package contains implementation for following activation functions (✅ - implemented functions, 🕑 - functions to be implemented soon, :white_large_square: - function is implemented in the original deep learning package):

#	Function	Equation	PyTorch	TensorFlow-Keras	TensorFlow - Core
1	Weighted Tanh	$weightedtanh(x) = tanh(x * weight)$	✅	✅	🕑
2	Swish	$SwishX(x, \beta) = xsigmoid(\betax) = \frac{x}{(1+e^{-\beta*x})}$	✅	✅	🕑
3	ESwish	$ESwish(x, \beta) = \betaxsigmoid(x)$	✅	✅	🕑
4	Aria2	$Aria2(x, \alpha, \beta) = (1+e^{-\beta*x})^{-\alpha}$	✅	✅	🕑
5	ELiSH	$ELiSH(x) = \left\{\begin{matrix} x / (1+e^{-x}), x \geq 0 \\ (e^{x} - 1) / (1 + e^{-x}), x < 0 \end{matrix}\right.$	✅	✅	🕑
6	HardELiSH	$HardELiSH(x) = \left\{\begin{matrix} x \times max(0, min(1, (x + 1) / 2)), x \geq 0 \\ (e^{x} - 1)\times max(0, min(1, (x + 1) / 2)), x < 0 \end{matrix}\right.$	✅	✅	🕑
7	Mila	$mila(x) = x * tanh(ln(1 + e^{\beta + x})) = x * tanh(softplus(\beta + x))$	✅	✅	🕑
8	SineReLU	$SineReLU(x, \epsilon) = \left\{\begin{matrix} x , x > 0 \\ \epsilon * (sin(x) - cos(x)), x \leq 0 \end{matrix}\right$	✅	✅	🕑
9	Flatten T-Swish	$FTS(x) = \left\{\begin{matrix} \frac{x}{1 + e^{-x}} , x \geq 0 \\ 0, x < 0 \end{matrix}\right.$	✅	✅	🕑
10	SQNL	$SQNL(x) = \left\{\begin{matrix} 1, x > 2 \\ x - \frac{x^2}{4}, 0 \leq x \leq 2 \\ x + \frac{x^2}{4}, -2 \leq x < 0 \\ -1, x < -2 \end{matrix}\right.$	✅	✅	🕑
11	ISRU	$ISRU(x) = \frac{x}{\sqrt{1 + \alpha * x^2}}$	✅	✅	🕑
12	ISRLU	$ISRLU(x)=\left\{\begin{matrix} x, x\geq 0 \\ x * (\frac{1}{\sqrt{1 + \alpha*x^2}}), x <0 \end{matrix}\right.$	✅	✅	🕑
13	Bent's identity	$bentId(x) = x + \frac{\sqrt{x^{2}+1}-1}{2}$	✅	✅	🕑
14	Soft Clipping	$SC(x) = 1 / \alpha * log(\frac{1 + e^{\alpha * x}}{1 + e^{\alpha * (x-1)}})$	✅	✅	🕑
15	SReLU	$SReLU(x_i) = \left\{\begin{matrix} t_i^r + a_i^r(x_i - t_i^r), x_i \geq t_i^r \\ x_i, t_i^r > x_i > t_i^l\\ t_i^l + a_i^l(x_i - t_i^l), x_i \leq t_i^l \\ \end{matrix}\right.$	✅	✅	🕑
15	BReLU	$BReLU(x_i) = \left\{\begin{matrix} f(x_i), i \mod 2 = 0\\ - f(-x_i), i \mod 2 \neq 0 \end{matrix}\right.$	✅	✅	🕑
16	APL	$APL(x_i) = max(0,x) + \sum_{s=1}^{S}{a_i^s * max(0, -x + b_i^s)}$	✅	✅	🕑
17	Soft Exponential	$SoftExponential(x, \alpha) = \left\{\begin{matrix} - \frac{log(1 - \alpha(x + \alpha))}{\alpha}, \alpha < 0\\ x, \alpha = 0\\ \frac{e^{\alpha * x} - 1}{\alpha} + \alpha, \alpha > 0 \end{matrix}\right.$	✅	✅	🕑
18	Maxout	$maxout(\vec{x}) = max_i(x_i)$	✅	✅	🕑
19	Mish	$mish(x) = x * tanh(ln(1 + e^x))$	✅	✅	🕑
20	Beta Mish	$\beta mish(x) = x * tanh(ln((1 + e^{x})^{\beta}))$	✅	✅	🕑
21	RReLU	$RReLU_j_i = \left\{\begin{matrix} x_j_i ,\if \ x_j_i\geq 0\\ a_j_i * x_j_i, \if \ x_j_i < 0 \end{matrix}\right. a_j_i \sim U(l, u), \l< u \and \ l, u \in [0,1)$	⬜	🕑	🕑
22	CELU	$CELU(x, \alpha) = \left\{\begin{matrix} x , x \geq 0\\ \alpha * (e^{\frac{x}{\alpha}} - 1), x < 0 \end{matrix}\right.$	⬜	✅	🕑
23	HardTanh	$HardTanh(x)=\left\{\begin{matrix} 1, x > 1\\ -1, x < -1\\ x, -1 \leq x \leq 1 \end{matrix}\right.$	⬜	✅	🕑
24	GLU	$GLU(a,b)=a \otimes \sigma(b), \otimes - element \ wise \ product$	⬜	🕑	🕑
25	LogSigmoid	$LogSigmoid(x)=log(\frac{1}{1 + e^{-x}})$	⬜	✅	🕑
26	TanhShrink	$TanhShrink(x)=x - Tanh(x)$	⬜	✅	🕑
27	HardShrink	$HardShrink(x)=\left\{\begin{matrix} x, x > \lambda \\ x, x < - \lambda \\ 0, - \lambda \leq x \leq \lambda \end{matrix}\right.$	⬜	✅	🕑
28	SoftShrink	$SoftShrinkage(x)=\left\{\begin{matrix} x - \lambda, x > \lambda \\ x + \lambda, x < - \lambda \\ 0, - \lambda \leq x \leq \lambda \end{matrix}\right.$	⬜	✅	🕑
29	SoftMin	$Softmin(x_i)=\frac{e^{-x_i}}{\sum _j e^{-x_j}}$	⬜	✅	🕑
30	LogSoftmax	$LogSoftmax(x_i)=log(\frac{e^{x_i}}{\sum _j e^{x_j}})$	⬜	✅	🕑
31	Gumbel-Softmax		⬜	🕑	🕑
32	LeCun's Tanh		✅	✅	🕑
33	TaLU		🕑	✅	🕑
34	SiLU		✅	✅	🕑
35	GELU		🕑	🕑	🕑
36	NReLU		🕑	🕑	🕑
37	CReLU		🕑	✅	🕑
38	ProbAct		🕑	🕑	🕑
39	Noisy Activation Function		🕑	🕑	🕑
40	NLReLU		✅	✅	🕑

Repository Structure

The repository has the following structure:

- echoAI # main package directory
| - Activation # sub-package containing activation functions implementation
| |- Torch  # sub-package containing implementation for PyTorch
| | | - functional.py # script which contains implementation of activation functions
| | | - weightedTanh.py # activation functions wrapper class for PyTorch
| | | - ... # PyTorch activation functions wrappers
| |- TF_Keras  # sub-package containing implementation for Tensorflow-Keras
| | | - custom_activation.py # script which contains implementation of activation functions
| - __init__.py

- Observations # Folder containing other assets

- docs # Sphinx documentation folder

- LICENSE # license file
- README.md
- setup.py # package setup file
- Scripts #folder, which contains the Black and Flake8 automated test scripts
- Smoke_tests # folder, which contains scripts with demonstration of activation functions usage
- Unit_tests # folder, which contains unit test scripts

Setup Instructions

To install echoAI package from PyPI run the following command:

$ pip install echoAI

Code Examples:

Sample scripts are provided in Smoke_tests folder.

PyTorch:

# import PyTorch
import torch

# import activation function from echoAI
from echoAI.Activation.Torch.mish import Mish

# apply activation function
mish = Mish()
t = torch.tensor(0.1)
t_mish = mish(t)

TensorFlow Keras:

#import tensorflow
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.layers import Dense, Flatten

# import activation function from echoAI
from echoAI.Activation.TF_Keras.custom_activation import Mish

model = tf.keras.Sequential([
    layers.Flatten(),
    layers.Dense(128, input_shape=(784,)),
    Mish(), # use the activation function
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')])

# Compile the model
model.compile(optimizer = "adam", loss = "mean_squared_error", metrics = ["accuracy"])

# Fit the model
model.fit(x = X_train, y = y_train, epochs = 3, batch_size = 128)