morrigan/user-behavior-anomaly-detector

About

User anomaly detector based on logs generated by Osquery framework and machine learning to process those logs. Machine learnings alogirthms that are currently implemented are: Recurrent neural network - Long short-term memory (LSTM) and One-Class Support Vector Machine (OCSVM). This project is part of Master thesis at Faculty of electrical engineering and computing, University of Zagreb.

Note: Only Linux platforms are supported. Tested on Xubuntu 16.04.

The code includes:

dataset for different users to train and test models
preprocessing functions to properly tokenize the data
train and predict functions for existing (saved) data
training new model on raw osquery logs
saving and loading trained models
predict functions for incoming new data

Motivation

Existing security solutions are mostly based on preventing known malicious threats or a defined set of rules and therefore most outside and inside threats end as successful attacks. The idea was to build a system that is an adaptive user action identifier, so it can predict and detect anomalous behavior in real time.

Installation

osquery

Download osquery.
Move osquery pack of queries file user-behavior.conf and osquery configuration file osquery.conf from conf directory to osquery directory which is usually /etc/osquery/ or /usr/local/.

To capture syslog events additional configuration is required. (Ubuntu) Add the following to your rsyslog configuration files (usually located in /etc/rsyslog.conf or /etc/rsyslog.d/:

template(
name="OsqueryCsvFormat"
type="string"
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
)
*.* action(type="ompipe" Pipe="/var/osquery/syslog_pipe" template="OsqueryCsvFormat")

If you use syslog-ng or you can read about the configuration here. If no logs are available, read debugging suggestions.

Repository

Download the repository using git clone.
Install pip requirements with the following command: pip install -r requirements.txt. Python 2.7 is required. I don't guarantee that everything works with Python 3+ but please feel free to try.

How to use