EXIST (EXternal Information aggregation System against cyber Threat)

EXIST is a web application for aggregating and analyzing CTI (cyber threat intelligence).

EXIST is written by the following software.

Concept

EXIST is a web application for aggregating CTI to help security operators investigate incidents based on related indicators.

EXIST automatically fetches data from several CTI services and Twitter via their APIs and feeds. You can cross-search indicators via the web interface and the API.

If you have servers logging network behaviors of clients (e.g., logs of DNS and HTTP proxy servers, etc.), you will be able to analyze the logs by correlating with data on EXIST. If you implement some programs by using the API, you will realize automated CTI-driven security operation center.

overview

Use Cases

Case1: Investigate domain detected by IDS

Just type domain in the search form.

lookup_domain

Case2: Access the malicious URL on behalf of the user and acquire the display image of the browser and the contents to be downloaded

Just type url in the search form.

lookup_url

Case3: Monitor cyber threats

Just add keywords in the Threat Hunter or Twitter Hunter.

twitter_hunter

Features

Tracker

Tracker automatically collects data feeds from several CTI services.

Hunter

Hunter enables us to set queries for gathering data from several CTI services and Twitter.

Lookup

Lookup retrieves information related to specific information (e.g. IP address, domain) from several internet services (e.g. whois).

Web API

Provide data stored in the EXIST database by Web API.

Getting started

After that I assume the environment of CentOS 7 or Ubuntu 18.04 LTS. Please at your own when deploying to other environment.

Install python modules

$ sudo pip install -r requirements.txt

Install MariaDB

Run database

$ sudo systemctl start mariadb
$ sudo systemctl enable mariadb

Database setting

Migrate database

$ python manage.py makemigrations exploit reputation threat threat_hunter twitter twitter_hunter news news_hunter vuln
$ python manage.py migrate

Install Redis server

Reputation tracker uses redis as the Celery cache server backend.

Setup Celery

Reputation tracker uses Celery as an asynchronous task job queue.

# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/path/to/your/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="intelligence"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=YOUR_USER
Group=YOUR_GROUP
EnvironmentFile=/etc/sysconfig/celery
WorkingDirectory=/path/to/your/exist
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target
$ sudo mkdir /var/log/celery; sudo chown YOUR_USER:YOUR_GROUP /var/log/celery
$ sudo mkdir /var/run/celery; sudo chown YOUR_USER:YOUR_GROUP /var/run/celery
#Type  Path               Mode  UID        GID         Age  Argument
d      /var/run/celery    0755  YOUR_USER  YOUR_GROUP  -
$ sudo systemctl start celery.service
$ sudo systemctl enable celery.service

Run web server

$ python manage.py runserver 0.0.0.0:8000

Note: I recommend to use Nginx and uWSGI when running in production environment.

Collect feed

Scripts for inserting feed into database are scripts/insert2db/*/insert2db.py.

Configure insert2db

Run scripts

$ python scripts/insert2db/reputation/insert2db.py
$ python scripts/insert2db/twitter/insert2db.py
$ python scripts/insert2db/exploit/insert2db.py
$ python scripts/insert2db/threat/insert2db.py
$ python scripts/insert2db/news/insert2db.py
$ python scripts/insert2db/vuln/insert2db.py

Note: To automate information collection, write them to your cron.

Setting hunter

Twitter Hunter

Twitter Hunter can detect tweets containing specific keywords and user ID. And you can notify slack if necessary.

Threat Hunter

Threat Hunter can detect threat events containing specific keywords. And you can notify slack if necessary.

Other requirement tools and settings

VirusTotal API

EXIST uses VirusTotal API.

Note: You get more information if you have private API key.

GeoIP DB

Lookup IP / Domain uses GeoLite2 Database.

wkhtmltopdf and Xvfb

Lookup URL uses wkhtmltopdf and Xvfb.

$ sudo yum install xorg-x11-server-Xvfb

If you deploy EXIST on Ubuntu 18.04 LTS, you can install these packages by using apt.

$ sudo apt install wkhtmltopdf xvfb

Flush old data

Report vulnerability

Please e-mail us if you find any vulnerabilities.

Vulnerability Report Template

Credits

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

License

MIT License © Cybersecurity Laboratory, NICT