Insight_Project_Framework

Framework for machine learning projects at Insight Data Science.

Motivation for this project format:

Insight_Project_Framework : Put all source code for production within structured directory
tests : Put all source code for testing in an easy to find location
configs : Enable modification of all preset variables within single directory (consisting of one or many config files for separate tasks)
data : Include example a small amount of data in the Github repository so tests can be run to validate installation
build : Include scripts that automate building of a standalone environment
static : Any images or content to include in the README or web framework if part of the pipeline

Setup

Clone repository and update python path

repo_name=Insight_Project_Framework # URL of your new repository
username=mrubash1 # Username for your personal github account
git clone https://github.com/$username/$repo_name
cd $repo_name
echo "export $repo_name=${PWD}" >> ~/.bash_profile
echo "export PYTHONPATH=$repo_name/src:${PYTHONPATH}" >> ~/.bash_profile
source ~/.bash_profile

Create new development branch and switch onto it

branch_name=dev-readme_requisites-20180905 # Name of development branch, of the form 'dev-feature_name-date_of_creation'}}
git checkout -b $branch_name

Initial Commit

Lets start with a blank slate: remove .git and re initialize the repo

cd $repo_name
rm -rf .git   
git init   
git status

You'll see a list of file, these are files that git doesn't recognize. At this point, feel free to change the directory names to match your project. i.e. change the parent directory Insight_Project_Framework and the project directory Insight_Project_Framework: Now commit these:

git add .
git commit -m "Initial commit"
git push origin $branch_name

Requisites

List all packages and software needed to build the environment
This could include cloud command line tools (i.e. gsutil), package managers (i.e. conda), etc.

Dependencies

Streamlit

Installation

To install the package above, pleae run:

pip install -r requiremnts

Build Environment

Include instructions of how to launch scripts in the build subfolder
Build scripts can include shell scripts or python setup.py files
The purpose of these scripts is to build a standalone environment, for running the code in this repository
The environment can be for local use, or for use in a cloud environment
If using for a cloud environment, commands could include CLI tools from a cloud provider (i.e. gsutil from Google Cloud Platform)
```
# Example
```

Step 1

Step 2


## Configs
- We recommond using either .yaml or .txt for your config files, not .json
- **DO NOT STORE CREDENTIALS IN THE CONFIG DIRECTORY!!**
- If credentials are needed, use environment variables or HashiCorp's [Vault](https://www.vaultproject.io/)

## Test
- Include instructions for how to run all tests after the software is installed

Example

Step 1

Step 2


## Run Inference
- Include instructions on how to run inference
- i.e. image classification on a single image for a CNN deep learning project

Example

Step 1

Step 2


## Build Model
- Include instructions of how to build the model
- This can be done either locally or on the cloud

Example

Step 1

Step 2


## Serve Model
- Include instructions of how to set up a REST or RPC endpoint
- This is for running remote inference via a custom model

Example

Step 1

Step 2


## Analysis
- Include some form of EDA (exploratory data analysis)
- And/or include benchmarking of the model and results