Project: numerflow (GitHub Link)

numerflow

Data workflows for the numer.ai machine learning competition

Tasks

Currently implemented:

fetch and extract the datasets
train and predict
automatic upload

Task Documentation

`FetchAndExtractData`

Fetches the dataset zipfile and extracts the contents to output-path.

Parameters

output-path: where the datasets should be saved eventually (defaults to ./data/)
dataset-path: URI of the remote dataset

`TrainAndPredict`

Trains a Bernoulli Naïve Bayes classifier and predicts the targets. Output file is saved at output-path with a custom, timestamped file name.

Parameters

output-path: where the datasets should be saved eventually (defaults to ./data/)
dataset-path: URI of the remote dataset

`UploadPredictions`

Uploads the predictions of not already uploaded.

Parameters

output-path: where the datasets should be saved eventually (defaults to ./data/)
dataset-path: URI of the remote dataset
usermail: user email
userpass: user password
filepath: path to the file ought to be uploaded

Usage

Prepare the project:

pip install -r requirements.txt --ignore-installed

If not alread done create an API key here with at least the following permissions:

Upload submissions.
View historical submission info.
View user info, (e.g. balance, withdrawal history)

To run the complete pipeline:

env PYTHONPATH='.' luigi --local-scheduler --module workflow Workflow --secret="YOURSECRET" --public-id="YOURPUBLICID"