NOTE this project has been renamed from discover_feature_relationships to beyond_correlation. Please check the updated examples. Pypi/Conda packages have not been created/updated.
Attempt to discover 1D relationships between all columns in a DataFrame using scikit-learn (RandomForests) and standard correlation tests (Pearson, Spearman and Kendall via Pandas).
The goal is to see if we can better understand the data in a DataFrame by learning which features (1 column at a time) predict each other column. This code attempts to learn a predictive relationship between the Cartesian product (all pairs) of all columns.
Rather than just learning which column(s) predict a target column, we might want to know what other relationships exist (e.g. during Exploratory Data Analysis) and whether some predictive features are driven by other less-predictive features (to help us find new & better features or data sources). We might also sense-check out data by checking that certain relationships exist.
By default it assumes every target column is a regression challenge. You can provide a list of columns to treat as classification challenges. For regression we cap negative scores at 0 (r^2 can be arbitrarily negative, we cap at 0 to make this a little easier to interpret).
Text-encoded columns are automatically LabelEncoded (this is a sensible default but may not reveal information in your case, you might need to provide your own smarter encoding). This adds to the correlation plots in YellowBrick and Pandas Profiling where the text columns are not auto-encoded.
We might want to use this tool alongside:
The project (and the examples) live on GitHub:
Information on the number of dropped columns per (feature, target) tuple can be returned upon request.
This is generated using:
df = pd.read_csv("titanic_train.csv")
...
import beyond_correlation as bc
df_results = bc.discover(cols, classifier_overrides, df)
df_results.pivot(index='target', columns='feature', values='score').fillna(1) \
.style.background_gradient(cmap="viridis", low=0.3, high=0.0, axis=1) \
.set_precision(2)conda install scikit-learn pandas jupyter pytest seaborn
conda install -c conda-forge watermarkThanks to the following colleagues for renaming the project, adding a third example and improving the code at the Man AHL 2019 hackathon:
pip install discover_feature_relationships
https://pypi.org/project/discover-feature-relationships/
First check-out from GitHub, then install with python setup.py install, then cd into the examples folder and run the Notebooks.
discover.py for a simple test that the code is working pytest to run test_discover.py for a single unit test (use pytest -s to see print outputs)These notes are for the Man AHL Hackathon 2019
# use the same environment as we used for the previous session
conda create -n hackathon_ipython_memory_usage python=3.7
conda activate hackathon_ipython_memory_usage
# note needed # conda install ipython numpy memory_profiler
conda install scikit-learn pandas jupyter seaborn watermark
mkdir hackathon_ipython_memory_usage
cd hackathon_ipython_memory_usage/
# FORK this repo to your own account
# and then UPDATE the line below for your own git checkout
git clone git@github.com:ianozsvald/beyond_correlation.git
# note "develop" and not the usual "install" here, to make the local folder editable!
python setup.py develop
Environment: . ~/anaconda3/bin/activate discover_feature_relationships
To push to PyPI I need to follow https://docs.python.org/3/distributing/index.html#distributing-index - specifically python se tup.py sdist bdist_wheel and twine upload dist/*. This uses https://pypi.org/project/twine/ .