This project allows numerical features to be reduced down to fewer dimensions for plotting using unsupervised machine learning. Features can be taken simply as face value numbers from a spreadsheet (csv) file, or they can be extracted from images using a pre-trained model.
All functions in this package can be imported for use in your own python scripts, or run as stand-alone commands in a CLI.
In order to deal with all inputs in a standardardised fashion, csv files are parsed using
While this is done automatically for CLI commands, if you're writing your own scripts you should parse your csv data in
through this first. It essentially puts your data in a pd.DataFrame, where the first column is always a unique ID key column.
python cli.py features <image-or-directory-of-images> <output-csv-path>
python cli.py tsne <image-or-directory-of-images> <output-csv-path> <feature-cols> <unique-col>
python cli.py umap <image-or-directory-of-images> <output-csv-path> <feature-cols> <unique-col>
Running 'features' will extract the numerical features of a directory of images, and save them (with the unique IDs) to the output path.
Running 'tsne' or 'umap' will reduce such features (or features from a regular csv) into fewer dimensions, and save these (with the unique IDs) to the output path.
These reduction functions will accept a
--model argument, allowing you to specify one of several common pre-trained models to be used. I will probably add a command
to specify your own custom model soon.
It is worth reiterating:
As stated in the above, it is imperetive you parse any data you want to reduce using
parse_data first if you're accessing these functions in your own scripts, and not using the CLI.
This project uses keras and tensorflow 2. I run these commands using tensorflow-gpu, so for the smoothest experience I recommend setting CUDA up from NVIDIA's website, I have not tested regular tensorflow (cpu).
I have some folder full of images at path
./images that I want extract features from, and reduce into 2 dimensions using t-SNE.
python cli.py features "./images" "features.csv" python cli.py tsne "features.csv" "tsne-results.csv" --feature-cols all --unique-col A
from features import extract_features from tsne_reducer import tsne features = extract_features('./images') reduced = tsne(features, write_to='./tsne_features.csv')
I have a csv file called
data.csv containing a unique ID column (called "name") at column D, and the important columns containing numbers are at A, C, H, and AB.
python cli.py umap "./data.csv" "./tsne.csv" --feature-cols A,C,H,AB --unique-col D
from parse_data import parse_data from umap_reducer import umap data = parse_data('./data.csv', feature_cols=['A', 'C', 'H', 'AB'], unique_col='D') reduced = umap(data, write_to='./umap_features.csv')