Deep Bilateral Learning for Real-Time Image Enhancements

Siggraph 2017

Visit our Project Page.

Michael Gharbi Jiawen Chen Jonathan T. Barron Samuel W. Hasinoff Fredo Durand

Maintained by Michael Gharbi (gharbi@mit.edu)

Tested on Python 2.7, Ubuntu 14.0, gcc-4.8.

Disclaimer

This is not an official Google product.

Setup

Dependencies

To install the Python dependencies, run:

cd hdrnet
pip install -r requirements.txt

Build

Our network requires a custom Tensorflow operator to "slice" in the bilateral grid. To build it, run:

cd hdrnet
make

To build the benchmarking code, run:

cd benchmark
make

Note that the benchmarking code requires a frozen and optimized model. Use hdrnet/bin/scripts/optimize_graph.sh and hdrnet/bin/freeze.py to produce these.

To build the Android demo, see dedicated section below.

Test

Run the test suite to make sure the BilateralSlice operator works correctly:

cd hdrnet
py.test test

Download pretrained models

We provide a set of pretrained models. One of these is included in the repo (see pretrained_models/local_laplacian_sample). To download the rest of them run:

cd pretrained_models
./download.py

Usage

To train a model, run the following command:

./hdrnet/bin/train.py <checkpoint_dir> <path/to_training_data/filelist.txt>

Look at sample_data/identity/ for a typical structure of the training data folder.

You can monitor the training process using Tensorboard:

tensorboard --logdir <checkpoint_dir>

To run a trained model on a novel image (or set of images), use:

./hdrnet/bin/run.py <checkpoint_dir> <path/to_eval_data> <output_dir>

To prepare a model for use on mobile, freeze the graph, and optimize the network:

./hdrnet/bin/freeze_graph.py <checkpoint_dir>
./hdrnet/bin/scripts/optimize_graph.sh <checkpoint_dir>

You will need to change the ${TF_BASE} environment variable in ./hdrnet/bin/scripts/optimize_graph.sh and compile the necessary tensorflow command line tools for this (automated in the script).

Android prototype

We will add it to this repo soon.

Known issues and limitations

The BilateralSliceApply operation is GPU only at this point. We do not plan on releasing a CPU implementation.
The provided pre-trained models were updated from an older version and might slightly differ from the models used for evaluation in the paper.
The pre-trained HDR+ model expects as input a specially formatted 16-bit linear input. In summary, starting from Bayer RAW:
1. Subtract black level.
2. Apply white balance channel gains.
3. Demosaic to RGB.
4. Apply lens shading correction (aka vignetting correction).
Our Android demo approximates this by undoing the RGB->YUV conversion and white balance, and tone mapping performed by the Qualcomm SOC. It results in slightly different colors than that on the test set. If you run our HDR+ model on an sRGB input, it may produce uncanny colors.