Substra-Backend Build Status

Backend of the Substra platform

substra-backend celeryworker celerybeat flower
Docker Cloud Build Status Docker Cloud Build Status Docker Cloud Build Status Docker Cloud Build Status


You will note substra-backend use a versioned REST API with the header protocol. Current is 0.0.

Getting started 1: Prepare the django app

  1. Clone the repo:
    git clone
  2. If you are on Linux, follow the Linux user namespaces instructions
  3. Install dependencies (might be useful to create a virtual environment before, eg using virtualenv and virtualenvwrapper):
    • For numpy, scipy, and pandas (for Ubuntu & Debian users): sudo apt-get install python-numpy python-scipy python-pandas
    • pip install -r requirements.txt
  4. Setup the database:
  5. Create a main postgresql use with password:
    $> sudo su postgres
    $> psql
  6. Create two databases for both orgs: owkin and chu-nantes. A shell script is available, do not hesitate to run it. It will drop the databases if they are already created, then create them and grant all privileges to your main user backend. (If this is the first time you create the databases, you will see some warnings which are pointless):

    $> ./scripts/
  7. We will populate data:
Clean environment (recommanded)
With fixtures (fixtures container has been run from hlf-k8s, old behavior for testing)

data in fixtures are relative to the data already set in the ledger if the fixtures container instance succeeded

Two solutions:

Populate media files

  $> ./scripts/

It will clean the medias folders and create the owkin and chu-nantes folders in the medias folder.

  1. Optional: Create a superuser in your databases:
    BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 ./backend/ createsuperuser
    BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 ./backend/ createsuperuser

Getting started 2: Linking the app with Hyperledger Fabric

Make the hlf-k8s available to the app

See here.

Install rabbitmq

sudo apt-get install rabbitmq-server

Launch celery workers/scheduler and celery beat

Execute this command in the backend/backend folder.

Note the use of the development settings. BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E -A backend worker -l info -B -n owkin -Q owkin,scheduler,celery --hostname owkin.scheduler BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E -A backend worker -l info -B -n owkin -Q owkin,owkin.worker,celery --hostname owkin.worker BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E -A backend worker -l info -B -n chunantes -Q chu-nantes,scheduler,celery --hostname chu-nantes.scheduler BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E -A backend worker -l info -B -n chunantes -Q chu-nantes,chu-nantes.worker,celery --hostname chu-nantes.worker
DJANGO_SETTINGS_MODULE=backend.settings.common celery -A backend beat -l info

Launch the servers

Go in the backend folder and run the server locally: :warning:

Be very careful, --settings is different here, server is needed.


Generate nodes authentication

For working with node to node authentication, you need to generate and then load some fixtures

python ./backend/node/
BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 ./ init_nodes ./backend/node/nodes/owkinMSP.json
BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 ./ init_nodes ./backend/node/nodes/chu-nantesMSP.json

Create a default user

A django admin command is available for registering a user:


The file will use for each organization credentials substra/p@$swr0d44 for connection. Create these users with:

BACKEND_ORG=owkin ./backend/ add_user substra 'p@$swr0d44'
BACKEND_ORG=chu-nantes ./backend/ add_user substra 'p@$swr0d44'

Test with unit and functional tests

    make test
    make coverage  # For shell report
    coverage html  # For HTML report

Test by creating a traintuple

Clean environment

Run the script which will create data in the ledger with a traintuple. dependencies are defined in backend/requirements-dev.txt. Check the status of the created traintuple http://localhost:8000/traintuple/ If everything run correctly, its status should pas from todo->train->trained->testing->tested->done.

When you want to re-run the testing process:

With fixtures

You can test your environment by creating a traintuple:

curl -H "Accept: text/html;version=0.0, */*;version=0.0" -H "Content-Type: application/json" -d '{"algo_key":"da58a7a29b549f2fe5f009fb51cce6b28ca184ec641a0c1db075729bb266549b","model_key":"","train_data_sample_keys":["62fb3263208d62c7235a046ee1d80e25512fe782254b730a9e566276b8c0ef3a","42303efa663015e729159833a12ffb510ff92a6e386b8152f90f6fb14ddc94c9"]}' -X POST http://localhost:8001/traintuple/?format=json

It will try to create a traintuple with creator: chu-nantes (localhost:8001). The chu-nantes celery worker will try to add the traintuple to the ledger. You can check your traintuple has been corectly added by visiting http://localhost:8000/traintuple/ or http://localhost:8001/traintuple/ As the trainDataSample passed are also created by chu-nantes, the chu-nantes celery worker will try to update the traintuple status to training and save some data if needed. You can check http://localhost:8000/traintuple/ or http://localhost:8001/traintuple/ to check if the status of your newly created traintuple is set to training after 10 sec (celery worker periodic task period). You can also check a new objective has been added in medias/chu-nantes with a file but not file.

Launching with docker

As for hlf-k8s, you can launch all the services in docker containers.

First, Make sure you've generated some nodes artifacts:

$> python ./backend/node/

Then, build the images:

$> sh

Then, go to the docker dir and run (-d means dev settings):

$> python -d --no-backup

Check your services are correctly started with docker ps -a.

Expiry token period

Two global environment variables ACCESS_TOKEN_LIFETIME and EXPIRY_TOKEN_LIFETIME expressed in minutes can be set for dealing with expiry token period. The first one ACCESS_TOKEN_LIFETIME deals with JWT Authentication. THe second one EXPIRY_TOKEN_LIFETIME deals with simple token expiration. By default, set to 24*60 min i.e 24h.

Testing fabric-sdk-py

A directory named fabric-sdk-py_tests is available to the root of this project. If you launch a hlf-k8s setup, you will be able to play with theses tests. For, be sure to have run the script for producing the network.json file needed.



This project is developed under the Apache License, Version 2.0 (Apache-2.0), located in the LICENSE file.