HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data

Build Status


HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object based storage such as AWS S3, Azure Blob Storage, or OpenIO . HSDS can be run a single machine using Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure) The commercial offering based on this code is known as Kita™. More info at: https://www.hdfgroup.org/solutions/hdf-kita/.


Other useful resources

Quick Start

Make sure you have Python 3, docker, docker-compose installed, then:

  1. Setup password file: $ cp admin/config/passwd.template admin/config/passswd.txt
  2. Start server: $ ./runall.sh
  3. To run test suite, set an environment variable for the admin password: $ export ADMIN_PASSWORD=admin
  4. Run the test suite: $ ./testall.sh

To shut down the server, run: $ ./stopall.sh

Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image.

Detailed Install Instructions


See: docs/docker_install_aws.md for complete install instructions.

See: docs/kubernetes_install.md for setup on Kubernetes.

On Azure (BETA)

For complete instructions to install on a single Azure VM:

For complete instructions to install on Azure Kubernetes Service (AKS):


For complete instructions to install on DCOS:

On Prem (POSIX-based storage) (BETA)

For complete instructions to install on a desktop or local server:

Writing Client Applications

As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).

The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.

For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest.


HSDS only modifies the storage location that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket/Azure Container/directory files.

Reporting bugs (and general feedback)

Create new issues at http://github.com/HDFGroup/hsds/issues for any problems you find.

For general questions/feedback, please use the Kita™ forum: https://forum.hdfgroup.org/c/kita.


HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.

Integration with JupyterHub

The HDF Group provides access to an HSDS instance that is integrated with JupyterLab: Kita™ Lab. Kita™ Lab is a hosted Jupyter environment with these features:

Sign up for Kita™ Lab here: https://www.hdfgroup.org/hdfkitalab/.

AWS Marketplace

The HDF Group provides an AWS Marketplace product, Kita™ Server, which provides simple installation of HSDS and related AWS resources. Kita™ offers these features:

Kita™ Server for AWS Marketplace can be found here: https://aws.amazon.com/marketplace/pp/B07K2MWS1G.