Simple File Server

Overview

Features

Mailing Lists

Latest release

The most recent release of sfs is release-1.20170106133707.

To run release-1.20170106133707

docker run ... pitchpointsolutions/simple-file-server:release-1.20170106133707

To run the latest release-1

docker pull pitchpointsolutions/simple-file-server:release-1
docker run ... pitchpointsolutions/simple-file-server:release-1

Snapshots

Snapshots of sfs are built from the master branch

To run the latest snapshot

docker pull pitchpointsolutions/simple-file-server:latest
docker run ... pitchpointsolutions/simple-file-server:latest

Quickstart on Linux

docker run -d -P --name sfs_example_elasticsearch elasticsearch:2.4.1 -Des.cluster.name=sfs_example_elasticsearch
HOSTNAME=`hostname` && export HOST_IP=`ping -c1 -n ${HOSTNAME} | head -n1 | sed "s/.*(\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)).*/\1/g"`;
export DOCKER_ES_PORT=`docker port sfs_example_elasticsearch 9300/tcp | sed -E 's/(.+):(.+)/\2/'`
docker run -d -P --add-host localhost:127.0.0.1 -e SFS_HTTP_LISTEN_ADDRESSES=0.0.0.0:80 -e SFS_HTTP_PUBLISH_ADDRESSES=127.0.0.1:80 -e SFS_REMOTENODE_SECRET=YWJjMTIzCg== -e SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com -e SFS_KEYSTORE_AWS_KMS_KEY_ID=arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab -e SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE -e SFS_KEYSTORE_AWS_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAWSEXAMPLEKEY -e SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://yourvaultname.vault.azure.net -e SFS_KEYSTORE_AZURE_KMS_KEY_ID=6603bbb5-cf2e-4367-8327-43ba49ba74b0 -e SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=a14970c2-397c-4af2-867e-b3480f9eaac6 -e SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAZUREEXAMPLEKEY -e SFS_ELASTICSEARCH_CLUSTER_NAME=sfs_example_elasticsearch -e SFS_ELASTICSEARCH_NODE_NAME=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_MULTICAST_ENABLED=false -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_ENABLED=true --detach --name sfs_example_middlware -P pitchpointsolutions/simple-file-server
export DOCKER_SFS_PORT=`docker port sfs_example_middlware 80/tcp | sed -E 's/(.+):(.+)/\2/'`
curl -v -XGET "http://localhost:${DOCKER_SFS_PORT}/admin/001/healthcheck"
curl -XPOST -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account"
curl -XPUT -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container"
curl -XPUT -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XGET -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object"

Quickstart using Docker Machine

docker run -d -P --name sfs_example_elasticsearch elasticsearch:2.4.1 -Des.cluster.name=sfs_example_elasticsearch
export HOST_IP=`docker-machine ip`;
export DOCKER_ES_PORT=`docker port sfs_example_elasticsearch 9300/tcp | sed -E 's/(.+):(.+)/\2/'`
docker run -d -P --add-host localhost:127.0.0.1 -e SFS_HTTP_LISTEN_ADDRESSES=0.0.0.0:80 -e SFS_HTTP_PUBLISH_ADDRESSES=127.0.0.1:80 -e SFS_REMOTENODE_SECRET=YWJjMTIzCg== -e SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com -e SFS_KEYSTORE_AWS_KMS_KEY_ID=arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab -e SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE -e SFS_KEYSTORE_AWS_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAWSEXAMPLEKEY -e SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://yourvaultname.vault.azure.net -e SFS_KEYSTORE_AZURE_KMS_KEY_ID=6603bbb5-cf2e-4367-8327-43ba49ba74b0 -e SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=a14970c2-397c-4af2-867e-b3480f9eaac6 -e SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAZUREEXAMPLEKEY -e SFS_ELASTICSEARCH_CLUSTER_NAME=sfs_example_elasticsearch -e SFS_ELASTICSEARCH_NODE_NAME=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_MULTICAST_ENABLED=false -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_ENABLED=true --detach --name sfs_example_middlware -P pitchpointsolutions/simple-file-server
export DOCKER_SFS_PORT=`docker port sfs_example_middlware 80/tcp | sed -E 's/(.+):(.+)/\2/'`
curl -v -XGET "http://${HOST_IP}:${DOCKER_SFS_PORT}/admin/001/healthcheck"
curl -XPOST -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account"
curl -XPUT -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container"
curl -XPUT -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XGET -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object"

Master Keys

Container Keys

Object Encryption

Metadata Replication

Object Data Replication

Backups

Volume Block Allocation

Access Control

Adding a new AuthProvide

  1. Create a new project or module and include sfs-server as a dependency (my-organization-sfs-server).
  2. Implement org.sfs.auth.AuthProvider and make priority() return a value higher than 0
  3. Create a file named my-organization-sfs-server/src/main/resources/META-INF/services/org.sfs.auth.AuthProvider
  4. Put the name of your implementation class into the my-organization-sfs-server/src/main/resources/META-INF/services/org.sfs.auth.AuthProvider file
  5. Using sfs-docker/main/docker/Dockerfile as an example create a new artifact (my-organization-sfs-docker) that uses your new server artifact (my-organization-sfs-server) as a dependency instead of sfs-server

Caveats

Building

Oracle Java 8 and Maven 3 should be used for building.

Environment Variables
SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com
SFS_KEYSTORE_AWS_KMS_KEY_ID=...
SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=$...
SFS_KEYSTORE_AWS_KMS_SECRET_KEY=...

SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://....vault.azure.net
SFS_KEYSTORE_AZURE_KMS_KEY_ID=...
SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=...
SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=...
Building, testing and assembling Docker image (from the root project directory)
mvn clean package
Building, testing (from the sfs-server directory)
mvn clean package    
Building, testing and regenerating the protobuf files (from the sfs-server directory)
mvn clean package -Pprotoc     

Running (Requires Elasticsearch 2.4)

Running the docker image you just built (See sample configuration and logback configuration)
docker run -it --add-host "es-host:${HOST_IP}" --add-host "localhost:127.0.0.1" -e "INSTANCES=200" -e "HEAP_SIZE=512m" -p 8092:80 -v ${PWD}:/data -v ${PWD}/sample-config.json:/etc/vertx-conf.json -v ${PWD}/sample-logback.xml:/etc/vertx-logback.xml pps-sfs:latest
Sample Configuration
{
    "auth": {
        "admin": [
            {
                "id": 1,
                "password": "admin",
                "username": "admin"
            }
        ],
        "user": [
            {
                "id": 2,
                "password": "user",
                "username": "user"
            }
        ]
    },
    "elasticsearch.cluster.name": "elasticsearch_samplecluster",
    "elasticsearch.defaultdeletetimeout": 30000,
    "elasticsearch.defaultgettimeout": 30000,
    "elasticsearch.defaultindextimeout": 30000,
    "elasticsearch.defaultsearchtimeout": 30000,
    "elasticsearch.discovery.zen.ping.multicast.enabled": false,
    "elasticsearch.discovery.zen.ping.unicast.enabled": true,
    "elasticsearch.discovery.zen.ping.unicast.hosts": [
        "es-host:9300"
    ],
    "elasticsearch.node.name": "simple-file-server-client",
    "elasticsearch.replicas": 0,
    "elasticsearch.shards": 1,
    "fs.home": "/data",
    "http.listen.addresses": [
        "0.0.0.0:80"
    ],
    "http.maxheadersize": 40960,
    "http.publish.addresses": [
        "${docker_image_ip_address}:8092"
    ],
    "keystore.aws.kms.access_key_id": "${aws_kms_access_key_id}",
    "keystore.aws.kms.endpoint": "https://kms.us-east-1.amazonaws.com",
    "keystore.aws.kms.key_id": "${aws_kms_key_id}",
    "keystore.aws.kms.secret_key": "${aws_kms_access_key_id}",
    "keystore.azure.kms.access_key_id": "${aws_azure_access_key_id}",
    "keystore.azure.kms.endpoint": "https://....vault.azure.net",
    "keystore.azure.kms.key_id": "${aws_azure_key_id}",
    "keystore.azure.kms.secret_key": "${aws_azure_secret_key}",
    "node.data": true,
    "node.master": true,
    "number_of_object_replicas": 1,
    "remotenode.connectimeout": 5000,
    "remotenode.maxpoolsize": 200,
    "remotenode.responsetimeout": 10000,
    "remotenode.secret": "YWJjMTIzCg==",
    "threadpool.background.size": 200,
    "threadpool.io.size": 200
}
Sample Logback Configuration
<?xml version="1.0" encoding="UTF-8" ?>

<configuration scan="true">

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned by default the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
        <encoder>
            <pattern>%d [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="org.apache" level="TRACE" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger> d

    <logger name="io.vertx" level="TRACE" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

    <logger name="io.netty" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

    <logger name="org.sfs" level="INFO" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

</configuration>

Accounts

Create/Update Account
curl -XPOST -u admin:admin "http://localhost:8092/openstackswift001/my_account"
Get Account Metadata
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account"
List Containers in Account
curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account"
Delete Account (if it's empty)
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account"    

Containers

Create a Container
curl -XPUT -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 3" "http://localhost:8092/openstackswift001/my_account/my-container"    
Get Container Metadata
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"
List Objects in Container
 curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container?format=xml&prefix=&limit=10000&delimiter=%2F"
List Objects in subfolder
curl -XGET -u admin:admin "http://localhost:8092/sfs/openstackswift001/my_account/my-container?format=xml&prefix=subfolder%2F&limit=10000&delimiter=%2F
Create a Container the by default encrypts objects and retains at most 3 object revisions
curl -XPUT -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 3" "http://localhost:8092/openstackswift001/my_account/my-container"
Create a Container and control the number of object index shards, the number of object index replicas and the number of object replicas
curl -XPUT -u admin:admin -H "x-sfs-object-index-shards: 12" -H "x-sfs-object-index-replicas: 2" -H "x-sfs-object-replicas: 2" "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container so that by default it encrypts objects and retains at most 2 object revisions
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 2" "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container so that by default it doesn't encrypt objects and retains at most 1 object revisions
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: false" -H "X-Container-Meta-Max-Object-Revisions: 1" "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container so that one object index replica in maintained for each object index shard and one each object replicated once
curl -XPOST -u admin:admin -H "x-sfs-object-index-replicas: 1" -H "x-sfs-object-replicas: 1" "http://localhost:8092/openstackswift001/my_account/my-container"
Delete a Container (if it's empty)
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"  

Object Storage

Upload and object
curl -XPUT -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object" -d 'abc123'
Upload and object and have it be stored encrypted
curl -XPUT -u admin:admin -H "X-Server-Side-Encryption: true" "http://localhost:8092/openstackswift001/my_account/my-container/my_object" -d 'abc123'
Get object metadata
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object"
Get object
curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object" 
Delete an object (will create delete marker)
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object" 
Really Delete and object
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=all"     
Really Delete and object version
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=1"   
Really Delete and multiple object version
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=1,2,3" 

Container Import and Export

Export a container
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" "http://localhost:8092/export_container/my_account/my-container"
Export a container and compress the export
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" -H "x-sfs-compress: true" "http://localhost:8092/export_container/my_account/my-container"   
Export a container, compress and encrypt the export
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" -H "x-sfs-compress: true" -H "x-sfs-secret: YWJjMTIzCg==" "http://localhost:8092/export_container/my_account/my-container"       
Import a container into a container named target_container
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" "http://localhost:8092/import_container/my_account/target_containers"    
Import an encrypted container into a container named target_container
curl -XPOST -u admin:admin -H "x-sfs-src-directory: /data/my-container_export" -H "x-sfs-secret: YWJjMTIzCg==" "http://localhost:8092/import_container/my_account/target_container"       

Health Check

Your load balancers in front of sfs should use this url to see if sfs is alive
curl -XGET -u admin:admin "http://localhost:8092/admin/001/healthcheck"

Misc. Administration

Verify/Repair an object
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects/my_account/my-container/my_object"
Execute Verify/Repair a container
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects/my_account/my-container"  
Wait for Verify/Repair of a container for 30 seconds
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects/my_account/my-container"      
Stop Verify/Repair of a container waiting 30 seconds for it to stop
curl -XDELETE -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects/my_account/my-container"     
Verify/Repair all container
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects"         
Wait for Verify/Repair of all container
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects"  
Stop for Verify/Repair of all container
curl -XDELTE -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects"              
Verify and repair master keys (if amazon web services or azure spontaneously vanish from the face of planet earth)
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_masterkeys" 
Wait for Verify and repair master keys (if amazon web services or azure spontaneously vanish from the face of planet earth)
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_masterkeys"         
Re Encrypt Container Keys if they're older than 30 days
curl -XPOST -u admin:admin "http://localhost:8092/reencrypt_containerkeys"   
Wait for Re Encrypt Container Keys if they're older than 30 days
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/reencrypt_containerkeys"     
Re Encrypt Master Keys if they're older than 30 days
curl -XPOST -u admin:admin "http://localhost:8092/reencrypt_masterkeys"     
Wait for Re Encrypt Master Keys if they're older than 30 days
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/reencrypt_masterkeys"
Get Object Index Entry
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_objects/my_account/my-container/my_object"
Get Container Index Entry
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_containers/my_account/my-container"
Get Account Index Entry
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_accounts/my_account"   

Road Map