apls777/spotty

Project: spotty (GitHub Link)

spotty-master
- .github
  - FUNDING.yml
- spotty
  - providers
    - abstract_instance_manager.py
    - gcp
      - helpers
        rtc_client.py
        deployment.py
        dm_resource.py
        gsutil.py
        sync.py
        ssh_key.py
        gcp_credentials.py
        dm_client.py
        __init__.py
        ce_client.py
        gs_client.py
      - commands
        create_image.py
        delete_image.py
        __init__.py
      - instance_manager.py
      - config
        image_url.py
        validation.py
        __init__.py
        instance_config.py
      - deployment
        image_deployment.py
        abstract_gcp_deployment.py
        project_resources
        image_stack.py
        disk_volume.py
        bucket.py
        __init__.py
        instance_stack.py
        instance_deployment.py
        dm_templates
        image_template.py
        instance
        cloud_init.yaml
        template.yaml
        instance_template.py
        __init__.py
        image
        cloud_init.yaml
        template.yaml
        __init__.py
      - __init__.py
      - errors
        image_not_found.py
        __init__.py
      - gcp_resources
        snapshot.py
        image.py
        disk.py
        instance.py
        __init__.py
        stack.py
    - test
      - deployment
        test_volume.py
        __init__.py
      - __init__.py
    - __init__.py
    - instance_manager_factory.py
    - aws
      - helpers
        spot_prices.py
        download.py
        aws_cli.py
        sync.py
        __init__.py
      - commands
        delete_ami.py
        spot_prices.py
        create_ami.py
        clean_logs.py
        __init__.py
      - aws_resources
        snapshot.py
        image.py
        volume.py
        instance.py
        subnet.py
        __init__.py
        stack.py
        vpc.py
      - instance_manager.py
      - config
        validation.py
        __init__.py
        instance_config.py
      - deployment
        ami_deployment.py
        abstract_aws_deployment.py
        project_resources
        ami_stack.py
        key_pair.py
        bucket.py
        instance_profile_stack.py
        __init__.py
        ebs_volume.py
        instance_stack.py
        instance_deployment.py
        cf_templates
        instance_profile_template.py
        ami_template.py
        data
        instance.yaml
        lambdas
        set_logs_retention.js
        create_ami
        search_ami.js
        delete_ami.js
        ami.yaml
        instance_profile.yaml
        instance_template.py
        __init__.py
        __init__.py
        checks.py
      - __init__.py
      - errors
        volume_not_found.py
        __init__.py
  - helpers
    - run.py
    - ssh.py
    - print_info.py
    - commands.py
    - __init__.py
  - commands
    - run.py
    - start.py
    - ssh.py
    - abstract_provider_command.py
    - stop.py
    - download.py
    - abstract_command.py
    - sync.py
    - writers
      - null_output_writrer.py
      - abstract_output_writrer.py
      - __init__.py
      - output_writrer.py
    - aws.py
    - gcp.py
    - abstract_config_command.py
    - __init__.py
    - status.py
  - config
    - project_config.py
    - container_config.py
    - validation.py
    - __init__.py
    - abstract_instance_config.py
  - deployment
    - container_deployment.py
    - abstract_instance_volume.py
    - __init__.py
  - __init__.py
  - configuration.py
  - utils.py
  - errors
    - instance_not_running.py
    - __init__.py
- LICENSE
- CONTRIBUTING.md
- setup.py
- setup.cfg
- README.md
- tests
  - providers
    - gcp
      - config
        image_url.py
        __init__.py
    - __init__.py
    - aws
      - project_resources
        key_pair.py
        bucket.py
        __init__.py
      - config
        instance_config_validation.py
        __init__.py
      - __init__.py
  - container_deployment.py
  - container_config.py
  - __init__.py
- bin
  - spotty
- .gitignore
- docs
  - safari-pinned-tab.svg
  - _config.yml
  - Gemfile
  - _includes
    - head
      - custom.html
    - nav.html
    - head.html
  - CNAME
  - index.md
  - _layouts
    - default.html
  - browserconfig.xml
  - favicon.ico
  - 404.html
  - site.webmanifest
  - .gitignore
  - docs
    - faq.md
    - _dev_docs
      - gcp
        image_creation.md
    - configuration.md
    - gcp
      - account_preparation.md
      - volumes.md
      - instance_parameters.md
      - gcp.md
      - commands.md
    - commands.md
    - aws
      - aws.md
      - volumes.md
      - instance_parameters.md
      - commands.md
  - assets
    - images
    - js
      - search-data.json

PyPI - Python Version PyPI - License

Spotty drastically simplifies training of deep learning models on AWS and GCP:

it makes training on GPU instances as simple as training on your local machine
it automatically manages all necessary cloud resources including images, volumes, snapshots and SSH keys
it makes your model trainable in the cloud by everyone with a couple of commands
it uses tmux to easily detach remote processes from their terminals
it saves you up to 70% of the costs by using AWS Spot Instances and GCP Preemtible VMs

Documentation

See the documentation page.
Read this article on Medium for a real-world example.

Installation

Requirements:

Python >=3.5
AWS CLI (see Installing the AWS Command Line Interface) if you're using AWS
Google Cloud SDK (see Installing Google Cloud SDK) if you're using GCP

Use pip to install or upgrade Spotty:

$ pip install -U spotty

Get Started

Prepare a spotty.yaml file and put it to the root directory of your project:
- See the file specification here.
- Read this article for a real-world example.
Start an instance:
```
$ spotty start
```
It will run a Spot Instance, restore snapshots if any, synchronize the project with the running instance and start the Docker container with the environment.
Train a model or run notebooks.

To connect to the running container via SSH, use the following command:
```
$ spotty ssh
```
It runs a tmux session, so you can always detach this session using Ctrl + b, then d combination of keys. To be attached to that session later, just use the spotty ssh command again.

Also, you can run your custom scripts inside the Docker container using the spotty run <SCRIPT_NAME> command. Read more about custom scripts in the documentation: Configuration: "scripts" section.

Contributions

Any feedback or contributions are welcome! Please check out the guidelines.

License