PyWren for IBM Cloud

What is PyWren

PyWren is an open source project whose goals are massively scaling the execution of Python code and its dependencies on serverless computing platforms and monitoring the results. PyWren delivers the user’s code into the serverless platform without requiring knowledge of how functions are invoked and run.

PyWren and IBM Cloud

This repository is based on PyWren main branch and adapted for IBM Cloud Functions and IBM Cloud Object Storage. PyWren for IBM Cloud is not, however, just a mere reimplementation of PyWren’s API atop IBM Cloud Functions. Rather, it is must be viewed as an advanced extension of PyWren. See the complete design overview.

PyWren for IBM Cloud provides great value for the variety of uses cases, like processing data in object storage, running embarrassingly parallel compute jobs (e.g. Monte-Carlo simulations), enriching data with additional attributes and many more. In extending PyWren to work with IBM Cloud Object Storage, we also added a partition discovery component that allows PyWren to process large amounts of data stored in the IBM Cloud Object Storage. See changelog for more details.

IBM Cloud for Academic institutions

IBM Academic Initiative is a special program that allows free trial of IBM Cloud for Academic institutions. This program is provided for students and faculty staff members, and allow up to 12 months of free usage. You can register your university email and get a free of charge account.

Initial Requirements

PyWren Setup

  1. Install PyWren from the PyPi repository:

    $ pip install pywren-ibm-cloud
  2. Navigate into config/ and follow the instructions to configure PyWren.

  3. Test PyWren by simply running the next command:

    $ pywren-ibm-cloud test

    or by running the next code:

    import pywren_ibm_cloud as pywren
    
    def hello(name):
       return 'Hello {}!'.format(name)
    
    pw = pywren.ibm_cf_executor()
    pw.call_async(hello, 'World')
    print(pw.get_result())

How to use PyWren for IBM Cloud

The primary object in PyWren is the executor. The standard way to get everything set up is to import pywren_ibm_cloud, and call one of the available functions to get a ready-to-use executor.

The available executors are:

The available calls within an executor are:

API Call Type Description
call_async() Async. Method used to spawn one function activation
map() Async. Method used to spawn multiple function activations
map_reduce() Async. Method used to spawn multiple function activations with one (or multiple) reducers
wait() Sync. Wait for the function activations to complete. It blocks the local execution until all the function activations finished their execution (configurable)
get_result() Sync. Method used to retrieve the results of all function activations. The results are returned within an ordered list, where each element of the list is the result of one activation
plot() Sync. Method used to create execution plots
clean() Async. Method used to clean the temporary data generated by PyWren

Additional information and examples:

Verify - Unit Testing

To test that all is working, use the command:

$ pywren-ibm-cloud verify

or

$ python -m pywren_ibm_cloud.tests

Notice that if you didn't set a local PyWren's config file, you need to provide it as a json file path by -c <CONFIG> flag.

Alternatively, for debugging purposes, you can run specific tests by -t <TESTNAME>. use --help flag to get more information about the test script.

Additional resources

Acknowledgements

image

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184.