PyWren for IBM Cloud
PyWren is an open source project whose goals are massively scaling the execution of Python code and its dependencies on serverless computing platforms and monitoring the results. PyWren delivers the user’s code into the serverless platform without requiring knowledge of how functions are invoked and run.
This repository is based on PyWren main branch and adapted for IBM Cloud Functions and IBM Cloud Object Storage. PyWren for IBM Cloud is not, however, just a mere reimplementation of PyWren’s API atop IBM Cloud Functions. Rather, it is must be viewed as an advanced extension of PyWren. See the complete design overview.
PyWren for IBM Cloud provides great value for the variety of uses cases, like processing data in object storage, running embarrassingly parallel compute jobs (e.g. Monte-Carlo simulations), enriching data with additional attributes and many more. In extending PyWren to work with IBM Cloud Object Storage, we also added a partition discovery component that allows PyWren to process large amounts of data stored in the IBM Cloud Object Storage. See changelog for more details.
IBM Academic Initiative is a special program that allows free trial of IBM Cloud for Academic institutions. This program is provided for students and faculty staff members, and allow up to 12 months of free usage. You can register your university email and get a free of charge account.
Install PyWren from the PyPi repository:
$ pip install pywren-ibm-cloudNavigate into config/ and follow the instructions to configure PyWren.
Test PyWren by simply running the next command:
$ pywren-ibm-cloud testor by running the next code:
import pywren_ibm_cloud as pywren
def hello(name):
return 'Hello {}!'.format(name)
pw = pywren.ibm_cf_executor()
pw.call_async(hello, 'World')
print(pw.get_result())The primary object in PyWren is the executor. The standard way to get everything set up is to import pywren_ibm_cloud, and call one of the available functions to get a ready-to-use executor.
The available executors are:
ibm_cf_executor(): IBM Cloud Functions executor.knative_executor(): Knative executor. See additional information.openwhisk_executor(): Vanilla OpenWhisk executor. See additional information.local_executor(): Localhost executor to run functions by using local processes.docker_executor(): Docker executor to run functions by using processes within a local or remote Docker container.function_executor(): Generic executor based on the compute backend specified in configuration.The available calls within an executor are:
| API Call | Type | Description |
|---|---|---|
| call_async() | Async. | Method used to spawn one function activation |
| map() | Async. | Method used to spawn multiple function activations |
| map_reduce() | Async. | Method used to spawn multiple function activations with one (or multiple) reducers |
| wait() | Sync. | Wait for the function activations to complete. It blocks the local execution until all the function activations finished their execution (configurable) |
| get_result() | Sync. | Method used to retrieve the results of all function activations. The results are returned within an ordered list, where each element of the list is the result of one activation |
| plot() | Sync. | Method used to create execution plots |
| clean() | Async. | Method used to clean the temporary data generated by PyWren |
Additional information and examples:
To test that all is working, use the command:
$ pywren-ibm-cloud verifyor
$ python -m pywren_ibm_cloud.testsNotice that if you didn't set a local PyWren's config file, you need to provide it as a json file path by -c <CONFIG> flag.
Alternatively, for debugging purposes, you can run specific tests by -t <TESTNAME>. use --help flag to get more information about the test script.
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184.
