Python Data Streams

Build Status GitHub issues Coverage PyPI PyPI Docs

Tributary is a library for constructing dataflow graphs in python. Unlike many other DAG libraries in python (airflow, luigi, prefect, dagster, dask, kedro, etc), tributary is not designed with data/etl pipelines or scheduling in mind. Instead, tributary is more similar to libraries like mdf, pyungo, streamz, or pyfunctional, in that it is designed to be used as the implementation for a data model. One such example is the greeks library, which leverages tributary to build data models for options pricing.

Installation

Install from pip:

pip install tributary

or from source

python setup.py install

Stream Types

Tributary offers several kinds of streams:

Streaming

These are synchronous, reactive data streams, built using asynchronous python generators. They are designed to mimic complex event processors in terms of event ordering.

Functional

These are functional streams, built by currying python functions (callbacks).

Lazy

These are lazily-evaluated python streams, where outputs are propogated only as inputs change. They are implemented as directed acyclic graphs.

Examples

Graph Visualization

You can visualize the graph with Graphviz. All streaming and lazy nodes support a graphviz method.

Streaming and lazy nodes also support ipydagred3 for live update monitoring.

Streaming

Here green indicates executing, yellow indicates stalled for backpressure, and red indicates that StreamEnd has been propogated (e.g. stream has ended).

Lazy

Here green indicates executing, and red indicates that the node is dirty. Note the the determination if a node is dirty is also done lazily (we can check with isDirty whcih will update the node's graph state.

Sources and Sinks

Sources

Sinks

Transforms

Modulate

Calculations

Arithmetic Operators

Boolean Operators

Comparators

Math

Converters

Python Builtins

Rolling

Node Type Converters