Python Projects Related with spark

1. TensorFlowOnSpark

under Apache License 2.0 license
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
2. haipproxy

under MIT License license
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
3. spark-deep-learning

under Apache License 2.0 license
Deep Learning Pipelines for Apache Spark
4. MusicBox

under MIT License license
:blush: :musical_note: MusicPlayer 一站式收听多平台音乐(网易云, 虾米, QQ)的跨平台音乐播放器，尽情享受吧~:sparkles:
5. elephas

under MIT License license
Distributed Deep learning with Keras & Spark
6. koalas

under Apache License 2.0 license
Koalas: pandas API on Apache Spark
7. spark-sklearn

under Apache License 2.0 license
(Deprecated) Scikit-learn integration package for Apache Spark
8. sparkit-learn

under Apache License 2.0 license
PySpark + Scikit-learn = Sparkit-learn
9. dist-keras

under GNU General Public License v3.0 license
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
10. petastorm

under Apache License 2.0 license
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
11. lopq

under Apache License 2.0 license
Training of Locally Optimized Product Quantization (LOPQ) models for approximate nearest neighbor search of high dimensional data in Python and Spark.
12. Frank-Kanes-Taming-Big-Data-with-Apache-Spark-and-Python

under MIT License license
Frank Kane's Taming Big Data with Apache Spark and Python, published by Packt
13. spark-recommendation-engine

under Apache License 2.0 license
14. vidgear

under Apache License 2.0 license
High-performance cross-platform Video Processing Python framework powerpacked with unique trailblazing features :sparkles:
15. instapy-cli

under MIT License license
:sparkles: Python library and CLI to upload photo and video on Instagram. W/o a phone!
16. KNN

under MIT License license
Spark Knn Recommender
17. sparklingpandas

under Apache License 2.0 license
Sparkling Pandas
18. aztk

under MIT License license
AZTK powered by Azure Batch: On-demand, Dockerized, Spark Jobs on Azure
19. findspark

under BSD 3-Clause "New" or "Revised" License license
20. spark-ml-streaming

under Apache License 2.0 license
Visualize streaming machine learning in Spark
21. LearningApacheSpark

under MIT License license
LearningApacheSpark
22. spark-df-profiling

under MIT License license
Create HTML profiling reports from Apache Spark DataFrames
23. cc-pyspark

under MIT License license
Process Common Crawl data with Python and Spark
24. spark-ui-proxy

under MIT License license
Lightweight proxy to expose the UI of an Apache Spark cluster that is behind a firewall
25. ardupilot-balloon-finder

under GNU General Public License v3.0 license
Code meant to be run on an Odroid to allow an ArduCopter Pixhawk based multicopter to find red balloons for Sparkfun's AVC 2014 competition
26. pyspark-csv

under MIT License license
An external PySpark module that works like R's read.csv or Panda's read_csv, with automatic type inference and null value handling. Parses csv data into SchemaRDD. No installation required, simply include pyspark_csv.py via SparkContext.
27. pyspark-cassandra

under Apache License 2.0 license
PySpark Cassandra brings back the fun in working with Cassandra data in PySpark.
28. tdigest

under MIT License license
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
29. airflow-pipeline

under Apache License 2.0 license
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
30. sk-dist

under Apache License 2.0 license
Distributed scikit-learn meta-estimators in PySpark
31. ZhiShiWenDa-Helper

under MIT License license
百万英雄/冲顶大会/知识超人答题助手瞬间使用Chrome打开百度
32. sparkflow

under MIT License license
Easy to use library to bring Tensorflow on Apache Spark
33. plugin.video.sparkle

under GNU General Public License v3.0 license
Kodi addon for finding acestream links
34. python_moztelemetry

under Mozilla Public License 2.0 license
Spark bindings for Mozilla Telemetry
35. spark-pycon15

under The Unlicense license
Materials for my Spark tutorial at Pycon 2015
36. momoapi-python

under MIT License license
MTN MoMo API Client Library for Python
37. dbt-spark

under Apache License 2.0 license
spark plugin for dbt
38. airflow-spark-operator-plugin

under Apache License 2.0 license
A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator
39. Duckyspark

under MIT License license
Translator from USB-Rubber-Ducky payloads to a Digispark code.
40. sqlflow

under Apache License 2.0 license
SQLflow based on python development, support to Spark, as the underlying distributed computing engine, through a set of unified configuration file to complete the batch, flow calculation, the Rest service development.
41. pyspark-stubs

under Apache License 2.0 license
Apache (Py)Spark type annotations (stub files).
42. pyspark-twitter-stream-mining

under MIT License license
Real-time Machine Learning with Apache Spark on Twitter Public Stream
43. Spark-in-Finance-Quantitative-Investing

under Apache License 2.0 license
Materials for IBM Spark contest. About the real-world application of big data and spark.
44. so-many-pyspark-examples

under MIT License license
Spark and Python (PySpark) Examples
45. bpr-spark

under MIT License license
Bayesian Personalized Ranking for Spark
46. spark-openstack

under Apache License 2.0 license
Scripts to setup Spark cluster (any version) in any Openstack environment with optional useful tools.
47. SMV

under Apache License 2.0 license
Spark Modularized View
48. SparkADMM

under Apache License 2.0 license
Generic Implementation of Consensus ADMM over Spark
49. pylivy

under MIT License license
A Python client for Apache Livy, enabling use of remote Apache Spark clusters.
50. spark-workshop

under MIT License license
Data and code for "Fast Data Applications with Spark and Python"
51. mmtf-pyspark

under Apache License 2.0 license
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
52. Desktop-PickAndPlace-CHMT36VA

under GNU General Public License v3.0 license
Tools and conversion scripts for the CHM-T36VA Desktop Pick and Place from SparkFun
53. pyspark2pmml

under GNU Affero General Public License v3.0 license
Python library for converting Apache Spark ML pipelines to PMML
54. emr-bootstrap-pyspark

under MIT License license
Quickstart PySpark with Anaconda on AWS/EMR
55. emr-bootstrap-pyspark

under MIT License license
Quickstart PySpark with Anaconda on AWS/EMR
56. pyspark-cassandra

under Apache License 2.0 license
pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4
57. sublime-standard-format

under MIT License license
:sparkles: Runs standard --fix against the javascript in your ST3 window on save or manually.
58. cloudformation-environmentbase

under BSD 2-Clause "Simplified" License license
Troposphere-based environment generator
59. pyspark-distributed-kmodes

under MIT License license
60. dataShark

under GNU General Public License v3.0 license
dataShark is a Security & Network Event Analytics Framework built on Apache Spark
61. pytest-spark

under MIT License license
pytest plugin to run the tests with support of pyspark
62. blogs

under Apache License 2.0 license
:sparkles: My personal blog :sparkles:
63. spark_using_python

under Apache License 2.0 license
64. Learning_Python

under MIT License license
本库托管了协程、SMTP邮件发送协议、 Python连接远程HBase、异步爬虫代码和快速上手中英文词云图等代码，如果你觉得对你有用，别忘了star我哦。
65. spark-gpu

under Apache License 2.0 license
GPU Acceleration for Apache Spark
66. pyCiscoSpark

under MIT License license
Python Library to Interface to Cisco Spark REST API
67. LSTM-TensorSpark

under MIT License license
Implementation of a LSTM with TensorFlow and distributed on Apache Spark
68. Hanhan-Spark-Python

under MIT License license
Used Spark core python, Spark sql, Spark MLlib, Spark Streaming
69. sparklingml

under Apache License 2.0 license
Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)
70. realtime-dashboard

under MIT License license
Real-time report dashboard with Apache Kafka, Apache Spark Streaming and Node.js
71. spark_gce

under Apache License 2.0 license
Spark GCE Script Helps you deploy Spark cluster on Google Cloud.
72. spark-streaming-twitter-kafka

under MIT License license
Ingest tweets with Kafka. Use Spark to track popular hashtags and trendsetters for each hashtag
73. sparktorch

under MIT License license
Train and run Pytorch models on Apache Spark.
74. spark-tree-plotting

under MIT License license
A simple tool for plotting Spark ML's Decision Trees
75. pyspark-cassandra

under Apache License 2.0 license
Utilities and examples to asssist in working with PySpark and Cassandra.
76. pyspark-examples

under GNU General Public License v3.0 license
pyspark sample scripts
77. DummyRDD

under BSD 3-Clause "New" or "Revised" License license
A pure python mock of pyspark's RDD
78. pyspark_dist_explore

under MIT License license
Data Exploration in PySpark made easy - Pyspark_dist_explore provides methods to get fast insights in your Spark DataFrames.
79. ozymandias

under MIT License license
Real-time image processing at scale using Kafka and Spark Streaming
80. stac-validator

under Apache License 2.0 license
Validator for the stac-spec
81. cursewords

under GNU Affero General Public License v3.0 license
:pencil: Terminal-based crossword puzzle solving interface
82. spark-cluster-deployment

under Apache License 2.0 license
Automates Spark standalone cluster tasks with Puppet and Fabric.
83. led-speech-edison

under MIT License license
Speech-activated LEDs using Intel Edison, SparkFun blocks, Python, and CMU Sphinx
84. django-monthfield

under BSD 3-Clause "New" or "Revised" License license
Custom field for storing month (YYYY-MM) as a django field.
85. darwin-sdk

under Apache License 2.0 license
Client library to use Darwin services in Python
86. complex_network

under GNU General Public License v2.0 license
The source code involved in complex network analysis.
87. data_analysis

under GNU General Public License v3.0 license
Source codes involved in data analysis.
88. pyspark-elastic

under Apache License 2.0 license
PySpark for Elastic Search
89. EurekaTrees

under MIT License license
Visualizes the Random Forest debug string from the MLLib in Spark using D3.js
90. sparksteps

under Apache License 2.0 license
:star: CLI tool to launch Spark jobs on AWS EMR
91. spark-package-cmd-tool

under Apache License 2.0 license
A command line tool for Spark packages
92. digiduck

under MIT License license
This program compiles Ducky Script into something usable for the Digispark ATTiny 85 chip. Get yourself a $1 USB Rubber Ducky.
93. spark-submit-with-pyspark-template

under MIT License license
94. SLQA

under Apache License 2.0 license
An Unofficial Pytorch Implementation of Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering
95. DigiSparkStealer

under MIT License license
Script for Digispark Attiny85, ATMEGA32U4 to steal passwords, cookies and send to your mail
96. joblib-spark

under Apache License 2.0 license
Joblib Apache Spark Backend
97. flask-spark-docker

under MIT License license
Just a boilerplate for PySpark and Flask
98. elsa

under Apache License 2.0 license
Elastic Sentiment Analysis (using Apache Mesos, Marathon and Apache Spark)
99. python-spark-recsys

under MIT License license
An example Python ALS recommender system
100. cassandra-spark-analytics

under MIT License license
Supercharge your analysis of Cassandra data with Apache Spark
101. NetSpark-Scripts

under GNU General Public License v3.0 license
Netmiko-based scripts to assist the Network Administrators and Engineers of the world!
102. Digispark-Duckduino

under GNU General Public License v2.0 license
Poor man's rubber ducky
103. django-dialogflow

under MIT License license
[Maintainer Required] Dialogflow Django is a web client to chat :sparkling_heart:
104. revelation

under MIT License license
Make awesome reveal.js presentations with revelation :sparkler:
105. CriteoDisplayCTR-TFOnSpark

under Apache License 2.0 license
106. maggy

under Apache License 2.0 license
Distribution transparent Machine Learning experiments on Apache Spark
107. aliyun-spark-deploy-tool

under Artistic License 2.0 license
Spark on ECS
108. dvorakjp-romantable

under MIT License license
Google 日本語入力用DvorakJPローマ字テーブル / DvorakJP Roman Table for Google Japanese Input
109. sparselandtools

under MIT License license
:sparkles: A Python package for sparse representations and dictionary learning, including matching pursuit, K-SVD and applications.
110. spyrk

under GNU Lesser General Public License v3.0 license
Python module for Spark devices (see spark.io)
111. python-spark

under MIT License license
An Earley-Algorithm Context-free grammar Parser Toolkit
112. Co-occurrence-Matrix

under MIT License license
使用Python构建共现矩阵，并以三元组形式存储到csv文件。