PyLearn

PyLearn is a resource for Bayesian inference and machine learning in Python.

Introduction

How do we infer and learn from experience ?

Edwin Jaynes, in his influential How does the brain do plausible reasoning ?, wrote

One of the most familiar facts of our experience is this: that there is such a thing as common sense, which enables us to do plausible reasoning in a fairly consistent way. People who have the same background of experience and the same amount of information about a proposition come to pretty much the same conclusions as to its plausibility. No jury has ever reached a verdict on the basis of pure deductive reasoning. Therefore the human brain must contain some fairly definite mechanism for plausible reasoning, undoubtedly much more complex than that required for deductive reasoning. But in order for this to be possible, there must exist consistent rules for carrying out plausible reasoning, in terms of operations so definite that they can be programmed on the computing machine which is the human brain.

Jaynes went on to show that these "consistent rules" are just the rules of Bayesian probability theory, supplemented by Laplace's principle of indifference and, its generalization, Shannon's principle of maximum entropy. This key observation implies that a computer can be programmed to "reason", or, update probabilities based on data. Given some very minimal desiderata, the rules of Bayesian probability are the only ones which conform to what, intuitively, we recognize as rationality. Such probability update rules can be used recursively to impute causal relationships between observations, that is, a machine can be programmed to "learn".

Inference and machine learning, then, is the creative application of Bayesian probability to problems of rational inference and causal knowledge discovery based on data.

I wrote a short article in The Hindu about learning from experience, together with my colleague at The Institute of Mathematical Sciences, Rahul Siddharthan. We jointly organized the Bayes by the Bay where distinguished international colleagues lectured on applications of Bayesian inference to physics, biology, earth sciene, brain function and more. These lectures are available on YouTube.

Here, I list reading material, algorithms and software, and tutorial slides with example codes from the ongoing workshop series co-organized with Dorai Thodla.

I welcome contributions - clone this repository and send me a pull request!

Inference and machine learning quotes

Online courses

Slide decks

Reading list

This is my reading list for Bayesian probability and its application to machine learning problems. The sections covered in this list are:

Classics of Bayesian probability

Bayesian inference in statistical analysis

Bayesian networks

The last reference includes the classic Tutorial on Learning with Bayesian Networks by David Heckerman.

Bayesian decision theory

Bayesian risk analysis

Antecedents

Algorithms and software

Since this is a resource for machine learning in Python, this basic toolchain is sine qua non!

The most popular machine learning library in Python is scikits-learn but few of its algorithms are explicitly probablistic. Probablistic graphical models (PGMs) are implemented in three good Python libraries listed below.

Python has a several good IDEs. Two of my favourites are