Python nltk.probability() Examples

The following are 2 code examples for showing how to use nltk.probability(). These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example.

You may check out the related API usage on the sidebar.

You may also want to check out all available functions/classes of the module nltk , or try the search function .

Example 1
Project: atap   Author: foxbook   File:    License: Apache License 2.0 5 votes vote down vote up
def entropy(self, text):
        Calculate the approximate cross-entropy of the n-gram model for a
        given text represented as a list of comma-separated strings.
        This is the average log probability of each word in the text.
        normed_text = (self._check_against_vocab(word) for word in text)
        entropy = 0.0
        processed_ngrams = 0
        for ngram in self.ngram_counter.to_ngrams(normed_text):
            context, word = tuple(ngram[:-1]), ngram[-1]
            entropy += self.logscore(word, context)
            processed_ngrams += 1
        return - (entropy / processed_ngrams) 
Example 2
Project: atap   Author: foxbook   File:    License: Apache License 2.0 4 votes vote down vote up
def logscore(self, word, context):
        For a given string representation of a word, and a word context,
        computes the log probability of this word in this context.
        score = self.score(word, context)
        if score == 0.0:
            return float("-inf")

        return log(score, 2)