Python nltk.probability() Examples
The following are 2 code examples for showing how to use nltk.probability(). These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example.
You may check out the related API usage on the sidebar.
def logscore(self, word, context): """ For a given string representation of a word, and a word context, computes the log probability of this word in this context. """ score = self.score(word, context) if score == 0.0: return float("-inf") return log(score, 2)
def entropy(self, text): """ Calculate the approximate cross-entropy of the n-gram model for a given text represented as a list of comma-separated strings. This is the average log probability of each word in the text. """ normed_text = (self._check_against_vocab(word) for word in text) entropy = 0.0 processed_ngrams = 0 for ngram in self.ngram_counter.to_ngrams(normed_text): context, word = tuple(ngram[:-1]), ngram[-1] entropy += self.logscore(word, context) processed_ngrams += 1 return - (entropy / processed_ngrams)