Python Examples

The following are code examples for showing how to use They are extracted from open source Python projects. You can vote up the examples you like or vote down the ones you don't like. You can also save this page to your account.

Example 1
Project: nmtrain   Author: philip30   File:    (license) View Source Project 5 votes vote down vote up
def __init__(self, codes, separator='@@'):            
        with, encoding='utf-8') as codes:
            self.bpe_codes = [tuple(item.split()) for item in codes]
        # some hacking to deal with duplicates (only consider first instance)
        self.bpe_codes = dict([(code,i) for (i,code) in reversed(list(enumerate(self.bpe_codes)))])

        self.separator = separator