Java Code Examples for org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter

The following examples show how to use org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter. These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}
 
Example 2
Source Project: lucene-solr   Source File: DutchAnalyzer.java    License: Apache License 2.0 6 votes vote down vote up
public DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable, CharArrayMap<String> stemOverrideDict) {
  this.stoptable = CharArraySet.unmodifiableSet(CharArraySet.copy(stopwords));
  this.excltable = CharArraySet.unmodifiableSet(CharArraySet.copy(stemExclusionTable));
  if (stemOverrideDict.isEmpty()) {
    this.stemdict = null;
  } else {
    // we don't need to ignore case here since we lowercase in this analyzer anyway
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    CharArrayMap<String>.EntryIterator iter = stemOverrideDict.entrySet().iterator();
    CharsRefBuilder spare = new CharsRefBuilder();
    while (iter.hasNext()) {
      char[] nextKey = iter.nextKey();
      spare.copyChars(nextKey, 0, nextKey.length);
      builder.add(spare.get(), iter.currentValue());
    }
    try {
      this.stemdict = builder.build();
    } catch (IOException ex) {
      throw new RuntimeException("can not build stem dict", ex);
    }
  }
}
 
Example 3
Source Project: crate   Source File: StemmerOverrideTokenFilterFactory.java    License: Apache License 2.0 6 votes vote down vote up
static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}
 
Example 4
@Inject
public StemmerOverrideTokenFilterFactory(Index index, IndexSettingsService indexSettingsService, Environment env, @Assisted String name, @Assisted Settings settings) throws IOException {
    super(index, indexSettingsService.getSettings(), name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }
    
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}
 
Example 5
Source Project: crate   Source File: StemmerOverrideTokenFilterFactory.java    License: Apache License 2.0 5 votes vote down vote up
StemmerOverrideTokenFilterFactory(IndexSettings indexSettings, Environment env, String name, Settings settings) throws IOException {
    super(indexSettings, name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }

    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}
 
Example 6
@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}
 
Example 7
Source Project: crate   Source File: StemmerOverrideTokenFilterFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}