org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter Java Examples

The following examples show how to use org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: StemmerOverrideTokenFilterFactory.java    From Elasticsearch with Apache License 2.0 6 votes vote down vote up
static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}
 
Example #2
Source File: DutchAnalyzer.java    From lucene-solr with Apache License 2.0 6 votes vote down vote up
public DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable, CharArrayMap<String> stemOverrideDict) {
  this.stoptable = CharArraySet.unmodifiableSet(CharArraySet.copy(stopwords));
  this.excltable = CharArraySet.unmodifiableSet(CharArraySet.copy(stemExclusionTable));
  if (stemOverrideDict.isEmpty()) {
    this.stemdict = null;
  } else {
    // we don't need to ignore case here since we lowercase in this analyzer anyway
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    CharArrayMap<String>.EntryIterator iter = stemOverrideDict.entrySet().iterator();
    CharsRefBuilder spare = new CharsRefBuilder();
    while (iter.hasNext()) {
      char[] nextKey = iter.nextKey();
      spare.copyChars(nextKey, 0, nextKey.length);
      builder.add(spare.get(), iter.currentValue());
    }
    try {
      this.stemdict = builder.build();
    } catch (IOException ex) {
      throw new RuntimeException("can not build stem dict", ex);
    }
  }
}
 
Example #3
Source File: StemmerOverrideTokenFilterFactory.java    From crate with Apache License 2.0 6 votes vote down vote up
static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}
 
Example #4
Source File: StemmerOverrideTokenFilterFactory.java    From Elasticsearch with Apache License 2.0 5 votes vote down vote up
@Inject
public StemmerOverrideTokenFilterFactory(Index index, IndexSettingsService indexSettingsService, Environment env, @Assisted String name, @Assisted Settings settings) throws IOException {
    super(index, indexSettingsService.getSettings(), name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }
    
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}
 
Example #5
Source File: StemmerOverrideTokenFilterFactory.java    From crate with Apache License 2.0 5 votes vote down vote up
StemmerOverrideTokenFilterFactory(IndexSettings indexSettings, Environment env, String name, Settings settings) throws IOException {
    super(indexSettings, name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }

    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}
 
Example #6
Source File: StemmerOverrideTokenFilterFactory.java    From Elasticsearch with Apache License 2.0 4 votes vote down vote up
@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}
 
Example #7
Source File: StemmerOverrideTokenFilterFactory.java    From crate with Apache License 2.0 4 votes vote down vote up
@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}