## org.apache.lucene.analysis.cn.smart
Class SmartChineseAnalyzer

    java.lang.Object
      org.apache.lucene.analysis.Analyzer
          org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

All Implemented Interfaces:
  Closeable

    public final class SmartChineseAnalyzer

extends org.apache.lucene.analysis.Analyzer

SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

WARNING: This API is experimental and might change in incompatible ways in the next release.

Constructor Summary
`SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion)` Create a new SmartChineseAnalyzer, using the default stopword list.
`SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion, boolean useDefaultStopWords)` Create a new SmartChineseAnalyzer, optionally using the default stopword list.
`SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion, Set stopWords)` Create a new SmartChineseAnalyzer, using the provided `Set` of stopwords.

Method Summary
`static org.apache.lucene.analysis.CharArraySet` `getDefaultStopSet()` Returns an unmodifiable instance of the default stop-words set.
`org.apache.lucene.analysis.TokenStream` `reusableTokenStream(String fieldName, Reader reader)`
`org.apache.lucene.analysis.TokenStream` `tokenStream(String fieldName, Reader reader)`

Methods inherited from class org.apache.lucene.analysis.Analyzer
`close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### SmartChineseAnalyzer

    public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion)

  Create a new SmartChineseAnalyzer, using the default stopword list.

### SmartChineseAnalyzer

    public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion,
                                boolean useDefaultStopWords)

  Create a new SmartChineseAnalyzer, optionally using the default stopword list.

  The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

  Parameters:
    `useDefaultStopWords` - true to use the default stopword list.

### SmartChineseAnalyzer

    public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion,
                                Set stopWords)

  Create a new SmartChineseAnalyzer, using the provided `Set` of stopwords.

  Note: the set should include punctuation, unless you want to index punctuation!

  Parameters:
    `stopWords` - `Set` of stopwords to use.

Method Detail

### getDefaultStopSet

    public static org.apache.lucene.analysis.CharArraySet getDefaultStopSet()

  Returns an unmodifiable instance of the default stop-words set.

  Returns:
    an unmodifiable instance of the default stop-words set.

### tokenStream

    public org.apache.lucene.analysis.TokenStream tokenStream(String fieldName,
                                                              Reader reader)

  Specified by:
    `tokenStream` in class `org.apache.lucene.analysis.Analyzer`

### reusableTokenStream

    public org.apache.lucene.analysis.TokenStream reusableTokenStream(String fieldName,
                                                                      Reader reader)
                                                               throws IOException

  Overrides:
    `reusableTokenStream` in class `org.apache.lucene.analysis.Analyzer`
  Throws:
    `IOException`