## Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer
org.apache.lucene.analysis: API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.ar: Analyzer for Arabic.
org.apache.lucene.analysis.cjk: Analyzer for Chinese, Japanese, and Korean, which indexes bigrams (overlapping groups of two adjacent Han characters).
org.apache.lucene.analysis.cn: Analyzer for Chinese, which indexes unigrams (individual chinese characters).
org.apache.lucene.analysis.cn.smart: Analyzer for Simplified Chinese, which indexes words.
org.apache.lucene.analysis.icu.segmentation: Tokenizer that breaks text into words with the Unicode Text Segmentation algorithm.
org.apache.lucene.analysis.in: Analysis components for Indian languages.
org.apache.lucene.analysis.ja: Analyzer for Japanese.
org.apache.lucene.analysis.ngram: Character n-gram tokenizers and filters.
org.apache.lucene.analysis.path: Analysis components for path-like strings such as filenames.
org.apache.lucene.analysis.ru: Analyzer for Russian.
org.apache.lucene.analysis.standard: Standards-based analyzers implemented with JFlex.
org.apache.lucene.analysis.wikipedia: Tokenizer that is aware of Wikipedia syntax.

Uses of Tokenizer in org.apache.lucene.analysis

Subclasses of Tokenizer in org.apache.lucene.analysis
`class` `CharTokenizer` An abstract base class for simple, character-oriented tokenizers.
`class` `EmptyTokenizer` Emits no tokens
`class` `KeywordTokenizer` Emits the entire input as a single token.
`class` `LetterTokenizer` A LetterTokenizer is a tokenizer that divides text at non-letters.
`class` `LowerCaseTokenizer` LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.
`class` `MockTokenizer` Tokenizer for testing.
`class` `WhitespaceTokenizer` A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

Fields in org.apache.lucene.analysis declared as Tokenizer
`protected Tokenizer` `ReusableAnalyzerBase.TokenStreamComponents.source`

Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
`ReusableAnalyzerBase.TokenStreamComponents(Tokenizer source)` Creates a new `ReusableAnalyzerBase.TokenStreamComponents` instance.
`ReusableAnalyzerBase.TokenStreamComponents(Tokenizer source, TokenStream result)` Creates a new `ReusableAnalyzerBase.TokenStreamComponents` instance.

Uses of Tokenizer in org.apache.lucene.analysis.ar

Subclasses of Tokenizer in org.apache.lucene.analysis.ar
`class` `ArabicLetterTokenizer` Deprecated. (3.1) Use `StandardTokenizer` instead.

Uses of Tokenizer in org.apache.lucene.analysis.cjk

Subclasses of Tokenizer in org.apache.lucene.analysis.cjk
`class` `CJKTokenizer` Deprecated. Use StandardTokenizer, CJKWidthFilter, CJKBigramFilter, and LowerCaseFilter instead.

Uses of Tokenizer in org.apache.lucene.analysis.cn

Subclasses of Tokenizer in org.apache.lucene.analysis.cn
`class` `ChineseTokenizer` Deprecated. Use `StandardTokenizer` instead, which has the same functionality. This filter will be removed in Lucene 5.0

Uses of Tokenizer in org.apache.lucene.analysis.cn.smart

Subclasses of Tokenizer in org.apache.lucene.analysis.cn.smart
`class` `SentenceTokenizer` Tokenizes input text into sentences.

Uses of Tokenizer in org.apache.lucene.analysis.icu.segmentation

Subclasses of Tokenizer in org.apache.lucene.analysis.icu.segmentation
`class` `ICUTokenizer` Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

Uses of Tokenizer in org.apache.lucene.analysis.in

Subclasses of Tokenizer in org.apache.lucene.analysis.in
`class` `IndicTokenizer` Deprecated. (3.6) Use `StandardTokenizer` instead.

Uses of Tokenizer in org.apache.lucene.analysis.ja

Subclasses of Tokenizer in org.apache.lucene.analysis.ja
`class` `JapaneseTokenizer` Tokenizer for Japanese that uses morphological analysis.

Uses of Tokenizer in org.apache.lucene.analysis.ngram

Subclasses of Tokenizer in org.apache.lucene.analysis.ngram
`class` `EdgeNGramTokenizer` Tokenizes the input from an edge into n-grams of given size(s).
`class` `NGramTokenizer` Tokenizes the input into n-grams of the given size(s).

Uses of Tokenizer in org.apache.lucene.analysis.path

Subclasses of Tokenizer in org.apache.lucene.analysis.path
`class` `PathHierarchyTokenizer` Tokenizer for path-like hierarchies.
`class` `ReversePathHierarchyTokenizer` Tokenizer for domain-like hierarchies.

Uses of Tokenizer in org.apache.lucene.analysis.ru

Subclasses of Tokenizer in org.apache.lucene.analysis.ru
`class` `RussianLetterTokenizer` Deprecated. Use `StandardTokenizer` instead, which has the same functionality. This filter will be removed in Lucene 5.0

Uses of Tokenizer in org.apache.lucene.analysis.standard

Subclasses of Tokenizer in org.apache.lucene.analysis.standard
`class` `ClassicTokenizer` A grammar-based tokenizer constructed with JFlex
`class` `StandardTokenizer` A grammar-based tokenizer constructed with JFlex.
`class` `UAX29URLEmailTokenizer` This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.

Uses of Tokenizer in org.apache.lucene.analysis.wikipedia

Subclasses of Tokenizer in org.apache.lucene.analysis.wikipedia
`class` `WikipediaTokenizer` Extension of StandardTokenizer that is aware of Wikipedia syntax.