org.apache.lucene.analysis.standard

## Class StandardTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.standard.StandardTokenizer

All Implemented Interfaces:
  Closeable, AutoCloseable

    public final class StandardTokenizer
    extends Tokenizer

A grammar-based tokenizer constructed with JFlex.

As of Lucene version 3.1, this class implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

You must specify the required `Version` compatibility when creating StandardTokenizer:

- As of 3.4, Hiragana and Han characters are no longer wrongly split from their combining characters. If you use a previous version number, you get the exact broken behavior for backwards compatibility.
- As of 3.1, StandardTokenizer implements Unicode text segmentation. If you use a previous version number, you get the exact behavior of `ClassicTokenizer` for backwards compatibility.

### Nested Class Summary

### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.AttributeFactory, AttributeSource.State`

### Field Summary

Fields
`static int` `ACRONYM` Deprecated. (3.1)
`static int` `ACRONYM_DEP` Deprecated. (3.1)
`static int` `ALPHANUM`
`static int` `APOSTROPHE` Deprecated. (3.1)
`static int` `CJ` Deprecated. (3.1)
`static int` `COMPANY` Deprecated. (3.1)
`static int` `EMAIL`
`static int` `HANGUL`
`static int` `HIRAGANA`
`static int` `HOST` Deprecated. (3.1)
`static int` `IDEOGRAPHIC`
`static int` `KATAKANA`
`static int` `NUM`
`static int` `SOUTHEAST_ASIAN`
`static String[]` `TOKEN_TYPES` String token types that correspond to token type int constants

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Constructor Summary

Constructors
Constructor and Description
`StandardTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader input)` Creates a new StandardTokenizer with a given `AttributeSource.AttributeFactory`
`StandardTokenizer(Version matchVersion, AttributeSource source, Reader input)` Creates a new StandardTokenizer with a given `AttributeSource`.
`StandardTokenizer(Version matchVersion, Reader input)` Creates a new instance of the `StandardTokenizer`.

### Method Summary

Methods
`void` `end()`
`int` `getMaxTokenLength()`
`boolean` `incrementToken()`
`void` `reset()`
`void` `setMaxTokenLength(int length)` Set the max allowed token length.

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`close, correctOffset, setReader`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### ALPHANUM

    public static final int ALPHANUM

See Also:
  Constant Field Values

#### APOSTROPHE

    @Deprecated
    public static final int APOSTROPHE

Deprecated. (3.1)

See Also:
  Constant Field Values

#### ACRONYM

    @Deprecated
    public static final int ACRONYM

Deprecated. (3.1)

See Also:
  Constant Field Values

#### COMPANY

    @Deprecated
    public static final int COMPANY

Deprecated. (3.1)

See Also:
  Constant Field Values

#### EMAIL

    public static final int EMAIL

See Also:
  Constant Field Values

#### HOST

    @Deprecated
    public static final int HOST

Deprecated. (3.1)

See Also:
  Constant Field Values

#### NUM

    public static final int NUM

See Also:
  Constant Field Values

#### CJ

    @Deprecated
    public static final int CJ

Deprecated. (3.1)

See Also:
  Constant Field Values

#### ACRONYM_DEP

    @Deprecated
    public static final int ACRONYM_DEP

Deprecated. (3.1)

See Also:
  Constant Field Values

#### SOUTHEAST_ASIAN

    public static final int SOUTHEAST_ASIAN

See Also:
  Constant Field Values

#### IDEOGRAPHIC

    public static final int IDEOGRAPHIC

See Also:
  Constant Field Values

#### HIRAGANA

    public static final int HIRAGANA

See Also:
  Constant Field Values

#### KATAKANA

    public static final int KATAKANA

See Also:
  Constant Field Values

#### HANGUL

    public static final int HANGUL

See Also:
  Constant Field Values

#### TOKEN_TYPES

    public static final String[] TOKEN_TYPES

String token types that correspond to token type int constants

### Constructor Detail

#### StandardTokenizer

    public StandardTokenizer(Version matchVersion,
                     Reader input)

Creates a new instance of the `StandardTokenizer`. Attaches the `input` to the newly created JFlex scanner.

Parameters:
  `input` - The input reader See http://issues.apache.org/jira/browse/LUCENE-1068

#### StandardTokenizer

    public StandardTokenizer(Version matchVersion,
                     AttributeSource source,
                     Reader input)

Creates a new StandardTokenizer with a given `AttributeSource`.

#### StandardTokenizer

    public StandardTokenizer(Version matchVersion,
                     AttributeSource.AttributeFactory factory,
                     Reader input)

Creates a new StandardTokenizer with a given `AttributeSource.AttributeFactory`

### Method Detail

#### setMaxTokenLength

    public void setMaxTokenLength(int length)

Set the max allowed token length. Any token longer than this is skipped.

#### getMaxTokenLength

    public int getMaxTokenLength()

See Also:
  `setMaxTokenLength(int)`

#### incrementToken

    public final boolean incrementToken()
                                 throws IOException

Specified by:
  `incrementToken` in class `TokenStream`
Throws:
  `IOException`

#### end

    public final void end()

Overrides:
  `end` in class `TokenStream`

#### reset

    public void reset()
               throws IOException

Overrides:
  `reset` in class `TokenStream`
Throws:
  `IOException`