## org.apache.lucene.facet.index.streaming
Class CategoryTokenizerBase

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.TokenFilter
                  org.apache.lucene.facet.index.streaming.CategoryTokenizerBase

All Implemented Interfaces:
  Closeable

Direct Known Subclasses:
  CategoryListTokenizer, CategoryTokenizer

    public abstract class CategoryTokenizerBase

extends org.apache.lucene.analysis.TokenFilter

A base class for all token filters which add term and payload attributes to tokens and are to be used in `CategoryDocumentBuilder`. Contains three attributes: `CategoryAttribute`, `CharTermAttribute` and `PayloadAttribute`.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State`

Field Summary
`protected CategoryAttribute` `categoryAttribute` The stream's category attributes.
`protected FacetIndexingParams` `indexingParams` Indexing params for creating term text
`protected org.apache.lucene.index.Payload` `payload` The object used for constructing payloads.
`protected org.apache.lucene.analysis.tokenattributes.PayloadAttribute` `payloadAttribute` The stream's payload attribute.
`protected org.apache.lucene.analysis.tokenattributes.CharTermAttribute` `termAttribute` The stream's term attribute.

Fields inherited from class org.apache.lucene.analysis.TokenFilter
`input`

Constructor Summary
`CategoryTokenizerBase(org.apache.lucene.analysis.TokenStream input, FacetIndexingParams indexingParams)` Constructor.

Method Summary
`abstract boolean` `incrementToken()`

Methods inherited from class org.apache.lucene.analysis.TokenFilter
`close, end, reset`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### categoryAttribute

    protected CategoryAttribute categoryAttribute

  The stream's category attributes.

### payloadAttribute

    protected org.apache.lucene.analysis.tokenattributes.PayloadAttribute payloadAttribute

  The stream's payload attribute.

### termAttribute

    protected org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAttribute

  The stream's term attribute.

### payload

    protected org.apache.lucene.index.Payload payload

  The object used for constructing payloads.

### indexingParams

    protected FacetIndexingParams indexingParams

  Indexing params for creating term text

Constructor Detail

### CategoryTokenizerBase

    public CategoryTokenizerBase(org.apache.lucene.analysis.TokenStream input,
                                 FacetIndexingParams indexingParams)

  Constructor.

  Parameters:
    `input` - The input stream, either `CategoryParentsStream` or an extension of `CategoryTokenizerBase`.
    `indexingParams` - The indexing params to use.

Method Detail

### incrementToken

    public abstract boolean incrementToken()
                                    throws IOException

  Specified by:
    `incrementToken` in class `org.apache.lucene.analysis.TokenStream`
  Throws:
    `IOException`