# Package org.apache.lucene.analysis

Support for testing analysis components.

See: Description

Interface Summary
Interface | Description
--- | ---
BaseTokenStreamTestCase.CheckClearAttributesAttribute | Attribute that records if it was cleared or not.
CannedBinaryTokenStream.BinaryTermAttribute | An attribute extending `TermToBytesRefAttribute` but exposing `CannedBinaryTokenStream.BinaryTermAttribute.setBytesRef(org.apache.lucene.util.BytesRef)` method.

Class Summary
Class | Description
--- | ---
BaseTokenStreamTestCase | Base class for all Lucene unit tests that use TokenStreams.
BaseTokenStreamTestCase.CheckClearAttributesAttributeImpl | Attribute that records if it was cleared or not.
CannedBinaryTokenStream | TokenStream from a canned list of binary (BytesRef-based) tokens.
CannedBinaryTokenStream.BinaryTermAttributeImpl | Implementation for `CannedBinaryTokenStream.BinaryTermAttribute`.
CannedBinaryTokenStream.BinaryToken | Represents a binary token.
CannedTokenStream | TokenStream from a canned list of Tokens.
CollationTestBase | Base test class for testing Unicode collation.
LookaheadTokenFilter<T extends LookaheadTokenFilter.Position> | An abstract TokenFilter to make it easier to build graph token filters requiring some lookahead.
LookaheadTokenFilter.Position | Holds all state for a single position; subclass this to record other state at each position.
MockAnalyzer | Analyzer for testing
MockBytesAnalyzer | Analyzer for testing that encodes terms as UTF-16 bytes.
MockBytesAttributeFactory | Attribute factory that implements CharTermAttribute with `MockUTF16TermAttributeImpl`
MockCharFilter | the purpose of this charfilter is to send offsets out of bounds if the analyzer doesn't use correctOffset or does incorrect offset math.
MockFixedLengthPayloadFilter | TokenFilter that adds random fixed-length payloads.
MockGraphTokenFilter | Randomly inserts overlapped (posInc=0) tokens with posLength sometimes > 1.
MockHoleInjectingTokenFilter | Randomly injects holes (similar to what a stopfilter would do)
MockPayloadAnalyzer | Wraps a whitespace tokenizer with a filter that sets the first token, and odd tokens to posinc=1, and all others to 0, encoding the position as pos: XXX in the payload.
MockRandomLookaheadTokenFilter | Uses `LookaheadTokenFilter` to randomly peek at future tokens.
MockReaderWrapper | Wraps a Reader, and can throw random or fixed exceptions, and spoon feed read chars.
MockTokenFilter | A tokenfilter for testing that removes terms accepted by a DFA.
MockTokenizer | Tokenizer for testing.
MockUTF16TermAttributeImpl | Extension of `CharTermAttributeImpl` that encodes the term text as UTF-16 bytes instead of as UTF-8 bytes.
MockVariableLengthPayloadFilter | TokenFilter that adds random variable-length payloads.
TokenStreamToDot | Consumes a TokenStream and outputs the dot (graphviz) string (graph).
ValidatingTokenFilter | A TokenFilter that checks consistency of the tokens (eg offsets are consistent with one another).
VocabularyAssert | Utility class for doing vocabulary-based stemming tests

## Package org.apache.lucene.analysis Description

Support for testing analysis components.

The main classes of interest are:

- `BaseTokenStreamTestCase`: Highly recommended to use its helper methods, (especially in conjunction with `MockAnalyzer` or `MockTokenizer`), as it contains many assertions and checks to catch bugs.
- `MockTokenizer`: Tokenizer for testing. Tokenizer that serves as a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test it wrapping this tokenizer instead for extra checks.
- `MockAnalyzer`: Analyzer for testing. Analyzer that uses MockTokenizer for additional verification. If you are testing a custom component such as a queryparser or analyzer-wrapper that consumes analysis streams, its a great idea to test it with this analyzer instead.