Lucene++ - a full-featured, c++ search engine
API Documentation


 All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Macros Pages
Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Protected Attributes
Lucene::StandardTokenizer Class Reference

A grammar-based tokenizer. More...

#include <StandardTokenizer.h>

+ Inheritance diagram for Lucene::StandardTokenizer:

Public Member Functions

 StandardTokenizer (LuceneVersion::Version matchVersion, ReaderPtr input)
 Creates a new instance of the StandardTokenizer. Attaches the input to the newly created scanner. More...
 
 StandardTokenizer (LuceneVersion::Version matchVersion, AttributeSourcePtr source, ReaderPtr input)
 Creates a new StandardTokenizer with a given AttributeSource. More...
 
 StandardTokenizer (LuceneVersion::Version matchVersion, AttributeFactoryPtr factory, ReaderPtr input)
 Creates a new StandardTokenizer with a given AttributeSource.AttributeFactory. More...
 
virtual ~StandardTokenizer ()
 
virtual String getClassName ()
 
boost::shared_ptr
< StandardTokenizer
shared_from_this ()
 
void setMaxTokenLength (int32_t length)
 Set the max allowed token length. Any token longer than this is skipped. More...
 
int32_t getMaxTokenLength ()
 
virtual bool incrementToken ()
 
virtual void end ()
 This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature. More...
 
virtual void reset (ReaderPtr input)
 Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer. More...
 
bool isReplaceInvalidAcronym ()
 
void setReplaceInvalidAcronym (bool replaceInvalidAcronym)
 
- Public Member Functions inherited from Lucene::Tokenizer
virtual ~Tokenizer ()
 
boost::shared_ptr< Tokenizershared_from_this ()
 
virtual void close ()
 By default, closes the input Reader. More...
 
virtual int32_t correctOffset (int32_t currentOff)
 Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOffset, else returns currentOff. More...
 
- Public Member Functions inherited from Lucene::TokenStream
virtual ~TokenStream ()
 
boost::shared_ptr< TokenStreamshared_from_this ()
 
virtual void reset ()
 Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()). More...
 
- Public Member Functions inherited from Lucene::AttributeSource
 AttributeSource ()
 An AttributeSource using the default attribute factory DefaultAttributeFactory. More...
 
 AttributeSource (AttributeSourcePtr input)
 An AttributeSource that uses the same attributes as the supplied one. More...
 
 AttributeSource (AttributeFactoryPtr factory)
 An AttributeSource using the supplied AttributeFactory for creating new Attribute instances. More...
 
virtual ~AttributeSource ()
 
boost::shared_ptr
< AttributeSource
shared_from_this ()
 
AttributeFactoryPtr getAttributeFactory ()
 returns the used AttributeFactory. More...
 
template<class ATTR >
boost::shared_ptr< ATTR > addAttribute ()
 This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned. More...
 
void addAttribute (const String &className, AttributePtr attrImpl)
 Adds a custom Attribute instance. More...
 
bool hasAttributes ()
 Returns true if this AttributeSource has any attributes. More...
 
template<class ATTR >
bool hasAttribute ()
 Returns true, if this AttributeSource contains the passed-in Attribute. More...
 
template<class ATTR >
boost::shared_ptr< ATTR > getAttribute ()
 Returns the instance of the passed in Attribute contained in this AttributeSource. More...
 
void clearAttributes ()
 Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation. More...
 
AttributeSourceStatePtr captureState ()
 Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource. More...
 
void restoreState (AttributeSourceStatePtr state)
 Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation). More...
 
virtual int32_t hashCode ()
 Return hash code for this object. More...
 
virtual bool equals (LuceneObjectPtr other)
 Return whether two objects are equal. More...
 
virtual String toString ()
 Returns a string representation of the object. More...
 
AttributeSourcePtr cloneAttributes ()
 Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using AttributeSource(AttributeSource)) More...
 
Collection< AttributePtrgetAttributes ()
 Return a vector of attributes based on currentState. More...
 
- Public Member Functions inherited from Lucene::LuceneObject
virtual ~LuceneObject ()
 
virtual void initialize ()
 Called directly after instantiation to create objects that depend on this object being fully constructed. More...
 
virtual LuceneObjectPtr clone (LuceneObjectPtr other=LuceneObjectPtr())
 Return clone of this object. More...
 
virtual int32_t compareTo (LuceneObjectPtr other)
 Compare two objects. More...
 
- Public Member Functions inherited from Lucene::LuceneSync
virtual ~LuceneSync ()
 
virtual SynchronizePtr getSync ()
 Return this object synchronize lock. More...
 
virtual LuceneSignalPtr getSignal ()
 Return this object signal. More...
 
virtual void lock (int32_t timeout=0)
 Lock this object using an optional timeout. More...
 
virtual void unlock ()
 Unlock this object. More...
 
virtual bool holdsLock ()
 Returns true if this object is currently locked by current thread. More...
 
virtual void wait (int32_t timeout=0)
 Wait for signal using an optional timeout. More...
 
virtual void notifyAll ()
 Notify all threads waiting for signal. More...
 

Static Public Member Functions

static String _getClassName ()
 
static const Collection< String > TOKEN_TYPES ()
 String token types that correspond to token type int constants. More...
 
- Static Public Member Functions inherited from Lucene::Tokenizer
static String _getClassName ()
 
- Static Public Member Functions inherited from Lucene::TokenStream
static String _getClassName ()
 
- Static Public Member Functions inherited from Lucene::AttributeSource
static String _getClassName ()
 

Static Public Attributes

static const int32_t ALPHANUM
 
static const int32_t APOSTROPHE
 
static const int32_t ACRONYM
 
static const int32_t COMPANY
 
static const int32_t EMAIL
 
static const int32_t HOST
 
static const int32_t NUM
 
static const int32_t CJ
 
static const int32_t ACRONYM_DEP
 

Protected Member Functions

void init (ReaderPtr input, LuceneVersion::Version matchVersion)
 
- Protected Member Functions inherited from Lucene::Tokenizer
 Tokenizer ()
 Construct a tokenizer with null input. More...
 
 Tokenizer (ReaderPtr input)
 Construct a token stream processing the given input. More...
 
 Tokenizer (AttributeFactoryPtr factory)
 Construct a tokenizer with null input using the given AttributeFactory. More...
 
 Tokenizer (AttributeFactoryPtr factory, ReaderPtr input)
 Construct a token stream processing the given input using the given AttributeFactory. More...
 
 Tokenizer (AttributeSourcePtr source)
 Construct a token stream processing the given input using the given AttributeSource. More...
 
 Tokenizer (AttributeSourcePtr source, ReaderPtr input)
 Construct a token stream processing the given input using the given AttributeSource. More...
 
- Protected Member Functions inherited from Lucene::TokenStream
 TokenStream ()
 A TokenStream using the default attribute factory. More...
 
 TokenStream (AttributeSourcePtr input)
 A TokenStream that uses the same attributes as the supplied one. More...
 
 TokenStream (AttributeFactoryPtr factory)
 A TokenStream using the supplied AttributeFactory for creating new Attribute instances. More...
 
- Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr getAttribute (const String &className)
 The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it. More...
 
bool hasAttribute (const String &className)
 Returns true, if this AttributeSource contains the passed-in Attribute. More...
 
void computeCurrentState ()
 
- Protected Member Functions inherited from Lucene::LuceneObject
 LuceneObject ()
 

Protected Attributes

StandardTokenizerImplPtr scanner
 A private instance of the scanner. More...
 
bool replaceInvalidAcronym
 
int32_t maxTokenLength
 
TermAttributePtr termAtt
 
OffsetAttributePtr offsetAtt
 
PositionIncrementAttributePtr posIncrAtt
 
TypeAttributePtr typeAtt
 
- Protected Attributes inherited from Lucene::Tokenizer
ReaderPtr input
 The text source for this Tokenizer. More...
 
CharStreamPtr charStream
 

Detailed Description

A grammar-based tokenizer.

This should be a good tokenizer for most European-language documents:

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

You must specify the required Version compatibility when creating StandardAnalyzer:

Constructor & Destructor Documentation

Lucene::StandardTokenizer::StandardTokenizer ( LuceneVersion::Version  matchVersion,
ReaderPtr  input 
)

Creates a new instance of the StandardTokenizer. Attaches the input to the newly created scanner.

Parameters
inputThe input reader
Lucene::StandardTokenizer::StandardTokenizer ( LuceneVersion::Version  matchVersion,
AttributeSourcePtr  source,
ReaderPtr  input 
)

Creates a new StandardTokenizer with a given AttributeSource.

Lucene::StandardTokenizer::StandardTokenizer ( LuceneVersion::Version  matchVersion,
AttributeFactoryPtr  factory,
ReaderPtr  input 
)
virtual Lucene::StandardTokenizer::~StandardTokenizer ( )
virtual

Member Function Documentation

static String Lucene::StandardTokenizer::_getClassName ( )
inlinestatic
virtual void Lucene::StandardTokenizer::end ( )
virtual

This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Reimplemented from Lucene::TokenStream.

virtual String Lucene::StandardTokenizer::getClassName ( )
inlinevirtual

Reimplemented from Lucene::Tokenizer.

int32_t Lucene::StandardTokenizer::getMaxTokenLength ( )
virtual bool Lucene::StandardTokenizer::incrementToken ( )
virtual
See Also
TokenStream::next()

Implements Lucene::TokenStream.

void Lucene::StandardTokenizer::init ( ReaderPtr  input,
LuceneVersion::Version  matchVersion 
)
protected
bool Lucene::StandardTokenizer::isReplaceInvalidAcronym ( )
Returns
true if StandardTokenizer now returns these tokens as Hosts, otherwise false
Deprecated:
Remove in 3.X and make true the only valid value
virtual void Lucene::StandardTokenizer::reset ( ReaderPtr  input)
virtual

Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Reimplemented from Lucene::Tokenizer.

void Lucene::StandardTokenizer::setMaxTokenLength ( int32_t  length)

Set the max allowed token length. Any token longer than this is skipped.

void Lucene::StandardTokenizer::setReplaceInvalidAcronym ( bool  replaceInvalidAcronym)
Parameters
replaceInvalidAcronymSet to true to replace mischaracterized acronyms as HOST.
Deprecated:
Remove in 3.X and make true the only valid value
boost::shared_ptr< StandardTokenizer > Lucene::StandardTokenizer::shared_from_this ( )
inline
static const Collection<String> Lucene::StandardTokenizer::TOKEN_TYPES ( )
static

String token types that correspond to token type int constants.

Field Documentation

const int32_t Lucene::StandardTokenizer::ACRONYM
static
const int32_t Lucene::StandardTokenizer::ACRONYM_DEP
static
Deprecated:
this solves a bug where HOSTs that end with '.' are identified as ACRONYMs.
const int32_t Lucene::StandardTokenizer::ALPHANUM
static
const int32_t Lucene::StandardTokenizer::APOSTROPHE
static
const int32_t Lucene::StandardTokenizer::CJ
static
const int32_t Lucene::StandardTokenizer::COMPANY
static
const int32_t Lucene::StandardTokenizer::EMAIL
static
const int32_t Lucene::StandardTokenizer::HOST
static
int32_t Lucene::StandardTokenizer::maxTokenLength
protected
const int32_t Lucene::StandardTokenizer::NUM
static
OffsetAttributePtr Lucene::StandardTokenizer::offsetAtt
protected
PositionIncrementAttributePtr Lucene::StandardTokenizer::posIncrAtt
protected
bool Lucene::StandardTokenizer::replaceInvalidAcronym
protected
StandardTokenizerImplPtr Lucene::StandardTokenizer::scanner
protected

A private instance of the scanner.

TermAttributePtr Lucene::StandardTokenizer::termAtt
protected
TypeAttributePtr Lucene::StandardTokenizer::typeAtt
protected

The documentation for this class was generated from the following file:

clucene.sourceforge.net