### Home Classes Methods

### In Files

- ext/r_analysis.c

### Parent

Ferret::Analysis::Analyzer

### Methods

- ::new

### Files

- MIT-LICENSE
- README
- TODO
- TUTORIAL

### Class/Module Index

Quicksearch

- Ferret
- Ferret::Analysis
- Ferret::Analysis::Analyzer
- Ferret::Analysis::AsciiLetterAnalyzer
- Ferret::Analysis::AsciiLetterTokenizer
- Ferret::Analysis::AsciiLowerCaseFilter
- Ferret::Analysis::AsciiStandardAnalyzer
- Ferret::Analysis::AsciiStandardTokenizer
- Ferret::Analysis::AsciiWhiteSpaceAnalyzer
- Ferret::Analysis::AsciiWhiteSpaceTokenizer
- Ferret::Analysis::HyphenFilter
- Ferret::Analysis::LetterAnalyzer
- Ferret::Analysis::LetterTokenizer
- Ferret::Analysis::LowerCaseFilter
- Ferret::Analysis::MappingFilter
- Ferret::Analysis::PerFieldAnalyzer
- Ferret::Analysis::RegExpAnalyzer
- Ferret::Analysis::RegExpTokenizer
- Ferret::Analysis::StandardAnalyzer
- Ferret::Analysis::StandardTokenizer
- Ferret::Analysis::StemFilter
- Ferret::Analysis::StopFilter
- Ferret::Analysis::Token
- Ferret::Analysis::TokenStream
- Ferret::Analysis::WhiteSpaceAnalyzer
- Ferret::Analysis::WhiteSpaceTokenizer
- Ferret::BlankSlate
- Ferret::BoostMixin
- Ferret::Browser
- Ferret::Browser::Controller
- Ferret::Browser::Delegator
- Ferret::Browser::DocumentController
- Ferret::Browser::ErrorController
- Ferret::Browser::HelpController
- Ferret::Browser::HomeController
- Ferret::Browser::TermController
- Ferret::Browser::TermVectorController
- Ferret::Browser::ViewHelper
- Ferret::Document
- Ferret::Field
- Ferret::FieldSymbol
- Ferret::FieldSymbolMethods
- Ferret::FileNotFoundError
- Ferret::Index
- Ferret::Index::FieldInfo
- Ferret::Index::FieldInfos
- Ferret::Index::Index
- Ferret::Index::IndexReader
- Ferret::Index::IndexWriter
- Ferret::Index::LazyDoc
- Ferret::Index::LazyDoc::LazyDocData
- Ferret::Index::TermDocEnum
- Ferret::Index::TermEnum
- Ferret::Index::TermVector
- Ferret::Index::TermVector::TVOffsets
- Ferret::Index::TermVector::TVTerm
- Ferret::ParseError
- Ferret::QueryParser
- Ferret::QueryParser::QueryParseException
- Ferret::Search
- Ferret::Search::BooleanQuery
- Ferret::Search::BooleanQuery::BooleanClause
- Ferret::Search::ConstantScoreQuery
- Ferret::Search::Explanation
- Ferret::Search::Filter
- Ferret::Search::FilteredQuery
- Ferret::Search::FuzzyQuery
- Ferret::Search::Hit
- Ferret::Search::MatchAllQuery
- Ferret::Search::MultiSearcher
- Ferret::Search::MultiTermQuery
- Ferret::Search::PhraseQuery
- Ferret::Search::PrefixQuery
- Ferret::Search::Query
- Ferret::Search::QueryFilter
- Ferret::Search::RangeFilter
- Ferret::Search::RangeQuery
- Ferret::Search::Searcher
- Ferret::Search::Sort
- Ferret::Search::SortField
- Ferret::Search::Spans
- Ferret::Search::Spans::SpanFirstQuery
- Ferret::Search::Spans::SpanMultiTermQuery
- Ferret::Search::Spans::SpanNearQuery
- Ferret::Search::Spans::SpanNotQuery
- Ferret::Search::Spans::SpanOrQuery
- Ferret::Search::Spans::SpanPrefixQuery
- Ferret::Search::Spans::SpanTermQuery
- Ferret::Search::TermQuery
- Ferret::Search::TopDocs
- Ferret::Search::TypedRangeFilter
- Ferret::Search::TypedRangeQuery
- Ferret::Search::WildcardQuery
- Ferret::StateError
- Ferret::Store
- Ferret::Store::Directory
- Ferret::Store::FSDirectory
- Ferret::Store::Lock
- Ferret::Store::Lock::LockError
- Ferret::Store::RAMDirectory
- Ferret::Utils
- Ferret::Utils::BitVector
- Ferret::Utils::MultiMapper
- Ferret::Utils::PriorityQueue
- WEBrick
- WEBrick::FerretBrowserHandler
- Date
- DateTime
- FileUtils
- Float
- Integer
- String
- Symbol
- Time

No matching classes.

# Ferret::Analysis::StandardAnalyzer

## Summary

The StandardAnalyzer is the most advanced of the available analyzers. If it were implemented in Ruby it would look like this;

    class StandardAnalyzer
      def initialize(stop_words = FULL_ENGLISH_STOP_WORDS, lower = true)
        @lower = lower
        @stop_words = stop_words
      end

      def token_stream(field, str)
        ts = StandardTokenizer.new(str)
        ts = LowerCaseFilter.new(ts) if @lower
        ts = StopFilter.new(ts, @stop_words)
        ts = HyphenFilter.new(ts)
      end
    end

As you can see it makes use of the StandardTokenizer and you can also add your own list of stopwords if you wish.

### Public Class Methods

new(stop_words = FULL_ENGLISH_STOP_WORDS, lower=true) click to toggle source
→ analyzer

Create a new StandardAnalyzer which downcases tokens by default but can optionally leave case as is. Lowercasing will be done based on the current locale. You can also set the list of stop-words to be used by the StopFilter.

lower | set to false if you don't want the field's tokens to be downcased
stop_words | list of stop-words to pass to the StopFilter

    static VALUE
    frb_standard_analyzer_init(int argc, VALUE *argv, VALUE self)
    {
        bool lower;
        VALUE rlower, rstop_words;
        Analyzer *a;
    #ifndef POSH_OS_WIN32
        if (!frb_locale) frb_locale = setlocale(LC_CTYPE, "");
    #endif
        rb_scan_args(argc, argv, "02", &rstop_words, &rlower);
        lower = ((rlower == Qnil) ? true : RTEST(rlower));
        if (rstop_words != Qnil) {
            char **stop_words = get_stopwords(rstop_words);
            a = mb_standard_analyzer_new_with_words((const char **)stop_words, lower);
            free(stop_words);
        } else {
            a = mb_standard_analyzer_new(lower);
        }
        Frt_Wrap_Struct(self, NULL, &frb_analyzer_free, a);
        object_add(a, self);
        return self;
    }

[Validate]

Generated with the Darkfish Rdoc Generator 2.