Character Set | Zend_Search_Lucene Programmer's Reference Guide | Interoperating with Java Lucene

# Extensibility

# Text Analysis

The Zend_Search_Lucene_Analysis_Analyzer class is used by the indexer to tokenize document text fields.

The Zend_Search_Lucene_Analysis_Analyzer::getDefault() and `Zend_Search_Lucene_Analysis_Analyzer::setDefault()` methods are used to get and set the default analyzer.

You can assign your own text analyzer or choose it from the set of predefined analyzers: Zend_Search_Lucene_Analysis_Analyzer_Common_Text and Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive (default). Both of them interpret tokens as sequences of letters. Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive converts all tokens to lower case.

To switch between analyzers:

1. Zend_Search_Lucene_Analysis_Analyzer::setDefault(
2. new Zend_Search_Lucene_Analysis_Analyzer_Common_Text());
3. ...
4. $index->addDocument($doc);

The Zend_Search_Lucene_Analysis_Analyzer_Common class is designed to be an ancestor of all user defined analyzers. User should only define the reset() and nextToken() methods, which takes its string from the $_input member and returns tokens one by one (a `NULL` value indicates the end of the stream).

The nextToken() method should call the normalize() method on each token. This will allow you to use token filters with your analyzer.

Here is an example of a custom analyzer, which accepts words with digits as terms:

Example #1 Custom text Analyzer

1. /**
2. * Here is a custom text analyser, which treats words with digits as
3. * one term
4. */
6. class My_Analyzer extends Zend_Search_Lucene_Analysis_Analyzer_Common
7. {
8. private $_position;
10. /**
11. * Reset token stream
12. */
13. public function reset ()
14. {
15. $this->_position = 0;
16. }
18. /**
19. * Tokenization stream API
20. * Get next token
21. * Returns null at the end of stream
22. *
23. * @return Zend_Search_Lucene_Analysis_Token|null
24. */
25. public function nextToken()
26. {
27. if ($this->_input === null) {
28. return null;
29. }
31. while ($this->_position < strlen ($this->_input)) {
32. // skip white space
33. while ($this->_position < strlen ($this->_input) &&
34. !ctype_alnum ( $this->_input[$this->_position] )) {
35. $this->_position++;
36. }
38. $termStartPosition = $this->_position;
40. // read token
41. while ($this->_position < strlen ($this->_input) &&
42. ctype_alnum ( $this->_input[$this->_position] )) {
43. $this->_position++;
44. }
46. // Empty token, end of stream.
47. if ($this->_position == $termStartPosition) {
48. return null;
49. }
51. $token = new Zend_Search_Lucene_Analysis_Token(
52. substr ($this->_input,
53. $termStartPosition,
54. $this->_position -
55. $termStartPosition),
56. $termStartPosition,
57. $this->_position);
58. $token = $this->normalize($token);
59. if ($token !== null) {
60. return $token;
61. }
62. // Continue if token is skipped
63. }
65. return null;
66. }
67. }
69. Zend_Search_Lucene_Analysis_Analyzer::setDefault(
70. new My_Analyzer());

# Tokens Filtering

The Zend_Search_Lucene_Analysis_Analyzer_Common analyzer also offers a token filtering mechanism.

The Zend_Search_Lucene_Analysis_TokenFilter class provides an abstract interface for such filters. Your own filters should extend this class either directly or indirectly.

Any custom filter must implement the normalize() method which may transform input token or signal that the current token should be skipped.

There are three filters already defined in the analysis subpackage:

- Zend_Search_Lucene_Analysis_TokenFilter_LowerCase

- Zend_Search_Lucene_Analysis_TokenFilter_ShortWords

- Zend_Search_Lucene_Analysis_TokenFilter_StopWords

The `LowerCase` filter is already used for Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive analyzer by default.

The `ShortWords` and `StopWords` filters may be used with pre-defined or custom analyzers like this:

1. $stopWords = array ('a', 'an', 'at', 'the', 'and', 'or', 'is', 'am');
2. $stopWordsFilter =
3. new Zend_Search_Lucene_Analysis_TokenFilter_StopWords($stopWords);
5. $analyzer =
6. new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
7. $analyzer->addFilter($stopWordsFilter);
9. Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);

1. $shortWordsFilter = new Zend_Search_Lucene_Analysis_TokenFilter_ShortWords();
3. $analyzer =
4. new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
5. $analyzer->addFilter($shortWordsFilter);
7. Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);

The Zend_Search_Lucene_Analysis_TokenFilter_StopWords constructor takes an array of stop-words as an input. But stop-words may be also loaded from a file:

1. $stopWordsFilter = new Zend_Search_Lucene_Analysis_TokenFilter_StopWords();
2. $stopWordsFilter->loadFromFile($my_stopwords_file);
4. $analyzer =
5. new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
6. $analyzer->addFilter($stopWordsFilter);
8. Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);

This file should be a common text file with one word in each line. The '#' character marks a line as a comment.

The Zend_Search_Lucene_Analysis_TokenFilter_ShortWords constructor has one optional argument. This is the word length limit, set by default to 2.

# Scoring Algorithms

The score of a document d for a query q is defined as follows:

`score(q,d) = sum( tf(t in d) * idf(t) * getBoost(t.field in d) * lengthNorm(t.field in d) ) * coord(q,d) * queryNorm(q)`

tf(t in d) - Zend_Search_Lucene_Search_Similarity::tf($freq) - a score factor based on the frequency of a term or phrase in a document.

idf(t) - Zend_Search_Lucene_Search_Similarity::idf($input, $reader) - a score factor for a simple term with the specified index.

getBoost(t.field in d) - the boost factor for the term field.

lengthNorm($term) - the normalization value for a field given the total number of terms contained in a field. This value is stored within the index. These values, together with field boosts, are stored in an index and multiplied into scores for hits on each field by the search code.

Matches in longer fields are less precise, so implementations of this method usually return smaller values when numTokens is large, and larger values when numTokens is small.

coord(q,d) - Zend_Search_Lucene_Search_Similarity::coord($overlap, $maxOverlap) - a score factor based on the fraction of all query terms that a document contains.

The presence of a large portion of the query terms indicates a better match with the query, so implementations of this method usually return larger values when the ratio between these parameters is large and smaller values when the ratio between them is small.

queryNorm(q) - the normalization value for a query given the sum of the squared weights of each of the query terms. This value is then multiplied into the weight of each query term.

This does not affect ranking, but rather just attempts to make scores from different queries comparable.

The scoring algorithm can be customized by defining your own Similarity class. To do this extend the Zend_Search_Lucene_Search_Similarity class as defined below, then use the Zend_Search_Lucene_Search_Similarity::setDefault($similarity); method to set it as default.

1. class MySimilarity extends Zend_Search_Lucene_Search_Similarity {
2. public function lengthNorm($fieldName, $numTerms) {
3. return 1.0/sqrt ($numTerms);
4. }
6. public function queryNorm($sumOfSquaredWeights) {
7. return 1.0/sqrt ($sumOfSquaredWeights);
8. }
10. public function tf($freq) {
11. return sqrt ($freq);
12. }
14. /**
15. * It's not used now. Computes the amount of a sloppy phrase match,
16. * based on an edit distance.
17. */
18. public function sloppyFreq($distance) {
19. return 1.0;
20. }
22. public function idfFreq($docFreq, $numDocs) {
23. return log ($numDocs/(float)($docFreq+1)) + 1.0;
24. }
26. public function coord($overlap, $maxOverlap) {
27. return $overlap/(float)$maxOverlap;
28. }
29. }
31. $mySimilarity = new MySimilarity();
32. Zend_Search_Lucene_Search_Similarity::setDefault($mySimilarity);

# Storage Containers

The abstract class Zend_Search_Lucene_Storage_Directory defines directory functionality.

The Zend_Search_Lucene constructor uses either a string or a Zend_Search_Lucene_Storage_Directory object as an input.

The Zend_Search_Lucene_Storage_Directory_Filesystem class implements directory functionality for a file system.

If a string is used as an input for the Zend_Search_Lucene constructor, then the index reader (Zend_Search_Lucene object) treats it as a file system path and instantiates the Zend_Search_Lucene_Storage_Directory_Filesystem object.

You can define your own directory implementation by extending the Zend_Search_Lucene_Storage_Directory class.

Zend_Search_Lucene_Storage_Directory methods:

1. abstract class Zend_Search_Lucene_Storage_Directory {
2. /**
3. * Closes the store.
4. *
5. * @return void
6. */
7. abstract function close();
9. /**
10. * Creates a new, empty file in the directory with the given $filename.
11. *
12. * @param string $name
13. * @return void
14. */
15. abstract function createFile($filename);
17. /**
18. * Removes an existing $filename in the directory.
19. *
20. * @param string $filename
21. * @return void
22. */
23. abstract function deleteFile($filename);
25. /**
26. * Returns true if a file with the given $filename exists.
27. *
28. * @param string $filename
29. * @return boolean
30. */
31. abstract function fileExists($filename);
33. /**
34. * Returns the length of a $filename in the directory.
35. *
36. * @param string $filename
37. * @return integer
38. */
39. abstract function fileLength($filename);
41. /**
42. * Returns the UNIX timestamp $filename was last modified.
43. *
44. * @param string $filename
45. * @return integer
46. */
47. abstract function fileModified($filename);
49. /**
50. * Renames an existing file in the directory.
51. *
52. * @param string $from
53. * @param string $to
54. * @return void
55. */
56. abstract function renameFile($from, $to);
58. /**
59. * Sets the modified time of $filename to now.
60. *
61. * @param string $filename
62. * @return void
63. */
64. abstract function touchFile($filename);
66. /**
67. * Returns a Zend_Search_Lucene_Storage_File object for a given
68. * $filename in the directory.
69. *
70. * @param string $filename
71. * @return Zend_Search_Lucene_Storage_File
72. */
73. abstract function getFileObject($filename);
75. }

The getFileObject($filename) method of a Zend_Search_Lucene_Storage_Directory instance returns a Zend_Search_Lucene_Storage_File object.

The Zend_Search_Lucene_Storage_File abstract class implements file abstraction and index file reading primitives.

You must also extend Zend_Search_Lucene_Storage_File for your directory implementation.

Only two methods of Zend_Search_Lucene_Storage_File must be overridden in your implementation:

1. class MyFile extends Zend_Search_Lucene_Storage_File {
2. /**
3. * Sets the file position indicator and advances the file pointer.
4. * The new position, measured in bytes from the beginning of the file,
5. * is obtained by adding offset to the position specified by whence,
6. * whose values are defined as follows:
7. * SEEK_SET - Set position equal to offset bytes.
8. * SEEK_CUR - Set position to current location plus offset.
9. * SEEK_END - Set position to end-of-file plus offset. (To move to
10. * a position before the end-of-file, you need to pass a negative value
11. * in offset.)
12. * Upon success, returns 0; otherwise, returns -1
13. *
14. * @param integer $offset
15. * @param integer $whence
16. * @return integer
17. */
18. public function seek($offset, $whence=SEEK_SET) {
19. ...
20. }
22. /**
23. * Read a $length bytes from the file and advance the file pointer.
24. *
25. * @param integer $length
26. * @return string
27. */
28. protected function _fread($length=1) {
29. ...
30. }
31. }

Character Set | Zend_Search_Lucene Programmer's Reference Guide | Interoperating with Java Lucene