Class DTokenizer

Inherits from:
Object
Declared in:
DTokenizer.h

Class Hierarchy

    Object
      |
      +---DTokenizer

Class Description

The class DTokenizer implements a (simple) lexical scanner. The tokenizer scans a text stream for whitespace, strings, keywords, comments, operators, numbers and three to be implemented formats. The scanner scans the text on a line basis. The tokenizer does not use regular expressions, but it uses for every token a method to scan for the token. The method can be overridden for a specific implementation of the token scanner. There are two methods: nextToken and checkToken. The first method checks for a token and if there is a match, the scanner location is moved to the next location in the source. The second method only returns the result of the match, the client must call the service 'next' to move the scanner location. The token methods return one of the following: DTK_UNKNOWN, DTK_EOFF, DTK_EXTRA1, DTK_EXTRA2, DTK_EXTRA3, DTK_WHITESPACE, DTK_COMMENT, DTK_NUMBER, DTK_KEYWORD, DTK_STRING, DTK_OPERATOR.

Last modified:
31-Oct-2007 (DTokenizer.h)

Instance Variables

private DList *_sources
the stack with the sources
private DSource *_source
the source for the tokenizer
private DText *_text
the last scanned text
private BOOL _eoff
is end of file reached ?
private unsigned _scanned
the length of the scanned text
private BOOL _skipWhiteSpace
should whitespace be skipped ? (def. YES)

Method Index


generated 05-Nov-2007 by ObjcDoc 3.0.0