Package libxyz :: Package parser :: Module lexer :: Class Lexer
[hide private]
[frames] | no frames]

Class Lexer

source code

object --+
         |
        Lexer


Lexical analyzer

Lexer rules:
-----------
* Blank chars are usually ignored. Except from in quotes.
* Quote can be one-line: "quoted value", or multiline:
  '''quoted value1,
     quoted value2,
  '''
* New-line char ends commented line if any.
* Values can be provided as simple literals or quoted ones.
* If value contains spaces or any other non-alphanumeric values it is better
  to quote it or escape it using escapechar.
* Variable can take list of values, separated by comma
* Escaping can only be used in rval position.

Macros:
------
Macros are special internal variables that get expanded upon parsing.
Macro definition is similar to variable definition, but macro char
(default '&') is prepended to var name:
&macro = value
var = &macro

Instance Methods [hide private]
 
__init__(self, source, tokens, comment=u"#", macro=u"&")
x.__init__(...) initializes x; see x.__class__.__doc__ for signature
source code
 
lexer(self)
Scan input for lexemes and return to parser
source code
 
get_idt(self)
Return current state of token buffer
source code
 
done(self)
Order lexer to stop processing
source code
 
unget(self, token)
Put read token back to input stream
source code
 
escaping_on(self)
Enable escaping
source code
 
escaping_off(self)
Disable escaping
source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Class Variables [hide private]
  TOKEN_IDT = 0
  TOKEN_MACRO = 1
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, source, tokens, comment=u"#", macro=u"&")
(Constructor)

source code 

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Parameters:
  • source (string, file-like object or SourceData object) - Parsing source. If file object is passed, it must be closed by caller function after parsing completes.
  • tokens (sequence) - List of tokens
  • comment - Comment char
  • macro - Macros char
Overrides: object.__init__

lexer(self)

source code 

Scan input for lexemes and return to parser

Returns:
typle (token_type, token_value)