yagdoc.tokenizer.Tokenizer

Package yagdoc :: Module tokenizer :: Class Tokenizer

[show private | hide private]

Type Tokenizer

object --+
         |
        Tokenizer

Tokenizer and preprocessor.

Attributes are only available after tokenize.

Method Summary
	`__init__(self)`
	`add_range_type(self, name, conditions)` Add a special range determined by preprocessor symbols.
	`assume(self, conditions)` Set preprocessor symbols assumed to be defined/undefined.
	`tokenize(self, source)` Perform source file tokenization and preprocessing.
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Instance Variable Summary
`list` of `TokenizerError`	`errors`: Errors encountered.
`list` of `str`	`lines`: Saved source lines (after trigraph resolution and stripping of trailing whitespace).
`list` of `Macro`	`macros`: Macro definitions.
`bool`	`private_header`: `True` if private_header directive is encountered.
	`ranges`: Dictionary, indexed by range type names, of special range lists of items `(start, end)`, where `start` and `end` are token numbers.
`str`	`source`: Set to `name` attribute of the source, or `'<unknown>'` is source has no such attribute.
`list` of `Token`	`tokens`: Source code tokens (in file order).

Method Details

add_range_type(self, name, conditions)

Add a special range determined by preprocessor symbols.

If the range already exists, the conditions are added to it.

Parameters:

name - The range type identifier that will be used as self.ranges key.
(type=arbitrary hashable, identifier-like string is recommended)

conditions - Preprocessor conditions It can be of positive or negative form:

'SYMBOL' for #ifdef SYMBOL or #if defined(SYMBOL)
'!SYMBOL' for #ifndef SYMBOL or #if !defined(SYMBOL).

Example: '!G_DISABLE_DEPRECATED G_ENABLE_BROKEN'.
(type=iterable of strings or a single string (will be split on whitespace))

assume(self, conditions)

Set preprocessor symbols assumed to be defined/undefined.

The specified conditions are added to those already present.

Unlisted preprocessor conditions do not cause any code exclusion, i.e. both branches are always processed.

By default two conditions are known:

'!0', excluding code in #if 0
'YAGDOC_FOR_PRESIDENT', excluding code in #ifndef YAGDOC_FOR_PRESIDENT

Parameters:: conditions - Preprocessor conditions, the notation and interpretation is the same as in add_range_type.
(type=iterable of strings or a single string (will be split on whitespace))

tokenize(self, source)

Perform source file tokenization and preprocessing.

If file is a private header (marked with

/*< private_header 
  >*/

yagdoc directive), all output attributes are set to values corresponding to an empty source and private_header is set to True.

Parameters:: source - Source code, its name attribute determines names shown in error messages.
(type=an iterable returning lines of code, typically a file instance)

Instance Variable Details

errors

Errors encountered.

Type:: list of TokenizerError

lines

Saved source lines (after trigraph resolution and stripping of trailing whitespace).

Type:: list of str

macros

Macro definitions.

Type:: list of Macro

private_header

True if private_header directive is encountered.

Type:: bool

ranges

Dictionary, indexed by range type names, of special range lists of items (start, end), where start and end are token numbers.

source

Set to name attribute of the source, or '<unknown>' is source has no such attribute.

Type:: str

tokens

Source code tokens (in file order).

Type:: list of Token

Generated by Epydoc 2.1 on Fri Oct 5 00:36:53 2007

http://epydoc.sf.net