stanhebben.zenscript.parser

Class TokenStream

java.lang.Object

stanhebben.zenscript.parser.TokenStream

All Implemented Interfaces:

java.util.Iterator<Token>

Direct Known Subclasses:

ZenTokener

public class TokenStream
extends java.lang.Object
implements java.util.Iterator<Token>

Represents a token stream. A token stream reads characters from a reader and presents it as a series of tokens.

Token classes with a negative class are considered to be whitespace.

Constructor Summary

Constructors

Constructor and Description
TokenStream(java.io.Reader reader, CompiledDFA dfa, java.lang.String fileNameFallback) Creates a token stream using the specified reader and DFA.
TokenStream(java.lang.String data, CompiledDFA dfa) Creates a token stream which reads data from the specified string.

Method Summary

Modifier and Type	Method and Description
ZenParsedFile	getFile()
int	getLine()
int	getLineOffset()
boolean	hasNext()
boolean	isNext(int type)
Token	next()
Token	optional(int... types)
Token	optional(int type)
Token	peek()
void	popMark() Pops a mark from the mark stack without reset.
protected Token	process(Token token) Can be implemented by subclasses to post-process tokens.
void	pushMark() Pushes a mark on the mark stack.
void	remove()
Token	required(int type, java.lang.String error)
void	reset() Pops a mark from the mark stack and resets the stream's position to it
void	setFile(ZenParsedFile file)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

TokenStream

public TokenStream(java.io.Reader reader,
                   CompiledDFA dfa,
                   java.lang.String fileNameFallback)
            throws java.io.IOException

Creates a token stream using the specified reader and DFA.

Parameters:

reader - reader to read characters from

dfa - DFA to tokenize the stream

Throws:

java.io.IOException

TokenStream

public TokenStream(java.lang.String data,
                   CompiledDFA dfa)
            throws java.io.IOException

Creates a token stream which reads data from the specified string.

Parameters:

data - data to read

dfa - DFA to tokenize the stream

Throws:

java.io.IOException

Method Detail

getFile

public ZenParsedFile getFile()

setFile

public void setFile(ZenParsedFile file)

getLine

public int getLine()

getLineOffset

public int getLineOffset()

peek

public Token peek()

isNext

public boolean isNext(int type)

optional

public Token optional(int... types)

optional

public Token optional(int type)

required

public Token required(int type,
                      java.lang.String error)

pushMark

public void pushMark()

Pushes a mark on the mark stack.

popMark

public void popMark()

Pops a mark from the mark stack without reset.

reset

public void reset()

Pops a mark from the mark stack and resets the stream's position to it

hasNext

public boolean hasNext()

Specified by:

hasNext in interface java.util.Iterator<Token>

next

public Token next()

Specified by:

next in interface java.util.Iterator<Token>

remove

public void remove()

Specified by:

remove in interface java.util.Iterator<Token>

process

protected Token process(Token token)

Can be implemented by subclasses to post-process tokens.

Parameters:

token -

Returns:

processed token