public class TokenStream extends java.lang.Object implements java.util.Iterator<Token>
Token classes with a negative class are considered to be whitespace.
| Constructor and Description |
|---|
TokenStream(java.io.Reader reader,
CompiledDFA dfa,
java.lang.String fileNameFallback)
Creates a token stream using the specified reader and DFA.
|
TokenStream(java.lang.String data,
CompiledDFA dfa)
Creates a token stream which reads data from the specified string.
|
| Modifier and Type | Method and Description |
|---|---|
ZenParsedFile |
getFile() |
int |
getLine() |
int |
getLineOffset() |
boolean |
hasNext() |
boolean |
isNext(int type) |
Token |
next() |
Token |
optional(int... types) |
Token |
optional(int type) |
Token |
peek() |
void |
popMark()
Pops a mark from the mark stack without reset.
|
protected Token |
process(Token token)
Can be implemented by subclasses to post-process tokens.
|
void |
pushMark()
Pushes a mark on the mark stack.
|
void |
remove() |
Token |
required(int type,
java.lang.String error) |
void |
reset()
Pops a mark from the mark stack and resets the stream's position to it
|
void |
setFile(ZenParsedFile file) |
public TokenStream(java.io.Reader reader,
CompiledDFA dfa,
java.lang.String fileNameFallback)
throws java.io.IOException
reader - reader to read characters fromdfa - DFA to tokenize the streamjava.io.IOExceptionpublic TokenStream(java.lang.String data,
CompiledDFA dfa)
throws java.io.IOException
data - data to readdfa - DFA to tokenize the streamjava.io.IOExceptionpublic ZenParsedFile getFile()
public void setFile(ZenParsedFile file)
public int getLine()
public int getLineOffset()
public Token peek()
public boolean isNext(int type)
public Token optional(int... types)
public Token optional(int type)
public Token required(int type, java.lang.String error)
public void pushMark()
public void popMark()
public void reset()
public boolean hasNext()
hasNext in interface java.util.Iterator<Token>public void remove()
remove in interface java.util.Iterator<Token>