jfun.parsec.tokens
Class Tokenizers

java.lang.Object
  extended by jfun.parsec.tokens.Tokenizers

public class Tokenizers
extends java.lang.Object

The facade class for all pre-built Tokenizer implementations.

Since:
version 0.6
Author:
Ben Yu Apr 27, 2006 7:40:16 PM

Constructor Summary
Tokenizers()
           
 
Method Summary
static Tokenizer forChar()
          Creates a tokenizer that's gonna tokenize a single quoted character literal possibly with escape character '\'
static Tokenizer forDecimal()
          Get the Tokenizer object that creates a decimal number literal token.
static Tokenizer forDecLong()
          Get the Tokenizer object that interprets the recognized character range as a decimal integer and translate it to a long value.
static Tokenizer forHexLong()
          Get the Tokenizer object that interprets the recognized character range as a hex integer and translate it to a long value.
static Tokenizer forInteger()
          Get the Tokenizer object that creates an integer literal token.
static Tokenizer forOctLong()
          Get the Tokenizer object that interprets the recognized character range as a oct integer and translate it to a long value.
static Tokenizer forQuotedString(char open, char close)
          Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing characters.
static Tokenizer forQuotedString(java.lang.String open, java.lang.String close)
          Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing strings.
static Tokenizer forReservedWord()
          Get the Tokenizer object that creates a reserved word token.
static Tokenizer forSimpleStringLiteral()
          Get the Tokenizer object that converts a string literal quoted by '"' to a string object.
static Tokenizer forSqlStringLiteral()
          Get the Tokenizer object that converts a sql string literal quoted by single quote to a string object.
static Tokenizer forString()
          Get the Tokenizer object that creates a string object.
static
<T> Tokenizer
forTypedToken(T type)
          Create a tokenizer that tokenizes the recognized character range to a TypedToken object.
static Tokenizer forWord()
          Get the Tokenizer object that creates a word token.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Tokenizers

public Tokenizers()
Method Detail

forTypedToken

public static <T> Tokenizer forTypedToken(T type)
Create a tokenizer that tokenizes the recognized character range to a TypedToken object.

Parameters:
type - the token type.
Returns:
the Tokenizer object.

forReservedWord

public static Tokenizer forReservedWord()
Get the Tokenizer object that creates a reserved word token.


forWord

public static Tokenizer forWord()
Get the Tokenizer object that creates a word token.


forInteger

public static Tokenizer forInteger()
Get the Tokenizer object that creates an integer literal token.


forDecimal

public static Tokenizer forDecimal()
Get the Tokenizer object that creates a decimal number literal token.


forChar

public static Tokenizer forChar()
Creates a tokenizer that's gonna tokenize a single quoted character literal possibly with escape character '\'

Returns:
the tokenizer instance.

forString

public static Tokenizer forString()
Get the Tokenizer object that creates a string object.


forDecLong

public static Tokenizer forDecLong()
Get the Tokenizer object that interprets the recognized character range as a decimal integer and translate it to a long value.


forHexLong

public static Tokenizer forHexLong()
Get the Tokenizer object that interprets the recognized character range as a hex integer and translate it to a long value.


forOctLong

public static Tokenizer forOctLong()
Get the Tokenizer object that interprets the recognized character range as a oct integer and translate it to a long value.


forSimpleStringLiteral

public static Tokenizer forSimpleStringLiteral()
Get the Tokenizer object that converts a string literal quoted by '"' to a string object. back-slash character is escaped.


forSqlStringLiteral

public static Tokenizer forSqlStringLiteral()
Get the Tokenizer object that converts a sql string literal quoted by single quote to a string object. double single quote is interpreted as one single quote.


forQuotedString

public static Tokenizer forQuotedString(char open,
                                        char close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing characters. The Tokenizer result is a TokenQuoted object.


forQuotedString

public static Tokenizer forQuotedString(java.lang.String open,
                                        java.lang.String close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing strings. The Tokenizer result is a TokenQuoted object.