com.sun.labs.minion.document.tokenizer
Class HandyTokenizer

java.lang.Object
  extended by com.sun.labs.minion.document.tokenizer.HandyTokenizer

public class HandyTokenizer
extends java.lang.Object

A helper class to tokenize strings and return an iterator for the results.


Field Summary
protected  char[] c
          Characters to tokenize.
protected  TokenCollectorStage tcs
          A token collecting stage.
protected  UniversalTokenizer tok
          A tokenizer.
 
Constructor Summary
HandyTokenizer()
          Creates a new handy tokenizer.
 
Method Summary
 java.lang.String[] tokenizeToArray(java.lang.String s)
          Tokenize the given string and hand back an array of the tokens it produces.
 java.util.Iterator tokenizeToIterator(java.lang.String s)
          Tokenize the given string and hand back an iterator for the tokens it produces.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

tcs

protected TokenCollectorStage tcs
A token collecting stage.


tok

protected UniversalTokenizer tok
A tokenizer.


c

protected char[] c
Characters to tokenize.

Constructor Detail

HandyTokenizer

public HandyTokenizer()
Creates a new handy tokenizer.

Method Detail

tokenizeToIterator

public java.util.Iterator tokenizeToIterator(java.lang.String s)
Tokenize the given string and hand back an iterator for the tokens it produces. Note that the iterator is backed by an array that may change if you call tokenize again!

Parameters:
s - The string to tokenize.
Returns:
an Iterator for the tokens that were generated.

tokenizeToArray

public java.lang.String[] tokenizeToArray(java.lang.String s)
Tokenize the given string and hand back an array of the tokens it produces.

Parameters:
s - The string to tokenize.
Returns:
an array containing the tokens.